Deep Dive: nftables – One Table of Our Own, Sets with Timeouts, and the 300 Rules K3s Writes Next to It

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

nftables replaced iptables in the kernel years ago, but most guides still translate old iptables rules line by line and stop there. That works until a second program writes rules too – and on a K3s host, several do. This post explains the data model of nftables with the details that matter in practice, how the role common_firewall builds one table for the host, fail2ban, port-scan detection and the SSH country filter, what K3s puts next to it, and four bugs from the last months that each taught one of those details the hard way. The migration itself is described in an earlier post, the country filter in another one; this one is about how it all fits together.

🧭 How nftables decides

Tables, chains, hooks

In iptables, the tables (filter, nat, mangle) and their chains were fixed. In nftables, a table is just a namespace that you create, and it belongs to an address family: ip, ip6, inet (both), arp, bridge, netdev. A chain only sees packets if it is a base chain – one that is attached to a netfilter hook (prerouting, input, forward, output, postrouting) with a priority and a policy. Regular chains without a hook are only reached by jump or goto.

table inet filter {
  chain input {
    type filter hook input priority filter; policy drop;
    ...
  }
}

Priorities: several tables on one hook

Any number of base chains from any number of tables can hang on the same hook. The kernel runs them in order of their priority number – lower first: raw is −300, connection tracking runs at −200, mangle −150, dstnat −100, filter 0, srcnat 100. Two facts follow that are easy to miss:

• An accept is only final for its own chain. If our input chain accepts a packet, the next base chain on the same hook – for example the one K3s created – still sees it and may drop it. Only a drop ends the packet’s journey everywhere.
• Where a chain sits relative to conntrack matters. A chain at −300 runs before connection tracking and cannot know whether a packet is a reply. One at −150 runs after it and can use ct state. One of the bugs below came exactly from this difference.

Sets and maps

Sets are typed collections (ipv4_addr, ipv6_addr, port numbers, concatenations) stored in the kernel and looked up in constant time. With flags interval they hold ranges, with flags timeout every element expires on its own, and with flags dynamic the rule itself can add elements from the packet path (add @set, update @set). That is what made ipset unnecessary: a ban list with 24-hour expiry is three lines.

set banned4 {
  type ipv4_addr
  flags timeout
  timeout 1d
}
...
ip saddr @banned4 drop

Atomic replacement – and why “flush ruleset” is dangerous

nft -f file applies the whole file as one transaction: either everything is loaded or nothing is, and there is no moment without rules. Many example configurations start with flush ruleset. On a host where only one program writes rules that is fine. On a host where kube-proxy, flannel and the Kubernetes network policy controller keep their own tables, it deletes their rules too – they notice only on their next sync, seconds to minutes later.

iptables-nft and iptables-legacy

Most Kubernetes components still speak iptables. On current distributions, the iptables command is iptables-nft: same syntax, but it writes into nftables tables named ip filter, ip nat and so on, using a compatibility layer for old match modules. The other backend, iptables-legacy, writes into the old x_tables kernel structures. Both can be active at the same time – and then neither nft list ruleset nor iptables-legacy -L shows the whole picture.

🛡️ Why it is here

• One readable file for IPv4 and IPv6 (table inet), versioned in Git, rendered from one template.
• Default drop in both directions. Incoming only web, SSH and what K3s needs; outgoing only DNS, NTP, HTTP(S), mail submission and the cluster’s own traffic.
• Kernel-side ban lists with timeouts for fail2ban, the port-scan detection and the SSH country allowlist – no daemon in the packet path.

🔧 What this repository does

One table of our own

The role renders fw.nft.j2 to /etc/sysconfig/nftables.conf – the file the nftables unit on AlmaLinux actually reads (not /etc/nftables.conf). The file starts like this:

table inet filter
delete table inet filter

table inet filter {
  ...
}

The first line creates the table if it does not exist (so the delete cannot fail on a fresh boot), the second deletes it with everything in it, then it is rebuilt – all in one transaction. Only our table is replaced; the tables of K3s stay untouched. A systemd drop-in scopes the unit the same way: ExecReload loads the file, ExecStop runs nft delete table inet filter instead of the distribution’s default flush. Before the file replaces the live one, Ansible validates the rendered candidate with nft -c -f; a syntax error never reaches the firewall.

The input chain, in order

1. Loopback, established/related, drop invalid; drop XMAS and NULL packets, reset SYN+ACK on new connections, rate-limit RSTs.
2. Ban lists first: banned4/6 (fail2ban, 24 h) and scanban4/6 (port scanners, 1 h) – before any accept, so a banned address reaches nothing.
3. K3s exceptions: pods may reach the API server and kubelet (6443, 10250), the host’s own HTTP(S) for cert-manager’s self-check, and on the Nextcloud hosts MariaDB and Redis on the host.
4. Drop private and reserved source ranges from outside, allow the useful ICMP and the ICMPv6 types IPv6 cannot live without.
5. Web on 80/443 (no UDP 443 – the ingress controller has no HTTP/3), SSH on 10022 only from the country allowlist.
6. Port-scan detection: whatever is still here is about to hit the policy drop – a closed port. A source sending more than 30 SYNs per minute to closed ports is added to scanban4 by the rule itself.
7. Rate-limited log line, policy drop.

tcp flags & (fin|syn|rst|ack) == syn tcp dport != 22 \
  update @portscan4 { ip saddr limit rate over 30/minute burst 30 packets } \
  add @scanban4 { ip saddr } drop

fail2ban cannot do this: it only reads logs, and the drop log is limited to two lines a minute on purpose. Port 22 is excluded so that forgetting -p 10022 a few times never locks the admin out.

fail2ban writes into the same sets

fail2ban gets no table of its own. Its action is two lines:

actionban   = nft add element inet filter <nftset> { <ip> timeout <bantime>s }
actionunban = nft delete element inet filter <nftset> { <ip> } 2>/dev/null; true

Every ban carries its own timeout, so it expires in the kernel even if fail2ban is not running. On the blog, banned4 held 22 addresses at the time of writing, the SSH allowlist ssh_geo4 7,498 German address ranges (with flags interval and auto-merge, built weekly from RIPE data). Since 19 October 2026 the same list also guards the WordPress login: ssh-geo-update writes it a second time as an nginx geo file, and nginx in the blog pod answers /wp-login.php and /wp-admin/ with 403 for everything outside it. Since 20 October 2026 the same list goes to the ingress controller on every server as well, which answers 403 on /grafana outside it.

What K3s writes next to it

On the blog, nft list tables shows seven tables. Only inet filter is ours (49 rules). The others come from K3s through iptables-nft: ip filter with about 190 rules – kube-proxy’s KUBE-SERVICES, KUBE-NODEPORTS, KUBE-PROXY-FIREWALL and the network policy controller’s KUBE-ROUTER-*, KUBE-NWPLCY-* and one KUBE-POD-FW-* chain per pod –, ip nat with about 120 (Service DNAT, flannel’s masquerading), and small ip6 tables. Our forward chain therefore only needs to let the pod network pass; whether a pod may talk to another pod is decided in K3s’s chains on the same hook.

🐛 Four bugs, four details

1. “flush ruleset” broke the Services

The first version of the template began with flush ruleset. Every firewall reload deleted kube-proxy’s and flannel’s rules as well; Nextcloud lost its connection to MariaDB and Redis via ClusterIP for 30 to 90 seconds until kube-proxy re-synced. The fix is the table-scoped delete shown above.

2. A reload silently unbanned everyone

Replacing our table also replaces the ban sets – empty. fail2ban did not notice and kept the addresses as “banned” in its own database while the firewall let them in again. Found on 30 September 2026: after a playbook run, 15 of 17 bans on one Nextcloud server and 17 of 24 on the other existed only in fail2ban. The firewall handler now restarts fail2ban right after every reload (systemctl try-restart fail2ban), and fail2ban re-applies every still-active ban from its database.

3. The NodePort chain dropped the replies to our own traffic

kube-proxy makes every NodePort (30000–32767) reachable on every local address – including the public one – and that traffic is DNATed to a pod and then passes the forward chain, not input. The exporters were reachable from the internet. The first fix was a chain at priority raw (−300) that dropped everything to that port range on the public address. It worked – and also dropped about 4 % of the replies to outgoing pod connections: flannel masquerades pod traffic with --random-fully over the whole port range, so some connections got a source port between 30000 and 32767, and their SYN-ACK was thrown away before conntrack could recognise it as a reply. Symptom for two months: sporadic timeouts to GitHub and app stores from every K3s host. The chain now runs at −150, after conntrack, and drops only ct state new.

chain prerouting {
  type filter hook prerouting priority -150; policy accept;
  ip  daddr <public-ipv4> tcp dport 30000-32767 ct state new drop
  ip6 daddr <public-ipv6> tcp dport 30000-32767 ct state new drop
}

4. The policy engine that enforced nothing, and a module block too wide

For two months Calico ran next to K3s as the supposed NetworkPolicy engine. A deny-all test on 19 October 2026 showed that it never matched anything: Calico’s Felix only handles cali* interfaces – those of its own CNI – while K3s’s Flannel creates veth*. All enforcement had always come from K3s’s embedded policy controller (the KUBE-NWPLCY-* chains). Calico was removed; the blog, which had used iptables-legacy until then, moved to nftables only, and the legacy table modules were blocked with install … /bin/false in modprobe.d. The first version of that block also included ip_tables and ip6_tables – after the next reboot, K3s’s policy controller panicked, because iptables-nft still uses the old icmp match through the compatibility layer, and that needs those modules. The blog answered 502 for about five minutes. Only the iptable_* and ip6table_* table modules are blocked now.

🧰 Day-to-day administration

# our table only (the full ruleset includes ~300 K3s rules)
nft list table inet filter

# ban lists: who, and how long still
nft list set inet filter banned4
nft list set inet filter scanban4

# is an address in the SSH allowlist?
nft get element inet filter ssh_geo4 '{ 203.0.113.7 }'

# lift a ban (fail2ban's own record: fail2ban-client unban <ip>)
nft delete element inet filter banned4 '{ 203.0.113.7 }'

# what K3s wrote
nft list tables
iptables-nft-save | grep -c '^-A'

# follow one packet through every chain (add a trace rule first)
nft add rule inet filter prerouting ip saddr 203.0.113.7 meta nftrace set 1
nft monitor trace
nft flush chain inet filter prerouting   # removes the trace rule again - the role restores the chain on its next run

Manual changes and the next Ansible run

Elements added by hand to banned4 survive until they expire – but only until the next reload, which rebuilds the table. Rules added by hand are gone after the next run, on purpose: the template is the truth. Permanent exceptions belong in the template or the inventory (for example fw_ssh_geo_extra for an address outside the country list).

Reading the drop log

Dropped packets are logged with the prefixes nftables DROP IN:, DROP FWD: and DROP OUT:, at most two a minute per chain. Since October 2026 they are searchable in Loki: {job="journal", transport="kernel"} |= "nftables DROP". A DROP OUT line is the interesting one – it means something on the server tried to connect somewhere it is not allowed to.

🔭 What comes next

• Counters in Prometheus. Named counters on the drop rules and the set sizes as metrics would show attack volume and ban list growth over time.
• Flowtables for the forward path could offload established pod connections – not needed at this traffic level.
• Fewer K3s rules: kube-proxy has an nftables mode of its own; once K3s ships it as the default, the iptables-nft layer can go.

⚖️ Alternatives – and why plain nftables

The real lesson of the last months is not a syntax detail. It is that a firewall on a Kubernetes host is never alone: every assumption about who writes which rule, and in which order the kernel runs them, has to be tested – a deny-all test found more in one afternoon than two months of reading configuration files.