Deep Dive: fail2ban – From a Log Line to a Kernel Set Entry, and the Bans That Never Happened

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

fail2ban looks like the simplest tool on these servers: read a log, count failures, ban the address. On a K3s host every part of that sentence has a catch. The logs live in container files that change their name with every restart; the bans have to land in a firewall that other programs also write to; and an address family that nobody tested can make the whole thing a placebo. This post follows one bad request from the log line to the entry in the kernel, explains what part 5 of the tour only touches – and describes three cases in which fail2ban said “Ban” and the firewall let the address in anyway.

🧭 How fail2ban works

Jails, filters, actions

A jail ties three things together: a log to watch, a filter that recognises a bad line, and an action that bans. The filter is a set of regular expressions (failregex, with ignoreregex for exceptions) in which <HOST> marks the address. Every match is a failure for that address. If an address collects maxretry failures within findtime, the action runs and bans it for bantime.

[nginx-k3s-wp-login]
enabled  = true
filter   = nginx-k3s-wp-login
logpath  = /var/log/pods/ingress-nginx_ingress-nginx-*/nginx-ingress/*.log
backend  = polling
maxretry = 10
findtime = 600

Backends: how the log is read

• systemd reads the journal directly (filtered by unit) – robust, no files, no rotation. Used here for sshd.
• polling checks the file for new content every few seconds. Simple, works on any filesystem – and the only option when the path is a glob that changes.
• pyinotify reacts to file events immediately; faster, but it watches the files that existed when the jail started.

The timestamp matters as much as the address: fail2ban parses the date of each line (datepattern) and ignores lines older than findtime. A filter with the wrong date format matches nothing, silently.

The database and recidive

fail2ban stores current bans in an SQLite database (/var/lib/fail2ban/fail2ban.sqlite3). After a restart it re-applies every ban that has not expired yet. The recidive jail turns fail2ban on itself: its log is fail2ban.log, its filter counts “Ban” lines, and whoever is banned three times in a day is banned for 30 days on every port.

🛡️ Why it is here

• Cheap protection against automated attacks on SSH, the WordPress login, the Nextcloud login and Grafana – and against scanners that walk through lists of known-vulnerable paths.
• Bans in the kernel, not in the application. A banned address reaches nothing at all, not even the TLS handshake.
• Visibility. Who knocks, how often and on which door is in the ban log – since October 2026 searchable in Loki.

🔧 What this repository does

Seven jails, two log sources

All HTTP jails read the access log of the ingress controller, the one place every request passes, for blog and Nextcloud alike. The filters expect the containerd prefix in front of every line (2026-06-19T14:24:11Z stdout F <client> - - [time] "POST /wp-login.php …"); a filter written for a plain nginx log would never match.

The ban goes straight into nftables

fail2ban gets no table or chain of its own. Its action adds the address to a set that the firewall checks at the very top of its input chain – every jail into its own pair f2b_<jail>4/6 (why, see the fifth bug below):

[Definition]
actionban   = nft add element inet filter <nftset> { <ip> timeout <bantime>s }
actionunban = nft delete element inet filter <nftset> { <ip> } 2>/dev/null; true

[Init]
setname = banned
nftset  = <setname>4

[Init?family=inet6]
nftset  = <setname>6

# jail.local - every jail names its own set pair
[sshd]
action = nftables-banned-sets[setname=f2b_sshd]

Each element carries its own timeout, so a ban expires in the kernel even if fail2ban has died in the meantime. The unban is only a best-effort clean-up.

The log file that moves

The ingress log is not one file. It is /var/log/pods/<namespace>_<pod>_<uid>/nginx-ingress/N.log, and both the pod directory and the number change: a container restart writes 1.log after 0.log, a new pod gets a new directory. fail2ban expands the glob in logpath once, when the jail starts. A small timer (fail2ban-ingress-logwatch, every two minutes) compares the current list of files with the last one and reloads the nginx-k3s-* jails when it differs. Without it, see the first bug below.

🐛 Three times fail2ban said “Ban” and nothing happened

1. Watching a dead file for days

On 23 September 2026 the HTTP jails on both Nextcloud servers turned out to have banned nothing for days. After a reboot fail2ban had started before the ingress container, and after a container restart the jails kept tailing a file that no longer received lines. Everything looked healthy – the service ran, the jails were listed, no error anywhere. The log-watch timer described above is the fix. It is also one of the reasons Monit now checks that every expected jail is active (the story of the watchdog began with fail2ban dying quietly for two days).

2. A firewall reload that emptied the ban sets

The firewall replaces its own table on every reload – and with it the sets, empty. fail2ban kept its bans in its database and believed they were active. On 30 September 2026, after a playbook run, 15 of 17 bans on one server and 17 of 24 on the other existed only in fail2ban. The firewall handler now restarts fail2ban immediately after every reload, and fail2ban re-applies every ban that has not expired.

3. IPv6 bans that were never possible

Until 14 October 2026 every jail passed nftset = banned4 to the action – also for IPv6 addresses. nft refused to put an IPv6 address into an IPv4 set (“Address family for hostname not supported”), fail2ban logged the ban anyway, and the address was never blocked. On all three servers. The [Init?family=inet6] section above selects the set by address family; in the week to 20 October 2026, 15 of 444 bans were IPv6 addresses – few, but each of them would have walked straight through before.

A fourth one, found by Loki

When the logs went to Loki on 19 October 2026, the new alert for fail2ban errors fired at once. Most errors were harmless (KeyError on a pod log that had just been removed – the polling backend losing a file, the jail carries on). But on 13 and 14 October there were real ones: Failed to execute ban jail 'nginx-k3s-bad-paths' action 'nftables…' – bans that did not happen while the firewall reload was being reworked. Nobody had seen them, because fail2ban does not mail its own errors. The alert now does, minus the known harmless pattern.

A fifth one: one set for every jail

Until 21 October 2026 all jails wrote into the same set pair banned4/6. That looks tidy and has a catch: an address banned by two jails exists only once in the set. When the shorter ban ended – the scanner jail after a day, say – the address left the set, although recidive still meant to keep it out for 30 days. fail2ban listed it as banned; the kernel no longer did. A comparison that day found one to five such addresses per server. Since then every jail has its own pair (f2b_sshd4, f2b_recidive4, …) and adds and removes only there, and a Monit check (f2b_sets) regularly compares fail2ban’s list with the sets.

📊 What it bans

One week, 14 to 20 October 2026, all three servers:

The picture is clear: almost all bans come from scanners that look for .env files, Git directories, admin panels and web shells. Real login attacks are rare – the WordPress login saw two POSTs in three hours on the day the logs were analysed, and since 19 October 2026 /wp-login.php only answers addresses from the country list anyway. SSH bans are low because port 10022 only accepts German address ranges; what remains are a few addresses from inside that range.

🧰 Day-to-day administration

# all jails, then one in detail (currently banned, total)
fail2ban-client status
fail2ban-client status nginx-k3s-bad-paths

# the firewall's view: every jail has its own set pair f2b_<jail>4/6
nft list set inet filter f2b_bad_paths4
nft list set inet filter f2b_recidive4
fail2ban-client get sshd action nftables-banned-sets setname   # -> f2b_sshd

# lift a ban (both: fail2ban's record and the kernel set)
fail2ban-client set nginx-k3s-scanner unbanip 203.0.113.7

# does a filter match real lines? (against the current ingress log)
fail2ban-regex /var/log/pods/ingress-nginx_ingress-nginx-*/nginx-ingress/0.log \
    /etc/fail2ban/filter.d/nginx-k3s-bad-paths.conf

# which log file does a jail watch right now?
fail2ban-client get nginx-k3s-wp-login logpath

When a jail does not trigger

1. Is it watching the right file? get <jail> logpath against ls /var/log/pods/… – after a restart of the ingress controller the log-watch timer should have reloaded it within two minutes.
2. Does the filter match at all? fail2ban-regex against the real file. A line format change after an ingress upgrade shows up here.
3. Is the date understood? Lines older than findtime are ignored; a wrong datepattern makes every line “old”.
4. Does the ban reach the kernel? Compare status with nft list set. If they differ, look in fail2ban.log for “Failed to execute ban” – or in Loki: {job="fail2ban"} |= "ERROR".

Do not test from your own jump host

Requesting /.env once to see whether the bad-paths jail works is tempting. Two requests within an hour ban the address for 24 hours – the jump host included, and with it every SSH session to that server. The private ranges are in ignoreip, the public address of the admin network is not. Test with fail2ban-regex against a log line, or from an address you can afford to lose for a day.

🆘 Cheat sheet: symptom → command

The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders.

🔭 What comes next

• Bans in Grafana – done. Since 21 October 2026 the repository ships a dashboard “fail2ban – Bans”: bans per jail and hour, the totals for the chosen period and the latest ban lines, all from the log lines Loki already has.
• One ban list for all three servers. A scanner banned on the blog is usually the same one that tries the Nextcloud servers an hour later. Sharing would need a small sync job between the hosts.
• Reputation lists (CrowdSec-style) as an additional layer – at the price of an external service.

⚖️ Alternatives – and why fail2ban

fail2ban is old, simple and does exactly what it says. The work was never the tool; it was making sure that every “Ban” in its log also ended up as an entry in the kernel. Since October 2026 that is checked – by the firewall handler, by Monit and by a log alert – instead of assumed.