
🌐 Also available in: Deutsch · Français · Español
Part 5 of the Ansible Tour. Part 1 explained the framework, part 2 the server and the platform, parts 3 and 4 the blog and Nextcloud. This part collects every security role in one place, across both stacks. It describes release v3.3.0.
Defense in Depth described the idea: nine layers between the internet and the data, each one catching what the one before missed. That post said what the layers are. This one shows how they are built – role by role, including the handlers, the declared inputs, the templates and the mistakes that shaped them. Nine roles, from the outside in:
| Layer | Role | Protects against |
|---|---|---|
| Network | common_firewall | Everything that is not explicitly allowed, port scans, a back door through Kubernetes |
| Login | common_ssh | Password guessing, weak crypto, admin sessions that can do too much |
| Reaction | common_fail2ban | Repeated attempts – login guessing, scanners, known attack paths |
| Operating system | common_cis_hardening | Unused features, weak defaults, leaks of system information |
| Containment | common_selinux | A compromised process reaching files it has no business with |
| Detection | common_auditd, common_aide_refresh, common_rkhunter, common_clamav | Changes that should never happen, rootkits, malware in uploads – noticed, reported |
Two more layers were covered elsewhere and are only summarised at the end: the kernel’s network hardening (common_sysctl, part 2) and the hardening of the pods themselves (parts 3 and 4). And one has its own post: the SSH login alarm.
A note on detail: the code is public and the series explains it, so an attacker can read it too. That is fine – the protection must come from keys, layers and alarms, not from secrecy. But I keep exact thresholds vague here. They are in the repository for anyone who needs them.
🧱 common_firewall: one table, allow-list, and a few traps
The firewall is nftables, the successor of iptables in the Linux kernel. If you know iptables, the differences that matter here are: one tool (nft) for IPv4 and IPv6 instead of two rule sets; tables and chains you create yourself and attach to a point in the kernel’s packet path (a hook, with a priority); built-in sets of addresses with automatic expiry, instead of the separate ipset tool; and – most important for automation – a rule file is loaded as one atomic transaction: the old rules stay active until the new ones are completely in place. The migration from iptables is an older post.
The role, step by step
- Install nftables.
- DHCP or static? The role asks NetworkManager how the uplink gets its addresses and stores the answer in two facts (part 1 used this as the example for “detect instead of declare”). Only where DHCP is used does the template render the DHCP rules – without them, the lease renewal fails and the address is lost.
- The SSH country filter, if the host has
fw_ssh_geo_countriesset (all three servers allow SSH only from one country – the story is here). A script downloads the address ranges of that country from the regional registry and writes them into an nftables file; a systemd timer refreshes it weekly. This block runs before the rule set is written, on purpose: the rule set includes the file, and if the very first download fails, the play stops before any rule exists that could lock SSH out. The whole block is oneblockwith onewhen, and a mirror block removes all of it when the filter is switched off. - The rule set from
fw.nft.j2into/etc/sysconfig/nftables.conf– the file AlmaLinux’s service actually reads, not the/etc/nftables.confmany guides mention. Withvalidate: nft -c -f %s: Ansible writes the rendered file to a temporary path, runs the syntax check on it, and only replaces the real file if the check passes. A typo never reaches the server’s configuration. - A systemd drop-in that changes reload and stop. More on that below – it is the most important line in the role.
- Enable nftables, start it – except on the very first run.
- Clean up: an old iptables script from the time before this role (the blog ran two firewalls on top of each other for a while, and the old one dropped DHCP renewals), the unused
/etc/nftables.conf, and firewalld, which is masked rather than removed – removing the package would also remove nftables and fail2ban, because they depend on each other in the packaging.
Never “flush ruleset”
The packaged nftables service reloads with flush ruleset – delete every rule in the kernel – and then loads the file. On a Kubernetes node that is a trap. kube-proxy, Flannel and the network policy controller built into K3s write their own rules into nftables too, in their own tables. A flush deletes those as well, and for 30 to 90 seconds, until they notice and rewrite them, Kubernetes Services stop working – Nextcloud could not reach its database after every firewall reload. So the rule set replaces only its own table, and the drop-in teaches systemd to do the same:
# fw.nft.j2, first lines
table inet filter
delete table inet filter
table inet filter {
...
}
# /etc/systemd/system/nftables.service.d/10-own-table-only.conf
[Service]
ExecReload=
ExecReload=/sbin/nft -f /etc/sysconfig/nftables.conf
ExecStop=
ExecStop=/sbin/nft delete table inet filterThe first line looks odd: why declare a table only to delete it? Because delete fails if the table does not exist – on a fresh boot. Declaring it first makes the delete always safe, and since the whole file is one transaction, there is no moment without a firewall. In the drop-in, the empty ExecReload= clears the packaged command before setting the new one, the same trick as with the update timer in part 2.
What the rule set does
The template is long, but its structure is simple. Every chain has policy drop: whatever is not explicitly allowed is dropped – outgoing traffic included.
- Input: loopback; packets belonging to an existing connection (
ct state established,related– the connection tracker remembers what was allowed); then drop garbage (invalid state, packets with impossible flag combinations, forged handshake replies, RST floods); then the ban sets; then the exceptions the cluster needs (pods reaching the API server, and on Nextcloud hosts the host database and Redis – thestackexample from part 1); then web traffic, SSH (from the country list) and the ICMP types that IPv4 and especially IPv6 need to work. Last, before the policy drop, a rate-limited log line. - Ban sets:
banned4/banned6for fail2ban, with a timeout so entries expire by themselves, andscanban4/scanban6for the port-scan detector. - Port-scan detection, inside the firewall: a packet that reaches the end of the input chain is about to be dropped – a closed port. The rule counts such connection attempts per source address in a dynamic set; a source that exceeds a few dozen per minute lands in
scanbanfor an hour and is then blocked on all ports. A person mistyping a port sends a handful; a scanner sends hundreds. The SSH default port is not counted, so forgetting the custom port a few times never locks out the admin. fail2ban could not do this – it only reads logs, and the drop log is rate-limited on purpose. - Prerouting: closes a back door Kubernetes opens. kube-proxy forwards the NodePort range on every local address, the public one included, and that traffic goes through the forward chain, not input. The metrics exporters of parts 3 and 4 were reachable from the internet that way. This chain drops new connections to that range on the public addresses. Its priority is chosen precisely – after the connection tracker, before Kubernetes rewrites the destination – and that precision has a story: it first ran earlier, before connection tracking, and also dropped the replies to pods’ own outgoing connections whenever their randomly chosen source port fell into that range. About 4 percent of outgoing pod connections timed out for two months before an acceptance test of a new server found it.
- Forward: pod traffic in both directions; output: DNS, NTP, HTTP(S), mail submission, DHCP where needed, the pod network and the API server – nothing else leaves the server.
The first run: why the server reboots
On a fresh server, SSH still listens on port 22 and no firewall is active. The SSH role (below) moves SSH to its new port, this role writes the firewall. Activating both live would kill the connection Ansible is using: the existing connection is not in the connection tracker yet, matches no rule, and its packets are silently dropped – the play just hangs. So on the first run the role does not start nftables, and ends with a reboot instead:
- name: Firewall | First run only – reboot to activate the new SSH port and firewall together
ansible.builtin.include_tasks:
file: first_run_reboot.yml
apply:
tags: [common_firewall]
when: ssh_first_runInside first_run_reboot.yml: send systemctl reboot over the current port-22 connection with async: 1, poll: 0 (fire and forget – the connection will die), switch ansible_port to the new port with set_fact, drop the old connection with meta: reset_connection, and wait_for_connection until the server answers on the new port. The play then continues there. Three Ansible details make this work:
include_tasks, notimport_tasks:meta: reset_connectionignoreswhen. Inside a statically imported file with a condition, it ran on every run and printed a warning. A dynamic include evaluates its condition first and only then loads the file.apply: tags: tags on a dynamic include do not reach the tasks inside it.applypasses them on, so the include also works with--tags common_firewall.- The fact
ssh_first_runcomes from the SSH role, which ran just before:set_factvalues stay available to all later roles on that host. It is true when the play connected on port 22.
One thing stays manual, and the play prints a reminder: the new port must be written into the host’s vars.yml before the next run. An earlier version tried to do the reboot over the new port – and failed immediately, because task variables also apply to the connection that sends the reboot. The current order (reboot over the old port, then switch) came from that.
Handlers: reload, then restore the bans
- name: Reload nftables
ansible.builtin.service:
name: nftables
state: reloaded
- name: Re-apply fail2ban bans after the nftables reload
ansible.builtin.command: systemctl try-restart fail2ban
listen: Reload nftablesReplacing the table also replaces the ban sets – empty. fail2ban did not notice and kept those addresses as “banned” while the firewall let them in again: after one playbook run, most bans on two servers existed only in fail2ban’s memory. The second handler subscribes to the same notification with listen and runs after the first, because handlers run in the order they are defined. A restart makes fail2ban re-apply every active ban from its database. try-restart does nothing where fail2ban is not running.
Update (19 Oct 2026): nftables only, on every host. The blog still had the EPEL packages iptables-legacy and iptables-services, so iptables was the legacy backend and K3s kept its rules in legacy tables, invisible to nft list ruleset. common_firewall now removes those packages, and common_cis_hardening blocks the legacy table modules (iptable_*, ip6table_*). Not ip_tables itself: iptables-nft takes its ICMP match from it – blocking that too made K3s’s policy controller crash after a reboot and took the blog down for five minutes.
🔑 common_ssh: the door, and what you can do once inside
Port, SELinux and the configuration
SSH moves to a non-standard port. That is not security by itself – a scanner finds it – but it removes almost all of the automated noise aimed at port 22, which keeps the logs and fail2ban meaningful. SELinux only allows sshd to listen on ports labelled ssh_port_t, so the role reads the label list once and then either adds the label (semanage port -a) or modifies an existing one (-m) – two tasks with opposite conditions on the same registered output, one of them with a regex_search that checks whether the port is already listed under the right type.
The configuration file is strict: keys only, no passwords and no keyboard-interactive login, only root may log in (there are no other accounts that need SSH), very few attempts and sessions per connection, a short time to authenticate, no forwarding of ports, X11 or tunnels, and a short list of modern key exchange, cipher and MAC algorithms. LogLevel VERBOSE logs the fingerprint of the key that was used for each login.
First value wins
A lesson from two servers. sshd_config includes /etc/ssh/sshd_config.d/*.conf at the top, and for sshd the first value of a keyword wins. The provider’s image for the Nextcloud servers drops a 50-cloud-init.conf there that says PasswordAuthentication yes – and it silently beat the “no” in the main file. AlmaLinux’s own 50-redhat.conf did the same with X11 forwarding on all three servers. The fix is a drop-in named 00-hardening.conf: it sorts first, so its values win no matter what an image adds later.
The sandbox around sshd
A systemd drop-in restricts the sshd service – and everything started from it inherits the restrictions, every SSH session and every Ansible task included. That is the point and the risk at the same time:
[Service]
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictRealtime=yes
LockPersonality=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINKNobody who logs in over SSH can load a kernel module or change a kernel parameter – an attacker with a stolen key included. Legitimate changes go through systemd-run, which part 2 explained. The comment above the drop-in lists what was tried and removed, and each removal is a lesson:
AF_NETLINKmust be allowed: without it,nft,ipandssfail in every SSH session – including when you are trying to repair the firewall.- No
RestrictSUIDSGID: it blocked setting setuid bits even for root, and package installs that ship such files failed. - No
PrivateTmp: sshd’s private/tmpis inherited by every session, so files dropped in/tmpover SSH were invisible to cron and other services, and every sshd restart destroyed it under running sessions –dnffailed twice that way.
The general rule the role states: per-service sandboxing belongs on services that start no sessions. The MariaDB drop-in in part 4 is the example of doing it fully.
Restarting the server you are connected through
The most delicate part of the role is its handler chain. Restarting sshd kills the connection Ansible runs through. The handlers, in the order they are defined (which is the order they run):
| Handler | What it does |
|---|---|
Validate sshd config | sshd -t – a broken configuration stops the play before sshd is restarted with it |
Reload systemd units for sshd | re-read the drop-in; without it systemd would restart sshd from its cached old unit |
Reload sshd | the restart – with async: 30, poll: 0: start it and do not wait for an answer that can never arrive over a dying channel |
Reset connection after sshd restart | meta: reset_connection: throw away the cached, now dead SSH connection |
Confirm sshd is accepting connections again | wait_for_connection: reconnect; if sshd did not come back, the play fails here – loudly, while you presumably still have another session open |
It is a restart, not a reload, on purpose: systemd applies sandbox settings only when it starts a process. A reload would send a signal to the running sshd, and changes to the drop-in would never take effect. Before async was added, the restart task died with “Module result deserialization failed” – the module’s answer never arrived because its connection was gone.
One limitation is documented in the role: the main sshd_config only notifies the validation handler. A change to that one file is checked, but only becomes active with the next sshd restart – from a drop-in change or a reboot. The drop-ins, which carry the security-critical settings, restart sshd themselves.
The role ends with the login alarm (STEP 8), which has its own post, a warning banner, and a replaced /etc/issue that no longer tells anyone at the login prompt which OS and kernel version runs here.
🚫 common_fail2ban: reacting to what the logs say
fail2ban reads log files, recognises attacks with regular expressions (filters), counts them per address, and after too many hits within a time window runs an action – here: put the address into the firewall’s ban set. Seven jails combine a filter, a log and limits:
| Jail | Watches | Reacts to |
|---|---|---|
sshd | the system journal | failed SSH logins |
nginx-k3s-wp-login | the ingress log | failed WordPress logins |
nginx-k3s-nc-login | the ingress log | failed Nextcloud logins |
nginx-k3s-grafana-login | the ingress log | failed Grafana logins (stricter) |
nginx-k3s-scanner | the ingress log | bursts of requests typical for vulnerability scanners |
nginx-k3s-bad-paths | the ingress log | paths that are never legitimate here – /.env, /.git/, web shells, admin tools that do not exist – with almost zero tolerance and a long ban |
recidive | fail2ban’s own log | addresses banned again and again: a much longer ban on all ports |
The HTTP jails read the ingress controller’s log directly on the host – nginx runs in a pod, but its log lands in /var/log/pods/, in a format with a containerd prefix that the custom filters account for. That creates a problem of its own: the log file’s name changes with every container restart, and fail2ban expands the path only when a jail starts. After a reboot, fail2ban started before the ingress container and then tailed a dead file – the HTTP jails banned nothing for days. A small script on a systemd timer now reloads those jails whenever the set of log files changes; its state lives in /run, so the first check after every boot always reloads.
The ban action – and a bug found while writing this
[Definition]
actionban = nft add element inet filter <nftset> { <ip> timeout <bantime>s }
actionunban = nft delete element inet filter <nftset> { <ip> } 2>/dev/null; true
[Init]
nftset = banned4
[Init?family=inet6]
nftset = banned6The action writes the address straight into the firewall set, with the ban time as the element’s timeout – nftables removes it by itself; the unban is only cleanup. The two [Init] sections choose the set by address family. That part is new: while reading the role for this post, I found that the jails passed nftset=banned4 for every address. Every IPv6 ban therefore tried to put an IPv6 address into an IPv4 set, and nft refused with “Address family for hostname not supported” – on all three servers. fail2ban logged “Ban”, the firewall let the address in, banned6 stayed empty. The fix was tested against a copy of the live configuration (fail2ban-client -t and -d, and the nft command in check mode) before it went into the repository. The firewall’s own port-scan detector was never affected – it handles both families itself. The lesson for operators: a log line saying “Ban” is not proof that anything was blocked. Look at the set.
Two smaller details. fail2ban’s own log must exist before its first start, because the recidive jail reads it and fail2ban checks all logs before it creates its own – a fresh server failed exactly that way. The role creates it with state: touch and modification_time: preserve, so the task is not “changed” on every run. And the single handler restarts fail2ban once at the end of the play, however many files changed – which also re-applies all active bans.
Added 19 Oct 2026: the WordPress login now sits behind the same country list that guards SSH. nginx in the blog pod answers /wp-login.php and /wp-admin/ with 403 for addresses outside Germany – before PHP runs, so brute-force attempts from abroad reach neither WordPress nor fail2ban. ssh-geo-update writes the same RIPE blocks as an nginx geo file and reloads nginx after its weekly refresh; admin-ajax.php and admin-post.php stay open because visitor features use them. Logging in from abroad needs a VPN with a German exit or an entry in fw_ssh_geo_extra.
🛡️ common_cis_hardening: the benchmark, applied with judgement
The CIS benchmark is a long checklist for hardening an operating system. Running it against these servers produced over a hundred findings. This role applies the ones that are safe – and its header starts with a list of what it deliberately does not do, each with a reason. That list is the most useful part for anyone running a similar setup:
- PAM changes (password quality, lockout after failures): PAM was never under the management tool that the benchmark assumes; enabling it rewrites the authentication stack, and a mistake there locks out every local login. SSH is key-only anyway.
- A boot loader password: protects against physical access, which a VPS does not have, and complicates the provider’s rescue console.
- A custom crypto policy: changes TLS and SSH ciphers system-wide and could break things that are hard to test.
- IP forwarding and reverse-path filtering: required by Kubernetes, see part 2.
What it does apply, each item a small group of tasks:
- AIDE, a file integrity checker: it keeps a database of checksums and attributes of system files and reports every difference. The role installs it, excludes the pod log directories (they change with every pod start and are not a tamper target), builds the first database once (
stat→when: not ... exists, viasystemd-runbecause the SSH sandbox hides parts of the file system), and schedules two cron jobs: the check at midnight, which mails only when it finds something, and a baseline refresh at 04:00 – after the nightly updates and the reboot slot, so that a kernel update installed at night does not show up as thousands of “changed files” the next midnight. That false alarm happened once, with exactly that kernel update. - Kernel modules for old file systems and USB storage are blocked (
install <module> /bin/false). - journald keeps logs persistently and compressed, with a size cap; a handler restarts it when that file changes.
- No core dumps: a crash dump of a privileged process can contain passwords and session tokens.
- cron and at: restrictive permissions, and only root may use them.
- sudo: a pseudo-terminal, a log file and no cached credentials – written with
validate: "visudo -cf %s". This is the most important safety net in the role: a broken sudoers file can make sudo unusable, and the validation guarantees that a broken version never replaces the real one. - The image’s default user: the provider’s image for the Nextcloud servers creates a user with passwordless sudo. The role removes that sudo line (again with
visudovalidation) and gives the user no login shell. It cannot delete the user: the same numeric user ID is used by a process inside a pod, so the host shows a running process “owned” by it.getentreads the user database first, and(getent_passwd | default({})).get('almalinux') is not noneis a safe way to ask “does this user exist?” without an error when it does not. - Unused services and packages: Bluetooth, the rsync daemon (but not the rsync client, which the backups need – an earlier version removed the whole package and broke them), the NFS/Samba client stack the cloud image brings, and cloud-init after provisioning.
- Mount options:
/tmp,/var/tmpand/dev/shmwithnodev,nosuid,noexec– nothing can be executed from there.ansible.posix.mountwithstate: mountedwrites the line into/etc/fstaband remounts immediately;presentwould only write the file. This is why the K3s installer in part 2 is started assh /tmp/k3s-install.shinstead of being executed.
🏷️ common_selinux: labels that contain a break-in
SELinux gives every file and every process a type, and the policy says which process types may touch which file types. A web server process runs as one type and may read web content, but not, say, SSH keys – even if it runs as root and even if the file permissions would allow it. For containers, the rule is simple: container processes run as container_t and may write only files of type container_file_t. A container that breaks out of its own file system finds a host where almost nothing carries that label.
Found while writing this part (19 Oct 2026): for weeks, that rule was only half true here. SELinux was enforcing and the directories carried container_file_t, but K3s starts its container runtime without SELinux support unless its configuration says selinux: true – and it did not. Every pod ran as spc_t or unconfined_service_t, confined by nothing. A look at ps -eZ showed it; no check had ever asked. common_k3s now installs the k3s-selinux policy, sets the option and restarts the workloads once. Since then every application process runs as container_t with its own MCS categories – no pod runs as spc_t any more. The lesson for any SELinux setup: check the domain of the running processes, not just the mode and the labels.
That is also why the order of the playbook matters (part 1): the host directories that are mounted into the pods must carry container_file_t before the pods start. This role creates them – with the owners the containers expect, from the profile of the stack – and labels them:
- name: SELinux | Read the local file-context rules
ansible.builtin.command: semanage fcontext -l -C
register: semanage_fc_local
changed_when: false
- name: SELinux | Label the HostPath directories as container_file_t
ansible.builtin.command: >
semanage fcontext -a -t container_file_t "{{ item.path }}(/.*)?"
loop: "{{ selinux_profiles[stack] }}"
when: >-
(item.path ~ '(/.*)?') not in
(semanage_fc_local.stdout_lines | select | map('split') | map('first') | list)semanage fcontext -a adds a rule to the policy – “this path and everything below it is container_file_t” – which survives a full relabel of the system; restorecon -R then applies the rule to the files that already exist, and new files inherit it. The read-first logic has a recent story: an update of the SELinux tools stopped failing on an existing rule and printed “already defined, modifying instead” to the normal output instead – the old check looked for the error and reported “changed” on every run. Now the role reads the local rules (-C), takes the first column of each line with map('split') | map('first'), and adds only what is missing.
The role’s argument_specs show a feature worth knowing: stack is declared with choices: [wordpress, nextcloud] – a typo in the playbook’s play variable fails before the first task. The role also sets SELinux to enforcing with the module from ansible.posix (the old built-in name only still works through a redirect), and it is honest about one setting: a boolean whose comment used to claim it allowed outbound connections. It does not – outbound connections work without it – and whether anything still needs it is an open question, written down instead of guessed.
🔎 Detection: auditd, AIDE, rkhunter, ClamAV
The layers so far try to keep an attacker out. These four assume someone got in – or that something changed that should not have – and make sure it is noticed. Their common rule: silent when everything is fine, one mail when it is not. A detector that mails every night trains you to ignore it.
common_auditd: the kernel’s audit log, and four keys that mail
auditd records what the kernel is told to watch. The rules file watches writes to user and group files, sudo configuration, the SSH configuration, cron, the network configuration, the audit configuration itself and K3s’s configuration, tokens, certificates and auto-deploy directory; and it records every execution of the module-loading tools and of sudo. Each rule has a key, so ausearch -k <key> finds the events later. A handler compiles the rules into the kernel with augenrules --load when the file changes.
auditd only logs. A small script, every ten minutes from cron, turns four keys into mail: K3s’s auto-deploy directory (K3s applies anything written there – a perfect place for a cluster back door, and this repository never writes there), the cluster token, the cluster certificates, and the user database. A checkpoint per key makes sure each event is reported exactly once. The noisy keys – sudo, modules, SSH config, cron – are deliberately not mailed: they fire during normal administration and Ansible runs, and stay available for investigation.
Update (19 Oct 2026): every AIDE run logged a burst of SELinux denials, because AIDE calls statfs() on the vfat /boot/efi even though it is excluded. A one-rule local policy module, aide_dosfs (allow aide_t dosfs_t:filesystem getattr;), built with checkmodule/semodule_package and loaded only when its source changes, ended that – the hosts now log no AVC at all.
common_aide_refresh: the last role of every run
Every playbook run changes files AIDE watches – a package update alone does. Without a fresh baseline, the next midnight check reports the run’s own changes as an intrusion. So the very last role of each playbook refreshes the baseline, but only if something actually changed:
- name: AIDE refresh | Compare with the baseline and write the new database (one scan)
ansible.builtin.command: systemd-run --wait --pipe --quiet --collect -- aide --update
register: aide_update
changed_when: false
failed_when: false
- name: AIDE refresh | Activate the new baseline (differences were found)
ansible.builtin.command: mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
changed_when: true
when:
- aide_update.rc is defined
- aide_update.rc in range(1, 8)
- aide_db_new.stat.existsAIDE’s exit code is a bit mask: 1 means files were added, 2 removed, 4 changed – so 1 to 7 means “differences found and recorded”, which right after a playbook run is normal. 0 means identical (the new database is thrown away), and 14 and above is an error (reported, baseline left alone). range(1, 8) is Jinja for 1 to 7 – the end is exclusive. failed_when: false is essential: the first version treated a non-zero code as a failure and aborted the play before the new database could be activated. Only the mv reports “changed”, so a refreshed baseline appears exactly once in the recap. And one scan does both the comparison and the new database – the old version scanned everything twice.
There is a trade-off here that an operator should understand: refreshing the baseline accepts all current differences – including any that were not caused by the playbook. The midnight check before the run is what catches those, which is why its mails matter.
common_rkhunter: rootkits, nightly
rkhunter compares system binaries with a stored database of their properties and looks for traces of known rootkits. The role sets its options with lineinfile in a loop over key/value pairs, builds the database (--propupd), and schedules one nightly check that reports warnings only. Two details:
- A small dnf plugin, written by the role into dnf’s plugin directory, runs
--propupdafter every package transaction – nightly automatic updates and manual ones alike. Without it, every update would be a “file changed” warning the next night. - The package ships its own daily cron script, so every server ran two scans a night. Deleting it alone would not last – the next package update would put it back. A one-line rpm macro,
%_netsharedpath, tells rpm never to install that path again.
common_clamav: malware in other people’s files
A nightly ClamAV scan of what users upload – the blog’s media uploads, Nextcloud’s user data – with low CPU and disk priority, mailing only when something is found and never deleting anything by itself. Proving that it actually finds things is an older post. The role picks its settings from a profile like common_selinux, and has one nice piece of Jinja: the list of media extensions to skip is turned into a case-insensitive pattern, letter by letter – jpg becomes [jJ][pP][gG] – because the scanner has no flag for case-insensitive excludes. On Nextcloud, the generated previews are skipped (they are rendered from files that are scanned anyway, and with them a full scan took sixteen hours), and the scan runs at 04:30, after the reboot slot – at an earlier time, the reboot killed it on every update night.
📦 The layers from other parts, in one view
| Layer | Where | In short |
|---|---|---|
| Kernel network hardening | part 2, common_sysctl | no ICMP redirects, no source routing, SYN cookies, ASLR, restricted ptrace, kernel log for root only, RFC 1337, protected files in /tmp |
| Pod hardening | parts 3 and 4 | all capabilities dropped and only the needed ones back, no privilege escalation, read-only root file systems, non-root users where possible, no service-account tokens |
| NetworkPolicies | parts 3 and 4 | database and cache only from the application; Collabora cut off from both |
| Image policy | parts 3 and 4 | only approved image repositories may run |
| Pod Security Standards | parts 3 and 4 | warn and audit, not enforce – host directories would be rejected |
| Audit log of the cluster | part 2 | who did what in Kubernetes, with secrets logged as metadata only |
| Login alarm | own post | a mail for every interactive SSH login |
🛠️ Running it: security day to day
Looking at it
# fail2ban: which jails, how many bans
fail2ban-client status
fail2ban-client status nginx-k3s-bad-paths
# What is really blocked right now (fail2ban and the port-scan detector)
nft list set inet filter banned4
nft list set inet filter banned6
nft list set inet filter scanban4
# Audit events of one key
ausearch -k k3s_manifests -i
# The whole firewall as it is loaded
nft list table inet filterMails from this part
| Meaning | What to do | |
|---|---|---|
| SSH login alarm | Someone logged in interactively | If it was not you: act now – check last -i, rotate keys |
| AIDE found file integrity changes | System files differ from the baseline | Read the list. After a planned change: expected. Otherwise: investigate before the next playbook run refreshes the baseline |
| auditd event (k3s_manifests, token, tls, identity) | Something that should never change did | Treat as an incident until explained |
| rkhunter warnings | A binary or setting looks suspicious | Often a package change – compare with recent updates; otherwise investigate |
| ClamAV found something | A file in the uploads or user data matched a signature | Look at the file and who uploaded it; ClamAV deleted nothing |
When you lock yourself out
- Banned your own address:
fail2ban-client set <jail> unbanip <address>from another connection, or wait for the ban to expire. For the port-scan ban:nft delete element inet filter scanban4 { <address> }. Never test attack paths from the machine you administer from – I banned myself twice. - Outside the allowed country: add your address to
fw_ssh_geo_extraand run the firewall role from inside the allowed area – or use the provider’s web console. - SSH does not come back after a change: the provider’s web console with the root password. This is why part 2 says: test the console before you need it.
🧭 What comes next
Part 6 closes the tour with monitoring: the exporters, Prometheus, Grafana with its alarms, and the Monit watchdog – what each of them watches and what their mails mean.


