🌐 Also available in: Deutsch · Français · Español
An integrity checker is only as good as the mails it sends. If it reports hundreds of changes every morning, nobody reads them after a week – and the one real finding gets lost with them. That was my problem with AIDE and rkhunter on three K3s hosts: both worked, but mostly they reported things I had caused myself.
This deep dive shows what the two tools do, where the noise comes from on a Kubernetes host, and how the baseline follows every intended change.
🧾 What AIDE does
AIDE (Advanced Intrusion Detection Environment) builds a database: for every watched file it stores checksums, permissions, owner, size and timestamps. The daily check compares the file system with that baseline. If someone changes a program in /usr/bin or a config in /etc, it shows up – even if the attacker cleaned the logs.
Mine runs once a day and only mails when AIDE actually finds differences. Two details make it usable:
- Several threads: AIDE 0.19 can hash in parallel. With one thread a run took 38 seconds to one and a half minutes; now it uses one thread per vCPU, at low priority.
- The exit code is a bitmask: 1 = files added, 2 = removed, 4 = changed. Values from 1 to 7 therefore mean “differences found”; only 14 and above is an error of AIDE itself. Treat it like an ordinary error code and you abort exactly when AIDE has done its job – which is what my first version did.
The takeaway: AIDE does not say “good” or “bad”, it says “different from yesterday”. Whether that is a finding depends on whether the baseline is current.
🐳 Where the noise comes from on a K3s host
A Kubernetes host changes constantly without anything suspicious going on. Three places regularly produced false alarms for me:
/var/log/pods: K3s creates a subdirectory for every pod start and every CronJob run. That changes the parent directory’s link count – AIDE reported it practically every day./var/log/containers: one symlink per container pointing into/var/log/pods. If a CronJob pod is cleaned up during the check, the link dangles, and the mail started with “failed to access …”./boot/efi: a FAT file system that AIDE is not allowed to read under SELinux. It skipped the partition anyway – but every run produced a batch of SELinux denials, 13 per run on one host.
The first two are no target for tampering: they are ephemeral captures of container output, not programs and not configuration. So they are excluded, each with its reason in the code. For /boot/efi, excluding is not enough: AIDE still queries the file system’s properties with statfs(). A small local SELinux module allows exactly that one call – reading file system metadata, not a single file on it. The EFI files come from packages, so rpm -V checks them if needed.
The takeaway: exclusions are fine as long as each one has a reason. “It’s annoying” is not one – “it’s ephemeral and no attack target” is.
🔄 The baseline has to move along
Most false alarms, however, came from me. Every intended change – a package update, a new firewall rule, a new K3s binary – is a deviation to AIDE at first. On my hosts there are three routes by which watched files change, and each one needs its own refresh:
- Ansible runs: the last role of every playbook is
common_aide_refresh. It runsaide --update, which checks and writes the new database in a single pass. It used to be a check followed by an update, so two full scans per run. - Nightly security updates: dnf-automatic installs updates at night, without Ansible. After a kernel swap, AIDE reported 3,273 added, 3,273 removed and 364 changed files the next morning – old kernel out, new kernel in. Since then a separate refresh runs at 04:00, after the update window and the possible reboot at 03:30.
- Changes by hand: anyone who changes things outside these two routes has to trigger the refresh themselves. My rule: after every playbook run with
--tags, also run--tags common_rkhunter,common_aide_refresh– a tagged run skips the last role.
The third route is the tricky one. When I recently installed a kernel by hand in the afternoon, the midnight check – before the 04:00 refresh – would have reported exactly that kernel.
The takeaway: every route by which a system changes needs a matching baseline refresh. Miss one and the false alarm arrives reliably.
🦠 rkhunter: a second opinion with its own quirks
rkhunter (Rootkit Hunter) works in a similar way but with a different focus: it compares important system programs with its own property database and looks for traces of known rootkits. It also needs a current baseline, which rkhunter --propupd creates.
Three things were needed before it went quiet:
- A dnf plugin instead of a timer hook: after every dnf transaction that actually changed packages, a small plugin runs
rkhunter --propupd. My first solution hung off the dnf-automatic service, so it only fired for nightly updates, not for a manualdnf upgrade. - One scan per night: the EPEL package ships its own script in
/etc/cron.daily. Together with my own cron entry, every host ran two full scans a night – and a warning meant two mails. The packaged script is gone, and an rpm setting (%_netsharedpath) keeps the next rkhunter update from putting it back. - Disabled tests, each with a reason: some tests do not fit AlmaLinux 9 or K3s – the SSH configuration test with a known bug in rkhunter 1.4.6, for example, or the module tests that look in the wrong path after a kernel swap. Every disabled test is listed with its reason in the role.
One detail for the drift principle: --propupd refreshes a derived database, not a configuration. The task therefore never reports “changed” – otherwise every playbook run would look like drift.
The takeaway: a package sometimes brings its own idea of how to operate. If you schedule things yourself, switch the bundled schedule off deliberately – and make sure it does not come back.
💓 Who checks the checker?
A quiet integrity check is only good if it actually runs. “No mail” can mean two things: all is well – or the job no longer runs. So each of these jobs writes a timestamp (a heartbeat) for Prometheus at the end, and an alert rule fires when one of them gets too old.
Setting up the heartbeats brought two old mistakes to light:
- On one host a hand-made second AIDE check ran at 05:00 – in no role and no repository, single-threaded and redundant.
- Mails to the bare recipient
rootnever arrived. The relay accepted them, but msmtp has no local aliases, and without a domain the mail went nowhere. Since then every alert goes to a full address.
The takeaway: silence is no proof. A checking tool needs its own check that it runs and that its mails arrive.
🎯 The lessons
- An alarm has to mean something – otherwise nobody reads it.
- Exclusions only with a reason – ephemeral container logs yes, “annoying” no.
- Every change route needs its refresh – Ansible, nightly updates and manual work.
- Check bundled schedules – a package can bring a second scan.
- Monitor the checker – a heartbeat per job, mails to real addresses.
More background on SELinux on these hosts is in the SELinux deep dive, the job monitoring in part 6 of the Ansible tour.
