Same evening, different layer of the stack. 🥱➡️🤓 After the kube-bench post I got a very reasonable follow-up question in the comments-that-live-in-my-own-head: “okay but what about the actual Linux box underneath all this Kubernetes stuff?” Fair point. kube-bench only ever looks at the cluster — API server flags, RBAC, pod security. The AlmaLinux 9 host it’s all running on top of never got the same treatment. So, one more late-night detour. 🐧🔍
🧐 OpenSCAP and the CIS AlmaLinux 9 Benchmark, briefly
OpenSCAP is the Linux-distro equivalent of what kube-bench does for Kubernetes — except older, more established, and backed by an actual official standard (SCAP, Security Content Automation Protocol) rather than a single vendor’s opinionated checklist. Red Hat ships a package called scap-security-guide that bundles CIS, DISA STIG, PCI-DSS, HIPAA and a pile of other compliance profiles as machine-readable XCCDF content, and oscap xccdf eval walks through every rule in a chosen profile and tells you PASS, FAIL, or NOTAPPLICABLE. Same spirit as kube-bench: hundreds of pedantic, specific checks — file permissions, kernel parameters, PAM config, mount options — not a vulnerability scanner, a configuration auditor. 🛠️
I ran it against the CIS AlmaLinux OS 9 Benchmark, Level 1 – Server profile, read-only first (--profile, no --remediate), because unlike a throwaway test VM this box is currently serving the actual blog you’re reading this on, and I wanted to see the full picture before touching anything live. 🙏
🚦 Round one: 151 pass, 113 fail
293 checks total: ✅ 151 PASS, ❌ 113 FAIL, ⚪ 27 N/A, ⚠️ 2 ERROR. Noticeably worse ratio than the Kubernetes layer — makes sense, nobody had ever pointed a CIS benchmark at this specific host before, whereas the k3s side had already had a full pass. 113 is a lot to triage by hand, so here’s the tour: what got fixed, what got a hard pass with reasons, and — because apparently this is just how my evenings go now — the two genuinely interesting “wait, why did that just lie to me” moments along the way. 🕵️
1️⃣ 🗃️ AIDE — file integrity monitoring I somehow never had
I’ve got rkhunter for rootkits and ClamAV for malware, but nothing that would notice if someone quietly edited /etc/passwd or swapped out a system binary. AIDE (Advanced Intrusion Detection Environment) fills exactly that gap: it fingerprints the filesystem once, then compares against that baseline on a schedule and yells if anything unexpected changed. Installed it, built the initial database (a full filesystem walk, ~30 seconds, ~51,000 files fingerprinted), scheduled a daily check via cron. 📅
Small hiccup building the baseline cleanly: my first two attempts both reported 9,279 removed entries, all under one specific old kernel version’s module directory — despite that directory very much still existing on disk. Explanation, once I chased it down: this SSH session has ProtectKernelModules=yes applied by sshd’s own systemd hardening (yes, the same hardening I documented for this exact blog months ago), which makes /usr/lib/modules look completely empty from inside an interactive SSH session — a private, sandboxed illusion, not reality. I’d built the database correctly (escaping the sandbox via systemd-run), but then eyeballed it with a plain aide --check run directly in my SSH session, which saw the fake empty version and panicked about “missing” files that were never actually missing. 🙈 Once I re-ran the check the same escaped way as the init, it came back clean: zero differences. The lesson generalizes past AIDE: if a check run over SSH ever reports something bizarre about kernel modules or /proc/sys, ask whether you’re looking at the sandbox’s idea of the filesystem before you believe it.
2️⃣ 🧱 Kernel-level network hardening — and the one sysctl I deliberately did NOT touch
Added the standard battery: reject ICMP redirects, refuse source-routed packets, log “martian” packets with impossible source addresses, ignore broadcast ICMP echoes (smurf-attack mitigation), enable TCP SYN cookies, restrict ptrace to a process’s own descendants, full ASLR. All boring, all safe, all shipped. ✅
What I didn’t touch, on purpose, after actually checking first: net.ipv4.ip_forward and reverse-path filtering (rp_filter). CIS wants forwarding off and strict reverse-path filtering on. My cluster needs the exact opposite — Flannel and Calico require IP forwarding to route pod traffic at all, and I found this host’s rp_filter already sitting at 0 instead of the distro default of 1, almost certainly because strict reverse-path filtering is a well-documented way to make a Kubernetes node silently drop legitimate pod traffic. Flipping either of those “for compliance” would have been the fastest way to take the whole cluster offline for a checkbox. Left both alone, and I mean genuinely tested-and-verified alone: after applying the rest of the sysctl batch, a full sysctl --system re-apply cycle briefly reset the per-interface rp_filter values for the CNI interfaces back to 1 anyway (a distro default file reasserting itself), so I immediately fired a real pod-to-pod-to-database round trip through a live CronJob to check nothing broke. It came back clean. Kept an eye on it; didn’t have to revert anything, but I wanted the receipts before writing this sentence, not just the theory. 🧪
3️⃣ 🔒 Locking down /tmp, /var/tmp, /dev/shm — and the installer script it broke
Bind-remounted all three with nodev,nosuid,noexec — nothing should be running executables out of a world-writable scratch directory. Before flipping the switch I went hunting for anything in my own Ansible code that executes a file living in /tmp, because that’s exactly the kind of thing noexec silently breaks. Found one: the k3s installer bootstrap downloads get.k3s.io‘s install script to /tmp/k3s-install.sh and then runs it directly by path. Direct execution needs the exec bit on the mount; reading the same file as an argument to sh doesn’t. One-line fix (sh /tmp/k3s-install.sh instead of /tmp/k3s-install.sh) before flipping noexec on, so a future fresh install of this exact playbook doesn’t quietly break on a machine I’m not even watching. Verified post-change: read/write in /tmp still fine, kubectl apply -f /tmp/whatever.yml still fine (reading YAML as data, not executing it), and a deliberately-planted executable test script got a clean, correct “Permission denied.” 🎯
4️⃣ 🪵 journald, core dumps, cron/at permissions, sudo hardening, Bluetooth, rsync
The unglamorous-but-easy batch: journald now persists to disk (compressed, capped at 500MB so “persistent” doesn’t quietly become “fills the disk”), core dumps are disabled everywhere they could be generated (a core dump is just a memory snapshot, which for a box running a database and an admin-panel login is a memory snapshot that might contain a plaintext password mid-flight — not something I want lying around even locally), cron/at directories got tightened permissions and an explicit allow-list, sudo now logs to its own file, requires a pty, and asks for re-authentication every single time instead of caching it. Bluetooth got masked (a VPS has no Bluetooth hardware, the service was just running for no reason), and rsync got uninstalled after I actually went and checked — not a single script, cron job, or Ansible task on this host references it. If I ever need it again, it’s one dnf install away. 🧹
5️⃣ 🎭 The SSH findings that were already true, and the one systemd-run couldn’t survive
Here’s the fun one. Several SSH checks came back FAIL for settings I was pretty sure I already had right — root login restrictions, host-based auth, environment options. I double, then triple-checked the actual live sshd config via sshd -T (the command that shows what sshd would actually enforce, not just what’s written down), both from a plain SSH session and from one escaped via systemd-run to rule out yet another sandboxing illusion like the AIDE one above. Identical, correct results both times. So this wasn’t a sandbox lie — it turned out to be a genuinely different kind of gap: OpenSSH already defaults to the secure behavior for a few of these, but I’d never written the directive down explicitly, and a strict compliance scanner checks the literal config file, not “well, the default happens to be fine.” Added the three missing lines outright — belt-and-suspenders, and a future OpenSSH update changing a default wouldn’t silently change my posture without me noticing.
One I deliberately left exactly as-is: CIS wants PermitRootLogin no — root SSH access blocked completely, full stop. I run PermitRootLogin without-password — key-only, but still root. There’s no separate sudo-capable admin account on this box; Ansible connects as root directly. Disabling root login entirely without first introducing a dedicated admin user would have been an extremely efficient way to lock myself out of the only access path to a server that only I administer. Noted in the code, left alone.
And the actually-technical footnote for anyone who tries to reproduce this workflow: running the full OpenSCAP scan itself wrapped in systemd-run --wait --pipe (to dodge the SSH sandbox, same trick as the AIDE fix) reliably failed with “Remote peer disconnected” the moment I backgrounded the SSH command that launched it. Best guess: systemd-run‘s DBus session ended up tied to the login session PAM created for that SSH connection, and once I detached from it (even via plain nohup + disown at the shell level), the DBus link died with it. Long-running systemd-run jobs and “fire it and walk away over SSH” apparently don’t mix. Worked around it by verifying the handful of affected rules individually with short-lived systemd-run calls instead of forcing the whole multi-minute scan through it.
🤷 What I deliberately left alone
Same principle as the kube-bench post: a red result isn’t automatically a to-do, and none of the following hands an attacker anything they didn’t already have:
- 🔐 PAM / authselect / password-quality / account-lockout policy — the biggest cluster of remaining FAILs, and the one I’m most deliberately not touching yet.
authselectreported “no existing configuration detected” on this host, meaning PAM here has never been brought under its management at all. Enabling it now means structurally rewriting/etc/pam.d/system-authandpassword-auth— get that wrong and I could lock out every local login,su, andsudopath on a server I have no confirmed rescue-console access to. SSH itself is already key-only, so the actual security upside is modest next to a genuinely catastrophic downside. This gets its own carefully tested maintenance window, not a Thursday-evening drive-by. - 🔑 GRUB2 bootloader password — a control against physical console access, which mostly doesn’t apply to a rented VPS, and risks fighting with the hosting provider’s own rescue-console tooling.
- 🔏 Custom system-wide crypto policy for CIS — changes what TLS/SSH ciphers the whole box will accept. Real potential to break compatibility with something (an old client hitting the blog, my own tooling) without dedicated testing first. Not a Thursday-evening drive-by either.
- 📡 systemd-journal-remote — CIS wants it installed for centralized log shipping. I have nothing configured to receive those logs. Installing a daemon nothing uses doesn’t harden anything, it just adds a bigger attack surface for zero benefit.
🏁 The scoreboard
Started at 🌙 151 PASS / 113 FAIL / 27 N/A / 2 ERROR. Ended at 🎉 208 PASS / 56 FAIL / 27 N/A / 2 ERROR — roughly halved the FAIL count in one evening, with everything remaining either a deliberate, documented trade-off or a genuine “needs its own careful pass” item, not an oversight. Between this and the kube-bench pass, both layers of this stack — the cluster and the box it’s standing on — have now actually been audited against a real standard instead of “seems fine, ship it.” 📈
Same housekeeping note as last time: this is all sitting in my homelab GitLab and getting mirrored to GitHub as I go, so if you want to see the actual Ansible role behind any of this, it’s public. 🐙
Two audits down. CKS reading list, you’re still not off the hook. 📖✌️
