Back From Sweden, Straight Into a CIS Benchmark Rabbit Hole

🇸🇪 I’m back from Sweden. Tan: nonexistent (it’s Sweden). Mosquito bite count: illegal. Motivation to touch my homelab again: through the roof. There’s something about two weeks of forced screen-free time in a forest that makes you come home and immediately want to ssh into a Kubernetes cluster at 11pm on a Wednesday. Don’t diagnose me. 💻

So the last few evenings I’ve been doing what any well-adjusted person does after a vacation: 📚 reading into Kubernetes security material, specifically the Certified Kubernetes Security Specialist (CKS) curriculum. To be clear — I’m not currently planning to actually sit the exam, I’m just nerding out on the topic because it’s genuinely interesting and directly relevant to a cluster I run in production (well, “production” — it’s a blog). CKS is the “okay but can you actually secure the thing” syllabus in the Kubernetes world — RBAC hardening, supply chain, runtime security, network policy, the whole buffet of “things that are fine until they very much aren’t.” And somewhere in that rabbit hole I re-discovered a tool I’d used once years ago and completely forgotten existed: kube-bench 🔍.

🧐 What is kube-bench, and what’s a CIS Benchmark anyway?

kube-bench is a small Go binary from Aqua Security that checks your Kubernetes cluster’s configuration against the CIS Kubernetes Benchmark — a big, boring, extremely useful checklist published by the Center for Internet Security that says things like “hey, your API server’s audit logging should probably be turned on” or “your kubelet certificates shouldn’t be world-readable, genius.” It doesn’t hack anything, it doesn’t scan for CVEs, it just walks through hundreds of very specific, very pedantic configuration checks — file permissions, API server flags, RBAC bindings, pod security settings — and tells you PASS, FAIL, or WARN for each one.

The nice thing about kube-bench specifically is that it ships with distribution-aware profiles. Vanilla kubeadm clusters look different from EKS, which looks different from GKE, which looks very different from k3s (my distro of choice for this homelab, because who has RAM for a “real” control plane). I ran it with the k3s-cis-1.9 profile against the k3s single-node cluster that’s currently hosting, well, this exact blog.

🚦 Round one: 55 pass, 4 fail, 57 warn

First run came back with ✅ 55 PASS, ❌ 4 FAIL, ⚠️ 57 WARN, ℹ️ 14 INFO. Not catastrophic, but not nothing either. Quick PSA before I go further: most of those 57 WARNs are checks kube-bench marks “Manual” — meaning the tool literally cannot verify them automatically and just always prints WARN no matter how well-configured you actually are, because it needs an actual human to look at the thing and use judgement. So a big WARN count isn’t automatically “58 problems,” it’s more “58 things a human should glance at, some of which are already fine.” I did glance at a bunch of them. More on that below. 👇

Here’s the tour of what got fixed, what turned out to be a non-issue, and — because no infrastructure change survives contact with reality — the couple of genuinely funny ways I broke things while fixing other things. 🙃

1️⃣ 🔐 Certificate files with the permissions of a public park bench

k3s writes its internal PKI certificates (the ones under /var/lib/rancher/k3s/server/tls/) as 644 — world-readable. CIS wants 600. To be fair, these are the public certs, not the private keys (those were already locked down correctly), so the actual risk here was closer to “slightly untidy” than “gaping hole.” Fixed with a chmod, and since k3s occasionally regenerates some of these on restart, I made the fix idempotent in the Ansible role so it re-asserts itself on every playbook run instead of quietly reverting.

2️⃣ 🎫 ServiceAccount tokens mounted into pods that will never once call the Kubernetes API

This one’s a classic. By default, every pod gets a Kubernetes API token auto-mounted into its filesystem, “just in case.” My WordPress pod, MariaDB pod, and Redis pod never talk to the Kubernetes API in their entire lifecycle — they serve a blog, store rows, and cache things, respectively. But they all had a live, if low-privilege, API token sitting in /var/run/secrets/kubernetes.io/serviceaccount/ anyway, purely as a “why not.” If an attacker ever popped one of those containers (say, via a dodgy plugin RCE — it’s WordPress, it happens 😅), that token was free real estate for reconnaissance.

Set automountServiceAccountToken: false everywhere it wasn’t needed — on the pods themselves and, as I found out the hard way, on the underlying ServiceAccount objects too, because kube-bench’s automated check apparently inspects the ServiceAccount, not just whether some pod happens to override it. Patched that cluster-wide for every non-system namespace while I was at it. 🧹

3️⃣ 🕵️ RBAC deep-dive: expected to find something, found nothing

CIS has a whole batch of checks around RBAC minimization — wildcard permissions, who can impersonate whom, who can create pods, etc. I went in expecting to find at least one over-broad binding somebody (me, six months ago, at 1am 🌙) had left lying around. Nope. Every custom role in the cluster turned out to be either a completely unused stock Kubernetes default (admin/edit, bound to nobody) or a role installed by a Helm chart (cert-manager, ingress-nginx) scoped exactly to what that component needs and nothing more. Genuinely satisfying to confirm rather than assume. ✨

4️⃣ 🧱 NetworkPolicies: the one I’d been putting off

Here’s a fun bit of homelab history: my Nextcloud stack has had NetworkPolicies enforced via Calico for ages, but the WordPress/blog stack never got the same treatment. The reasoning at the time was that Collabora (Nextcloud’s document editor) has an obvious, clean threat model — it parses untrusted documents and has zero legitimate reason to ever touch the database, so locking it down was a no-brainer. WordPress felt murkier: the WordPress pod does legitimately need to talk to both MariaDB and Redis, so “just deny everything” isn’t the right shape of policy.

But murkier isn’t the same as pointless. The actual value here isn’t isolating WordPress from its own database — it’s making sure that MariaDB and Redis are reachable only from WordPress, not from every other pod that might ever land in this cluster, and capping WordPress’s own blast radius if it ever gets popped. So I finally wrote three NetworkPolicies: WordPress can reach MariaDB, Redis, DNS, and the open internet on 443 (for plugin/core updates — more on why that last one can’t be locked down tighter in a second); MariaDB and Redis can only be reached by WordPress and nothing else.

One honest limitation I’ll cop to publicly, because hiding it would be dishonest and also pointless: standard Kubernetes NetworkPolicy can only filter by IP/CIDR, not by domain name, and my Calico install runs in the free “policy-only” mode with no fancy domain-aware rule sets. WordPress core, its plugins, and my WP-CLI bootstrap all need outbound HTTPS to a rotating cast of CDN IPs (wordpress.org, GitHub’s release CDN, etc.) that I simply cannot enumerate. So port 443 to the open internet stays allowed. What this policy actually buys me is blocking lateral movement inside the cluster — a compromised WordPress pod can’t go poke at cert-manager or the ingress controller’s admin surface — not a full outbound lockdown. I’d rather tell you the honest scope of a mitigation than let you assume it does more than it does. 🙏

5️⃣ 🐛🐛 Two bugs that only showed up because I actually tested things instead of trusting green checkmarks

This is the part of the post where infrastructure work stops being a checklist and starts being an actual investigation, and honestly it’s the part I had the most fun with. 🕵️‍♂️

After deploying the NetworkPolicies, I manually triggered the WordPress cron job (a Kubernetes CronJob that runs wp-cron.php every 5 minutes) to make sure nothing broke. It reported STATUS: Complete ✅. Great, ship it — except I actually checked the logs instead of trusting the green status, and found a full WordPress fatal error page: “Error establishing a Redis connection.” 💥 Every single cron run. Silently. Forever, apparently, because a PHP wp_die() exits with status 0, so Kubernetes has absolutely no idea anything went wrong. “Complete” just meant the process ended, not that it succeeded. Lesson relearned the hard way: a green status is a claim, not a proof.

Digging in, there turned out to be two separate bugs stacked on top of each other:

• 🐛 Bug #1 — the real, pre-existing one, unrelated to anything I did today: the CronJob’s container definition never had the WORDPRESS_CONFIG_EXTRA environment variable set — the block that defines WP_REDIS_HOST and friends — unlike the main WordPress deployment. So WP_REDIS_HOST was undefined on every single cron run since forever, the Redis object-cache plugin quietly fell back to its own default of 127.0.0.1, and obviously nothing’s listening on localhost inside that pod. This had nothing to do with NetworkPolicies, Calico, or anything from today — it’s been broken since the CronJob was written, and it just never surfaced because “Complete” lied to everyone, including me, for who knows how long 🙈. Fixed by copying the exact same config block from the main deployment.
• 🐛 Bug #2 — the genuinely new one, caused by the NetworkPolicy I’d just added: once I fixed bug #1 and pointed the cron pod at the correct Redis hostname, it still failed — but this time with a plain “Connection refused” instead of a WordPress fatal error, which is a very different flavour of failure and immediately smelled like a network problem rather than a config problem. Turns out: a brand-new pod’s very first outbound connection attempt, within roughly its first second of existing, can get rejected because Calico’s Felix agent hasn’t finished programming that specific pod’s firewall rules yet ⏱️. Long-lived pods never notice this because their readiness probes have a generous 20-second grace period before anyone checks on them. A CronJob pod that tries to talk to Redis in the first half-second of its life has no such luxury. I reproduced this reliably with an isolated test (bare TCP connect, no WordPress involved) — fails instantly on a fresh pod, succeeds every time with a 5-second sleep first. Fix: a sleep 5 && stuck in front of the actual cron command. Not elegant. Extremely effective. 🛠️

✅ Both fixes are now live and verified against a real, unforced, actually-scheduled cron tick, not just a manual test run.

6️⃣ 🤦 The one where I accidentally undid a security default while adding a different one

My favourite self-own of the whole exercise. I added the EventRateLimit admission plugin to the API server to stop a crash-looping pod from being able to flood the event stream. Standard vanilla Kubernetes treats --enable-admission-plugins as additive — it adds your plugin to whatever’s already enabled by default. k3s, it turns out, does not share this philosophy: setting that flag replaces its default list wholesale. Which means my one-line “let’s add some rate limiting” change silently switched off NodeRestriction, a default k3s admission plugin that stops a compromised kubelet from tampering with API objects it shouldn’t touch. I only caught this because I re-ran kube-bench afterward as a sanity check and watched a previously-passing check flip straight to FAIL 📉. Moral of the story: always list your defaults explicitly when you touch a flag like this, and always re-run your validation tool after every single change, not just once at the very end.

7️⃣ 🗝️ Secrets as files instead of environment variables

Database passwords were sitting in plain environment variables inside the containers — readable via /proc/<pid>/environ, kubectl describe, crash dumps, or any error handler dumb enough to dump its environment on a fatal error. Both the official WordPress and MariaDB Docker images support pulling secrets from a file path instead (the _FILE suffix convention), so I switched everything over. The one non-obvious gotcha ⚠️: MariaDB’s own health-check probes were reading the root password straight out of that same environment variable. Convert the variable to a file without touching the probes, and MariaDB would’ve started reporting itself as permanently unhealthy the moment I rolled it out. Caught it during review, updated the probes to read the file directly, tested it live before calling it done. 👍

8️⃣ 📦 Image provenance, the honest way

CIS wants “Image Provenance using ImagePolicyWebhook” — which is a real Kubernetes feature, but implementing it means standing up and hosting an entire external webhook service to approve or deny images at admission time. That’s a legitimate amount of new infrastructure for a homelab blog, and I wasn’t about to build a whole microservice just to make a compliance checkbox happy 🙅. Instead I used ValidatingAdmissionPolicy — a native, built-into-the-API-server mechanism (GA since Kubernetes 1.30, no extra moving parts) — to reject any pod in the WordPress namespace whose container image isn’t on an explicit allowlist of the exact repositories this stack actually uses. Same control objective, zero new infrastructure to babysit at 2am. 😴

🤷 What I deliberately left as-is

Not everything that shows up red needs fixing, and I want to be upfront about what’s still flagged and why — because “we fixed literally everything” is usually a sign nobody looked closely enough, and because none of the following actually hands an attacker anything useful:

• 🗄️ An “etcd” check that doesn’t apply to me at all. This is a single-node k3s cluster using its embedded SQLite datastore, not etcd. A CIS check asking about an etcd CA file is checking for a component that literally isn’t running here. Not a finding, just a benchmark being generic.
• 🧩 A wildcard RBAC permission that belongs to Calico itself, needed for it to manage its own custom resources. It’s the visible cost of turning on the NetworkPolicy enforcement described above — hand-trimming it risks the network policy engine itself failing to start, which felt like a worse trade than “one wildcard grant on a component that’s already trusted with cluster networking.”
• 🔧 A handful of components that still have their Kubernetes API tokens mounted — specifically cert-manager, my ingress controller, and Calico’s own controllers. All three of them need that access to do their actual jobs (watching for certificates, ingress objects, and network policies respectively). Ripping those out wouldn’t harden anything, it would just break TLS issuance.

🏁 The scoreboard

Started the evening at ❌ 55 PASS / 4 FAIL / 57 WARN / 14 INFO. Ended it at ✅ 63 PASS / 3 FAIL / 50 WARN / 14 INFO, with the 3 remaining FAILs all consciously accepted for the reasons above rather than silently ignored. Along the way I fixed one security regression I’d introduced myself within the same session, found and fixed one completely unrelated pre-existing bug that’s probably been silently failing for months, and learned that “Job: Complete” and “the thing inside actually worked” are two very different claims that Kubernetes will happily let you conflate if you don’t check. 🎯

📤 One more housekeeping note for anyone following along at home: I’m currently also doing a bit of restructuring on the GitLab side of my homelab, so the Ansible code and documentation changes described here are sitting in my private repo for the moment — I’ll get them pushed up to my public GitHub mirror over the next few days once that’s sorted.

Back to the CKS reading list now. 📖 Turns out reading about Kubernetes security and actually going and finding real findings in your own cluster hit different. Would recommend. ✌️