Ten Points, One Branch: Taking My Ansible Code From “It Works” to “It Is Clean”

🌐 Also available in: Deutsch · Français · Español

It started with a slightly vain question. My Ansible repository runs this blog and two Nextcloud servers: 34 roles, four playbooks, about 9,000 lines of YAML, almost 200 commits. I didn’t write most of those lines myself – my AI co-admin (Claude) did, while I decided, tested and ran the playbooks. So I asked it: if I understand most of this code, am I an advanced Ansible user?

The answer was honest and a little uncomfortable: advanced at reading and operating it, yes. Advanced at writing it from scratch, not proven yet. And then it listed ten things that would make the code itself better. I said: do all ten. Test them properly. Then write about it. This is that post.

Ground rules, same as always: everything happens on a separate Git branch, nothing touches a production server until I run it, and every claim gets tested. “Tested” turned out to be the interesting part.

📋 The ten points

Halfway through I added an eleventh wish: every version pin in one place, so I can see at a glance what runs where. More on that under point 10.

🧪 First problem: how do you test Kubernetes code without touching production?

There used to be a test server in the inventory. When Claude tried to reach it, SSH refused: host key verification failed. The IP probably belongs to someone else by now, so it deliberately did not connect. And I don’t hand out my vault password in a chat, so no playbook could run against the real servers either.

The solution was neat: a throwaway K3s in rootless podman on my jump host. A full K3s failed immediately (failed to find cpuset cgroup – rootless containers don’t get that controller delegated). But to test whether manifests are applied idempotently, you don’t need running pods at all, only the API server. K3s has a flag for exactly that:

podman run -d --name k3s-test --privileged -p 127.0.0.1:16443:6443 \
  docker.io/rancher/k3s:v1.36.5-k3s1 server --disable-agent \
  --disable traefik --disable servicelb --disable metrics-server

A small stand-in script plays k3s kubectl: it forwards everything to the container and answers the commands that would need running pods (exec, rollout status, wait) with “fine”. The test then does what matters: apply everything with the old code from main (that’s the state the servers are in), then run the new code on top – dry run, real run, real run again, dry run again.

1️⃣ kubernetes.core instead of kubectl apply

The old pattern appeared 29 times: render a template to /tmp, kubectl apply -f, delete the file, and guess “changed” by searching the output for the word configured. The new pattern renders in memory and talks to the API directly:

- name: Redis | Apply Deployment + Service
  kubernetes.core.k8s:
    kubeconfig: "{{ k3s_kubeconfig }}"
    state: present
    apply: true
    definition: "{{ item }}"
  loop: "{{ lookup('ansible.builtin.template', 'redis-deployment.yml.j2') | ansible.builtin.from_yaml_all | select | list }}"
  loop_control:
    label: "{{ item.kind }}/{{ item.metadata.name }}"

Three things only came out because of the test:

• The first dry run said “changed” for almost everything – without a diff. It turns out the module shows no diff when you hand it a list of objects. One loop iteration per object fixed it, and now the diff says exactly image: redis:7.4.11-alpine → redis:7.4.12-alpine.
• The one-time migration only rewrites the last-applied-configuration annotation (kubectl writes an empty "annotations":{} into it, the module doesn’t). Spec and pod template stay identical, so no pod restarts – checked via the ReplicaSet revisions, all still at 1.
• Helm: helm upgrade --install used to run on every play and stacked up a new revision each time. kubernetes.core.helm compares chart version and values and does nothing if they match. Catch: Helm 4 is only supported from collection version 6.5.0; the distro package shipped 6.1.0.

Side effect I like a lot: Kubernetes Secrets no longer pass through a file in /tmp on the server.

2️⃣ A dry run that means something

In check mode, Ansible skips every shell/command task – including the harmless ones that only read something. Later tasks that evaluate those results then fail or decide on nothing. 94 pure queries (version checks, occ config:system:get, systemctl is-active …) now carry check_mode: false. Everything that writes stays skipped – and where that would hide a pending change, a small extra task reports it.

3️⃣ Lint, secret scan, hooks

The first ansible-lint run reported 811 findings. About 600 were style rules that contradict deliberate choices (aligned columns, shared variable names without a role prefix) – those are switched off with a written reason. The rest got fixed. Two real finds:

• ansible.builtin.ini_file doesn’t exist. It only worked through a legacy redirect to community.general – a collection that wasn’t even in requirements.yml. On a controller with plain ansible-core, the Nextcloud playbook would have stopped right there.
• 27 shell pipes without pipefail. Mostly harmless, but in the version check, k3s --version | awk … || echo 'not installed' could never print “not installed”: without pipefail only awk‘s exit code counts.

gitleaks scanned all 191 commits of history: one hit, a PASTE_CONTENT_OF_id_rsa_HERE placeholder between two key markers in an old example file. Accepted on purpose, documented. The hooks run on every commit now – and the first commit they blocked was Claude’s own, a few hours later (wrong YAML indentation in generated files). Exactly what they’re for.

4️⃣ Idempotency, tested

The podman test is now part of the repo (tests/k8s/run.sh). It fails if the second run still changes anything, a pod would be rolled out, a Helm release gets a new revision – or if a deliberately changed image tag is not detected. That last one matters: a test that can never fail proves nothing.

For the real servers there is scripts/check-drift.sh <playbook> <host>: a dry run that prints only what would change, with a readable diff. Right after a real run it must say NO DRIFT.

On the way, a few tasks that reported “changed” on every run got fixed (signature download, a database user, a dnf module). The sneakiest find was the opposite: Nextcloud’s nine SMTP settings were rewritten on every run and always reported “ok”. A changed mail password would never have shown up as a change. Now they are read first, compared, and only differing values are written – passed via argv, so a quote in a password can’t break the command.

5️⃣ When an update fails halfway

Claude’s first suggestion was “leave maintenance mode cleanly in the rescue”. On closer thought it corrected itself: after an aborted occ upgrade, switching maintenance mode off puts users on a half-upgraded instance. The logic now:

• A pre-flight check outside the block: installed, not in maintenance mode, no pending DB upgrade – otherwise the playbook refuses to start. Outside on purpose: the rescue must only deal with states the run itself caused.
• The rescue collects occ status, pods, events and the last lines of nextcloud.log, and switches maintenance mode off only if needsDbUpgrade is false. Then it stops, so no second server gets touched.
• Bonus: the update playbooks used to restart Nextcloud and Collabora on every run. A small new role compares the digest of each freshly pulled image with what the pods are running and restarts only what was actually rebuilt – exporters and notify_push included.

Tested with fake kubectl/crictl in 14 scenarios. And then the important bit: deliberately break the rescue (always switch maintenance off) and run the test again. It turned red at exactly the right scenario. Before that, the same test had once shown “all green” for the wrong reason – two scenarios with the same name shared a directory and mixed up their logs. Green doesn’t mean much until you’ve seen the test fail.

6️⃣ to 9️⃣ The quieter ones

• 6 – split: the WordPress role (912 lines) is now seven thematic files, Grafana three, the verify roles two each. Proof that nothing changed: --list-tasks --list-tags before and after, identical.
• 7 – argument specs: 23 roles declare their inputs. “Required” is computed per host type: shared roles read blog_hostname on one server and nextcloud_hostname on the other, so neither may be mandatory. Checked against the real variables of all three servers. Vault values are no_log – verified that they appear neither with -vvv nor in a validation error.
• 8 – merge: one ClamAV role and one SELinux role with a profile per stack, plus 16 shared smoke checks. Trap avoided: Ansible identifies cron entries by name, so the names had to stay exactly the same – otherwise the old job would have kept running next to the new one.
• 9 – pins: >=3.0.0 became 3.15.0. Renovate’s dry run showed why that matters: majors 5.x and 13.x were waiting.

🔟 Renovate, and all versions in one file

All version pins now live in inventory/group_vars/all/versions.yml: the values compactly on top, the reasons and history below. Renovate gets regex managers for exactly these lines and proposes updates as pull requests – weekly, never merged automatically. A local dry run with the Renovate container found all 21 tracked pins and promptly suggested nginx 1.31. That’s the mainline branch; I run the stable one (even minor). So there are guard rails now:

• nginx stays on even minor versions.
• Majors (Nextcloud 35, MariaDB, Redis 8) and K3s/Calico minors only after approval on the dependency dashboard, with a note on what to do.
• WP-CLI PRs remind me to update the SHA-512 checksum too – Renovate can’t compute it, and without it the blog’s init container would reject the download.

📚 And the documentation

New in the repo: “How the Ansible code works” in German, English and French – variables and their precedence, what happens during a run, how the Kubernetes modules work, idempotency patterns, error handling, the tests, and a learning path. Updating the existing guides turned up some stale facts as a bonus: an old image version, a Calico description that still said “Nextcloud only”, a variable table pointing to the wrong file.

🚀 The rollout

The local tests were green. Then I gave Claude the vault password file (created by me in the terminal, never in the chat) and it ran the drift check against the real servers. The very first run against the blog crashed – Claude’s own script tripped over the dnf module, which returns its results as a list of strings instead of objects. Fixed, run again, and then the real finds came in. Most of them were in code that had been running for months.

What only the real servers showed

• The Grafana role couldn’t do a dry run at all. It asks the API “does the datasource exist?” via the uri module – which Ansible skips in check mode – and the next task then evaluated a result that didn’t exist. Harmless in real runs, fatal for --check.
• My own point-4 fix was incomplete. I had added creates: so that freshclam only runs once – but left changed_when: true in place, which overrides the “skipped, file exists” result. It still reported “changed” every time while doing nothing.
• An ownership ping-pong: the playbook set publickey.asc to root, something else set it back to www-data – on every run, unnoticed for months.
• Nextcloud’s monitoring binaries were downloaded again after every reboot. The archives live in /tmp, a reboot empties it, and the download task had no condition. Now it checks the installed version first.
• A renamed privilege: the MariaDB exporter user was re-granted on every run. The code asked for REPLICATION CLIENT, MariaDB 10.11 reports the same privilege as BINLOG MONITOR, and the module saw a difference that wasn’t one.
• 46 deprecation warnings for top-level fact variables (ansible_kernel instead of ansible_facts['kernel']) – fine today, broken in a future Ansible release.

One question needed more than a log: would the Nextcloud Deployment change? Its task is no_log because it carries secrets, so the diff is hidden. A restart means a short outage (Recreate strategy). Claude compared the live object with a server-side dry run of the apply and printed only the names of differing fields, never values: identical apart from the last-applied annotation. No restart.

Faster, while we were at it

I asked whether roles could run in parallel to speed things up. Honest answer: not on one host – Ansible works through the tasks of a host one after the other, and the real race conditions would be dnf locks and AIDE/rkhunter needing the final state. The safe wins were elsewhere: AIDE now compares and writes its baseline in one scan instead of two (30 seconds to 3 minutes per run on the blog), and the Kubernetes objects of a template go to the API in one call on normal runs – per object only with --diff, where the readable diff matters. Calico alone dropped from about 30 seconds to under 2.

The real runs

• Blog: MariaDB and Redis untouched, WordPress restarted once – I checked why: the only difference between old and new pod template was the restartedAt stamp from the deliberate restart after plugin updates. The Helm releases stayed at revision 87 (cert-manager) and 72 (ingress-nginx). Those numbers say it all: the old code had added one revision per run. Drift check afterwards: NO DRIFT.
• sofie: no pod restarted, Helm revisions unchanged, notify_push self-test green, NO DRIFT.
• cvjm (the big Nextcloud): same picture – no pod restarted, Helm revisions unchanged, the exporter user now holds BINLOG MONITOR and is no longer re-granted, NO DRIFT.

🎯 Takeaway

In numbers: 16 commits, 165 files touched, 0 lint failures, two new test suites, three servers converted – each one ending with NO DRIFT. The numbers aren’t the point, though. Most of the real finds came from tests that were allowed to fail: the hidden collection dependency, the mail password that never showed as a change, the missing diff, the test that was green for the wrong reason – and, once real servers were in the loop, half a dozen small lies in code that had been running quietly for months.

And my starting question? The repository has a learning path now. Step three says: write a role yourself, without a template. That one’s on me.

Related posts

• Two Minutes Forty-Five: My First Nextcloud App Update via Ansible
• Bare Metal to K3s: A Nextcloud Migration Battle Log
• Germany Only: Geo-Fencing SSH with nftables and Ansible
• A+ on SSL Labs: What My Blog’s TLS Actually Speaks