Update day: Nextcloud 34 → 35, a two-week rule and kernel updates

🌐 Also available in: Deutsch · Français · Español

Every Monday a small script mails me a version report: which pinned versions are outdated, which images were rebuilt under the same tag, which packages are pending on the hosts? This time the list was long – including the jump from Nextcloud 34 to 35 on both instances. By the end of the afternoon everything was current, nothing was broken, and the drift check showed changed=0.

This post walks through the stations in order. Each one comes with its own small lesson – and one of them was a bug in my own tooling.

Table of Contents

📬 The report informs, it does not act

The check script runs on a server in my homelab and is strictly read-only. It compares the pins in versions.yml with Docker Hub and GitHub, checks whether an image was rebuilt under the same tag, compares the PHP version in the pods with php.net, and counts pending packages per host. It changes nothing – I apply updates myself, deliberately, one host at a time.

This time the report said: Nextcloud 35.0.2 available, Trivy 0.76.1 available, the WordPress image “rebuilt”, and six kernel packages on all three hosts.

How the report works: the script is a Python program on HP3, a server in my homelab that also pulls the nightly backups of the three servers – not on the servers themselves. A cron entry starts it every Monday at 07:17, and it sends the mail through its own mail account. In five steps:

It reads versions.yml from the local copy of the repository – every pinned version lives in that one place.
It asks Docker Hub about every image and GitHub about the latest release of every binary, and compares them with the pin.
It logs in to each server via SSH, lists the images actually present with crictl images, and compares them with the current build under the same tag.
It reads the PHP version inside the running pods and compares it with php.net.
It counts pending packages per host with dnf check-update.

All findings end up in one mail – together with the commands for the update runs in the right order. Running them is up to me.

📦 To rebuild it: weekly-update-check.zip (script, cron wrapper and a short README, 4 KB). The hosts are placeholders at the top of check.py; the pins come from versions.yml in my public repository.

The takeaway: a report that only informs can afford to be thorough. Deciding what goes in, and when, stays with a human.

☁️ Nextcloud 34 → 35: check first, then jump

A major upgrade is never automatic with Nextcloud. You cannot skip versions, and app-store apps need a release that matches the new major version. So before the jump I checked:

Does the image exist? nextcloud:35.0.2-fpm was on Docker Hub.
Do the apps fit? The app store API lists the available apps per platform version. All nine store apps on each instance – Collabora, Deck, notify_push and Groupfolders among them – had a release for 35.
Do the requirements fit? Nextcloud 35 needs at least MariaDB 10.11 and drops PHP 8.2. The database already ran 10.11, and PHP ships inside the container image.
Is there a way back? On top of the nightly backup I took a fresh database dump right before the upgrade.

I set the new pin for one instance only first, in its host variables. The second instance and the global default stayed on 34.0.5 until the first one ran clean. Only then did 35.0.2 move into versions.yml, and both host pins were removed again.

The takeaway: a major upgrade does not start with the upgrade. It starts with three questions: is everything there, does everything fit, and can I go back?

⚙️ What actually happens during the upgrade

Ansible does not run the upgrade itself – the Nextcloud image does. On start, its entrypoint compares the installed version with the one in the image. If the image is newer, it copies the new code and runs occ upgrade, which migrates the database. Nextcloud is in maintenance mode meanwhile.

The playbook makes sure of two things:

Recreate instead of a rolling update: with a new image, the old pod stops completely before the new one starts. Otherwise old and new code would work from the same directory at the same time.
A patient startup probe: PHP-FPM only starts after the upgrade. The startupProbe allows up to ten minutes, so Kubernetes does not mistake a long database migration for a crash.

It took 40 and 47 seconds. The log, however, had lines that look alarming at first: “Disabled incompatible app: richdocuments” – and so on for every store app. That is expected: during a major upgrade Nextcloud disables every app whose installed version does not match the new major release. The following run of nextcloud-update.yml fetches the matching releases, re-enables them and restarts the pod, so OPcache stops running old bytecode.

Afterwards all apps were back, most of them in a new version (Collabora 11 → 12, Groupfolders 22 → 23). New is the bundled sharing app for the unified sharing in Nextcloud 35.

The takeaway: a major upgrade has two steps that belong together – the core, then the apps. Stop after the first and you have a Nextcloud without Office.

🔍 How I define “done”

“The page loads” is not my acceptance test. After each instance I checked:

the Nextcloud log: no errors since the upgrade, only three known warnings about apps that are disabled on purpose
the web server logs: no 5xx responses
the notify_push self-test
on the host: no SELinux denials, MariaDB and Redis in their own domains, no failed units, every Prometheus target up
and finally the drift check: a playbook run in check mode that must not change anything any more

Both instances reported changed=0. Only then do I consider an upgrade finished: the server is exactly what the code describes.

The takeaway: “it works” is a feeling, changed=0 is a finding.

🧪 The drift check in detail

The drift check is a second, entirely different script: scripts/check-drift.sh in the public repository. It does not run on a schedule but by hand, on the machine I run Ansible from – always right after a real playbook run and after every upgrade.

The idea: Ansible describes what a server should look like. Run the same playbook again in check mode and there must be nothing left to do. The script does exactly that:

It calls ansible-playbook … --check --diff for one host. In check mode Ansible changes nothing; it only reports what it would do.
It takes the output as JSON and filters the tasks that would report “changed” – with their diff. For Kubernetes objects it hides the server’s bookkeeping fields, so only real differences remain.
The exit code tells the result: 0 = no drift, 1 = drift, 2 = the check run itself failed.

This is what it looked like today after the upgrade:

$ scripts/check-drift.sh nextcloud-k3s.yml <host>

<host>: changed=0 failed=0
NO DRIFT

And this is what a hit looks like – an earlier case in which a task reported “changed” on every run although nothing had changed:

### would change: rkhunter | Build / refresh file-properties database

<host>: changed=1 failed=0
DRIFT: 1 task(s) would change

A hit right after a real run always means one of two things: someone changed something on the server by hand, or a task in the code is not idempotent. You want to know either before the next run quietly “fixes” it.

The takeaway: the drift check is the final inspection against the blueprint. It does not prove that everything works – it proves that the server matches the code exactly.

⏳ A two-week wait for Trivy

Trivy, my nightly image scanner, had a new version. I did not install it. In March 2026 Trivy’s release pipeline was compromised and shipped a manipulated build. Since then I have a rule: Trivy only in releases that are at least two weeks old, always pinned with a SHA-256 checksum taken from the release’s checksum file.

Version 0.76.1 came out on 21 October, so it is allowed from 4 November. I checked the checksum in advance – the downloaded archive matches the official checksum file. The commit waits one more day.

The takeaway: for security tools, “new” is not automatically “safer”. A waiting period costs little, and in March it would have helped.

🧩 The false alarm: a new index, the same image

The report flagged wordpress:7.1.4-php8.5-fpm as “rebuilt upstream”. The update playbook pulled it – and restarted nothing. The containerd log showed why: it read about 21 KB, and the image ID stayed the same.

The explanation: my script had compared the index digest. A tag on Docker Hub points to an index, which lists the images for every platform. If the image for another architecture changes, or just an attestation, the index gets a new digest – but the amd64 image I run stays identical, byte for byte.

The fix was small: the script now resolves the index, picks the linux/amd64 entry and compares its config digest with the image ID on the host. That is exactly what the container runs. Since then: “none – every pinned tag is current”.

The takeaway: a new digest does not mean a new image. Compare what actually runs.

🐧 Why Ansible never installs my kernels

That left the kernel packages. My package playbook installs every update – except kernels, on purpose. The SSH service is hardened with systemd, among other things with ProtectKernelModules=yes. Every SSH session, including Ansible’s, therefore runs in its own mount namespace in which /usr/lib/modules appears empty and read-only. A kernel install over SSH fails there every time.

So kernels take a different route:

dnf-automatic installs security updates only, at night, as a systemd service – outside any SSH session. The new kernel was such a security update.
At 03:30 the host reboots if needs-restarting -r asks for it.
At 04:00 AIDE refreshes its baseline, so the new kernel is not an alarm the next day.

This time I did not want to wait for the night. Instead of installing the package over SSH, I started dnf-automatic-install.service by hand on each host – the same route as at night, just earlier. Then a reboot, one host after the other. Each one was back after 40 to 50 seconds, with the new kernel, no SELinux denials and every pod running.

The early timing had one side effect: the daily AIDE check runs at midnight, before the 04:00 refresh. Left alone, it would have reported the new kernel as a change. So each host needs the AIDE and rkhunter refresh afterwards.

The takeaway: hardening has side effects on your own automation. Instead of weakening the hardening, give the kernel update its own, fitting route.

🎯 The lessons

Reports inform, people decide – check automatically, apply deliberately.
Answer three questions before a major upgrade – is everything there, does everything fit, can I go back?
Core and apps are one upgrade – Nextcloud is done only when the apps run again.
Compare what actually runs – not the index, but the image for your own platform.
Done means changed=0 – the drift check closes every update day.