There’s a migration coming up that I’ve been circling for a while. A Nextcloud instance I administer still runs as a plain native installation — packages, webserver, PHP-FPM, all hand-managed on one box. It’s moving onto the K3s stack I’ve been building out in Ansible.
The honest reason for moving is that the native installation has aged. It still works, but it has accumulated a decade of small decisions nobody wrote down, and every update is an event rather than a routine. What I’m hoping to get from the K3s side is a duller kind of upgrade: the Nextcloud version and every container version pinned in the Ansible code, so an update becomes a changed tag in a file I can review, roll out the same way twice, and change back if it goes badly. The application and the data end up in genuinely separate places instead of interleaved in one tree. And Collabora becomes a container of its own, talking to Nextcloud across a defined interface rather than being wired into the same installation — one less thing that breaks because two applications are sharing a runtime.
None of that is a guess, which is the main reason I’m willing to do it. This blog has been running on the same K3s stack for a while now, and updating it has turned out to be exactly as dull as I hoped: change a container tag, run the playbook again, done. Because the playbook is idempotent, re-running it is a non-event — it is the same operation whether the server is fresh, half-configured, or already correct, and that removes most of the reasons to hesitate before running it. Everything else this blog accumulated along the way — the hardening, the monitoring, the ingress and certificate handling, the performance tuning, the small corrections that each cost an evening — is code I can simply point at the Nextcloud side. That’s really what this migration is: taking what the low-stakes project taught me and spending it on the one that matters.
Before doing that to an instance with real users on it, I ran the whole thing end to end on a throwaway server first: restore the data, restore the database, walk the version path, break things, find out why. Thirty-six files and eleven hundred lines of changes later, the Ansible code is meaningfully better than it was when I started.
All of it is on GitHub — playbooks, roles, hardening, verification. The Nextcloud side in particular got a lot of attention recently.
What I actually want to write about isn’t the fixes, though. It’s the shape of the problems.
🎭 Everything failed by pretending to be something else
Not one of these problems announced itself. Every single one arrived disguised as a different, more plausible problem — usually one that would have sent me looking in entirely the wrong place.
A package install failed with a permission error that had nothing to do with permissions. chmod: Operation not permitted, as root, with every capability present, on a filesystem mounted normally. I checked everything the message pointed at, and all of it was fine. The actual cause was a systemd hardening directive on the SSH daemon — RestrictSUIDSGID=yes — whose seccomp filter is inherited by every process descended from sshd. Which means every SSH session, and every command Ansible runs through one.
That’s the lesson worth keeping: hardening applied to sshd isn’t service-level hardening. sshd is a process factory for interactive sessions, so anything you restrict there, you restrict for every human and every tool that comes through the door. For a while I genuinely suspected the hosting provider had locked something down.
A playbook run “hung” at the firewall step. No error, no timeout, just silence for minutes. The role switches SSH to a non-standard port and enables nftables — on the very connection it’s reconfiguring. The new ruleset accepts established connections, which sounds complete, except that when nftables starts for the first time, conntrack has no record of a connection that predates it. So the session matched no rule at all and its packets were simply dropped. Not refused — dropped. To Ansible that looks identical to a task taking a long time.
A site was completely unreachable, and the browser said nothing useful. This ingress controller answers a hostname whose TLS certificate isn’t ready by refusing the handshake outright — no certificate at all, not even a self-signed one. From outside, that is indistinguishable from the server being down. The real cause was three layers away: an image allowlist was blocking cert-manager’s challenge pod, so the certificate could never be issued.
And my favourite: a brute-force protection that was protecting nothing. The fail2ban jails watching web logins read the ingress log through a path containing the pod’s ID. fail2ban resolves that path once, at startup. Every time the ingress pod is recreated — an image update, a helm upgrade, a reboot — the jails keep running against a path that no longer exists:
Status for the jail: nginx-k3s-nc-login
|- Filter
| |- Currently failed: 0
| `- File list:An empty file list. Enabled, running, and structurally incapable of banning anyone. The monitoring check that existed only asked whether the fail2ban process was alive, which it very much was.
🩹 What came out of it
Most of the fixes are unremarkable once you know what the problem was, which is usually how this goes:
- The SSH and firewall switchover now happens via a single reboot on a fresh server, so both come up together from a clean boot instead of being changed underneath a live connection.
- Any hostname without a real certificate yet gets a short-lived self-signed placeholder, so a browser warning replaces a dead socket. A warning tells you the server is alive and the problem is the certificate; a dead socket tells you nothing.
- Two new monitoring checks: one for certificates approaching expiry, one that verifies the fail2ban jails are actually watching a file rather than merely running.
- Plus the boring half — a mail sender assembled from the wrong half of an address (which took Grafana down entirely on startup), a verification role that aborted the whole play with a parser error exactly when something was broken, and security headers being set twice with two different values.
⬆️ One thing worth knowing before you plan a window
Nextcloud refuses to skip major versions. Moving a restored instance from 32 to 34 means stopping at 33 in between, with an upgrade step at each stop.
Two details the documentation doesn’t stress: the container runs its own upgrade on startup and won’t accept traffic until it’s finished, so running the upgrade manually just races the one already in progress — change the image tag and wait. And check the app list after each major, not once at the end. Some apps disable themselves temporarily and come back; others are genuinely incompatible and stay off. One in my set gets removed entirely rather than upgraded, because it’s installed by direct download instead of through the app store.
🧱 What I deliberately did not change
For a while I was tempted to do the AlmaLinux 9 → 10 upgrade in the same window. New server, fresh start, might as well begin on the current major, right?
I talked myself out of it, and I think the reasoning generalises. This migration already changes a lot at once: the deployment model (bare metal to K3s), the Nextcloud version, how Collabora is run, and the machine underneath all of it. Adding an OS major bump means adding a second, entirely unrelated variable. If something then misbehaves two weeks later — a performance oddity, a subtle permissions problem, something in SELinux — I would have no way to tell which of the two changes caused it. Debugging works by holding things constant, and a migration is precisely the moment when you have the fewest constants left.
There’s also no clock forcing it. AlmaLinux 9 is supported until 2032. That is not a deadline I need to plan around.
But the more interesting reason is the one that only became visible while building this stack. In a container setup, the host operating system carries much less weight than it used to. Everything that actually determines how the service behaves — the Nextcloud version, PHP, the web server, Collabora — lives in container images whose versions I pin in the Ansible code. The host provides a kernel, systemd, K3s, and the hardening around them. The interesting decisions have moved up a layer, and what’s underneath has become comparatively interchangeable.
Which is a slightly odd thing to realise: one of the benefits of this migration is that it makes the next migration less important. The OS upgrade becomes a separate, boring exercise I can do on its own terms, on a quiet evening, with nothing else in flight — which is exactly what an OS upgrade should be.
🔒 The one thing the rehearsal could not solve
There is one problem I could not work around on the test server at all: getting valid Let’s Encrypt certificates.
The reason is structural rather than technical. The HTTP-01 challenge works by having Let’s Encrypt resolve the domain and fetch a token over HTTP — from whatever address the public A record points at. That’s still the old server, and will be until the moment I switch DNS. I faked DNS locally with an /etc/hosts entry pointing at the new machine, which is enough to convince my own browser but does precisely nothing for Let’s Encrypt, which does its own lookup against public DNS. So the new server cannot hold a valid certificate for a hostname it doesn’t yet officially answer for.
There is a proper answer to this: the DNS-01 challenge proves control of the domain by publishing a TXT record at _acme-challenge.<domain> instead of serving a file. It doesn’t care where the A record points, so certificates could be issued in advance and be ready on day one. It needs a DNS provider with an API the ACME client can drive, and wiring that up is outside what I want to take on for this migration.
So I’m accepting the gap: for a few minutes after the DNS switch, the new server will be reachable without a valid certificate yet. This is exactly the situation the placeholder certificate from earlier in this post was built for — instead of a refused handshake that looks like an outage, visitors get a browser warning, which at least says the server is there and the certificate isn’t ready. Not elegant. But a known, bounded, visible gap beats an invisible one, which is more or less the theme of everything above.
📅 What happens next
The real migration is happening in the next few days.
The plan for the day itself is deliberately unremarkable:
- A day or two ahead: lower the DNS TTL, so the switch later propagates in minutes rather than hours.
- Old server into maintenance mode. Nothing changes underneath me from this point on, and no client can write to the instance I’m about to copy.
- Final backup of the old server — data and database — before touching anything.
- Final sync to the new server, closing the gap that opened since the rehearsal copy.
- Switch DNS — A and AAAA. Forgetting the IPv6 record is an excellent way to leave half your users on the old box while everything looks fine from your own laptop.
- Wait for the certificates, which can only be issued now (see above).
- Test: login, desktop sync in both directions, shares still intact, Collabora opening a document, outbound mail.
- Leave the old server exactly where it is, in maintenance mode, as the rollback.
That order matters more than it looks. Because the old server goes into maintenance mode rather than being shut down or wiped, it keeps sitting there complete and unchanged — data, database, configuration, all of it exactly as it was the moment I stopped it. If the new stack turns out to be a disaster for some reason I haven’t anticipated, the way back is to point the DNS records at the old machine again and lift maintenance mode. No restore, no data loss, nothing to reassemble, because nothing was ever taken apart. The old server isn’t decommissioned on migration day; it’s my rollback, and it stays that way until the new one has proven itself for a while.
The one part I genuinely can’t predict is the DNS propagation. The TTL tells resolvers how long they may cache a record, and lowering it a day or two in advance shortens the tail considerably — but it’s an upper bound, not a promise. Plenty of resolvers round it up, some ignore it, and a few clients hold onto an address until something restarts. So in practice there’s a window where some users reach the new server and some still land on the old one. That’s uncomfortable for a file sync service specifically, because writes could land on either side. It’s the main reason the old instance stays in maintenance mode rather than merely being left running: a user who hasn’t switched over yet sees a maintenance page, which is annoying but harmless, instead of successfully uploading a file to a server nobody is looking at any more.
I’d like to say the rehearsal means it will go smoothly, and I hope it does. But the honest version is that a rehearsal on a throwaway server with copied data is not the same as the live instance with real users, real sharing relationships, real sync clients that have opinions about what changed while they weren’t looking. There will be something. There always is.
So: I’ll report back here afterwards — what actually went wrong, what the rehearsal failed to predict, and what I had to change. And whatever that turns out to be will go straight back into the Ansible code in the repo, because a playbook that only works on the server you tested it against isn’t finished.
If the next post is short and boring, that’s the good outcome.
The pattern underneath
Looking back at all of it, there’s one thing connecting these.
None of these were errors. They were absences — a jail with no file, a certificate that never issued, a connection that stopped answering instead of refusing, a filter blocking a syscall nobody thought to look for. Every one of them produced output indistinguishable from everything being fine.
That’s what a rehearsal actually buys you. Not confidence that it will work — you can’t have that. Just a cheap opportunity to meet the silent ones before they cost something.
