Today I moved a production Nextcloud from a cozy native bare-metal install to a shiny K3s/container stack. On paper: pull a backup, restore it, flip DNS, done. In practice: a series of small landmines, each invisible until you step on it. Here’s the battle log. 🪖
Step 1: shovel the data 🚚
The heavy lifting was an rsync of ~360 GB of user data plus a ~330 MB SQL dump from the backup box into the new node. Because I pre-synced earlier in the day, the final run only shipped the delta — the whole thing came down to a fast incremental copy. Lesson old as time: pre-seed, then the cutover sync is tiny.
Step 2: the Major Version Gauntlet 🎮
The backup was Nextcloud 32.x. The target stack runs 34.x. Nextcloud has one hard rule: you cannot skip a major version. So it’s a walk, one step at a time:
32.0.14 → 33.0.9 → 34.0.4 (occ upgrade at every step)The container twist: the code lives in a persistent volume, and Nextcloud refuses to run older code against a newer data version (“downgrade not supported”). Restoring a v32 database onto a pod that still had v34 code = instant refusal. The fix is delightfully hacky: pin the image to the matching version, nudge the code volume’s version.php so the entrypoint re-lays the correct code, and let each occ upgrade finish (its readiness probe blocks until the DB migration is done — patience required).
The invisible landmines 💣
1. The antivirus socket of doom
After the restore, the Files app threw a bare “Internal Server Error” right after login — and the log stayed empty, because the crash happened below Nextcloud’s own logger. Cause: the restored config re-enabled files_antivirus in socket mode, pointing at a ClamAV daemon socket that exists on the old native box but not in the container. Fix: disable the app (this stack does nightly batch scans instead). Diagnosed only by noticing, not by any log line. Sneaky.
2. Let’s Encrypt jail 🔒
DNS wasn’t pointed at the new box yet, so cert-manager had spent days failing to issue certificates. When I finally flipped DNS, the certs still wouldn’t issue. Why? After enough failures, cert-manager parks the Certificate in an exponential back-off stored in the object’s status — next attempt scheduled ~24 hours out. Restarting the controller? Ignored. Deleting the secret? Ignored. The move that actually works: delete the Certificate object and let the ingress-shim recreate it fresh, with no failure history → instant issuance.
Bonus trap: Let’s Encrypt’s HTTP-01 check prefers IPv6. If you only move the A record and forget the AAAA, validation keeps hitting the old host and failing while everything “looks” switched. Move both. Always both.
The main event: the checkbox that lied ☑️
This one cost the most head-scratching. Collabora (Nextcloud Office) refused to open documents: “Nextcloud Office could not be loaded.” Meanwhile the admin panel’s Office connectivity test was reassuringly green: “Collabora Online server is reachable.” Server-side everything checked out — discovery 200, capabilities 200, valid certs, correct WOPI URL, WebSocket headers present, identical to a known-good reference host.
The difference turned out to be a single setting carried over from the old server: “Disable certificate verification (insecure)” was ticked (disable_certificate_verification=yes in richdocuments). On the old box it made sense — self-signed certs. But now, with valid Let’s Encrypt certificates, that “insecure” mode breaks the WOPI document session in richdocuments 11.x — while leaving the admin connectivity test perfectly green, because that test uses a different code path than the actual document handshake.
Why do I think it was the culprit? Because the fix was exactly that: untick the box (delete the key), retry, and documents opened immediately. My theory: the insecure/no-verify path and the WOPI token exchange disagree once real TLS is in play, and the failure surfaces only in the editor session — the classic “the health check is green but the feature is dead” situation that keeps sysadmins humble. Two more prod-leftover keys (public_wopi_url, doc_format) went out with it, matching the reference host’s minimal config.
The red herring: “why are so many apps disabled?” 🎣
Post-upgrade, a pile of apps showed as disabled (dashboard, comments, external storage, …) and I braced for a reconcile marathon. Then I diffed the current app set against the old server’s state (extracted from the pre-upgrade DB dump). Plot twist: they were disabled on the old box too. The new instance already matched production. The only genuine casualty was metadata, which simply has no version compatible with Nextcloud 34. Sometimes the scary diff is just… the truth.
Native vs. containers: the honest bit ⚖️
The new instance feels a touch slower than the old native one. RAM and CPU sit bored (plenty of headroom), and server-side response times are 50–210 ms — perfectly healthy. The difference is architectural: containers add hops (ingress proxy, overlay network, PHP-FPM↔nginx over the pod network, DB across the pod/host boundary). Per request it’s milliseconds; across the many requests a Nextcloud page fires, it adds up to “slightly less snappy.” That’s the tax you pay for reproducibility, isolation and painless upgrades — and for a small org it’s a trade worth making.
Leftovers in the database 🗃️
A fun stat from the restored DB: 26% of the entire file-cache index (~276k rows) belonged to a long-removed OnlyOffice bundle whose index entries outlived the app. It’s dead weight that mostly slows backups and future upgrades rather than daily use — a one-time cleanup during a migration, never a routine chore.
Verdict ✅
It’s live. Certificates valid, Office editing works, data intact, users none the wiser. The migration itself was the easy 20%; the other 80% was a scavenger hunt for settings and behaviors that only misbehave under exactly the new conditions. As always: the server was never the problem — the assumptions were. 😄
