Deep Dive: Nextcloud on K3s – PHP Pods, Host Databases, and Office in the Browser

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

Two Nextcloud servers run on this setup, one for an organisation with about 150 accounts and almost 400 GB of files, one for a small group. Both are built from the same roles (part 4 of the tour). This post explains how Nextcloud is put together, how the pieces are distributed between host and cluster, and what running it has taught so far.

🧭 How Nextcloud works

One PHP application, many entry points

Nextcloud is PHP. Every browser page, every WebDAV request of a sync client and every API call of the mobile apps is a PHP request through nginx to PHP-FPM. The files themselves sit in a data directory; the database knows about every file through the file cache table – name, size, ETag, permissions – so that listings never have to touch the disk.

Locking and caching

When two clients change the same file, transactional file locking decides who wins. The locks must be visible to every PHP process at once, which is why they live in Redis. Next to it, APCu caches per process and Redis caches across processes.

Background jobs

Cleanup, trash and version expiry, notifications, app updates checks, preview pre-generation – Nextcloud keeps a job queue in the database. Something has to run cron.php regularly to work through it; Nextcloud recommends every five minutes.

Push, previews, office

• notify_push is a small Rust daemon shipped inside an app. Clients keep a WebSocket open to it and learn about changes immediately instead of asking every 30 seconds.
• Previews are thumbnails, generated on first view or in advance by the Preview Generator app, and stored in the app data folder.
• Nextcloud Office is the app richdocuments plus a Collabora Online server. The browser loads the editor from Collabora; Collabora fetches and saves the document from Nextcloud through the WOPI protocol, with a token per session.

Apps and occ

Almost everything beyond files is an app – calendar, contacts, Talk, group folders. occ is the command-line tool for all administration: status, upgrades, maintenance mode, configuration, file scans, database repairs.

🛡️ Why it is here

• Files, calendars, contacts and office in one place, for people who should not need to know what WebDAV is.
• The data stays on the organisation’s own servers. No vendor decides about access, prices or features.
• An ecosystem. Apps for nearly every need, clients for every platform, a large community.

🔧 What this repository does

What runs where

The heavy, precious components – database and data – live on the host, with the full systemd sandbox and their own SELinux domains. The replaceable ones – PHP, Collabora, the push daemon – run in the cluster. Kubernetes Services without selectors, pointing at the node IP, make MariaDB and Redis reachable for the pods under the names mariadb and redis; nftables allows that port only from the pod network. The PHP code (/srv/nextcloud-www) and the data (/data) are hostPath volumes with their own SELinux labels.

Configuration in layers

Nextcloud reads config.php and then every *.config.php in the config directory in alphabetical order, later files winning. The repository uses that: custom.config.php from a ConfigMap carries the settings (caches, previews, proxies, logging), redis.config.php from a Secret carries the Redis password. Both are mounted with subPath, which never updates in a running pod; a checksum of everything mounted is an annotation on the pod, so a change rolls it. That checksum originally covered only the ConfigMaps – a new preview setting once sat in the ConfigMap for a day while the pod still ran the old file. Since this series it covers the Secret too, after the Redis deep dive found that a rotated password would not have reached the pod.

Collabora

extra_params: --o:ssl.enable=false --o:ssl.termination=true
              --o:num_prealloc_children=10 --o:per_document.max_concurrency=6

TLS ends at the ingress, so Collabora runs plain HTTP behind it and is told so. num_prealloc_children keeps that many document processes started in advance; opening a document takes one, and if none is ready the user waits for a fresh one to start. The occasional “failed to load, retry works” right after a Collabora restart is exactly that. The larger server keeps ten ready, the smaller one six. next_config sets the WOPI allowlist to the pod network – only Collabora may call the WOPI endpoints – and makes Nextcloud re-read Collabora’s discovery document after every Collabora change.

One installation detail cost an afternoon: occ app:install richdocuments reproducibly truncated one file of the app while extracting it, and the integrity check failed afterwards. The role therefore downloads the tarball from the app store itself, extracts it with tar, and verifies the app’s signatures with occ integrity:check-app.

Updates in two kinds

A version change is a new image tag in versions.yml and a run of the main playbook; the container entrypoint runs occ upgrade on start. Nextcloud itself refuses to skip a major version, so a jump from 32 to 34 is two steps. Rebuilt images – Docker Hub rebuilds unchanged tags for base-image fixes, and K3s never pulls again on its own – are the job of nextcloud-update.yml. It refuses to run without --limit (one server at a time), refuses to start on an instance that is in maintenance mode or has a pending database upgrade, restarts only Deployments whose image really changed, runs occ upgrade as a safety net, updates the apps, restarts a second time (OPcache does not check timestamps, see the PHP-FPM deep dive) and checks the status. If anything fails, a rescue block collects the state for a decision by hand. Both servers run Nextcloud 34.

Isolation

• A NetworkPolicy lets Collabora talk only to Nextcloud and DNS, and accept connections only from Nextcloud and the node. The Nextcloud pod and the push daemon have none yet – unlike the blog, where every pod is isolated. In a cluster that holds nothing else this is not an open door, but it is a layer missing; found while writing this article and proposed.
• An admission policy admits only images from an allowlist of registries – including the image of cert-manager’s ACME solver, which runs in this namespace; on the blog its absence once left a certificate request pending without any error.
• No service account tokens in the pods – nothing in them talks to the Kubernetes API.
• ClamAV scans the files in the background at 04:30, with previews and media formats excluded – before that change the scan was the cause of a server that showed up as “busy” for no visible reason.

📊 Measured

The file cache of the larger server had more than ten times as many entries after the migration from the old server: the restored database still contained the old server’s whole file index, and occ files:cleanup did nothing as long as the storage row of the old path existed. A quarter of what remained belonged to leftovers of a previously used office app. Both were removed – after a backup and after counting the live rows, because data of disabled apps is kept on principle, in case someone wants them back.

🧰 Day-to-day administration

occ() { k3s kubectl -n nextcloud exec deploy/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ "$@"; }

occ status
occ setupchecks                       # the admin overview, on the command line
occ maintenance:mode --on             # before anything that touches data
occ files:scan --path="/alice/files"  # after files were changed outside Nextcloud
occ db:add-missing-indices
occ background-job:list --limit=20
occ app:list --disabled

Expected warnings

• “Errors in the log” on the smaller server: one database error at 03:30 – the nightly reboot, during which a background job pod could not resolve mariadb. Harmless, but it stays in the overview until the log rotates.
• “Office failed to load, retry works” right after a restart: Collabora had no pre-started process yet.
• Integrity warnings after a manual change in the app directory are real: Nextcloud signs its code, and any edited file shows up. Never edit app code in place.

Backups and restore

Every night a backup machine pulls the database dump, config.php and the data directory over SSH and versions them for 365 days. A restore is the reverse plus occ maintenance:data-fingerprint, which tells the sync clients that the server state changed so they do not delete what they think is new.

🆘 Cheat sheet: symptom → command

The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders. occ and wp are the shell functions from the Nextcloud and WordPress deep dives.

🔭 What comes next

• Nextcloud 35, once the first point release is out, smaller server first.
• S3 primary storage would make the data independent of the host disk – only relevant with more than one node.
• The Talk high-performance backend, if video calls ever become more than occasional.

⚖️ Alternatives – and why Nextcloud

Nextcloud is not hard to install; it is demanding to run well, because it is a dozen services pretending to be one. Splitting them along “precious or replaceable” – database and files on the host, PHP and office in the cluster – made each part simpler to reason about. The mistakes in this post were almost all at the seams between them.