The Ansible Tour, Part 4: Nextcloud – Collabora, Push, Host Database and occ, Role by Role

🌐 Also available in: Deutsch · Français · Español

Part 4 of the Ansible Tour. Part 1 explained the framework, part 2 the server and the platform, part 3 the blog. This part is the second application: Nextcloud. It describes release v3.3.0. If you have read part 3, much will look familiar – I point out the differences and explain what is new, rather than repeating the shared patterns.

Nextcloud is a file cloud: files, calendars, contacts, shared folders, and – with Collabora – documents edited together in the browser. Two of the three servers here run it for groups of very different size, with the same code and different inventory values. The work is split over three roles plus the update playbook:

🏗️ What runs, and why

The request path is the same as for the blog: ingress controller (TLS) → nginx sidecar → PHP-FPM, in one pod. What is different is everything around it.

Background on this setup is spread over older posts: how the instances were run before K3s, and the migration to K3s, step by step.

🗄️ next_mariadb: a database on the host

The role installs MariaDB from the repository that next_packages set up (part 2), secures it and creates the Nextcloud database. Its header lists the order, and the order explains the role.

Sandbox before the first start

MariaDB gets a systemd drop-in with sandbox settings (from the role’s files/): which directories it may write, which kernel features it may not touch. The header comment makes a point worth repeating: the same kind of restriction was deliberately not applied to SSH in its strictest form, because sshd starts user sessions and everything a user or Ansible does would inherit it. MariaDB starts nothing, so a restriction reaches only the database. Since 2026-10-16 its /tmp is a private tmpfs of 4 GB, so temp tables and sort files stay in RAM. It must not be combined with PrivateTmp=yes: tested with systemd-run, that setting silently wins and keeps /tmp on disk.

The order of three tasks matters: write the drop-in → daemon_reload directly (not as a handler) → start the service. A handler would run at the end of the play – on a fresh server MariaDB would first start without the sandbox. Then wait_for waits until the socket file appears, the sign that MariaDB accepts connections.

Configuration, and a log directory nobody created

A config file in /etc/my.cnf.d/ sets the essentials: listen on all interfaces (the pods connect via the host address; the firewall limits port 3306 to the pod network), the InnoDB buffer pool (the cache for database pages, sized per host in the inventory – about a quarter of the RAM), flush behaviour, connection limits, the slow query log, and skip_name_resolve. That last one stops MariaDB from looking up a host name for every pod connection – a lookup that can never succeed for a pod address and only costs time.

Two stories hide in the tasks around it. The slow query log needs a directory that MariaDB does not create. Without it, MariaDB logs one error at startup and then runs with slow logging silently switched off – a diagnostic that looks enabled and collects nothing. So the role creates it, with the right SELinux type. And the log rotation: the package’s own logrotate file and an extra one for the slow log covered the same files, and logrotate failed every night with “duplicate log entry”. Now the role owns the package file (it is marked as a config file, so package updates leave a .rpmnew next to it instead of overwriting it) and uses copytruncate, which needs no database login.

One warning in the role is worth quoting for anyone who edits it: the comment lines inside the config file’s content: are part of the file on the server. Changing a comment there changes the file, and the handler restarts MariaDB.

First run only – and every run

- name: MariaDB | Check if the Nextcloud database exists
  ansible.builtin.shell: |
    set -o pipefail
    mysql --protocol=socket --socket="{{ mariadb_socket }}" -u root -p"..." \
        -e "SHOW DATABASES LIKE '{{ nextcloud_db }}';" | grep -Fx "{{ nextcloud_db }}"
  register: db_check
  failed_when: false
  changed_when: false
  no_log: true

- name: MariaDB | Create Nextcloud database user (host restricted to K3s pod CIDR)
  community.mysql.mysql_user:
    name: "{{ nextcloud_user }}"
    host: "{{ k3s_pod_cidr.split('.')[:2] | join('.') }}.%"
    password: "{{ nextcloud_pass }}"
    priv: "{{ nextcloud_db }}.*:ALL"
  when: db_check.rc != 0
  no_log: true

The first task decides whether this is a new server: does the Nextcloud database exist? grep -Fx matches the whole line literally, so “nextcloud” does not match “nextcloud_old”. If it does not exist, four tasks with when: db_check.rc != 0 set the root password, write a /root/.my.cnf for root, create the database and the user. The user is only allowed from the pod network: k3s_pod_cidr.split('.')[:2] | join('.') turns 10.42.0.0/16 into 10.42, and 10.42.% is MariaDB’s wildcard for “any address starting with that”. Even with a firewall mistake, the database would refuse that login from anywhere else.

These tasks use the community.mysql modules, which talk to MariaDB through Python and are idempotent: “present” creates or corrects, “absent” removes. They connect over the local socket file (login_unix_socket) – and a small set_fact with an inline if/else picks whichever of the two usual socket paths exists on this host.

Other tasks run on every play: removing the anonymous accounts and the test database that MariaDB ships (what mysql_secure_installation would do; some upgrades recreate them), and maintaining the exporter’s read-only user. That user used to share the root password, which meant the exporter’s config file effectively held root’s credentials. Now it has its own. Its privilege list carries a small idempotence lesson: MariaDB reports the privilege REPLICATION CLIENT under its newer name BINLOG MONITOR, and as long as the role asked for the old name, the module saw a difference and re-granted it on every run.

⚡ next_redis: three lines and a guard

Redis is installed by the package role; this role only edits /etc/redis/redis.conf with three lineinfile tasks – bind to localhost and the host’s public address (so the pods can reach it), protected mode on, and a password. backup: true keeps a timestamped copy of the file whenever a line changes.

The password line has the better comment. Redis listens on the public address; only one firewall rule stood between it and the internet. “One firewall rule silently failing is not a hypothetical here” – the comment refers to fail2ban sitting dead for two days on the blog without anyone noticing. An unauthenticated Redis holding Nextcloud’s file locks and sessions is not something to leave on a single layer. The same password goes to Nextcloud through the Kubernetes Secret, and an assert at the start of the role refuses to run if the vault does not have it.

- name: Redis | Flush handlers to apply config changes now
  ansible.builtin.meta: flush_handlers

- name: Redis | Wait for Redis to be ready on host IP
  ansible.builtin.wait_for:
    host: "{{ nextcloud_ipv4 }}"
    port: 6379
    timeout: 30

Again flush_handlers instead of waiting for the end of the play: the configuration role later in the same play talks to Redis through the pods. If Redis restarted only at the end, it would still be listening on localhost only when those commands run. wait_for then checks that the port really accepts connections on the address the pods use.

🚀 next_k3s_deploy: Nextcloud, Collabora and push as objects

Same structure as the blog’s deploy role: one task file per component in dependency order, each applied with the K8s apply pattern. One difference in behaviour: here the waits run only when: <apply>.changed – an unchanged run never waits for a rollout.

External services: a name for something outside the cluster

How does a pod reach a database that is not in the cluster? With a Service that has no pod selector, and an Endpoints object filled in by hand:

apiVersion: v1
kind: Service
metadata:
  name: mariadb
  namespace: nextcloud
spec:
  ports:
  - port: 3306
---
apiVersion: v1
kind: Endpoints
metadata:
  name: mariadb          # same name = belongs to the Service above
  namespace: nextcloud
subsets:
- addresses:
  - ip: <the host's address>
  ports:
  - port: 3306

Normally Kubernetes fills the Endpoints itself with the addresses of matching pods. Without a selector it leaves that to us, and we put the host’s address there. Inside the cluster, mariadb and redis are now ordinary DNS names; that they lead out of the cluster to the host is invisible to Nextcloud. The same trick gives Grafana on the host a name the ingress can route /grafana/ to.

The Nextcloud pod

The deployment template follows the blog’s pattern – init container for directory owners, three containers, capabilities dropped with comments per capability, configuration checksum in an annotation so a changed ConfigMap rolls the pod, startup/readiness/liveness probes. The differences say something about Nextcloud:

• Installation from environment variables. On first start against an empty database, the image installs Nextcloud from MYSQL_*, NEXTCLOUD_ADMIN_* and REDIS_* variables. Passwords again come as *_FILE variables pointing at mounted Secret files – the image supports that. Worth knowing for operations: the database and admin values are only read during that very first installation. Afterwards the credentials live in config.php, and changing the Secret alone changes nothing.
• A ten-minute startup gate. On a major version jump the entrypoint runs the upgrade before PHP-FPM starts, and that can take minutes. The startup probe allows up to sixty attempts at ten seconds; without it, the liveness probe would kill the container in the middle of its own upgrade.
• Three numbers that must agree. Uploads up to 2 GB need PHP_UPLOAD_LIMIT, nginx’s client_max_body_size in the sidecar and the same limit on the ingress – and PHP’s memory limit must be above the upload limit. The template says so where the values are set, because changing one of the three alone produces errors that point somewhere else.
• Configuration as a file. Nextcloud loads every *.php file in its config directory. The role mounts custom.config.php from a ConfigMap next to Nextcloud’s own config.php: language and region, session lifetime, the cache and locking setup, preview settings, log level, the trusted proxy (the pod network, so Nextcloud sees the visitor’s real address), trash retention of 180 days. A value set in this file takes precedence over the same key in config.php, so it cannot drift through the web interface.
• Caches sized for Nextcloud. OPcache gets 256 MB for up to 20,000 files – an installation with its apps has 15,500 to 17,800 PHP files, and the old limit of 10,000 let files fall out of the cache again. APCu, the per-pod cache all workers share, gets 128 MB instead of PHP’s 32 MB. The number of PHP-FPM workers is set per host: 30 on the large instance and 20 on the small one (measured peaks were 17 of 20 and 10 of 12), each with a container limit that fits.

Collabora and a key that has to exist once

Collabora signs every WOPI request with a proof key; Nextcloud verifies the signature. Older images generated that key in a start script. The current image starts its server directly and logged “Could not open proof RSA key” – it worked, but Nextcloud could no longer check that WOPI requests really come from this Collabora. The role now generates the key once and stores it as a Secret:

- name: Collabora | Does the WOPI proof key secret exist?
  kubernetes.core.k8s_info:
    kind: Secret
    namespace: nextcloud
    name: collabora-proof-key
  register: _collabora_proof_key
  no_log: true

- name: Collabora | Generate the WOPI proof key (first run only)
  ansible.builtin.shell: |
    set -o pipefail
    openssl genrsa -traditional 2048 2>/dev/null \
      | k3s kubectl -n nextcloud create secret generic collabora-proof-key --from-file=proof_key=/dev/stdin
  when: _collabora_proof_key.resources | length == 0
  no_log: true

The pipe is the point: openssl writes the key to its output, kubectl reads it from /dev/stdin into the Secret. The private key never touches the server’s disk and never appears in a log. The pod mounts it, and fsGroup makes it readable for Collabora’s user without making it readable for everyone. After a change to Collabora, next_config tells Nextcloud to re-read Collabora’s discovery document, so the new key is used immediately.

Two honest notes in the Collabora template are worth knowing as an operator. First, it is the one container that does not drop all capabilities: Collabora builds its own sandbox per document with chroot and namespaces and needs capabilities the others do not. Guessing that set from documentation could break document editing in a way that only shows when a user opens a file, so it is left until it can be tested properly. Second, the probes: readiness and liveness ask / (two bytes, “OK”), not the 43 KB discovery document. The kubelet reads only the first 10 KB of a probe response and closes the connection – and Collabora logged every such close as two errors, about 1,300 lines per two days per server. Only the startup probe still uses discovery, once, because it proves the whole service is up.

NetworkPolicy: Collabora is the risky one

The blog’s policy protects the database from everything but WordPress. Nextcloud’s policy has a different target: Collabora parses arbitrary documents that users upload – a more plausible entry point for an attacker than most PHP code – and it has no reason to ever talk to MariaDB or Redis. So the policy allows Collabora exactly what was verified to be needed: requests from the ingress and from the Nextcloud pod, DNS, and the WOPI callbacks to Nextcloud – port 443 to the host address, and explicitly not 3306 or 6379 on the same address. If Collabora were compromised, the database would still be out of reach.

Ingress: one host, four minions

The same master/minion structure as on the blog, with more minions: Nextcloud’s routes (with the security headers), Grafana, and – if push is enabled – /push/, with WebSocket headers and a long timeout. The push minion rewrites /push/x to /x, because notify_push serves its endpoints at its root, and its name matters: the controller orders regex locations of minions by Ingress name, and the wrong name let Nextcloud’s catch-all route answer first (“404 for /push/test/cookie” on the first rollout). Collabora gets its own master for its own host name.

The background CronJob

Every five minutes a pod runs cron.php as www-data, with the same volumes as Nextcloud. Two details: it keeps the capabilities SETUID/SETGID because it uses runuser to drop from root to www-data, which fails without them; and it gets the Redis password, because background jobs take the same file locks as the web workers.

apps.yml: only the deltas

Which apps are on or off should be reproducible – but not as a complete list. The bundled apps belong to the image and change between major versions. So the inventory only lists the deliberate differences: nextcloud_apps_disabled (forced off and kept off across upgrades) and nextcloud_apps_enabled (switched on – if possible).

- name: Apps | Ensure required apps are enabled (only if compatible with this major)
  ansible.builtin.command: >-
    k3s kubectl exec -n nextcloud {{ nc_pod.stdout }} -c nextcloud-fpm --
    php occ app:enable {{ item }}
  loop: "{{ nextcloud_apps_enabled }}"
  register: nc_app_enable
  changed_when: "'already enabled' not in (nc_app_enable.stdout | lower)"
  failed_when: false

- name: Apps | Report apps left disabled (no version compatible with this major)
  ansible.builtin.debug:
    msg: "App '{{ item.item }}' could not be enabled and stays disabled ..."
  loop: "{{ nc_app_enable.results | default([]) }}"
  when:
    - item.rc is defined
    - item.rc != 0

After a major upgrade an app may not have a compatible release yet. Nextcloud’s own upgrader simply leaves it disabled, and this role does the same: failed_when: false on the enable, then a second task loops over the results and reports exactly which apps stayed off. A run that would otherwise fail on one app now finishes and tells you what to look at. (occ here runs as root: since Nextcloud 34 the occ script switches to the owner of config.php by itself. The configuration role still uses runuser -u www-data explicitly.)

notify_push.yml: binary and app in lockstep

The push service is switched on per host (nextcloud_notify_push_enabled). The binary is shipped inside the notify_push app, so the order is: install and enable the app, read its version, deploy the pod for exactly that version, run the self-test. Two techniques make that robust:

• The version as an annotation. After an app update the running process still has the old binary. The installed app version goes into the pod template as an annotation – a new version changes the template, and Kubernetes rolls the pod. Binary and app can never drift apart.
• A loop that knows about when. Ansible evaluates a task’s loop before its when. The template needs the app version, which does not exist when push is disabled – so rendering it would fail even though the task is about to be skipped. The loop therefore returns an empty list when push is off: {{ [] if not (nextcloud_notify_push_enabled | bool) else ... }}. A detail you only learn by hitting it.

The last step had an idempotence bug of its own. occ notify_push:setup saves the endpoint and prints “configuration saved” every time, so it reported a change on every run. Now the role reads the stored endpoint first: if it is already correct, it runs only notify_push:self-test – the same checks, nothing written. A failed setup is harmless, by the way: Nextcloud only switches clients to push after the self-test passed; until then they keep polling.

⚙️ next_config: get, compare, set

Nextcloud is running; now its settings. The tool is occ, Nextcloud’s command line, run inside the pod as the owner of config.php:

k3s kubectl exec deployment/nextcloud -n nextcloud -c nextcloud-fpm \
  -- runuser -u www-data -- php /var/www/html/occ <command>

The role stores the first part once with set_fact (occ_exec) and uses it in every task. Then it applies one pattern to every value – the pattern its header calls “get, compare, set”: read the current value with occ config:...:get and register it; write only when it differs. A run that changes nothing reports nothing, and a value someone changed in the admin interface is put back and shows up as a change. That second part is what makes the drift check meaningful for application settings, not just for files.

nc_config.yml: domains, URL and mail

Trusted domains and the command-line URL are simple get/compare/set pairs. The mail settings are more interesting, because there are nine of them and one is a password:

- name: Config | Read current mail settings
  ansible.builtin.command: "{{ occ_exec }} ... config:system:get {{ item.key }}"
  loop: "{{ nc_mail_settings }}"
  register: nc_mail_current
  changed_when: false
  failed_when: false        # rc 1 = not set yet

- name: Config | Set mail settings that differ
  ansible.builtin.command:
    argv: "{{ ... + ['config:system:set', item.item.key, '--value', item.item.value] }}"
  loop: "{{ nc_mail_current.results }}"
  when: item.rc != 0 or (item.stdout | trim) != (item.item.value | string)
  no_log: true

- name: Config | Report mail settings a real run would change (--check only)
  ansible.builtin.debug: ...
  when: ansible_check_mode and (...same condition...)
  changed_when: true

Three ideas in one place. The write task loops over the results of the read task, so each item carries both the wanted value (item.item.value) and the current one (item.stdout). argv passes the values verbatim – a quote or a space in the password cannot break the command line. And because command tasks are skipped in a dry run, the pending change would be invisible there; the third task exists only for check mode and reports what a real run would change. The old version rewrote all nine values on every run and reported “ok” either way, so a changed SMTP password never showed up at all.

db_maintenance.yml, apps.yml, wopi.yml

• Database maintenance: occ db:add-missing-indices and occ maintenance:repair --include-expensive – the two things Nextcloud’s admin overview otherwise nags about after an upgrade. Both check what is missing and do nothing when everything is in place.
• Office and previews: Nextcloud Office (the richdocuments app) is downloaded and unpacked with Ansible’s own modules instead of occ app:install, because the latter reproducibly truncated one file of this app at the same byte offset every time. Bypassing occ also bypasses its signature check – so the role puts it back with occ integrity:check-app and the certificate shipped with the app, and warns on a mismatch. The preview generator app is installed and enabled normally.
• WOPI: the allowlist of addresses that may call Nextcloud’s WOPI endpoint (it must contain the pod network, because Collabora calls back from its pod address) and the Collabora URL – get, compare, set. One task uses a variable registered by next_k3s_deploy, an earlier role in the same play: registered variables stay available to all later roles on that host. collabora_apply is defined guards runs where that role was skipped with tags.

password_policy.yml: a lockout and a lesson

The last file switches on the password policy app and sets its rules: at least 12 characters, no common passwords, and a check against known leaked passwords. That check uses the Have I Been Pwned service with k-anonymity: only the first five characters of the password’s hash leave the server, and the service returns all leaked hashes with that prefix to compare locally – the password itself is never sent. Character-class rules (upper case, digits, special characters) are explicitly off: length and the leak check matter more, and class rules mostly produce “Password1!”.

This file also has the most instructive incident of the whole repository. When the policy app was enabled on one server, every user whose password was older than 300 days was locked out at the next login – including the admin. The cause was a leftover setting from the old server, expiration = 300, that had been sitting unnoticed in the database until the app that reads it was switched on. Since then the role sets every key of the policy explicitly, including the ones that should be off (expiration = 0), so no leftover can come back. The general lesson for operators: before you enable an app, read what configuration it already has.

A second detail is technical: Nextcloud 34 stores app settings with a type, and old values from earlier versions are untyped. Setting such a key with --type=boolean stops at an interactive “type yes to confirm” prompt – which aborts an Ansible run. So a differing old value is deleted first and then written fresh with its type.

✅ nextcloud_verify

The same structure as on the blog (part 3): checks that only collect, a report, a mail, and an assert at the end; host, Kubernetes and HTTP basics from common_verify. The Nextcloud-specific checks: MariaDB and Redis running on the host, all pods and Collabora running, status.php healthy, the login page and Collabora’s discovery reachable, config.php and the data directory not reachable from outside, background jobs in cron mode, Redis and MariaDB reachable from the Nextcloud pod, and the office URL configured correctly.

🔄 nextcloud-update.yml

Part 1 introduced its guard rails; here is the full sequence, because updating a file cloud that people depend on deserves care:

1. Refuse without --limit – one instance at a time.
2. Refuse an unhealthy instance: installed, not in maintenance mode, no pending database upgrade. Why that guard exists.
3. Rebuilt images: common_image_refresh finds Deployments whose image tag was rebuilt upstream; only those are restarted, in a fixed order.
4. occ upgrade – normally done by the entrypoint on start, called again as a safety net. Only “Update successful” in the output counts as a change.
5. occ app:update --all – app updates.
6. A second restart, and this one deserves an explanation. Nextcloud runs with opcache.validate_timestamps=0: PHP never checks whether a file changed, compiled code stays in memory for the life of the pod. That is fast – and it means that files overwritten by occ upgrade and the app updates would keep running as the old code until the next unrelated restart, security fixes included. So the pod is restarted once more after the updates.
7. Post-check: the same health conditions as before. The run fails if Nextcloud is not healthy afterwards.

Steps 3 to 7 are inside a block. The rescue shows the status, the pods and events, and the last lines of nextcloud.log. Then it makes one careful decision: it leaves maintenance mode only if the upgrade itself has completed (no pending database upgrade). With a pending upgrade, maintenance mode stays on – users would otherwise reach a half-upgraded instance. Then it stops and prints what to do next.

🛠️ Running it: Nextcloud day to day

Looking at it

# Pods: nextcloud, collabora, notify-push; cron pods come and go
k3s kubectl get pods -n nextcloud

# Nextcloud's own view: version, maintenance mode, pending upgrade
k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php occ status

# Nextcloud's log (warnings and errors), on the host
tail -f /data/nextcloud.log

# Background jobs: when did cron last run? (Admin settings → Basic settings shows it too)
k3s kubectl get jobs -n nextcloud

# Disk: user data, trash, versions
df -h /data

The routine

When something goes wrong

• “Office failed to load”, and a retry works: a short-lived hiccup between browser, Collabora and Nextcloud – usually harmless. If it happens often, Collabora needs more pre-started processes or more memory (collabora_num_prealloc and the memory limit, per host in the inventory). The large instance keeps 10 processes warm in 8 GiB, the small one 6 in 4 GiB.
• Nextcloud stuck in maintenance mode after a failed update: look at occ status. If an upgrade is pending, run occ upgrade and read its output; only then occ maintenance:mode --off. Never switch it off with an upgrade still pending.
• Users locked out by the password policy: check occ config:app:get password_policy expiration – it must be 0. The role pins it, but it is the first thing to look at.
• Disk filling up: the alarm comes at 80 percent (part 6). The trash keeps deleted files for 180 days and cleans itself; a single user’s trash can still be large. Previews can be regenerated and are not a real reserve. occ trashbin:cleanup --all-users empties every trash – with the users’ consent, not by surprise.
• The whole server is gone: set up a new one with the full playbook, then restore the database dump, the application directory (with config.php and the installed apps) and the user data from the external backup with the restore script, and run the playbook once more. The operations manual in the repository lists every step.

🧭 What comes next

With part 4 both applications are covered. Part 5 collects every security role in one place – SSH, the firewall, the CIS hardening, SELinux, audit, AIDE, rkhunter, ClamAV and fail2ban – and explains what each layer protects against. Part 6 is monitoring: exporters, Prometheus, Grafana and the Monit watchdog, and what each alarm means.