Nextcloud – Operations Guide

Overview

Nextcloud runs on a single-node K3s cluster on AlmaLinux 9. The entire infrastructure is set up and managed with an Ansible playbook (nextcloud-k3s.yml). The playbook is idempotent – it can be re-run at any time without modifying data or unnecessarily interrupting services.

Architecture

Overall Overview

Why This Approach?

Nextcloud Pod in Detail

Both containers share the same pod (= the same network namespace and the same volumes). This allows nginx to communicate with PHP-FPM via 127.0.0.1:9000 without needing a Kubernetes service.

Storage & Data

Why Are App and Data Separate?

Nextcloud app files (/srv/nextcloud-www) and user data (/data) are intentionally split into two separate HostPath volumes because:

• User data can be backed up independently (daily recommended)
• App files are replaced by the new image during an update
• For a fresh installation, an empty /srv/nextcloud-www is sufficient – Nextcloud installs itself on first start

Important Configuration File: config.php

Nextcloud automatically reads all *.php files from /var/www/html/config/. There are two sources:

• config.php – written by Nextcloud itself (initial installation)
• custom.config.php – from the Kubernetes ConfigMap nextcloud-custom-config, managed by Ansible. This file takes precedence and overrides values from config.php.

Network & TLS

IP Address Ranges

Required DNS Records

DNS records must be created at the domain registrar before the first playbook run. cert-manager requires them for the Let's Encrypt HTTP-01 challenge – without valid records certificate issuance will fail.

How Pods Reach the Host (MariaDB / Redis)

MariaDB and Redis run as host services. Pods connect to them via K8s ExternalServices: a service without a selector with manually maintained endpoints pointing to the public IP of the host.

Pod → mariadb.nextcloud.svc.cluster.local (10.43.x.x)
   → Endpoint: 82.165.165.230:3306
   → iptables ACCEPT for 10.42.0.0/16 on :3306

The MariaDB database user is restricted to the pod CIDR ('nextcloud'@'10.42.%'). A direct connection from the host only works as root via Unix socket.

TLS and Proxy Chain

Internet
  → nginx-ingress (F5) (Port 443, TLS terminated, sets X-Forwarded-For / -Proto)
    → nginx sidecar (Port 80, plain HTTP)
      → PHP-FPM (Port 9000, FastCGI)
        fastcgi_param HTTPS on
        fastcgi_param HTTP_SCHEME https

Nextcloud needs to know that requests are coming through a proxy:

'trusted_proxies'       => ['10.42.0.0/16'],
'forwarded_for_headers' => ['HTTP_X_FORWARDED_FOR'],
'overwriteprotocol'     => 'https',

WOPI (Collabora)

Collabora calls Nextcloud's WOPI API for document editing. These callbacks come from the pod IP of the Collabora pod (10.42.x.x) – not from the service IP. The wopi_allowlist must therefore contain the pod CIDR:

# inventory/host_vars/<host>/vars.yml
wopi_allowed_hosts:
  - "10.42.0.0/16"   # Pod CIDR (Collabora pod → Nextcloud WOPI)
  - "10.43.0.0/16"   # Service CIDR
  - "collabora.inspired-as-code.de"

Important Files in the Repository

Inventory & Variables

Ansible Roles (Nextcloud-specific)

Templates (generating the K8s manifests)

Variables & Secrets

Where Credentials Are Stored

All passwords are stored in inventory/host_vars/<host>/vault.yml, encrypted with Ansible Vault. This file must never be committed unencrypted to git.

# Encrypt before committing:
ansible-vault encrypt inventory/host_vars/test/vault.yml

# Decrypt for editing:
ansible-vault decrypt inventory/host_vars/test/vault.yml
# (re-encrypt immediately afterwards!)

# Open directly in editor (preferred):
ansible-vault edit inventory/host_vars/test/vault.yml

Password File for Unattended Runs

# .vault_pass (never commit – listed in .gitignore)
echo "my-vault-password" > .vault_pass
# Uncomment in ansible.cfg:
# vault_password_file = .vault_pass

Secrets in the K8s Cluster

The vault variables flow into the K8s secret nextcloud-secrets in the namespace nextcloud during the playbook run. The secret is injected as environment variables into the pod.

# Show secret (decoded):
k3s kubectl get secret nextcloud-secrets -n nextcloud \
  -o jsonpath='{.data}' | python3 -c \
  "import sys,json,base64; [print(k,base64.b64decode(v).decode()) \
   for k,v in json.load(sys.stdin).items()]"

Container Images & Versions

All image tags are centrally defined in inventory/group_vars/all.yml. Since imagePullPolicy defaults to IfNotPresent for tagged images, K3s only pulls a new image automatically when the tag in the manifest has changed.

# inventory/group_vars/all.yml
nextcloud_image_fpm:       "nextcloud:33-fpm"      # Nextcloud PHP-FPM
nextcloud_image_nginx:     "nginx:1.30-alpine"          # nginx Sidecar
nextcloud_image_collabora: "collabora/code:25.04.9.4.1" # Collabora CODE
nextcloud_image_busybox:   "busybox:1"             # Init container

Tag Strategy

crictl pull docker.io/library/nextcloud:33-fpm
k3s kubectl rollout restart deployment/nextcloud -n nextcloud

Nextcloud Configuration

Configuration is done in two ways:

1. custom.config.php (via Ansible ConfigMap) – static settings that are updated with every playbook run
2. occ commands (via kubectl exec) – dynamic settings such as trusted_domains, WOPI URL, app installation

custom.config.php – Overview of Key Values

// Language
'default_language'     => 'de',
'default_locale'       => 'de_DE',
'force_language'       => 'de',

// Session (1 hour, no "stay logged in")
'session_lifetime'               => 3600,
'auto_logout'                    => true,
'remember_login_cookie_lifetime' => 0,

// Redis (cache, locking, sessions)
'memcache.local'       => '\\OC\\Memcache\\Redis',
'memcache.distributed' => '\\OC\\Memcache\\Redis',
'memcache.locking'     => '\\OC\\Memcache\\Redis',
'redis' => ['host' => 'redis', 'port' => 6379],

// Reverse proxy
'trusted_proxies'       => ['10.42.0.0/16'],
'forwarded_for_headers' => ['HTTP_X_FORWARDED_FOR'],
'overwriteprotocol'     => 'https',

// Logging (in /data/nextcloud.log on the host)
'loglevel'        => 4,
'logtimezone'     => 'Europe/Berlin',
'log_rotate_size' => 104857600,   // 100 MB

// Maintenance window (1 a.m.)
'maintenance_window_start' => 1,

Verify Configuration After Playbook Run

k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ config:list system

k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ setupchecks

Run Playbook

Full Initial Run (New Server Installation)

1. Set SSH port to 22 in inventory/host_vars/test/vars.yml: ansible_port: 22
2. Enter vault passwords in vault.yml and encrypt
3. Install collections: ansible-galaxy collection install -r requirements.yml
4. Run playbook:

   ansible-playbook nextcloud-k3s.yml --limit test --ask-vault-pass

5. Change SSH port in vars.yml to 10022 (after SSH hardening by common_ssh)

Idempotent Re-Run (e.g. After Config Change)

ansible-playbook nextcloud-k3s.yml --limit test --ask-vault-pass

The playbook detects what is already correctly configured and only changes what is necessary. Kubernetes manifests are only re-rolled when the manifest has changed (changed: "configured" in output).

Run Only Specific Roles

# Only Nextcloud configuration (occ commands)
ansible-playbook nextcloud-k3s.yml --limit test --tags next_config --ask-vault-pass

# Only update K8s deployments
ansible-playbook nextcloud-k3s.yml --limit test --tags next_k3s_deploy --ask-vault-pass

Important Operations Commands

Pod Status

# All pods in the nextcloud namespace
k3s kubectl get pods -n nextcloud

# Detailed status (events on problems)
k3s kubectl describe pod -l app=nextcloud -n nextcloud

# Logs
k3s kubectl logs deployment/nextcloud -c nextcloud-fpm -n nextcloud --tail=50
k3s kubectl logs deployment/nextcloud -c nginx -n nextcloud --tail=50
k3s kubectl logs deployment/collabora -n nextcloud --tail=50

occ Commands (Nextcloud CLI)

# Prefix for all occ commands:
k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ <COMMAND>

# Examples:
... occ status
... occ check
... occ setupchecks
... occ maintenance:mode --on
... occ maintenance:mode --off
... occ files:scan --all
... occ db:add-missing-indices
... occ app:list
... occ config:list system
... occ config:system:get trusted_domains

Restart Pod

# Nextcloud (rolls gracefully due to Recreate strategy)
k3s kubectl rollout restart deployment/nextcloud -n nextcloud
k3s kubectl rollout status  deployment/nextcloud -n nextcloud --timeout=300s

# Collabora
k3s kubectl rollout restart deployment/collabora -n nextcloud

Namespace Overview

k3s kubectl get all -n nextcloud

MariaDB (Host)

# As root via socket (credentials in /root/.my.cnf):
mysql test_nextcloud

# Create DB dump:
mysqldump --single-transaction test_nextcloud > /tmp/dump.sql

Redis (Host)

# Test connection:
redis-cli ping

# Flush cache (e.g. after configuration change):
redis-cli FLUSHALL

Monitoring

The monitoring stack (Prometheus + Grafana + Node Exporter + mysqld_exporter + php-fpm_exporter) runs partly as host services, partly as sidecar containers in the Nextcloud K3s pod. Grafana is accessible via K3s ingress on the Nextcloud hostname under /grafana/.

MariaDB Slow Query Log

The host MariaDB writes slow queries (≥ 1 second) to /var/log/mariadb/slow.log. The file is created automatically by MariaDB and can be read via journalctl or directly.

# Read slow query log live
tail -f /var/log/mariadb/slow.log

# Filter queries from the last 24 hours
grep -A 3 "Query_time" /var/log/mariadb/slow.log | grep -v "^--$"

# Check log size
ls -lh /var/log/mariadb/slow.log

Configuration: /etc/my.cnf.d/nextcloud-k3s.cnf → slow_query_log = 1, long_query_time = 1

PHP-FPM Slow Log

PHP-FPM logs requests that take longer than 5 seconds to /proc/1/fd/2 (stderr of the pod's main process). Since ptrace is not available in containers, no stack traces are written – only the request timestamp.

# Monitor slow FPM requests
k3s kubectl logs -n nextcloud deployment/nextcloud -c nextcloud-fpm -f | grep -i "slow"

Configuration: ConfigMap nextcloud-fpm-pool-config → pm.status_path = /fpm-status, slowlog = /proc/1/fd/2, request_slowlog_timeout = 5s

Grafana Dashboards

Node Exporter Full (ID 1860) – Host Metrics

Shows the health of the entire server in real time.

MySQL Overview (ID 7362) – MariaDB Database Metrics

Shows the performance of the Nextcloud database (host MariaDB, port 9104).

PHP-FPM (ID 4912) – PHP Process Pool

Shows how well PHP-FPM handles incoming Nextcloud requests (K3s pod, NodePort 30254).

mysqld_exporter – Check Status

# Service status
systemctl status mysqld_exporter

# Logs
journalctl -u mysqld_exporter -n 50

# Fetch metrics directly (test)
curl -s http://127.0.0.1:9104/metrics | grep mysql_up

Updates

Operating System (AlmaLinux 9)

Security updates are automatically applied daily by dnf-automatic. For a full system update:

1. ssh -p 10022 root@server
   dnf upgrade -y

2. Check if a reboot is needed:

   needs-restarting -r

3. If needed: reboot the server. K3s starts automatically after the reboot, pods come back up on their own.

   reboot
   # Check afterwards:
   k3s kubectl get pods -n nextcloud

Container Update: Patch Within the Same Tag

When Docker Hub updates e.g. nextcloud:33-fpm from 33.0.2 to 33.0.3 and the tag in Ansible remains unchanged:

1. Pull new image manually:

   crictl pull docker.io/library/nextcloud:33-fpm

2. Restart pod (now uses the freshly pulled image):

   k3s kubectl rollout restart deployment/nextcloud -n nextcloud
   k3s kubectl rollout status  deployment/nextcloud -n nextcloud --timeout=300s

3. Ensure DB migrations:

   ansible-playbook nextcloud-k3s.yml --limit test --tags next_config --ask-vault-pass

Container Update: Major Version Jump (e.g. 33 → 34)

1. Change tag in inventory/group_vars/all.yml:

   nextcloud_image_fpm: "nextcloud:34-fpm"
   # if applicable also:
   nextcloud_image_collabora: "collabora/code:25.08"

2. Run playbook – K3s pulls the new image automatically and rolls out the pod:

   ansible-playbook nextcloud-k3s.yml --limit test --ask-vault-pass

3. The Nextcloud container runs occ upgrade automatically on startup. The playbook run then executes the next_config role (DB indices, WOPI config, app settings).
4. Check admin panel for warnings:

   k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
     runuser -u www-data -- php /var/www/html/occ setupchecks

Where to Check for New Versions

Backup & Restore

What Is Backed Up

Run Backup

The script expects the instance name as an argument. The instance controls which server is contacted and where the backup is stored locally.

# Example: back up test instance
cd ~/www_k3s
./scripts/nextcloud-backup.sh test

# Process (applies to all instances):
# 1. Enable maintenance mode  (kubectl exec occ maintenance:mode --on)
# 2. mysqldump --single-transaction → /data/nextcloud/<env>/db.sql
# 3. rsync /srv/nextcloud-www/      → /data/nextcloud/<env>/nextcloud/
# 4. rsync /data/                   → /data/nextcloud/<env>/data/
# 5. Disable maintenance mode

The script is located in the repository directory scripts/nextcloud-backup.sh. Sofie and CVJM are only usable once IP and hostname are entered in inventory/host_vars/<env>/vars.yml – the script checks this and aborts with an error message if changeme is still present.

Restore Workflow

The restore script also expects the instance name as an argument:

1. Set up server (infrastructure) – replace test with instance name:

   ansible-playbook nextcloud-k3s.yml --limit test --ask-vault-pass

2. Run restore script (overwrites the fresh installation):

   cd ~/www_k3s
   ./scripts/nextcloud-restore.sh test

   Process: enable maintenance → stop pod → rsync app files → rsync user data → SQL import → fix ownership → start pod → files:scan → disable maintenance

3. Run Ansible again (ensures occ config: trusted_domains, WOPI, apps):

   ansible-playbook nextcloud-k3s.yml --limit test --ask-vault-pass

   This step is mandatory on a hostname change, otherwise recommended.

Manual Individual Commands (Reference)

# Maintenance mode
k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ maintenance:mode --on

# Database dump (on the server)
mysqldump --single-transaction <db_name> > /tmp/nc.sql

# Fix ownership after manual rsync
chown -R 33:33 /srv/nextcloud-www /data
chmod 0755 /srv/nextcloud-www && chmod 0770 /data

# Rebuild file index
k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ files:scan --all

Troubleshooting

Pod Does Not Start / Stays in CrashLoopBackOff

k3s kubectl describe pod -l app=nextcloud -n nextcloud
k3s kubectl logs deployment/nextcloud -c nextcloud-fpm -n nextcloud --previous

Typical causes:

• DB not reachable: MariaDB service is not running or iptables rules are missing (after reboot?)
• config.php empty (0 bytes): First start could not write to config/ due to incorrect ownership. Fix: chown 33:33 /srv/nextcloud-www/config, delete empty file, restart pod.
• Redis not reachable: Check systemctl status redis on the host

Nextcloud Page Loads But CSS/JS Is Missing (Page Looks Broken)

# Check MIME types of assets:
curl -sI "https://nextcloud.example.com/core/css/server.css" | grep content-type
# Must be: text/css

curl -sI "https://nextcloud.example.com/dist/core-main.js" | grep content-type
# Must be: application/javascript

In the past, the cause was a types {} block in the nginx server context that overrode all inherited MIME types. The current configuration uses a dedicated location block for .mjs.

Admin Warning: "Reverse Proxy Header Is Incorrect"

trusted_proxies is missing from custom.config.php or does not contain the K3s pod CIDR (10.42.0.0/16).

k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ config:system:get trusted_proxies

Collabora: "Unauthorized WOPI Host"

The wopi_allowlist does not contain the pod CIDR (10.42.x.x). Collabora pods connect to Nextcloud WOPI from their pod IP.

k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ \
  config:app:get richdocuments wopi_allowlist

# Fix:
k3s kubectl exec -n nextcloud deployment/nextcloud -c nextcloud-fpm -- \
  runuser -u www-data -- php /var/www/html/occ \
  config:app:set richdocuments wopi_allowlist \
  --value="10.42.0.0/16,10.43.0.0/16,collabora.example.de,82.165.165.230"

Certificate Is Not Issued

# Check cert-manager logs:
k3s kubectl logs -n cert-manager deployment/cert-manager --tail=50

# Check certificate object:
k3s kubectl describe certificate nextcloud-tls -n nextcloud

# ClusterIssuer status:
k3s kubectl describe clusterissuer letsencrypt-prod

Common cause: port 80 is blocked by iptables or the provider. cert-manager requires port 80 for the HTTP-01 challenge.

CronJob Is Not Running

# Status of recent jobs:
k3s kubectl get cronjob nextcloud-cron -n nextcloud
k3s kubectl get jobs -n nextcloud --sort-by=.metadata.creationTimestamp | tail -5

# Logs of the last job pod:
k3s kubectl logs -n nextcloud -l job-name=nextcloud-cron-<id>

Security

Multi-Layered Protection Strategy

Firewall in Detail

The script /root/fw/fw_ipv4.sh (from common_firewall) is applied via @reboot cron job after every restart. K3s then inserts its own KUBE-* chains at the top of the INPUT/OUTPUT chains.

# Monitor blocked packets live
journalctl -k -f --grep "IPTables-Dropped"

# Show port scan blocklist
cat /proc/net/xt_recent/portscan | awk '{print $1}'

# Reload firewall rules (happens automatically after reboot)
/root/fw/fw_ipv4.sh && /root/fw/fw_ipv6.sh

Important Security Notes