🌐 Also available in: Deutsch · Français · Español
Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.
On 16 October 2026 the three servers got one tuning pass: the blog and two Nextcloud servers. None of them was slow. The question was a different one: there was free memory on all three – where would it actually help, and where would it only look busy? This post walks through the method, every change with the number behind it, the things left alone on purpose, and what the measurements four days later say.
🧭 The method
- Peaks over seven days, not snapshots. A look at
topshows one moment. Prometheus keeps 30 days of samples every 15 seconds;max_over_time(…[7d])gives the worst minute of the week, which is what limits have to survive. - Counts instead of assumptions. How many PHP files exist, how many temporary tables go to disk per day, how many inotify instances are in use – each one a single command.
- One reason per change. Every setting below has a measurement next to it. Settings that had none were not touched.
- Before and after. The same queries again after the rollout, plus container metrics that show whether a limit is now too tight.
The resources, and what to read from them
| Resource | Metric | Trap |
|---|---|---|
| CPU | usage per container against its limit; CFS throttling | A container can be throttled while the host is idle – the limit, not the machine, is the bottleneck. |
| Memory | working set against the limit | “Used” includes page cache; free RAM is not wasted, Linux uses it for files. |
| Worker memory | PSS instead of RSS | RSS counts shared memory (OPcache) in every process and overstates by a factor of five here. |
| Queues | FPM listen queue, max_children reached | Zero queue means enough workers – more would only cost memory. |
| Kernel limits | inotify instances and watches, file descriptors | They fail with odd errors (“too many open files”) long before anything looks full. |
🛡️ Why bother when nothing is slow
- Limits are cheap to raise before they bite and expensive to diagnose afterwards, when the symptom is a sync client that hangs once a week.
- Defaults are someone else’s guess. Two of them turned out to be wrong for Nextcloud by a factor of four.
- Knowing what not to change is half the result: bigger buffers in the wrong place only move the bottleneck or hide it.
🔧 What was changed
| Area | Measured | Changed to | Where |
|---|---|---|---|
| PHP-FPM workers (Nextcloud) | peak 17 of 20 and 10 of 12 busy at once | 30 and 20 workers; container limits 6 and 5 GiB | host variables |
| OPcache (Nextcloud) | 15,500 and 17,800 PHP files; limit 10,000 = 16,229 slots | 256 MB, 20,000 files (32,531 slots) | PHP config map |
| APCu (Nextcloud) | PHP default 32 MB, one segment for all workers | 128 MB | PHP config map |
| MariaDB temporary files | 2,000–3,500 temporary tables a day on disk | /tmp as tmpfs (host: 4 GB, blog pod: 512 MiB) | MariaDB unit / pod |
| Collabora | “office failed to load” on a cold start | 10 and 6 pre-started document processes | host variables |
| Hanging PHP requests | no wall-clock limit at all | request_terminate_timeout 300 s blog / 3,600 s Nextcloud | FPM pool |
| Kernel: inotify | default 128 instances per user; root already used 63 | 8,192 instances, 524,288 watches | common_sysctl |
| Kernel: write-back | dirty pages flushed at 10 / 30 % of RAM | 5 / 10 % | common_sysctl |
Why these and not others
- Workers: 17 of 20 is 85 % at peak – the next busy afternoon would have queued. The container limits followed the pools, not the other way round: PSS per worker times
max_children, plus OPcache and APCu, with room to spare. - OPcache: on the smaller Nextcloud server 1,500 files never fit and were compiled on every request. The number of files decides this, not a feeling.
- Temporary tables with TEXT columns can never use the memory engine, so tuning
tmp_table_sizewould have changed nothing. Moving the directory they are written to into RAM does. One trap on the way: combined withPrivateTmp=yes, systemd’s private directory on disk wins and the tmpfs is silently ignored – found by testing withsystemd-run, not by reading. And in the blog pod a memory-backed volume counts against the container’s memory limit, which therefore went from 768 MiB to 1,280 MiB. - inotify: K3s, containerd and every pod with a file watcher take instances as root. 63 of 128 is fine today and a mystery error after the next few pods.
- Write-back at 10 % instead of 30 % of 23 GB means at most 2.3 GB of unwritten data instead of 7 GB – shorter stalls when the kernel finally has to write, and less data lost in a hard crash.
tuned and sysctl: who wins
AlmaLinux runs tuned with the profile virtual-guest, which sets vm.dirty_ratio = 30 itself. Our 10 is what the kernel reports, because tuned re-applies /etc/sysctl.d/ after its own profile (reapply_sysctl = 1, the default). Without that, the profile would quietly undo the setting at every boot. Checking the live value with sysctl -n after a reboot is the only reliable proof.
🧊 What was deliberately left alone
| Area | Measured | Why not |
|---|---|---|
| InnoDB buffer pool | hit rate > 99.8 %, pools larger than the data (2 GB for 344 MB) | More memory cannot improve a cache that already holds everything. |
| Redis | 1–1.5 MB used | Nothing to gain. |
| Blog PHP-FPM | peak 6 of 12 workers, queue at most 1 | The pool was big enough. |
| Blog OPcache | ~4,600 PHP files, limit 10,000 | Fits. |
| MariaDB connections | peak 15 of 50 | Plenty of room. |
Free memory that nothing uses is not lost either: Linux fills it with file cache, which is exactly what a file server wants.
📊 Four days later
| Blog | Nextcloud A | Nextcloud B | |
|---|---|---|---|
| Peak active PHP workers (7 days) | 8 of 12 | 13 of 30 | 9 of 20 |
max_children reached | 0 | 0 | 0 |
| Container memory peak / limit | 538 MiB / 1 GiB | 763 MiB / 6 GiB | 277 MiB / 5 GiB |
| CPU periods throttled | 2.1 % | 2.2 % | 0.2 % |
| Slow PHP requests (> 5 s) | 0 | 0 | 0 |
| Slow database queries (> 1 s) | 0 | 0 | 0 |
| Memory available on the host | 5.0 of 7.5 GB | 19 of 23 GB | 8.5 of 11.4 GB |
No pool came near its limit, no container near its memory limit, throttling stays in the low single digits, and the hosts still have most of their memory free. The changes cost little and removed three real ceilings – workers, bytecode slots, a wall-clock limit – before anyone hit them.
The measuring mistake
While collecting the numbers above, the first query for the blog’s PHP container returned a peak of 940 MiB against a 1 GiB limit – which would have called for a higher limit at once. The query summed all series of the container: during a rollout the old and the new pod exist at the same time, and right after a container restart cAdvisor briefly reports the old and the new cgroup. The peak of a single pod was 538 MiB. Summing by pod, or taking the maximum per pod, is the correct reading; a plain sum by container name adds up things that never existed at the same moment in one process.
🧰 Repeating the exercise
# worst minute of the week, per pod (not summed over pods!)
max_over_time(max by (pod) (container_memory_working_set_bytes{container="nextcloud-fpm"})[7d:1m])
# share of CPU periods throttled
sum(increase(container_cpu_cfs_throttled_periods_total{container="nextcloud-fpm"}[7d]))
/ sum(increase(container_cpu_cfs_periods_total{container="nextcloud-fpm"}[7d]))
# PHP-FPM peak and queue
max_over_time(sum(phpfpm_active_processes)[7d:1m])
max_over_time(max(phpfpm_listen_queue)[7d:1m])- When: after a change in load (new users, a new app), after a major upgrade, and otherwise every few months.
- Where to look first: the “Container-Ressourcen” dashboard (limits, throttling, OOM kills) and the alerts on container memory and throttling, which mail when a limit gets tight.
- What to write down: the measurement next to the setting. The operations guide has a table for exactly that, so the next pass starts from numbers, not from memory.
🆘 Cheat sheet: symptom → command
The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders.
| Symptom | Command |
|---|---|
| Is a pod short of memory? | max_over_time(max by (pod) (container_memory_working_set_bytes{container="<c>"})[7d:1m]) |
| Is a pod short of CPU (throttled)? | sum(increase(container_cpu_cfs_throttled_periods_total{container="<c>"}[7d])) |
| Are there enough PHP workers? | max_over_time(sum(phpfpm_active_processes)[7d:1m]) |
| Was something killed for lack of memory? | journalctl -k | grep -i "out of memory" |
| What does the host have left? | free -h; uptime; df -h / |
| Where to look first | # Grafana: dashboard "Container-Ressourcen" (limits, throttling, OOM kills) |
🔭 What comes next
- Load tests with k6 against a test instance, to see where the ceilings are instead of waiting for them.
- Alert thresholds from measured peaks instead of round numbers – the first one, for the PHP-FPM queue, came from 14 days of data in October 2026.
- Capacity planning: data growth of the larger Nextcloud server per month against its disk.
⚖️ Alternatives – and why measure first
| Approach | Strength | Weakness |
|---|---|---|
| Tuning by checklist | Fast, no tooling needed | Changes what the author of the list needed, not what this server needs. |
| Bigger machine | Always helps a little | Hides the ceiling instead of moving it; costs every month. |
| Commercial APM tools | Traces down to a single function | Cost, an agent in every container, and overkill for these workloads. |
| Measure, then change one thing | Every change has a reason and a check | Needs metrics that are already there – which is why they came first. |
The most valuable result of the day was not a setting but a list: eight changes, each with a number, and five things left alone, each with a number too. The next tuning pass will start from it, and if a value turns out wrong, it will be clear what it was based on.
