🌐 Also available in: Deutsch · Français · Español
Version v3.6.0 of the Ansible code behind this blog and two Nextcloud servers is out. It is the last release with new features for a while – from here on the focus is maintenance. The common thread of the last days: almost every bug of the previous weeks had slipped through because the checks asked “is it configured and running?” instead of “does it actually work?”. This release changes that.
📦 The download that stopped at 1 GB
The newest bug was found this morning, and not by a test: an alert mail said a container was CPU-throttled more than half of the time. The container was the nginx in front of PHP on one of the Nextcloud servers – a component that usually idles.
Behind it was a user who tried to download a 13 GB archive. Every attempt broke off at about 1.08 GB. Two defaults met: the ingress buffers a response into a temporary file of at most 1 GB, then stops reading from the pod until the client catches up. The nginx in the pod waits 60 seconds for progress and then closes the connection. The client receives a broken download, and the ingress logs “upstream prematurely closed”. The timestamps matched on three attempts in a row: 1 GB in a few seconds, then exactly 67 seconds of silence.
The fix is one annotation that lets the ingress stream at the client’s pace instead of filling a temp file:
nginx.org/proxy-max-temp-file-size: "0"The CPU alert was only a side effect: copying 1 GB in a few seconds briefly pushed a container with a 0.5-core limit to its ceiling.
The takeaway: two sensible defaults can produce a bug together. An alert on a symptom is worth following even when the symptom itself looks harmless.
🧪 Tests that check the effect
The verify roles now test what a component is supposed to achieve, not just whether it exists:
- Ban chain: for every fail2ban jail a documentation address (TEST-NET) is banned, must appear in the right firewall set, is unbanned and must be gone again. Two jails ban the same address, one unbans – the other ban must stay.
- Filters: a canned log line in the current format runs through
fail2ban-regexfor every jail. A log format change can no longer silently blind a jail. - Data presence: key metrics must exist, and no Grafana alert rule may sit in an error state or show no data for good.
- SELinux domains: every host daemon must run in its own domain. “Active (running)” in the wrong domain counts as a failure.
The takeaway: “configured and running” is a state. “Does it work” is a test with an input and an expected result.
💓 Heartbeats and the mail path
Some failures are silent: a nightly job that no longer runs never sends an error. Every periodic job – AIDE, rkhunter, ClamAV, automatic updates, the login and ban notifiers – now writes a timestamp as a metric when it finishes. If a job is overdue, Grafana alerts.
That only helps if alert mails arrive. So one rule fires on purpose every week; if that mail does not come, the mail path is broken. Monit also watches the mail log for failed deliveries.
The takeaway: silence is not a signal. A job has to report that it ran, and the alerting has to prove that it can still reach you.
📉 An error budget for logs
Two exporters wrote errors for days without any metric showing it. A new Loki alert counts error and warning lines per service and fires above 100 an hour. Known harmless noise is on an allow-list with a reason.
The takeaway: a service that complains constantly is a finding, even if all its metrics look fine.
🔍 Upload probes in the access log – and what the first two days showed
On 21 October a critical hole in PHP’s handling of file uploads (multipart/form-data) became public: a crafted upload could make PHP run the sender’s code. Scanners started trying it within a day. In the access log such a probe looked like any form submission – just POST /.
The ingress now also logs the content type and the size of every request. That is enough for a simple pattern: a multipart upload of a few kilobytes to a page that never accepts uploads, such as the start or the login page, is a probe – not a visitor.
The first two days of the new log: scanners sent such uploads to all three servers, mostly from cloud hosting addresses. One fired 32 of them within a single second, about 6 KB each, switching its browser name every few requests to pass as normal visitors. The body is prepared form data meant to trip the upload parser. If that worked, the server would run the attacker’s code – usually a small web shell, a backdoor that accepts commands over HTTP.
Nothing got through. Every probe was answered with 400 (rejected), 404 or a redirect, and none received a response with content. Several senders were already banned by fail2ban for probing other forbidden paths. Normal uploads – the Nextcloud desktop clients, document previews, media in the blog admin – look clearly different: other paths, other content types, logged-in users.
The takeaway: you can only judge what your logs record. Two extra fields turned “probably an attack” into “certainly a probe, certainly rejected”.
🔧 The rest
- Blog on PHP 8.5.
- Blog PHP-FPM memory limit from 1 GiB to 1.5 GiB – twelve workers plus OPcache never fit into 1 GiB; the old comment assumed half the memory per worker. A mistake of my own, found by measuring.
- Rolling updates for the app pods: the new pod starts before the old one stops.
- Nextcloud servers on Redis 7 from the distribution’s module stream.
- Ingress access log with content type and request length – suspicious multipart uploads can now be told apart from normal form posts.
- Pressure stall information (PSI) enabled at boot, so the pressure panel shows data.
🧭 What comes next: nothing new – on purpose
For the coming weeks and months I have no further features or changes planned. The setup does what it should, and every new component would bring new failure modes. The time now goes into understanding what is already there – reading the configuration with fresh eyes, measuring instead of guessing – and documenting it: the deep dives, the operations guides and the lessons in the code comments.
Maintenance continues as usual: updates every Monday, certificates, security fixes. Bug fixes, yes; new features only when there is a real need.
The takeaway: a setup you understand is worth more than one more feature. Stability is a decision, too.
🎯 The lessons
- Test the effect, not the configuration.
- Every periodic job needs a heartbeat; silence is not “all good”.
- Prove the alert path regularly, with an alert that always fires.
- Follow symptoms – a harmless-looking CPU alert led to broken downloads.
- Defaults are decisions someone else made; check them for your own sizes.
The full list of changes is in the tag message of v3.6.0.
