Ten nginx vulnerabilities, an update that never came and release notes that were wrong

Golem reports a critical nginx vulnerability and I think: “dnf-automatic runs every night anyway.” One look at the server shows: my nginx had not been updated for weeks – and it was not the only service.

Comic: in a data centre aisle, a friendly maintenance robot loads only parcels with a red “SECURITY” stamp onto its conveyor belt; a glowing green parcel without a stamp lies forgotten on the floor, a small rabbit next to it; the man, horrified, holds up a newspaper with an exclamation mark and points at it

🌐 Also available in: Deutsch · Français · Español · Italiano · Português · Nederlands · Polski

In short: On 18 November nginx fixed ten security vulnerabilities, among them memory corruption in HTTP/2 and a bug in HTTP/3. At that point my blog was still running the vulnerable version 1.31.8 – although dnf-automatic installs security updates every night. The reason: on my servers dnf-automatic only installs what a repository explicitly marks as a security update, and the nginx repository marks nothing at all. The same applied to Grafana, to a Rancher package and to MariaDB. Now a small nightly job fetches all updates from exactly these vendor repos. On the Nextclouds I switched HTTP/2 off for half a day and waited for a fixed ingress controller – which in truth had been there all along. Why I missed it is explained further down.

Table of Contents

📰 The trigger: a headline

On Friday morning Golem (article in German only) ran the story: millions of web servers at risk, nginx vulnerability allows code execution without authentication. My first reaction was relaxed. My blog runs on nginx, but the server installs security updates every night with dnf-automatic. What could possibly happen?

Then I checked. On nginx.org there were ten new security advisories, all with the same boundary: vulnerable up to 1.31.8, fixed in 1.31.9 (mainline) and 1.30.7 (stable). Three of them hit exactly what my blog uses:

VulnerabilitySeverityaffects me because …
Memory corruption in HTTP/2 flow controlhighthe blog speaks HTTP/2
Use-after-free in HTTP/3mediumthe blog speaks HTTP/3
Infinite loop when reading over TLS 1.3highevery connection uses TLS 1.3

The package had been in the repository for two days. It was not installed.

🔍 The cause: “security updates only” means “only security updates that are marked as such”

My dnf-automatic runs with upgrade_type = security. That sounds sensible: at night only what closes vulnerabilities, everything else I do deliberately myself. But how does dnf know what a security update is?

From the repository metadata. The AlmaLinux repos ship a file called updateinfo in which every update is classified as security, bugfix or enhancement. dnf-automatic reads this file and installs only what is listed there as security.

The nginx.org repository does not have this file. I checked: the metadata contains only the package lists, no updateinfo. So for dnf not a single nginx update is a security update, not even the one that closes ten vulnerabilities. It simply waits until someone runs dnf upgrade by hand.

My own documentation said: “nginx, Grafana and all other system packages come from their repositories and are updated by dnf-automatic.” Well meant, and wrong.

🧹 Not just nginx

Once the cause was clear, I checked every repository on my servers that does not come from AlmaLinux itself:

RepositoryServerupdateinfo
nginx.org (mainline)blogmissing
Grafanablog and Nextcloudsmissing
Rancher (k3s-selinux)Nextcloudsmissing
MariaDB CommunityNextcloudsmissing

Four vendor repos, none with security tags. All of these packages would only have been updated at the next manual update – the database included.

🛠️ The fix: a job just for vendor repos

First the urgent part: raised nginx on the blog to 1.31.9 via Ansible, restarted, checked. Because HTTP/3 has hung on me after a restart before, I then made 60 requests over HTTP/3: 60 out of 60 successful.

Then the permanent fix. The vendors publish their security fixes as perfectly normal new versions. So a small script, update-vendor-repos, takes all updates from exactly these repositories every night – and from no other:

dnf -y -q --disablerepo='*' --enablerepo="nginx-mainline,grafana" \
    --setopt=install_weak_deps=False upgrade

Which repos these are is set in an Ansible variable per server. The job runs at 02:15, i.e. after the nightly dnf-automatic window and before two things that already existed: at 03:45 a script restarts every service that still has old program code in memory, and at 04:00 AIDE takes the changed files into its baseline. A mail only arrives if something was updated or an error occurred, and a heartbeat reports if the job stops running.

With MariaDB I hesitated briefly: an automatic database update means a nightly restart of the database. The repository pins the series (10.11), so only patch versions arrive. And at a quarter to four in the morning nobody uses the clubs’ Nextclouds. So MariaDB is in.

The obvious alternative would have been to set dnf-automatic to upgrade_type = default: then every update arrives every night, however it is flagged. Technically that is a single line, and the restarts at 03:30 and 03:45 already exist. I decided against it anyway. Plain bug fixes and new features would then also arrive unasked at night, and the jump to a new AlmaLinux minor release means hundreds of packages in one go. On the Nextclouds, the kernel, container-selinux or network packages can throw K3s off balance – and if that happens, it happens at half past three when nobody is watching, not on update day when I am sitting next to it. On top of that, packages whose version Ansible deliberately pins would simply move on overnight. The targeted job for the vendor repos closes the gap without buying me those side effects.

🧩 And the Nextclouds?

The Nextclouds run on Kubernetes, where nginx sits in containers in two places:

  • The ingress controller, which accepts the connections from the internet. It brings its own nginx, and at the time of the vulnerability the vendor had not yet released a version with nginx 1.31.9 – even the newest release from the same day still contained 1.31.8. Or so I thought. As a temporary fix I switched HTTP/2 off at the ingress: that way the most severe of the ten vulnerabilities, the memory corruption in HTTP/2 flow control, is no longer reachable from outside. Browsers and sync clients barely notice; they quietly fall back to HTTP/1.1. HTTP/3 was off there anyway. A small script was to ask GitHub every two hours whether the vendor had caught up. In the evening controller 5.8.0 with nginx 1.31.9 was running on both Nextclouds, and HTTP/2 was back on – more on that in a moment.
  • The nginx in the Nextcloud pod. It is only reachable from inside, over plain HTTP – the HTTP/2, HTTP/3 and TLS bugs never reach it. I updated it anyway, from 1.30.6 to 1.30.7.

The watcher script runs on my home server, not on the Nextclouds themselves. It reads the release notes of the last ten controller versions on GitHub and looks for the bundled nginx version in them. The core is a few lines of Python, standard library only:

FIXED = ((1, 31, 9), (1, 30, 7))      # first fixed mainline / stable

def fixed(v):
    v = tuple(int(x) for x in v.split("."))
    return v >= FIXED[0] or (v[:2] == FIXED[1][:2] and v >= FIXED[1])

rels = json.loads(get("https://api.github.com/repos/nginx/kubernetes-ingress/releases?per_page=10"))
for r in rels:
    if r.get("draft") or r.get("prerelease"):
        continue
    vers = re.findall(r"NGINX (?:OSS )?(?:to )?(1\.3[0-9]\.[0-9]+)", r.get("body") or "", re.I)
    if any(fixed(v) for v in vers):
        print(f"FIXED: controller {r['tag_name']} ships nginx {vers[0]}")
        sys.exit(0)
sys.exit(1)                            # not yet

A small shell script around it sends exactly one mail on the first hit and remembers that in a file, so the same message does not arrive every two hours. The recipient address is only in the crontab entry, not in the script:

# crontab -e  (as a normal user, no root needed)
7 */2 * * * WATCH_MAIL_TO=du@example.org $HOME/nginx-ingress-watch/run.sh

Watchers like this are deliberately throwaway tools for exactly one question. This one, however, would never have fired. The script is not badly programmed; it asks the wrong question.

⏱️ Side effect: what does falling back to HTTP/1.1 cost?

Because an end-to-end test makes a real visit to both Nextclouds every hour – log in, open a Word document in Collabora, log out –, I could measure what switching off HTTP/2 means in everyday use. With HTTP/1.1 the browser has to fetch the many small files of the interface over several connections instead of sending them bundled over one.

On 20 November HTTP/2 was off from about 14:55 to 20:40. The test measures the duration of the whole run and, as the most important value, the time until Collabora shows the document:

with HTTP/2 (before)HTTP/1.1 only (7 runs)Assessment
sofie, whole run10.2–11.0 s10.8–11.5 sabout +0.5 s, roughly 5 %
sofie, document loaded2.9–3.2 s3.0–3.4 sbarely measurable
cvjm, whole run12.9–25.2 s18.1–44.5 swidely scattered
cvjm, document loaded3.7–6.5 s4.0–13.2 swidely scattered

On sofie, where hardly anyone works, the difference is small: a good five per cent for the whole run, almost nothing for the document itself. On cvjm the numbers cannot be attributed cleanly. Even the last run with HTTP/2 in the morning took 25 seconds, and the outlier of 44 seconds in the afternoon falls into the time when club members work there. The run at 20:00, still on HTTP/1.1, was back to its old speed with 4 seconds for the document. So for a few hours HTTP/1.1 was an acceptable price – but I would not have wanted to leave it like that for good.

🤦 My mistake: release notes are not an inventory

In the evening I once more considered simply building the controller myself, with nginx 1.31.9. The source code is open, so I downloaded it at the state of version 5.8.0. The first line of the Dockerfile read:

ARG NGINX_OSS_VERSION=1.31.9

Not 1.31.8, as in the release notes. So I fetched the finished, official image and looked inside:

podman run --rm --entrypoint nginx docker.io/nginx/nginx-ingress:5.8.0 -v
# nginx version: nginx/1.31.9

So the fixed controller had been published since 18 November, as a container image and as Helm chart 2.9.0. The release notes contain the line “Update NGINX to 1.31.8” – that was an earlier pull request. The bump to 1.31.9 only came shortly before the release, in a catch-all commit called “security hardening and minor bug fixes”. That one is in the notes too, but without a word about it updating nginx.

My mistake was taking the release notes as proof. They are a summary someone maintains by hand, not an inventory of the image. The question “Which nginx version is inside the controller?” I should have asked the image, not the text about it. My watcher script makes the same mistake, just automated: it searches the release notes and would have waited for the next version, perhaps for weeks.

In the end the fix was a single line: raise the Helm chart from 2.8.0 to 2.9.0 and switch HTTP/2 back on. First sofie, then cvjm, followed each time by a real end-to-end test with login and a Word document in Collabora. Both worked, and both Nextclouds speak HTTP/2 again.

Afterwards I rebuilt the watcher. The second version no longer asks the release notes but the image’s build recipe: every release tag contains a Dockerfile, and one line in it says which nginx version goes into the image. The image does not even have to be downloaded for that:

RAW = "https://raw.githubusercontent.com/nginx/kubernetes-ingress/{tag}/build/Dockerfile"

m = re.search(r"^ARG NGINX_OSS_VERSION=(\S+)", get(RAW.format(tag=tag)), re.M)
built = m.group(1)
if fixed(built):
    sys.exit(0)

The output shows the contradiction I had overlooked in black and white:

v5.8.0   2026-11-18  image: nginx 1.31.9   (release notes: 1.31.8)
v5.7.1   2026-11-05  image: nginx 1.31.8   (release notes: 1.31.8)
v5.7.0   2026-10-14  image: nginx 1.31.7   (release notes: -)

🎯 Conclusion: trust is good, checking is better

  • “Automatic security updates” only apply to repos that actually mark security updates. For every third-party repo a look at the metadata pays off: if updateinfo is missing, nothing arrives from there at night.
  • The most dangerous vulnerabilities often sit exactly in the third-party repos. nginx, database, monitoring – these are the services you deliberately take from vendor repos because they are meant to be more up to date.
  • A headline is a good reason to check your own assumptions. Without the Golem article the vulnerability would only have been noticed at the next manual update.
  • Release notes are not an inventory. Whether a vulnerability is closed in an image only the image itself can tell you: nginx -v in the container takes ten seconds.
  • Documentation can be wrong, your own included. The sentence “is updated by dnf-automatic” sat in my operations documentation for months and felt good.

📚 Related posts:

Some text and images on this site are AI-assisted or AI-generated. Details