Deep Dive: Trivy – Scanning Every Running Image Each Night, and Reading 241 Findings Without Panic

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

Every image in this setup is pinned to an exact version. That is good for reproducibility and bad for security in one specific way: a pinned image does not get better on its own. The Debian or Alpine packages inside it age, the Go runtime that a binary was built with ages, and new vulnerabilities are published every day for software that has not changed at all. The host gets its security updates every night from dnf-automatic; the images did not get anything – until somebody looked. Trivy now looks every night. This post explains how it finds vulnerabilities, how it runs on the blog since 19 October 2026, what the first scan found – 241 fixable findings – and why that number is far less alarming than it sounds once you sort it.

🧭 How Trivy works

Inventory first, then a database lookup

Trivy does not run or attack anything. It reads an image layer by layer and builds an inventory: which operating system and release (Debian 13, Alpine 3.21, Ubuntu 22.04 …), which packages in which exact versions from the package manager’s database, and which language dependencies are compiled into binaries or listed in lock files – Go modules read from the build information that every Go binary carries, npm, pip, Composer, Java and so on. Then it compares that inventory with its vulnerability database.

The database is built by the Trivy project from many sources – the security trackers of the distributions, the GitHub Advisory Database, the Go vulnerability database and others – and downloaded as one file of a few hundred megabytes. For every package it knows which versions are affected by which CVE and, if one exists, in which version the problem is fixed.

Severity and “fixed version” – the two fields that matter

• Severity (CRITICAL, HIGH, MEDIUM, LOW, UNKNOWN) comes from the source – often the distribution’s own rating, otherwise a CVSS score. It describes the vulnerability in general, not its risk in this image. A CRITICAL bug in a library that the image never calls is still listed as CRITICAL.
• Fixed version says whether a fix exists at all. Without one, nothing can be done except accepting the finding or replacing the software. With one, the question becomes: who has to ship it – the distribution (already done), the image maintainer (rebuild needed), or the upstream project (new release needed)?

That distinction is the key to everything below. Trivy reports what is known; it cannot know whether the vulnerable code path is reachable. Reading the results is therefore always a second step done by a human.

Where the image comes from

Trivy can read an image from a registry (remote), from a local Docker or Podman daemon, from a tar file – or directly from containerd, the runtime that K3s uses. The last option means: no second download, the scan sees exactly the image the pods run. On this setup it needs the K3s socket and namespace:

CONTAINERD_ADDRESS=/run/k3s/containerd/containerd.sock \
CONTAINERD_NAMESPACE=k8s.io \
trivy image --image-src containerd docker.io/library/mariadb:10.11.19

🛡️ Why it is here

• Pinned images age silently. The image refresh in the update playbooks pulls rebuilt images under the same tag – but nothing told us when a rebuild was worth it, or when an image would never get one.
• The host was covered, the containers were not. dnf-automatic patches the host every night; the 14 images that make up the cluster were outside that loop.
• Decisions need data. “Should we switch this exporter?” or “Is a K3s update urgent?” are easier with a list of what each image actually carries.

🔧 What this repository does

A timer on the host, not an operator in the cluster

The usual way to run Trivy in Kubernetes is the Trivy Operator: it watches the cluster, starts a scan job per workload and stores the results as custom resources. On a single small node that would mean CRDs, a controller, scan pods – and for each of them exceptions in the image policy and the NetworkPolicies this setup enforces. The role common_trivy does the same job with one binary and one script: a systemd timer starts trivy_scan.py every night at 05:15, after the security updates (01:00–02:00), the reboot window (03:30) and the other night jobs.

The binary: pinned and checksummed

Trivy is installed from the GitHub release archive, with the version and its SHA-256 pinned in versions.yml. That second pin has a concrete reason: in March 2026 the Trivy release pipeline itself was compromised and briefly shipped a malicious build (advisory GHSA-69fq-xp46-6×23). A security scanner runs as root and reads everything – exactly the tool you want to be sure of. The rule in the notes: update only to a release that is at least two weeks old, and copy the checksum from that release’s checksum file. Ansible’s get_url refuses an archive whose hash does not match.

Two small things the installation ran into, both on purpose elsewhere: /tmp is mounted noexec by the CIS hardening, so the archive is unpacked from /root; and after unpacking, restorecon gives the binary its correct SELinux label – files written through a temporary path keep the label of that path otherwise, the same lesson as with the k3s binary.

The script

1. Update the vulnerability database once (--download-db-only); all image scans then run with --skip-db-update.
2. List the images of all running pods. Finished pods of Jobs and CronJobs are left out – the first version included them, and a helm-install job that ran once at cluster start contributed 32 findings for an image that does not run.
3. Scan each image from containerd. If that fails, try again from the registry (see below). Write one JSON report per image to /var/lib/trivy/reports.
4. Write metrics for node_exporter’s textfile collector: trivy_image_vulnerabilities{image, severity, fixable}, trivy_image_scan_ok, trivy_db_update_ok, trivy_last_run_timestamp_seconds. The file is written under a temporary name and renamed, so node_exporter never reads half of it.
5. Compare the fixable CRITICAL and HIGH findings with the previous night and mail only the new ones.

Why a mail for new findings and no alert

The obvious design would be a Grafana alert “critical vulnerability with fix available”. The first scan showed why that would be wrong: the MariaDB image contains gosu, a tiny helper written in Go, built with an older Go version that has a CRITICAL vulnerability in its standard library. Fixing it is up to the MariaDB image maintainers. An alert would fire, repeat every four hours, and keep doing so for days or weeks – until it is ignored, and with it every other alert. The mail says it once. A finding that stays does not mail again; one that disappears is counted as “gone”. Grafana only alerts when the scan itself is broken: database update or an image scan failed, or the last successful run is older than 36 hours.

Accepting a finding

Some findings will be accepted on purpose. They go into the variable trivy_ignore, each with a mandatory reason, and end up in a .trivyignore file that the scan uses. The list is empty on purpose: every entry is a decision, not a default.

SELinux and the textfile collector

node_exporter runs in its own SELinux domain since the same day (after a permissive learning phase, enforcing since 20/21 October). Its textfile directory /var/lib/node_exporter/textfile got its own type, homelab_node_exporter_textfile_t, with exactly two rules: the exporter may list the directory and read the files. The scan, running as root, writes them; new files inherit the directory’s type.

🔍 What the first runs found – about the tool itself

Layers that containerd had already thrown away

Two of 15 images failed in the first run: metrics-server and kube-state-metrics, with “content digest … not found”. containerd unpacks image layers into snapshots and may discard the compressed layer blobs afterwards; the image runs fine, but it can no longer be exported as a whole. Trivy offers --image-src containerd,remote as a fallback list – which did not help, because it falls back only when the image is missing in the first source, not when a layer is. The script now retries explicitly with remote for exactly those images.

An AVC, found by the log alerts of the same day

The first version of the scan unit had NoNewPrivileges=yes. The script calls k3s kubectl to list the pods, and the k3s binary transitions into the SELinux domain container_runtime_t. Under NoNewPrivileges, SELinux refuses that transition unless the policy explicitly allows it, and logs avc: denied { nnp_transition }. The new Loki alert for real SELinux denials fired within minutes. The setting is gone; the unit keeps the other systemd restrictions (read-only /usr and /etc, no home directories, private /tmp, lowest CPU and I/O priority, 2 GB memory cap).

📊 Reading 241 findings

After the fixes the scan reported 209 fixable CRITICAL/HIGH findings across 14 running images. Sorted by who can fix them, the picture changes completely:

Three lessons from that table:

• Most findings need no action by us. They are fixed somewhere upstream and will arrive with the next rebuild or release, which the update playbooks already pick up.
• One image is a real problem. An exporter that has not had a release in four years and still ships OpenSSL 1.1 is responsible for almost a third of all findings and nearly all CRITICAL ones. Its practical exposure is small – it only talks FastCGI to PHP-FPM inside the pod and serves metrics inside the cluster, behind NetworkPolicies – but “small exposure” is not a reason to keep abandoned software forever.
• “Fixed” does not mean “relevant”. The WordPress image lists linux-libc-dev, the kernel headers – a container uses the host’s kernel, the headers are only there for compiling. A candidate for trivy_ignore, with exactly that reason.

One question came up immediately: is there a newer WordPress image that fixes its 32 findings? No. WordPress 7.1.3 is the current release, and the image 7.1.3-php8.4-fpm on the blog has the same digest as the one in the registry. Debian has published the fixed packages, but the official image has not been rebuilt since. Building our own image with an extra apt-get upgrade would fix it today – at the price of maintaining an image build forever. Waiting a few days for the official rebuild is the better trade.

🧰 Day-to-day administration

# when did it run, how long, how much memory?
systemctl list-timers trivy-scan.timer
journalctl -u trivy-scan --since today

# run it now (oneshot, returns when done)
systemctl start trivy-scan.service

# the summary line of the last run
journalctl -u trivy-scan -n 20 | grep "images scanned"

# which findings are open right now (image, CVE, package, installed -> fixed)
python3 -c 'import json; [print(v["severity"], v["id"], v["pkg"], v["installed"], "->", v["fixed"], v["image"]) for v in json.load(open("/var/lib/trivy/last-findings.json")).values()]' | sort | less

# one image in detail
trivy image --image-src containerd --skip-db-update docker.io/library/redis:7.4.11-alpine

In Grafana (Explore → Prometheus), the same numbers are queryable:

# fixable CRITICAL per image
trivy_image_vulnerabilities{severity="CRITICAL", fixable="true"} > 0

# total fixable HIGH+CRITICAL over time - does it go down after an update?
sum(trivy_image_vulnerabilities{severity=~"CRITICAL|HIGH", fixable="true"})

# age of the last scan in hours
(time() - trivy_last_run_timestamp_seconds) / 3600

What to do with a mail

1. Is the image rebuilt upstream under the same tag? Then the next update run (image refresh) fixes it – or run the update playbook now.
2. Is there a newer release? Then raise the pin in versions.yml, ideally one that is a few days old.
3. Neither? Then it is a finding to accept (with a reason in trivy_ignore) or a reason to replace the software.

Cost

The vulnerability database and the layer cache take about 1.6 GB under /var/cache/trivy. A full nightly run including the database download takes under half a minute of CPU on the blog; later runs reuse the cached layer analysis and take a few seconds. The scan runs with the lowest CPU and I/O priority and a 2 GB memory limit, so it cannot slow the sites down.

🆘 Cheat sheet: symptom → command

The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders.

🔭 What comes next

• A decision on the PHP-FPM exporter by 20 January 2027, when its acceptance lapses: two one-person forks are active (one with automatic dependency updates); otherwise extend the acceptance.
• Configuration scanning – tried once in October 2026 against the live manifests of the blog: 21 kinds of findings, nearly all deliberate decisions (ingress on the host network, hostPath volumes, images that run as root). As a CI job it would need rendered templates and a long ignore list; not worth it for now.
• Secret scanning is already in CI: gitleaks checks the whole git history on every push, next to the pre-commit hook.
• A small dashboard with the trend of open findings per image.

⚖️ Alternatives – and why Trivy

The point of the tool is not the number it reports but the sorting it enables. 241 findings sounded like an emergency. Sorted by who can fix them, they turned into one real decision, one image refresh – and a lot of waiting for upstream, which the existing update routine already handles.