🌐 Also available in: Deutsch · Français · Español
Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.
Every server here has its own Prometheus and its own Grafana (part 6 of the tour shows the roles). This post is the reading guide: what the numbers are, what normal looks like on these machines, which panels matter when something is wrong – and how the alerting is built, including the mistake that kept it silent for weeks.
🧭 How Prometheus and Grafana work
Pull, time series, labels
Exporters are small programs that expose numbers over HTTP – the node exporter for the host, mysqld_exporter for MariaDB, php-fpm_exporter for PHP, the kubelet’s cAdvisor for containers, kube-state-metrics for Kubernetes objects. Prometheus fetches (“scrapes”) them every 15 seconds and stores each number as a time series, identified by a name and labels such as container="nextcloud-fpm". Grafana asks Prometheus in its query language PromQL and draws the answer.
Three functions that cover most panels
rate(x[5m]): per-second increase of a counter, averaged over five minutes – requests per second, CPU seconds per second (= cores).increase(x[1h]): how much a counter grew in an hour – restarts, OOM kills, throttled periods.max_over_time(…[7d])andsum by (…): worst value in a period, and adding up series along chosen labels. The second one is where most reading mistakes happen (see below).
Alerting without Alertmanager
Grafana evaluates alert rules itself: a query (A), a threshold (B), a duration (“for 5 minutes”). If the condition holds long enough, it sends a mail to the contact point. For one administrator and three servers that replaces a separate Alertmanager.
🛡️ Why it is here
- Trends instead of moments. 30 days of history answer “since when?” and “is this normal?”, which
topnever can. - One place for host, database, PHP, containers – and, since October 2026, the logs next to them (Loki).
- Open and standard. Prometheus and Grafana are what Kubernetes monitoring speaks; dashboards for every exporter already exist.
🔧 What this repository does
| Part | Detail |
|---|---|
| Prometheus | common_prometheus: 15 s scrape interval, 30 days retention, listens on 127.0.0.1; 5 scrape jobs per server, ~7,500 series, 0.5–0.8 GB on disk |
| Grafana | common_grafana: package from the Grafana repository, behind the ingress under /grafana, port 3000 open only for the node itself; log level warn |
| Data sources | Prometheus and Loki, created through the API with fixed UIDs |
| Dashboards | 1860 Node Exporter Full, 7362 MySQL Overview, 4912 PHP-FPM, plus the own “Container-Ressourcen” |
| Alerts | 11 metric rules and 4 log rules, one mail contact point |
| Hardening | own SELinux domain for Grafana, Prometheus and the exporters; Grafana login protected by a fail2ban jail and Grafana’s own lock-out |
Everything is provisioned through Grafana’s HTTP API by small Python scripts that Ansible writes and runs. Dashboards are imported with fixed UIDs, so links and bookmarks survive a re-import; dashboard 1860 additionally gets its variables patched, because the import API leaves its data source and node/job variables uninitialised, and every panel showed “No data” on first open.
📊 Reading the dashboards
Node Exporter Full (1860) – the host
| Panel | Normal here | Look closer when |
|---|---|---|
| CPU Busy | average 5 % (blog), 9 % and 16 % (Nextcloud); 95th percentile below 30 % | sustained above 70 % – then “CPU Basic” shows whether it is user, system or iowait |
| Sys Load | below the number of cores, short peaks slightly above (reboot, nightly scans) | above the core count for more than a few minutes |
| RAM Used | 16–32 % on average, at most 39 % | rising for days without falling back – a leak, not load |
| SWAP Used | near zero | growing steadily |
| Root FS Used | well below 80 % | the alert fires at 80 % |
| Disk IOps / Read/Write | short peaks at night (scans, reboot) | high iowait together with high I/O during the day |
| Network Traffic | low, spikes from uploads and backups | sustained outbound traffic nobody can explain |
One pattern repeats every night: a gap of a few minutes around 03:30. That is not an outage but the planned reboot after updates; a gap at any other time is worth a look. And the larger Nextcloud server shows several times the CPU time of the blog for K3s itself, for the same work – a property of its processor generation, explained in the K3s deep dive, not a misconfiguration.
MySQL Overview (7362) – the database
| Panel | Normal here | Look closer when |
|---|---|---|
| MySQL Connections | a handful, peaks 6–15 | close to max_connections (50) |
| MySQL Questions / Queries | tens per second, follows the daytime | a jump without more users |
| InnoDB Buffer Pool | large share unused, hit rate > 99 % | free pages near zero and reads from disk rising |
| Temporary Objects | some disk temporary tables (TEXT columns) | disk tables growing faster than queries |
| Slow Queries | 0 | anything above zero – then the slow log says which |
PHP-FPM (4912) – the workers
| Panel | Normal here | Look closer when |
|---|---|---|
| Active / idle processes | a few active, most idle | active close to max_children |
| Listen queue | 0 | anything above zero for more than a minute |
| Max children reached | 0 | increases – the pool is too small |
| Slow requests | 0 | increases – the pod log names the script |
Container-Ressourcen – the own dashboard
Eight panels per namespace: CPU and memory as a share of the container’s limit, CPU throttling, absolute cores and working set, restarts and OOM kills per hour, the last termination reason per container, and the configured limits and requests. It answers the one question the other dashboards cannot: is a container held back by its own limit while the host has room? Throttling of a few percent is normal for bursty PHP; a container that sits above 50 % throttled is the one to give more CPU.
The reading mistake
While writing the performance deep dive, a query for the blog’s PHP container showed a memory peak of 940 MiB against a 1 GiB limit. It had summed all series with that container name – during a rollout the old and the new pod overlap, and after a restart cAdvisor briefly reports the old and the new cgroup. The real peak of a single pod was 538 MiB. Panels that sum by (namespace, pod, container), as the own dashboard does, are safe; a sum over the container name alone is not.
🚨 Alerts – and why they were silent
| Alert | Fires when (for 5 min) |
|---|---|
| Disk root / disk data | more than 80 % used |
| Memory | more than 90 % used |
| CPU load | more than 85 % busy |
| Node down | the node exporter does not answer |
| Container memory | a container above 90 % of its memory limit |
| Container throttling | more than 50 % of CPU periods throttled |
| Container OOM / crash loop | an OOM kill, or more than 2 restarts in 15 minutes |
| Trivy stale / failed | no vulnerability scan for 36 hours, or the last scan failed |
| Logs (Loki) | SELinux denial in enforcing mode, fail2ban error, kernel OOM, no logs at all |
In September 2026 a check on the larger Nextcloud server showed that none of these rules had ever fired – not because nothing happened, but because they could not. The rules were created through the API without the field condition, which names the query that decides the alert. Grafana logged “condition must not be empty” on every evaluation, and the rules were set to execErrState: OK, which turned every evaluation error into “all fine”. Two settings, each reasonable on its own, together made the alerting look healthy while it was dead. Since then every rule names condition: "B". The lesson: after a change, check the state of each rule in Grafana – “Normal” without an evaluation error – instead of taking the absence of mail as proof.
One gap remains by design: noDataState and execErrState are still OK, so that a restart of Prometheus does not send a dozen mails. If Prometheus itself stopped, every rule would quietly stay green. That case is covered from outside Grafana: Monit watches Prometheus, Grafana, the node and MySQL exporters, Loki and Alloy, and mails if one of them stops.
🧰 Day-to-day administration
# Prometheus directly, from the host
curl -s 'http://127.0.0.1:9090/api/v1/targets' | jq '.data.activeTargets[] | {job: .labels.job, health}'
curl -s --get 'http://127.0.0.1:9090/api/v1/query' \
--data-urlencode 'query=max_over_time(sum(phpfpm_active_processes)[7d:1m])' | jq '.data.result'
# Grafana state
systemctl status grafana-server prometheus
journalctl -u grafana-server --since -1h # warnings and errors only (log level warn)- Changing a threshold means changing it in the role and running the playbook; a change made in the web interface would be overwritten by the next run, and that is intended.
- A new panel that should stay goes into the dashboard JSON in the repository (the own dashboard) – for imported community dashboards, a copy with its own UID.
- Backups: everything that matters – data sources, dashboards, alert rules – is recreated from the repository. The metrics themselves are not backed up; 30 days of history is the deliberate horizon.
🆘 Cheat sheet: symptom → command
The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders.
| Symptom | Command |
|---|---|
| Grafana or Prometheus not reachable | systemctl status grafana-server prometheus |
| 403 on /grafana | # country filter in the ingress: is the address in the list? |
| A graph is empty | curl -s http://127.0.0.1:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}' |
| Ask Prometheus directly | curl -s --get http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up' | jq '.data.result' |
| No alert mails, although something is wrong | # Grafana → Alerting → Alert rules: state "Error" or "No data"? |
| Prometheus itself stopped | monit status svc_prometheus # Monit mails, Grafana cannot |
| Threshold or dashboard changed in the web interface | # overwritten by the next playbook run - change it in the role |
🔭 What comes next
- Logs and metrics in one view: Loki is there since October 2026; linking a spike in a panel to the log lines of the same minute is the next step.
- Done on 20/21 October 2026: an alert on the PHP-FPM listen queue (threshold from 14 days of measurements), and
/grafanabehind the same country list as SSH and the WordPress admin – the ingress controller answers 403 from anywhere else. - Dashboards as code (Jsonnet/Grafonnet) once the own dashboards grow beyond one.
⚖️ Alternatives – and why Grafana + Prometheus
| Option | Strength | Why not here |
|---|---|---|
| Netdata | No configuration, per-second resolution | Per host, short history by default, less suited to Kubernetes metrics. |
| Zabbix / Icinga | Classic check-based monitoring, mature alerting | Checks say “up or down”, not “how much since when”; container metrics need extra work. Icinga was tested in a lab. |
| Cloud monitoring services | Nothing to run | Metrics leave the server; costs per host and series. |
| Prometheus + Grafana | Standard for Kubernetes, pull model, open, dashboards for every exporter | Chosen. |
A dashboard is a set of questions someone else asked. The useful ones here are those whose normal state is known – which is why this post lists “normal here” next to every panel. And alerting is only as good as its last test: the most important line in this post is the one about rules that looked green for weeks without ever being able to fire.
