The Ansible Tour, Part 6: Monitoring – Exporters, Prometheus, Grafana and a Monit Watchdog

Part 6 of the Ansible Tour, and the last one. Part 1 explained the framework, part 2 the server and the platform, parts 3 and 4 the two applications, part 5 security. This part is about knowing what is going on. It describes release v3.3.0.

A server that nobody watches fails silently. The setup here has two kinds of watching, and it helps to keep them apart:

Both send mail, but differently: Grafana when a metric crosses a threshold, Monit when a state changes. And both follow the rule from part 5 – silent while healthy, so every mail means something.

📈 How metrics flow

The model is pull: programs called exporters offer their numbers as a text page over HTTP, and Prometheus fetches that page every 15 seconds, stores every value with a timestamp and keeps it for 30 days. Grafana asks Prometheus and draws the curves. Nothing pushes data anywhere, and nothing leaves the server.

Everything listens on localhost or on NodePorts that the firewall drops from outside (part 5). Prometheus binds to localhost too; only Grafana is reachable from outside, and only through the ingress under /grafana/, with its own login.

🧰 The exporters: the pinned binary pattern, once more

common_node_exporter and common_mysqld_exporter are small and almost identical, and they are the clearest example of a pattern part 2 introduced:

1. A system user without a login shell and without a home directory – the process must not run as root, and nobody ever logs in as this user.
2. Read the installed version (node_exporter --version | awk ..., with changed_when: false, check_mode: false, failed_when: false – on a fresh host there is no binary and the empty answer means “install”).
3. Download, unpack, install only if it differs from the pin in versions.yml, with the dry-run guard not ansible_check_mode on the steps that need the downloaded file.
4. A systemd unit plus a sandbox drop-in, written with copy and content:, and two handlers in a fixed order: reload systemd, then restart the service.

A small story sits in step 3. The archive used to be the marker for “already installed”: download only if /tmp/node_exporter.tar.gz is missing. But /tmp is emptied by every reboot – so every run after a reboot downloaded again and reported a change, although the right version was installed. Comparing the installed version with the pin is the honest test.

Node Exporter has no configuration file; all settings are command-line flags in the unit. The role points that out, together with a warning that applies to every role that writes a file with content:: comment lines inside the content are part of the file on the server. Editing a comment there changes the unit and restarts the service.

The MariaDB exporter adds one thing: credentials. It reads them from a file only it can read (mode 0600, owned by its own user), written with no_log: true. The user it logs in with is the read-only account from part 4 – with its own password, no longer root’s. Its meta/argument_specs.yml is the one the other roles point to as the explained example: when the role starts, Ansible checks every listed option, required ones must exist, each value must match its type, and no_log on an option keeps it out of the validation error message.

🗄️ common_prometheus: the time-series database

Prometheus is installed the same way as the exporters – system user, directories for the database, pinned binary, systemd unit, sandbox drop-in. The role’s header comment is the most beginner-friendly in the repository: it explains register, loop, handlers, check mode and the difference between template and copy from scratch. Every task that changes something Prometheus reads notifies the same handler, restart prometheus – a new binary and a new configuration still mean one restart.

One scrape configuration, two stacks

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["127.0.0.1:9100"]
        labels:
          instance: "{{ inventory_hostname }}"
{% if stack == 'nextcloud' %}
  - job_name: "mysqld_nextcloud"
    static_configs:
      - targets: ["127.0.0.1:9104"]
  - job_name: "php_fpm_nextcloud"
    static_configs:
      - targets: ["127.0.0.1:<nodeport>"]
{% endif %}
{% if stack == 'wordpress' %}
  - job_name: "mysqld_wordpress"
    ...
  - job_name: "php_fpm_wordpress"
    ...
{% endif %}

The stack variable from part 1 again. Until the jobs were made stack-specific, all four application jobs were rendered on every server, and each server showed two targets permanently “down” – the other stack’s ports. Harmless, but noise in exactly the view where you look for real problems.

Alert rules without a mail

The template also defines alert rules: disk below 20 and below 10 percent free, memory below 10 percent, CPU above 90 percent for ten minutes, Node Exporter unreachable. Prometheus evaluates them and shows them on its /alerts page – but it does not send mail. That would need a second component, Alertmanager, which is deliberately not installed. The mails come from Grafana and Monit instead, and the file says so in its first lines, so nobody waits for a mail that cannot come.

📊 common_grafana: curves, behind a sub-path, and mail

Grafana is the only part of the monitoring with a web interface, and the role that configures it is a good study in talking to an application that has no Ansible module. Its three files run in order, because the second and third need a running Grafana:

install.yml: the package and a few ini lines

Grafana comes from its official repository, with beta versions excluded. Then lineinfile sets single lines in grafana.ini: the admin user and password (from the vault, with no_log), no forced password change on first login, listening on all interfaces – because the ingress reaches it from the pod network through the external Service of parts 3 and 4 – and the two settings for running under a sub-path, root_url and serve_from_sub_path. Without them, Grafana would generate links to /login instead of /grafana/login.

- name: Grafana | Wait for API to become available
  ansible.builtin.uri:
    url: "http://127.0.0.1:3000/api/health"
    status_code: 200
  register: grafana_health
  check_mode: false
  until: grafana_health.status == 200
  retries: 30
  delay: 3

The uri module makes HTTP requests from the server – here to Grafana itself, on localhost. With until it polls the health endpoint until Grafana answers, at most 90 seconds.

dashboards.yml: check, then create

The Grafana API has no idempotence of its own: “create data source” twice creates two data sources. The role makes it safe with a pair of requests:

- name: Grafana | Check if Prometheus datasource already exists
  ansible.builtin.uri:
    url: "http://127.0.0.1:3000/api/datasources/name/Prometheus"
    user: "..."
    password: "..."
    force_basic_auth: true
    status_code: [200, 404]
  register: grafana_ds_check

- name: Grafana | Create Prometheus datasource
  ansible.builtin.uri:
    url: "http://127.0.0.1:3000/api/datasources"
    method: POST
    body_format: json
    body: { name: Prometheus, type: prometheus, url: "http://127.0.0.1:9090", ... }
  when: grafana_ds_check.status == 404

status_code: [200, 404] declares both answers as expected – “exists” and “does not exist” – so only a real error (401, 500) fails the task. The create request runs only on 404. That is “ask, then act” for an HTTP API.

The three dashboards (host metrics, MySQL, PHP-FPM) come from grafana.com and are imported through the API. The downloaded JSON is read back with slurp – base64, hence | b64decode | from_json – and merged with a fixed ID using the combine filter, so every import overwrites the same dashboard instead of adding a copy. The whole import block is skipped in check mode. A small Python script then adjusts one dashboard’s variables to match this setup’s job names. One limitation is stated in the role: uri never reports “changed” by itself, so API writes do not appear in the recap – the check/create pairs only keep them from repeating. A fourth dashboard, Container-Ressourcen, is not downloaded: its JSON lives in the repository (files/container-resources.json), carries its own fixed ID and is posted as it is.

alerting.yml: mail through the same relay

For mail, Grafana needs SMTP settings, and the role uses the same relay as the server’s own mails (part 2), so there is no second mail account. The settings are written with blockinfile: it manages a whole block of lines between two marker comments and, on the next run, replaces exactly that block. Then meta: flush_handlers restarts Grafana right away, because the next tasks configure alerting through the running Grafana, which must already know how to send mail. A second health wait follows the restart.

Then, again with check-then-create: an e-mail contact point, the default notification policy pointing to it, a folder for the rules, and the rules themselves – written by a Python script that the role places on the server and runs, with the credentials passed as environment variables for that one task. The rules:

“For 5 minutes” matters: a single spike – a backup, a preview run – does not wake anybody up.

📦 common_kube_metrics: one curve per container

Node Exporter sees the host as a whole. Whether the PHP container is throttled at its CPU limit while the host is idle, or a pod creeps towards its memory limit, stays invisible – on 16 October 2026 there was not a single per-container series. Since release v3.3.0 two more sources close that gap:

• cAdvisor is built into the kubelet, nothing to install. It reports CPU, memory, throttling and OOM events for every container on https://127.0.0.1:10250/metrics/cadvisor – but only to a caller with a token. The role creates a ServiceAccount prometheus-scraper whose only right is get on nodes/metrics, plus a Secret of type service-account-token. common_prometheus reads it with k8s_info (with retries, until the token controller has filled it) and writes it to /etc/prometheus/kubelet-token.
• kube-state-metrics turns the API server’s objects into numbers: limits and requests, restarts, the reason a container last ended. It runs in kube-system with read-only permissions, as non-root with a read-only root file system, behind NodePort 30808 – local only, like the other exporters.

  - job_name: "kubelet_cadvisor"
    scheme: https
    metrics_path: /metrics/cadvisor
    authorization:
      credentials_file: /etc/prometheus/kubelet-token
    tls_config:
      insecure_skip_verify: true   # K3s's self-signed kubelet certificate, 127.0.0.1 only
    static_configs:
      - targets: ["127.0.0.1:10250"]
    metric_relabel_configs:
      - source_labels: [container]
        regex: "|POD"              # pause containers and pod-level sums
        action: drop

The relabel rule drops about half of all series before they are stored. The fourth Grafana dashboard, Container-Ressourcen, sets every container against its own limit – CPU share, throttling, memory, restarts, OOM kills – and four mail alerts watch the same: memory above 90 % of the limit, throttling above 50 %, an OOM kill, more than two restarts in 15 minutes.

🐕 common_monit: the watchdog

Monit exists because of an incident told in its own post: fail2ban had died on the blog and nobody noticed for two days. Metrics did not help – a dead fail2ban does not show up as a curve. Monit checks states: every two minutes it runs a list of checks, and it mails only when a check changes from ok to failed, or back. A healthy server sends nothing.

The role, step by step

1. assert the mail settings. Monit sends its own mail through the relay; without credentials it would be a watchdog that cannot bark. Better to fail at the start of the role with a message that says which vault values are missing.
2. Find out which services exist here. The same role runs on both stacks, but not every service exists on every host (the MariaDB exporter only on Nextcloud). So:

- name: Monit | Detect which candidate services exist on this host
  ansible.builtin.command: "systemctl cat {{ item }}.service"
  register: monit_unit_check
  changed_when: false
  failed_when: false
  loop: "{{ monit_candidate_services }}"

- name: Monit | Build the list of present services
  ansible.builtin.set_fact:
    monit_present_services: >-
      {{ monit_candidate_services
         | zip(monit_unit_check.results)
         | selectattr('1.rc', 'equalto', 0)
         | map(attribute='0') | list }}

systemctl cat prints a unit file and exits non-zero if the unit does not exist – a read-only way to ask “is it installed?”. The second task is a compact piece of Jinja: zip pairs each candidate name with its result (item 0 = name, item 1 = result), selectattr('1.rc', 'equalto', 0) keeps the pairs whose result had exit code 0, and map(attribute='0') takes the names back out. The configuration template then writes a check only for those.

• Step 3 – check scripts and configuration from files/ and two templates. The scripts are small and each answers one question with its exit code: is this service active? Is the firewall table loaded with its ban sets? Is any pod crash-looping or failing to pull its image (brief “creating” states are ignored)? Are the fail2ban jails reading the current ingress log? Is any certificate closer than three weeks to expiry?
• Step 4 – validate, then start. monit -t checks the configuration; the role starts Monit and confirms it is active – with a failed_when that reads the output of systemctl is-active.

What Monit watches

Two checks have a schedule of their own, in cron syntax: certificates and backups once an hour instead of every cycle. There is nothing to gain from opening a TLS connection to every certificate every two minutes. The backup check is a nice trick: the external backup job touches a file on the server after every successful run, and Monit alerts when that file’s timestamp gets too old. The server cannot see the backup, but it can see that nobody told it about one.

The certificate threshold has a story. Once, an image policy (part 3) silently blocked cert-manager’s challenge solver, and no certificate could be renewed. Nobody noticed until a certificate was close to expiring. Now Monit notices three weeks ahead.

A self-test that sends real mail

How do you know the watchdog barks? tasks/selftest.yml stops Node Exporter for a moment, waits until Monit reports the failure, and starts it again – which sends a real alarm mail and a real recovery mail. That is disruptive, so it only runs on request, from the verify roles:

ansible-playbook blog.yml --limit <host> -e monit_selftest=true

Added 19 Oct 2026: since this part was written, the monitoring grew. common_selinux_monitoring gives Prometheus, the exporters and Grafana their own SELinux domains (permissive learning phase first), Monit runs in a systemd sandbox, and Grafana logs at warn instead of info. Two new roles, on the blog first: common_loki (Loki + Grafana Alloy: journal, pod logs, audit and fail2ban log, 14 days, with mails for blocked SELinux access, fail2ban errors, kernel OOM kills and missing logs) and common_trivy (a nightly scan of every running image; a mail only for new fixable CRITICAL/HIGH findings). Both get their own deep dive.

🛠️ Running it: who mails what

All the mails of the whole tour, in one place – this is the table to keep next to the inbox:

Looking at it

# Monit's view of everything it watches
monit summary

# Prometheus: are all targets up?
curl -s http://127.0.0.1:9090/api/v1/targets | python3 -m json.tool | grep -E '"job"|"health"'

# Prometheus: which of its own alert rules are firing?
curl -s http://127.0.0.1:9090/api/v1/alerts | python3 -m json.tool

# Grafana: https://<your host>/grafana/ – dashboards "Node Exporter Full", MySQL, PHP-FPM, Container-Ressourcen

Reading the curves

• Disk: look at the slope, not the level. A flat line at 70 % is fine for months; a line rising two percent a week reaches the 80 % alarm on a predictable day.
• Memory: Linux uses free memory as a file cache, so “used” is always high. What matters is “available”. If that shrinks steadily, something is growing.
• PHP-FPM: if all workers are busy and requests queue up, the site is slow no matter how fast the server is. More workers need more memory – see the sizing in part 2.
• MariaDB: slow queries show up in the slow query log, not only in Grafana. Measure first, optimise second is a post about exactly that.

🏁 The end of the tour

That is every role of the repository – 36 of them, four playbooks, the inventory that ties them together and the tests around them. If you have read all six parts, you know what runs on these servers and why, how each piece got there, what the code does at each step, and what to do when a mail arrives. That was the goal: not to make you copy this setup, but to make it understandable enough that you could run it – or build your own and avoid the mistakes that are written down in its comments.

The roles will keep changing. The links in this series point to release v3.3.0, so they keep showing the code the text describes. Some topics deserve more than a section – nftables, fail2ban, SELinux and K3s each get a post of their own, about the tool itself rather than this repository. They will be linked here.