Deep Dive: Loki and Grafana Alloy – Searchable Logs Next to the Metrics, and What the First Hour Found

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

Metrics tell you that something is wrong: memory is at 90 percent, a container restarted three times, the disk fills up. They rarely tell you why. The why is in the logs – and on a K3s host the logs are spread over the systemd journal, a directory per pod under /var/log/pods, the audit log and a handful of application files. Until now, finding something meant SSH, journalctl, kubectl logs and grep, one source at a time. This post explains how Loki and Grafana Alloy turn that into one searchable place next to the dashboards, what exactly runs on the blog since 19 October 2026 – and what the new log alerts found within their first hour, before anybody had looked for it.

🧭 How Loki and Alloy work

The idea: index the labels, not the words

Classic log systems such as Elasticsearch build a full-text index: every word of every line becomes searchable, and the index is often larger than the logs themselves. Loki does the opposite. It indexes only a small set of labels per stream – for example job="journal", unit="sshd.service", namespace="wordpress" – and stores the lines themselves compressed in chunks. A query first selects streams by label (cheap, via the index) and then scans only the chunks of those streams for the text (brute force, but on a small amount of data). That is the same model Prometheus uses for metrics, which is why the two fit together so well.

The consequence is a rule that decides whether Loki stays fast: labels must have few values. Host, job, systemd unit, namespace, container – a few dozen values each – are good labels. A user name, an IP address, a request ID are not: each new value creates a new stream, and thousands of tiny streams make the index large and the chunks useless. Such values stay in the line and are filtered at query time.

Streams, chunks, index, compactor

• Stream: all lines with exactly the same label set. {job="journal", unit="k3s.service", level="info"} is one stream.
• Chunk: a compressed block of lines of one stream, written to the filesystem when it is full or old enough.
• Index (TSDB, schema v13): maps label sets to chunks, one index file per day.
• Compactor: merges the daily index files and – important – is the component that actually deletes data once it is older than the retention period. Without retention_enabled the retention setting only hides old data from queries.

LogQL in two steps

A LogQL query always starts with a stream selector in braces, followed by optional filters. With a range and a function around it, the same query becomes a metric – that is what alerts use.

# all lines of the k3s unit that contain "error" (case-insensitive)
{job="journal", unit="k3s.service"} |~ "(?i)error"

# as a number: how many such lines in the last 15 minutes?
sum(count_over_time({job="journal", unit="k3s.service"} |~ "(?i)error" [15m]))

|= means “contains”, != “does not contain”, |~ and !~ are the regex versions. Parsers such as | json or | logfmt turn parts of a line into temporary labels that only exist during the query – the cheap way to filter by a value that must never become a real label.

Grafana Alloy: the collector

Alloy is Grafana’s agent, the successor of Promtail and the Grafana Agent. Its configuration is a pipeline of components: sources read logs (journal, files), processors change them (parse, relabel, set the timestamp, drop), and writers send them to Loki. Each component declares where its output goes (forward_to), so the configuration reads like a data flow diagram. Alloy remembers in its own data directory how far it has read each file and the journal, so a restart neither loses nor repeats lines.

🛡️ Why it is here

• One place for all logs. Journal, every pod, the audit log and fail2ban, searchable in Grafana next to the metrics, for the last 14 days – also for pods that no longer exist, whose logs kubectl logs cannot show any more.
• Alerts on things metrics cannot see. An SELinux denial, a failed fail2ban ban or a kernel OOM kill is a log line, not a number. Before, they were only found when somebody happened to look.
• Cheap enough for a small server. Loki and Alloy together need about half a gigabyte of RAM and a few megabytes of disk per hour on the blog – a full-text stack would need several times that.

🔧 What this repository does

The role common_loki runs after common_grafana (whose package repository and API it uses) and before common_monit (which then watches both services). Since 19 October 2026 it ran on the blog; the two Nextcloud servers followed on 20 October after a day of observation.

Packages, pinned

Loki and Alloy come as RPMs from the official Grafana repository, the same one Grafana itself is installed from. The versions are pinned in versions.yml like every other component (loki_version, alloy_version); the end-of-run smoke test reads their build_info metric and reports if the running version is not the pinned one.

Loki: single binary, local disk, 14 days

The package ships a demo configuration that stores everything under /tmp/loki – gone after every reboot – with debug logging. The role replaces it completely. The important parts:

server:
  http_listen_address: 127.0.0.1      # only Grafana on the same host reads
  grpc_listen_address: 127.0.0.1
common:
  path_prefix: /var/lib/loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory                  # one process, no cluster
schema_config:
  configs:
    - from: 2026-10-01
      store: tsdb
      object_store: filesystem
      schema: v13
limits_config:
  retention_period: 336h               # 14 days
compactor:
  retention_enabled: true              # the part that really deletes
analytics:
  reporting_enabled: false

The template is checked before it replaces the real file: Ansible’s validate runs loki -verify-config on the candidate. A typo therefore fails the playbook run instead of a service that no longer starts.

Alloy: four sources, a handful of labels

• Journal with the labels unit, level and transport. transport="kernel" separates the kernel’s own messages – that is where the OOM killer writes.
• Pod logs from /var/log/pods/<namespace>_<pod>_<uid>/<container>/N.log. A relabel rule cuts namespace, pod and container out of the path; stage.cri removes containerd’s prefix (time, stream, partial flag) and joins split lines.
• audit.log. SELinux denials are not in the journal on these hosts – auditd owns the kernel’s audit socket – so Alloy reads the file.
• fail2ban.log, because a failed ban is only logged there.

All streams get the label host; the automatic filename label is dropped because it would only duplicate information that is already in the other labels.

Reading root’s logs without being root

The package runs Alloy as the user alloy in the groups adm and systemd-journal. That is enough for the journal, but not for audit.log (root, mode 0600), fail2ban.log and the pod logs (root, 0640). The lazy fix would be to run Alloy as root. The role gives it a single Linux capability instead, in a systemd drop-in:

[Service]
AmbientCapabilities=CAP_DAC_READ_SEARCH
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/alloy
ProtectHome=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
...

CAP_DAC_READ_SEARCH bypasses the read permission checks on files and directories – and nothing else. Alloy can read every log, but it cannot write outside its own data directory, cannot change ownership, cannot load modules. Loki gets a similar drop-in without any capability at all. systemd-analyze security rates the two units at 4.0 and 3.8 – “OK” on its scale, where an unhardened service sits around 9.

Grafana: datasource, alerts and a self-test

The role creates the datasource “Loki” through Grafana’s API (check first, create only if missing) and four alert rules in the folder that already holds the metric alerts:

At the end, the role tests the whole chain: it writes a unique marker with logger into the journal and queries Loki until the marker shows up. If Alloy or Loki is broken, the playbook run fails right there.

🔍 The first hour: four findings nobody had looked for

Within an hour of the first rollout, the new alerts had sent mails. None of them was noise – each one pointed at something real, and two of them at mistakes of my own.

1. Old lines, new timestamps

The first two alerts fired immediately: an AVC and a fail2ban error. The fail2ban line was from 11:13 that morning. What had happened: on its first start Alloy reads every file from the beginning, and without further instructions it stamps each line with the time it read it. The whole backlog of audit.log and fail2ban.log landed in Loki as “now” – and the alerts, which count lines in the last 15 minutes, saw hours of old errors at once.

The fix is to take the timestamp from the line itself. The audit log carries the epoch in msg=audit(1792426019.999:37103), fail2ban writes local time without a zone:

loki.process "files" {
  stage.match {
    selector = "{job=\"audit\"}"
    stage.regex     { expression = "msg=audit\\((?P<ts>[0-9]+[.][0-9]+):" }
    stage.timestamp { source = "ts"  format = "Unix" }
  }
  stage.match {
    selector = "{job=\"fail2ban\"}"
    stage.regex     { expression = "^(?P<ts>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9:]{8},[0-9]{3})" }
    stage.timestamp { source = "ts"  format = "2006-01-02 15:04:05,000"  location = "UTC" }
  }
  ...
}

The time zone is not hard-coded in the role: it reads the host’s zone with timedatectl and passes it into the template, so the same role is right on a host that runs in local time.

2. The first real AVC: a package installer

The AVC itself was genuine and brand new: during the installation of the Alloy RPM, useradd -m tried to create .bash_logout in the new home directory /var/lib/alloy. That directory has the type var_lib_t, and the policy does not let useradd_t create user files there. Harmless – Alloy does not need a shell profile – but exactly the kind of event that used to disappear in a 6 MB audit log.

3. fail2ban errors that had been there for days

Looking at the fail2ban errors in Loki showed two kinds. One was harmless: KeyError('/var/log/pods/…'), the polling backend losing track of a pod log that was removed on a pod restart; the jail carries on. The other was not: on 13 and 14 October, Failed to execute ban jail 'nginx-k3s-bad-paths' action 'nftables…' – bans that did not happen, during the days the firewall reload was being reworked. Nobody had seen them. The alert now excludes only the known harmless pattern.

4. An AVC from the next new tool

An hour later, Trivy was installed (its own deep dive follows). Its scan unit had NoNewPrivileges=yes, and the script calls k3s kubectl. The k3s binary transitions into the domain container_runtime_t, and under NoNewPrivileges SELinux refuses such a transition unless the policy explicitly allows it: avc: denied { nnp_transition }. The Loki alert fired within minutes of the first scan. Not the first time this mechanism caused trouble here – MariaDB and Grafana had run into it before – but the first time it was found by a machine instead of by a failing service.

🧰 Day-to-day administration

LogQL recipes

# everything SELinux denied today, without the learning-phase domains
{job="audit"} |= "type=AVC" |= "permissive=0"

# what was banned in the last hour, and why
{job="fail2ban"} |= "Ban "

# logs of a pod that no longer exists (namespace + container are labels)
{namespace="wordpress", container="wordpress-fpm"} |= "Fatal"

# kernel messages: OOM, segfaults, hardware
{job="journal", transport="kernel"}

# ssh logins with a key, by user (parsed at query time, not a label)
{job="journal", unit="sshd.service"} |= "Accepted publickey" | regexp "for (?P<user>\\S+) from"

# which units log the most? (finds chatty services)
topk(10, sum by (unit) (count_over_time({job="journal"}[1h])))

Is everything arriving?

# Loki ready, Alloy ready (both only on 127.0.0.1)
curl -s 127.0.0.1:3100/ready
curl -s 127.0.0.1:12345/-/ready

# lines per source in the last 10 minutes
for j in journal pods audit fail2ban; do
  curl -sG 127.0.0.1:3100/loki/api/v1/query \
    --data-urlencode "query=sum(count_over_time({job=\"$j\"}[10m]))"
done

# which namespaces Loki knows
curl -s 127.0.0.1:3100/loki/api/v1/label/namespace/values

# Alloy's own view: components and their health
journalctl -u alloy -p warning --since -1h

Disk and memory

After the first two hours on the blog: Loki about 240 MB RSS, Alloy about 275 MB, together about two minutes of CPU time, and 13 MB in /var/lib/loki – about 150 MB a day, so roughly 2 GB at the full 14 days. The Nextcloud servers have more pods and more traffic; that is why the role stayed on the blog for a day first and the numbers were compared before it moved on. If the disk ever grows faster than expected, the first question is which stream is large – topk by unit or namespace answers it – and the second is whether that stream needs to be kept at all; Alloy can drop lines with stage.drop before they ever reach Loki.

Things to know

• Time zone in the browser. Grafana shows times in the browser’s zone, the servers run in UTC. A line “from 11:13” in a file is 13:13 in Grafana in summer.
• Old lines are rejected. reject_old_samples_max_age: 168h – a file with lines older than a week (for example a restored backup) will not be imported. That is intended.
• Restart is safe. Alloy keeps its read positions in /var/lib/alloy/data. Deleting that directory makes it read everything again – with correct timestamps now, but as duplicates.

🆘 Cheat sheet: symptom → command

The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders.

🔭 What comes next

• An SELinux domain for Loki and Alloy, like the monitoring stack got: today they run as unconfined_service_t, confined only by systemd.
• Logs and metrics in one view. Grafana can show the matching log lines below a metric panel for the same time range – useful for the container dashboard.
• Fewer, better labels for the pods if the number of streams grows: the pod name changes with every restart and could become structured metadata instead of a label.

⚖️ Alternatives – and why Loki

The decisive point was not features but fit. Prometheus and Grafana were already there, the alert mails already worked, and the label model is the same one the metrics use. Loki added the missing half – the why behind the numbers – for about half a gigabyte of RAM.