The Ansible Tour, Part 1: One Repository, Three Servers – How the Code Is Organised and What Happens During a Run

🌐 Also available in: Deutsch · Français · Español

This is the first part of a small series that walks through the Ansible code behind this blog and two Nextcloud servers – not variable by variable, but at the level of “what happens, in which order, and why is it built that way”. The code itself is public: github.com/aptupgrademe/www_k3s. The whole series describes release v3.1.0 (October 2026), and every link to a file points to that release – the roles will keep changing, the links will keep showing the code the text talks about.

A few months ago I open-sourced the stack, and since then the repository has grown up: a quality pass made every role idempotent, nine layers of defense went in, and every role now opens with a comment that explains it for someone who has never seen it. What was still missing is the big picture. If you open the repo today you find 34 roles, four playbooks, about 10,000 lines of tasks and another 12,000 lines of templates. Where do you even start?

Here. This post is the map. It explains how the repository is organised, how one set of roles serves two very different applications, what actually happens when I type ansible-playbook, and which safety nets catch my mistakes before a server sees them. The later parts then zoom in, from the general to the specific – and two of them cut across everything for one topic only: security and monitoring.

🗺️ The tour at a glance

So whichever application you are interested in, you read at most five posts: this one, the platform, your application, security and monitoring.

The goal of the series is practical: someone with basic IT knowledge who reads it carefully should understand enough to run such an environment safely – not just how the code works, but why each piece is there, what to look at, and what to do when something goes wrong. Every part therefore ends with a section “Running it”: the commands, the routine, the alarm mails and the first steps when something breaks.

🧩 One repository, two applications

Three servers are managed from this repository: the blog (WordPress) and two Nextcloud instances. On the surface they have little in common – a PHP blog with MariaDB in a container, and a file cloud with an office suite, its database on the host and a few hundred gigabytes of user data. Underneath, they are almost identical: the same AlmaLinux 9, the same hardened SSH, the same firewall, the same single-node K3s with the same ingress, the same monitoring, the same nightly scanners.

That is exactly how the roles are split. The prefix of a role tells you who uses it:

Two thirds of the code is shared. A fix in the firewall or in the SSH hardening lands on all three servers with the next run, and a lesson learned on one Nextcloud protects the blog too. The price is that a shared role has to know where it runs – how it finds out is one of the central ideas of the repository and gets its own section below.

📁 How the repository is laid out

www_k3s/
├── blog.yml                 full run for the blog
├── blog-update.yml          updates only (plugins, core, rebuilt images)
├── nextcloud-k3s.yml        full run for Nextcloud
├── nextcloud-update.yml     updates only (rebuilt images, occ upgrade, apps)
├── ansible.cfg              how Ansible connects and behaves
├── requirements.yml         Ansible collections (kubernetes.core, ...)
├── inventory/
│   ├── hosts.yml            which host belongs to which group
│   ├── group_vars/all/      values for every host (main.yml, versions.yml)
│   └── host_vars/<host>/    values for one host (vars.yml, vault.yml)
├── roles/<role>/
│   ├── tasks/               what the role does
│   ├── handlers/            what runs once, at the end, if something changed
│   ├── templates/           Jinja templates (config files, manifests)
│   ├── files/               files copied as they are (scripts)
│   ├── defaults/            the role's own default values
│   └── meta/                dependencies + argument_specs (declared inputs)
├── scripts/                 drift check, backup and restore helpers
├── tests/                   K8s apply test, update scenario test
└── docs/                    operations manuals (DE/EN/FR)

Nothing exotic – this is the standard Ansible layout. Two points are worth knowing, because the rest of the series relies on them. First, the playbooks are thin: they contain almost no tasks, only the list of roles in the right order. Everything that happens lives in a role. Second, a role never contains host-specific values. No IP address, no domain, no password appears in roles/. All of that comes from the inventory.

📒 The inventory: the switchboard

If the roles are the machinery, the inventory is the switchboard that decides what the machinery does on which server. It answers three questions: which servers exist, which values apply to all of them, and which apply to just one. I will not go through every variable – there are a few hundred – but the structure is worth understanding, because it explains how one repository can describe three different servers.

Who belongs where: hosts.yml

hosts.yml contains nothing but topology – which host belongs to which group:

all:
  children:
    blog:                         # WordPress servers
      hosts:
        www.example.de: {}
        test: {}
    nextcloud:                    # Nextcloud servers
      hosts:
        instance1: {}
        instance2: {}
        test: {}                  # shared with the blog test server

The group is the most important fact about a host. blog.yml starts with hosts: blog, nextcloud-k3s.yml with hosts: nextcloud – the playbook picks its servers by group, and inside a shared role the group decides what is rendered (more on that in a moment). Note that not even the IP address is here: it lives in the host’s own variables, so it exists in exactly one place.

What applies to everyone: group_vars/all/

Values that are the same on every server live in group_vars/all/ – the pod network of K3s, the paths of the data directories, the default resource limits. The most important file there is versions.yml: every version the playbooks install or deploy, in one table. K3s, Helm, Calico, the Helm charts, every container image, the monitoring binaries:

# --- Kubernetes platform ---------------------------------------
k3s_version:                 "v1.xx.x+k3s1"   # github.com/k3s-io/k3s/releases
helm_version:                "v4.x.x"         # github.com/helm/helm/releases
calico_version:              "v3.xx.x"        # (policy-only, see notes)

# --- WordPress containers --------------------------------------
blog_image_wordpress:        "wordpress:x.y.z-php8.4-fpm"
blog_image_mariadb:          "mariadb:10.11.xx"   # 10.11 LTS

Everything is pinned exactly – never latest, never a floating tag. That is not pedantry: K3s only pulls an image if it is not on the node yet, so a floating tag silently stays on whatever build happens to be cached. With an exact pin, “which version runs where” has one answer, and an update is a one-line change that shows up in git log. A bot (Renovate) watches the upstream releases and proposes those one-line changes as pull requests; I review them and roll them out with the playbook.

What applies to one server: host_vars/<host>/

Each server has a directory with two files. vars.yml holds the ordinary per-host values: addresses, host names, sizing (the two Nextcloud servers have very different hardware), feature switches like the SSH country filter. vault.yml holds the secrets – database passwords, admin logins, mail credentials – and is encrypted with ansible-vault. Ansible decrypts it in memory during the run; on disk and in every backup it is ciphertext.

Neither file is in the public repository. What is public are the .example files next to them: the same structure with placeholder values, so anyone can see which keys a host needs without seeing mine. The vault example looks like this, minus the comments:

email: ...
mysql_root_password: ...
nextcloud_admin_user: ...
nextcloud_admin_pass: ...
blog_db_pass: ...
blog_admin_pass: ...

Who wins: the order of precedence

The same variable can be defined in several places. Ansible resolves that with a fixed order of precedence; for this repository four levels matter, from weakest to strongest:

1. Role defaults (roles/<role>/defaults/main.yml) – the role’s sensible starting point.
2. group_vars – what all hosts share.
3. host_vars – what one host needs differently.
4. Facts set during the run (set_fact) – values the role computes itself, for example after asking the server something.

A concrete example: the SSH country filter. Its default in common_firewall is an empty list – off. A server that should only accept SSH from Germany sets fw_ssh_geo_countries: [DE] in its vars.yml, and that wins. The role itself never changes; the switch is in the inventory. That is the pattern behind most per-host differences in this repository.

The role checks its inputs: argument_specs

A missing value in the inventory used to show up the hard way: halfway through a run, as a cryptic template error. Today 23 roles declare their inputs in meta/argument_specs.yml – name, type, required or not, and whether it is secret. Ansible checks them before the role’s first task runs. A forgotten server_ipv4 now stops the run at the start of the firewall role with “required option missing”, not after the ruleset has been half-written, and a value marked no_log never appears in the output.

🔀 One role, two stacks: three patterns

This is the heart of the shared roles. A common_* role runs on the blog and on Nextcloud, but the two need slightly different things: Nextcloud has MariaDB and Redis on the host, the blog has them in containers; the blog scans uploads, Nextcloud scans user data. Three patterns handle that, and once you know them you find them everywhere.

1. Ask the group

Every host knows the groups it belongs to (group_names). A template or a task can simply ask. The firewall ruleset, for example, only opens the host database ports to the pod network on Nextcloud servers, because only there do the pods need to reach a database on the host:

{% if 'nextcloud' in group_names %}
    # Allow K3s pods to reach MariaDB on the host
    ip saddr {{ k3s_pod_cidr }} tcp dport 3306 accept
    # Allow K3s pods to reach Redis on the host
    ip saddr {{ k3s_pod_cidr }} tcp dport 6379 accept
{% endif %}

On the blog these lines simply do not exist in the rendered file. Same role, same template, two different results – decided by the inventory.

2. Pass a profile from the playbook

Some roles do the same kind of work on both stacks, only with a different set of paths. Those take a profile name as a role parameter, directly in the playbook:

# blog.yml
- { role: common_selinux, selinux_target: wordpress }
- { role: common_clamav,  clamav_target:  wordpress }

# nextcloud-k3s.yml
- { role: common_selinux, selinux_target: nextcloud }
- { role: common_clamav,  clamav_target:  nextcloud }

Inside the role, the profile selects a block of settings from its defaults – for SELinux, the directories that are mounted into the pods and must be labelled for containers:

selinux_profiles:
  wordpress:
    - { label: WordPress www dir, path: "{{ blog_k3s_www_dir }}",  owner: 33,  group: 33 }
    - { label: MariaDB data dir,  path: "{{ blog_k3s_db_dir }}",   owner: 999, group: 999 }
  nextcloud:
    - { label: Nextcloud www dir, path: "{{ nextcloud_k3s_www_dir }}", owner: 33, group: 33 }
    - { label: Nextcloud data,    path: "{{ nextcloud_data_dir }}",    owner: 33, group: 33 }

The role’s tasks then loop over “the directories of my profile” and do not care which stack that is. Adding a third kind of application would mean adding a profile, not a role.

3. Detect instead of declare

The third pattern avoids a variable altogether: the role asks the server. The firewall needs to know whether the server gets its address by DHCP – if it does and the DHCP rules are missing, the address is lost at the next lease renewal. That could be a flag in the inventory, but a flag can be wrong or go stale when a provider changes something. So the role asks NetworkManager at the start of every run:

- name: Firewall | Detect whether IPv4/IPv6 are configured by DHCP
  ansible.builtin.shell: |
    dev=$(ip -4 route show default | awk '{print $5; exit}')
    con=$(nmcli -g GENERAL.CONNECTION device show "$dev")
    nmcli -g ipv4.method,ipv6.method connection show "$con"
  register: _fw_ipmethods
  changed_when: false
  check_mode: false

- name: Firewall | Remember the address configuration for the template
  ansible.builtin.set_fact:
    fw_dhcp4: "{{ ... == 'auto' }}"

The answer goes into a fact, the template renders the DHCP rules only where they are needed. On my servers the result differs – one host gets IPv4 by DHCP, the others are fully static – and nobody has to remember which is which. The general rule in this repository: whatever the server can tell us, we ask the server; the inventory is for decisions, not for facts.

📜 The four playbooks

There are two pairs: a full playbook and an update playbook per application.

The full playbook: order matters

A full playbook is a list of about 25 roles. The order is not alphabetical, it is a chain of dependencies. Here is the blog’s, shortened:

roles:
  - common_version_check   # report: what is installed, pinned, available?
  - blog_packages          # repositories, system update, base packages
  - common_ssh             # SSH on its new port, key-only
  - common_firewall        # nftables
  - common_sysctl          # kernel parameters
  - common_cis_hardening   # CIS benchmark items
  - common_selinux         # label the data dirs  <- before the deploy!
  - common_k3s             # K3s, Helm, cert-manager, ingress
  - common_calico          # NetworkPolicy enforcement
  - blog_k3s_deploy        # all Kubernetes objects of WordPress

  # host-side services
  - common_node_exporter
  - common_prometheus
  - common_grafana
  - common_fail2ban
  - ...                    # mail, updates, scanners, audit, watchdog

  - blog_wordpress         # configure WordPress inside the running pod
  - blog_verify            # smoke tests
  - common_aide_refresh    # always last

A few of those positions are there for a reason worth knowing:

• SELinux before the deploy. The pods mount directories from the host. If they start before those directories carry the container label, SELinux blocks every write and MariaDB refuses to start.
• Monitoring and protection after the deploy. These are host services, but several of them watch what the deploy just created: Prometheus scrapes the exporters of the application pods, fail2ban reads the ingress logs, Monit checks that the pods are running.
• WordPress configuration near the end. blog_wordpress talks to WordPress inside its pod, so it needs a running pod. The deploy role makes sure of that: it waits until MariaDB, Redis and WordPress report ready before it finishes.
• AIDE always last. AIDE compares system files against a baseline every night. A playbook run changes watched files on purpose, so the very last role refreshes the baseline if – and only if – this run changed something. Otherwise the next night would report my own changes as an intrusion.

Tags: one role instead of all of them

Every role in the playbook carries a tag with its own name. That turns a full playbook into a precise tool:

# only the host package updates (e.g. a new Grafana package)
ansible-playbook blog.yml --limit <host> --tags blog_packages

# only the Kubernetes objects (e.g. after an image version bump)
ansible-playbook blog.yml --limit <host> --tags blog_k3s_deploy

This is also why the update playbooks do not install host packages: they are deliberately limited to the application. A new Grafana package arrives with the next full run, or right away with --tags blog_packages – and security updates do not wait for either, they are installed every night by dnf-automatic (part 2).

The update playbooks: guard rails first

The update playbooks restart the application, so they are written defensively. The Nextcloud one, in order:

1. Refuse to run without --limit. hosts: nextcloud would otherwise update every instance at once. Updates are deliberate, one host at a time.
2. Refuse to update an unhealthy instance. If Nextcloud is already in maintenance mode or waiting for a database upgrade, an update would bury the original problem. That guard rail has its own story.
3. Do the work inside block / rescue. Find the images that were rebuilt upstream, restart only those deployments, run the database upgrade, update the apps. If any step fails, the rescue part shows the status, pods, events and the last log lines, leaves maintenance mode only where that is safe, and stops – a defined failure path instead of a half-updated instance.

The blog version follows the same idea with WordPress specifics: it runs the same nightly update job right now, with a database dump first, and restarts only what actually changed. The layered update strategy behind it is described in an older post; part 3 shows the current code.

⚙️ What happens during a run

Now the mechanics: what Ansible does between pressing Enter and the final summary. Understanding this explains almost every design decision in the roles.

Connecting and gathering facts

Ansible connects over SSH as root, with a key – SSH accepts nothing else. Connections are reused across tasks, scripts are piped through the existing connection instead of being copied first, and Ansible asks for no terminal. That last detail matters for the SSH login alarm: without a terminal a playbook run leaves no entry in the login records, so the alarm only fires for real interactive sessions.

Then Ansible gathers facts about the server – but only the minimal set. The roles use a handful (distribution, kernel, host name, date); the full set would cost time for interfaces and hardware details nobody reads.

A report before anything changes

The first role, common_version_check, changes nothing. It compares three things for K3s, Helm, the ingress and cert-manager: what is installed on the server, what versions.yml pins, and what the newest release on GitHub is. The output is a small coloured table – before every run I see whether a pin is outdated or whether the server is behind the repository. The interesting part is how it works: two set_fact steps, the first one normalising all the different version formats into one, the second one comparing them. Collect, prepare, judge – the same three-step pattern returns in the verify roles at the end.

Task by task, host by host

A play runs task by task: task 1 on every host of the play, then task 2 on every host, and so on. When I run the Nextcloud playbook without --limit, both instances are configured in parallel, step for step. Each task reports one of three states, and the whole repository is built around making that report honest:

The goal is called idempotence: a second run directly after the first reports changed=0. That sounds like a cosmetic detail, but it is what makes the playbook usable as a checking tool. If a correct server reports zero changes, then every “changed” on a later run is a real deviation – someone edited a file by hand, a package changed a default, a value drifted. The quality pass was mostly about getting there.

A very common shape in the roles therefore looks like this (simplified) – read the state, change only if it differs:

- name: Read the current value
  ansible.builtin.command: occ config:system:get trusted_domains
  register: current
  changed_when: false          # reading changes nothing
  failed_when: false           # "not set yet" is not an error
  check_mode: false            # also run in --check, it is read-only

- name: Set the value only if it differs
  ansible.builtin.command: occ config:system:set ...
  when: current.stdout != wanted
  changed_when: true           # if it runs, it changed something

Handlers: once, at the end, only if needed

Many changes need a follow-up action: a new sshd_config needs an sshd restart, a new firewall ruleset a reload. Doing that inside the task would restart sshd five times if five files change. Ansible’s answer is the handler: a task notifies a handler by name, and the handler runs once, after all tasks of the play – and only if at least one notifying task actually reported “changed”.

# task
- name: Firewall | Upload nftables ruleset
  ansible.builtin.template:
    src: fw.nft.j2
    dest: /etc/sysconfig/nftables.conf
    validate: nft -c -f %s
  notify: Reload nftables

# handlers/main.yml
- name: Reload nftables
  ansible.builtin.service:
    name: nftables
    state: reloaded

- name: Re-apply fail2ban bans after the nftables reload
  ansible.builtin.command: systemctl try-restart fail2ban
  listen: Reload nftables

Two details from that example are used all over the repository. listen lets a second handler subscribe to the same notification: a firewall reload empties the ban lists, so fail2ban has to put its bans back – and handlers run in the order they are written, so that happens after the reload. And when a follow-up cannot wait until the end – Redis must be running with its new password before the next role connects to it – the role calls meta: flush_handlers, which runs all pending handlers right there.

The dry run: –check –diff

Every playbook can run in check mode: Ansible goes through all tasks, but changes nothing and shows the differences it would make. For this to be useful, the roles take care of two things. Read-only queries run anyway (check_mode: false), otherwise later tasks would compare against nothing. And steps that depend on something that was not really done – unpacking an archive that was not really downloaded – are skipped in check mode instead of failing.

On top of check mode sits a small script, scripts/check-drift.sh. It runs a playbook with --check --diff and prints only the tasks that would change, with their diff. Exit code 0 means: the server is exactly what the code describes. I run it before every real run, and right after a real run it must report nothing – everything it lists then is a task that is not idempotent yet.

scripts/check-drift.sh nextcloud-k3s.yml <host>
scripts/check-drift.sh blog.yml <host> --tags blog_k3s_deploy

The verdict: verify roles

A run that ends with “ok” only means that every task did what it was told. It does not mean the website works. So each playbook ends with a verify role – blog_verify and nextcloud_verify, both built on shared checks in common_verify. They test from the outside in: services running, pods ready, HTTPS answering with a valid certificate, security headers present, blocked paths really blocked, database and cache reachable.

They follow the collect-prepare-judge pattern from the version check, and that is deliberate. All checks run first and only record their results (failed_when: false). Then set_fact turns the raw results into true/false. Then the role prints a report and sends it by mail. Only at the very end does an assert fail the run – if a critical check is red. The advantage over failing at the first problem: one run shows all problems at once, not one per attempt.

🧪 The safety nets around the code

The servers are production – other people’s files, a blog with readers. So a change has to pass several nets before it reaches them, from fast and local to slow and real:

None of these needs a person to remember anything – that is the point. The one rule I keep by hand is the last one: real runs against the servers are started by me, one host at a time, after the dry run looked right.

🛠️ Running it: how a change reaches a server

This is the routine for every change, from a one-line version bump to a new role. It is the same seven steps each time, and the order is the point:

1. Change the role, the template or versions.yml.
2. Lint – runs anyway when you commit, but it is faster to see the findings now.
3. Test without a server where a test exists: tests/k8s/run.sh for the Kubernetes roles, tests/update/run.sh for the update playbooks.
4. Dry run: scripts/check-drift.sh <playbook> <host>. Read the list of changes and ask: is this exactly what I intended? Anything else on the list is a surprise to understand first.
5. Real run on one server, limited with --tags to what the change touches.
6. Drift check again. It must now say “no drift”. Anything still listed is a task that is not idempotent – fix it before it hides a real change on the next run.
7. Commit and push. The hooks check lint and secrets once more, the CI checks the whole history.

Two habits complete it. Secrets are only ever edited with ansible-vault edit inventory/host_vars/<host>/vault.yml – it decrypts into a temporary file for the editor, encrypts again on save and deletes the temporary copy. What you must not do is ansible-vault decrypt the file in place and forget it: then every password lies on disk in plain text, and ends up in every backup of that machine. (In this repository the real vault files are additionally kept out of git by .gitignore, and the secret scan in the hooks is a second net.) And a version update starts with the release notes of the component, not with the playbook: the bot’s pull request says that there is a new version, not whether it is safe to take.

📚 The vocabulary in one place

The later parts use these terms without explaining them again:

🔎 Where to see which Ansible feature

The later parts explain every role completely – not only its tasks, but also its handlers, the inputs it declares in meta/, the switches in its defaults, the logic inside its templates and the scripts it ships. Some Ansible features show up in many roles, but each has one place where it is easiest to understand. If you want to learn a particular feature, start there:

🧭 Where the tour goes next

Part 2 – the platform starts at the bottom: how a fresh AlmaLinux becomes a base system, how K3s, Helm, cert-manager and the ingress are installed with exact versions, and why Calico runs next to Flannel without doing any networking. Parts 3 and 4 then put the applications on top – WordPress and Nextcloud, each in its own post. Part 5 collects every security role in one place, from SSH to the nightly scanners, as the detailed companion to Defense in Depth. Part 6 does the same for monitoring.

Each of them follows the same grid per role: what happens, in which order – why it is built that way – and what went wrong along the way. The last point is often the most useful one. Most of the comments in this repository start with a date and the words “found on”.