The Ansible Tour, Part 2: The Platform – From a Bare AlmaLinux to a Kubernetes Cluster

🌐 Also available in: Deutsch · Français · Español

Part 2 of the Ansible Tour. Part 1 explained how the repository is organised and what happens during a run; this part starts at the bottom of the stack. Like the whole series, it describes release v3.2.0, and every file link points there.

Every server in this setup starts as a plain AlmaLinux 9 cloud image: a minimal Linux, a root login, nothing else. By the end of this part it is a patched, tuned base system that can send mail, keeps itself updated at night, rotates its logs – and runs a single-node Kubernetes cluster with Helm, automatic Let’s Encrypt certificates, an ingress controller and a policy engine. That is the platform: everything the blog and Nextcloud need underneath, and almost all of it shared by both.

The part covers eight roles. For each one I go through what it does step by step, which Ansible features it uses and why, and what went wrong on the way to the current version – including the handlers, the declared inputs in meta/ and the logic inside the templates. Where a Linux or Kubernetes concept is needed to understand the code, it is explained where it first appears.

Where these roles sit in the playbook matters, so a quick reminder of the order: the package role runs right after the version report, then SSH, the firewall and common_sysctl (those three security roles are covered in part 5), then hardening and SELinux, then common_k3s and common_calico, then the application. common_msmtp, common_dnf_automatic and common_logrotate run later, after the application has been deployed.

🖥️ Before any role runs: the server itself

Ansible needs something to talk to: a Linux server with a public address and SSH. All three servers here are virtual private servers (VPS, also called vServer) – virtual machines on a provider’s hardware, rented by the month. You get a fixed share of CPU cores, memory and disk, a public IPv4 and IPv6 address, root access, and a web console for emergencies. You do not get anyone who looks after the system: updates, security, backups and monitoring are yours. That is exactly the job of this repository.

Choosing a provider

The market is large and the offers look alike, so it pays to compare on the points that matter for running a service, not only the price:

The three servers here run at two different providers, all on KVM virtualisation. Spreading them is a small safety net: an outage or an account problem at one provider does not take everything down.

Choosing the distribution

The roles are written for AlmaLinux 9 and nothing else – the package names, the paths, SELinux and dnf are assumed everywhere. How I got from Debian to AlmaLinux is its own story; the reasons that matter for operating it are:

• Long support. AlmaLinux is binary-compatible with Red Hat Enterprise Linux and follows its release cycle: each major version gets security updates for about ten years. A server set up today does not need a major upgrade for years.
• Security updates are labelled. Every update carries an advisory with a type. That is what lets dnf-automatic (below) install only security fixes automatically and leave everything else to a planned run.
• SELinux enforcing by default, with a container policy that K3s supports. Part 5 shows why that matters.
• A large ecosystem. Documentation, the CIS benchmark and most tools exist for this family first.

A Debian or Ubuntu server would work just as well with different roles – but not with these.

How big?

Sizing is a guess at first and a measurement later. These are the three servers, with what they actually use on a quiet day:

Where the memory goes, measured on all three: the platform from this part – K3s itself, the container runtime, the cluster’s system pods, cert-manager and the ingress – takes about 1.3 to 1.5 GB. Prometheus and Grafana add about 0.5 GB. The rest is the application and the operating system’s file cache, which Linux fills with whatever memory is free (that is good, not a problem).

From that, a rough rule of thumb:

• A blog like this one needs at least 2 vCPU and 4 GB RAM; 4 GB is tight with monitoring, 8 GB is comfortable. Disk: a few gigabytes for WordPress, plus uploads, plus room for backups of the database.
• Nextcloud with an online office suite needs more: every open document in the office suite costs 150–200 MB, and file previews, the database and PHP all want memory. 4 vCPU and 8 GB are the minimum for a small group, more for many concurrent users. Disk is driven entirely by the users’ data – plan for growth and watch it (part 6 shows the alarms at 80 and 90 percent).
• No swap. None of the servers has swap. Kubernetes traditionally expects it to be off, and a server that starts swapping is already too small – better to see the memory alarm and resize.
• Upgrade, don’t guess big. Most providers let you add cores and memory later. Start a size smaller, measure with the monitoring from part 6, and grow when the numbers say so.

What the server needs before the first run

Very little, which is the point: a fresh minimal AlmaLinux 9 image, root login with an SSH key on port 22, the server’s addresses and host name in the inventory, and DNS records that point the domain at the server (cert-manager can only get a certificate for a name that resolves to it). The first run then moves SSH to its new port, switches on the firewall and reboots once – part 5 explains how that works without locking Ansible out.

📦 blog_packages and next_packages: from cloud image to base system

These two roles are the only pair in the repository that does the same job in two versions. The blog runs everything in containers, so its host needs very little. A Nextcloud host also runs MariaDB and Redis directly on the host (part 4 explains why), so it needs two more repositories and two more services. Everything else is the same, and I describe both together, pointing out the differences.

Step 1: a real host name

- name: Packages | Set system hostname to the public FQDN
  ansible.builtin.hostname:
    name: "{{ blog_hostname }}"

- name: Packages | Ensure /etc/hosts has a matching entry
  ansible.builtin.lineinfile:
    path: /etc/hosts
    regexp: '^127\.0\.1\.1'
    line: "127.0.1.1  {{ blog_hostname }}"

A fresh cloud image is called localhost. That sounds harmless until several servers send mail: every report from rkhunter, ClamAV or the nightly update job puts $(hostname -f) into its subject line, and five mails from “localhost” are useless. So the very first task sets the public host name from the inventory. The hostname module does what hostnamectl does – it changes the running name and makes it permanent.

The second task shows a module you will see a lot: lineinfile. It manages one line in a file. regexp says which line it is responsible for (here: the line starting with 127.0.1.1), line says what it should look like. If a matching line exists and is correct: ok. If it exists but differs: replaced. If none exists: appended. That is idempotent by construction, and it leaves the rest of the file alone – the right tool when a file belongs to the system and Ansible only owns a single line of it.

Where does blog_hostname come from? From the host’s vars.yml – and the role says so in meta/argument_specs.yml:

argument_specs:
  main:
    short_description: Minimal host-level packages for the blog server
    options:
      blog_hostname:
        description: Public hostname of the blog (e.g. www.example.de)
        type: str
        required: true

Before the first task runs, Ansible checks every option listed there. A host without blog_hostname fails right here with a clear “missing required arguments” – not three tasks later with an undefined-variable error inside a template. The Nextcloud version declares nextcloud_hostname the same way.

Step 2: repositories – ask first, then act

A repository is a package source that dnf downloads from. AlmaLinux ships with BaseOS and AppStream enabled. Many tools in this setup (fail2ban, Monit, rkhunter, the Kubernetes Python client) live in EPEL, the “Extra Packages for Enterprise Linux” repository, so both roles install epel-release – a small package whose only content is the repository definition.

The Nextcloud role needs two more. CRB (CodeReady Builder) ships with AlmaLinux but is disabled, and some EPEL packages depend on it. There is no Ansible module option for “enable this repository”, so the role uses a command – and makes it idempotent with a guard:

- name: Packages | Read list of currently enabled repositories (to check if CRB is active)
  ansible.builtin.command: dnf repolist --enabled
  register: enabled_repos
  changed_when: false
  check_mode: false

- name: Packages | Enable CRB repository (only if not already active)
  ansible.builtin.command: dnf -y config-manager --set-enabled crb
  when: "'crb' not in enabled_repos.stdout"
  changed_when: true

This “ask, then act” pair is the most common pattern in the whole repository, so it is worth reading slowly. The first task only reads: it runs dnf repolist and stores the result with register. changed_when: false tells Ansible that reading changes nothing (a command task would otherwise always report “changed”, because Ansible cannot know what a command does). check_mode: false lets it run even in a dry run – it is harmless, and without its output the second task could not decide anything. The second task only runs when CRB is missing from that output, and if it runs, it did change something (changed_when: true). Result: the first run reports one change, every later run reports none.

The third repository is MariaDB Community. AlmaLinux has its own MariaDB, but the Nextcloud hosts run a specific series from MariaDB directly, pinned in versions.yml. Three steps:

1. rpm_key imports the project’s GPG key. Every package is signed; with the key installed, dnf refuses packages that were tampered with on the way.
2. A copy task writes /etc/yum.repos.d/mariadb.repo with the pinned series in the URL – the variable even appears in the task name, so the run output shows which version it set up.
3. The AppStream “module” for MariaDB is disabled. AppStream modules are alternative package streams, and an enabled MariaDB module would hide the Community packages. Again “ask, then act”: dnf module list --disabled mariadb first, dnf module disable only if it is not disabled yet.

One more detail: EPEL gets module_hotfixes=1 in its repository file, via the community.general.ini_file module. A .repo file is an INI file, and ini_file sets exactly one option in one section – the INI equivalent of lineinfile.

Step 3: the system update – and why the kernel is left out

- name: Packages | Upgrade all system packages  # noqa: package-latest
  ansible.builtin.dnf:
    name: "*"
    state: latest
    update_cache: true
    exclude: kernel*

name: "*" with state: latest is a full dnf upgrade. ansible-lint normally warns about state: latest (it makes runs non-reproducible), and the # noqa comment tells it that here, a full update is exactly the point.

The interesting part is exclude: kernel*, and to understand it you need one piece of the SSH hardening from part 5. The SSH server runs with a systemd setting called ProtectKernelModules. systemd starts every SSH session in its own mount namespace – a private view of the file system – and in that view the kernel module directory /usr/lib/modules is an empty, read-only directory. Nobody who logs in over SSH can load a kernel module, which is the point. But Ansible logs in over SSH too, and installing a new kernel means writing its modules into exactly that directory. Inside the session, that always fails.

So kernels are excluded here, and they still arrive: dnf-automatic (further down in this post) runs as a systemd timer, outside any SSH session, and installs security updates – kernels included – every night.

Step 4: reboot only when it is really needed

Two situations require a reboot before the rest of the play can work. The first one is subtle: a cloud image sometimes already has a newer kernel installed when Ansible first connects, while the old one is still running. The old kernel’s module directory is gone, and the first task that needs a module – common_sysctl loading tcp_bbr, a few roles later – fails with a confusing “No such file or directory”. Better to notice it here:

- name: Packages | Check whether the running kernel's module directory still exists
  ansible.builtin.command: >-
    systemd-run --wait --pipe --quiet --
    test -d /lib/modules/{{ ansible_facts['kernel'] }}
  register: running_kernel_modules
  changed_when: false
  check_mode: false
  failed_when: false

- name: Packages | Reboot – running kernel's modules are gone
  ansible.builtin.reboot:
    reboot_timeout: 300
  when: running_kernel_modules.rc != 0

Why not simply stat the directory? Because of the same SSH sandbox: inside the session, /lib/modules always looks empty, and a plain check would “find” a missing kernel on every run and reboot for nothing. systemd-run is the way out. It asks systemd (PID 1) to start a short-lived service – a transient unit – which runs outside the SSH session’s namespace and sees the real file system. --wait waits for it to finish, --pipe passes its output and exit code back. The hardening stays fully in place; this one command runs where it is allowed to look. You will meet systemd-run again in common_sysctl, for the same reason.

ansible_facts['kernel'] is one of the few facts this repository uses – the running kernel version, gathered at the start of the play. failed_when: false turns “test returned 1” from an error into a result, and the reboot module does the rest: it reboots, waits up to five minutes until SSH answers again, and the play simply continues with the next task.

The blog role has a second check that the Nextcloud role does not: needs-restarting -r (from yum-utils) reports whether updated system libraries like glibc, systemd or OpenSSL are waiting for a reboot. It exits with 1 if so, and the blog reboots on the spot. The Nextcloud role does not have this second check; there, such reboots happen at night through the job described further down.

Step 5: packages and services

Then the actual package lists. On the blog: nftables, the SELinux tools, container-selinux (the SELinux policy that lets containers run), a few utilities and diagnostic tools. On Nextcloud additionally MariaDB-server, redis, rsync and rdiff-backup for the backups, and python3-PyMySQL – the Python library that Ansible’s MySQL modules need on the host. The Nextcloud role then enables and starts MariaDB and Redis.

The last block switches off what this server does not use. firewalld is stopped, disabled and on the blog also masked – masked: true links the unit to /dev/null, so it cannot be started at all, not even as a dependency of another service (the firewall role in part 5 masks it on all hosts). The NFS/RPC and LDAP helpers (rpcbind, sssd) go the same way, and gssproxy is masked because it comes back through socket activation even when disabled. All of these tasks carry failed_when: false: on a host where the service does not exist, “stop X” must not abort the run.

No handlers, no defaults: the package lists are written directly into the tasks, and every service is started where it is installed.

🧮 common_sysctl: kernel parameters

The Linux kernel exposes hundreds of settings under /proc/sys – how TCP behaves, how large queues are, whether the system accepts certain packets. The tool to set them is sysctl, and the permanent way is a file in /etc/sysctl.d/: every file there is read at boot. This role writes two such files and loads one kernel module.

Two templates, two purposes

99-webserver.conf is about speed:

• TCP BBR instead of the default CUBIC as the congestion control algorithm – the part of TCP that decides how fast to send. BBR measures bandwidth and round-trip time instead of waiting for packet loss, which helps on long or lossy connections. It needs the fq (fair queueing) packet scheduler, set in the line above it.
• TCP Fast Open, and no slow start after idle: for a website with sporadic visitors, a connection that was quiet for a moment should not start slowly again.
• Shorter timeouts for closing and idle connections, and larger queues for incoming connections, so a burst of visitors is not dropped at the door.

98-cis-hardening.conf is about security – the network items of the CIS benchmark (a story of its own): no ICMP redirects (a router telling the host “send it this way instead” is how an attacker on the path repoints traffic), no source routing, log packets with impossible source addresses, SYN cookies against SYN floods, full address space randomisation, a restricted ptrace and no core dumps from privileged processes.

The most instructive part of that file is what it leaves out. A stock CIS profile disables IP forwarding and enables strict reverse-path filtering. On a Kubernetes node, both would break the cluster: pods are reached through routing on the host, so forwarding must stay on, and strict reverse-path filtering is a well-known cause of dropped pod traffic. The template says so in a comment, with the date it was verified – so nobody “fixes” it later by following the benchmark blindly.

Loading BBR – through the sandbox again

- name: Sysctl | Load tcp_bbr kernel module
  ansible.builtin.command: systemd-run --wait --pipe --quiet -- modprobe tcp_bbr
  changed_when: false

- name: Sysctl | Persist tcp_bbr kernel module across reboots
  ansible.builtin.copy:
    dest: /etc/modules-load.d/tcp_bbr.conf
    content: |
      tcp_bbr

BBR is a kernel module, and loading modules is exactly what the SSH sandbox forbids. So the module is loaded through systemd-run, and a file in /etc/modules-load.d/ makes systemd load it at every boot. There is an Ansible module for this, community.general.modprobe, but it reads the module directory first to check the state – and inside the session that directory is empty, so it fails with a misleading error. Two small tasks that do the same thing are more honest than a module that cannot see.

Applying the values: a handler and a task

- name: Sysctl | Deploy web server kernel tuning
  ansible.builtin.template:
    src: 99-webserver.conf.j2
    dest: /etc/sysctl.d/99-webserver.conf
  notify: apply sysctl

- name: Sysctl | Apply settings immediately
  ansible.builtin.command: systemd-run --wait --pipe --quiet -- sysctl --system
  changed_when: false

Writing a file in /etc/sysctl.d/ only takes effect at the next boot; sysctl --system applies all files now. The role does that twice, on purpose. The handler apply sysctl runs at the end of the play if one of the files changed – the normal handler logic from part 1. The unconditional task above runs on every run, and it catches something no file change would notice: someone changing a value at runtime with sysctl -w. The file is still correct, so no handler fires, but the running kernel is not. This task puts it back. It cannot tell whether it changed anything, so it says changed_when: false. Both go through systemd-run: the SSH sandbox also makes /proc/sys read-only (ProtectKernelTunables).

Transparent Huge Pages

One setting is not a sysctl at all: Transparent Huge Pages, a memory feature that groups memory into large pages. Set to “always”, it causes latency spikes in databases and Redis, so it goes to madvise – only programs that ask for it get it. The value lives in /sys/kernel/mm/, and there is no config file for it. The role therefore writes a tiny systemd service that sets it at boot (Type=oneshot with RemainAfterExit=yes: run once, then count as “active”), enables it, and notifies a second handler that sets the value right away – again via systemd-run.

✉️ common_msmtp: a server that can send mail

Half of the safety mechanisms in this setup report by mail: the nightly scanners, the update job, the watchdog, the SSH login alarm. A full mail server would be overkill and another thing to secure. msmtp is the opposite: no daemon, no queue, no open port. A program that wants to send a mail calls sendmail (the msmtp package provides that command), msmtp connects to an external SMTP account, hands the mail over and exits.

The role is three tasks – package, configuration file, log file – but each of them shows something:

• Defaults as a translation layer. defaults/main.yml contains no real values, only mappings like msmtp_host: "{{ mail_smtphost }}". The inventory uses the same mail_* names that Nextcloud uses for its own mail settings, and the role translates them to its own names. One set of values, several consumers – Nextcloud, the Monit watchdog and msmtp.
• Secrets declared as secrets. The role’s argument_specs mark the SMTP login and password with no_log: true, so even the input check never prints them.
• no_log on the template task. The rendered /etc/msmtprc contains the password in clear text (mode 0600, root only). A dry run with --diff would print the file content – so the task hides its output entirely. The price is that a drift check only says “would change”, without the diff. For a file with a password in it, that is the right trade.
• A log file without false changes. file: state=touch creates the log, but it also updates the timestamps on every run, which counts as a change. modification_time: preserve and access_time: preserve keep the timestamps, so the task is “changed” only when the file had to be created.

🌙 common_dnf_automatic: updates while I sleep

dnf-automatic is part of AlmaLinux: a systemd timer (systemd’s version of a cron job) that runs dnf upgrade unattended. The role installs it, restricts it to security updates, and – this is the interesting part – fits it into a plan for the whole night.

Two lines in a config file

Two lineinfile tasks set upgrade_type = security (only packages marked as security fixes) and apply_updates = yes (install them, do not just download). Everything else stays a deliberate, manual step with the playbook.

The night has a schedule

Several jobs run at night, and their order matters. This is the plan, written down in the role:

The packaged timer fires at 06:00 – after the reboot slot and after the AIDE refresh. The consequence was found the hard way: every security update showed up as an AIDE alarm the next midnight, and a new kernel waited almost a day for the next 03:30 reboot. So the role moves the timer with a drop-in:

- name: dnf-automatic | Move the install timer into the 01:00-02:00 window
  ansible.builtin.copy:
    dest: /etc/systemd/system/dnf-automatic-install.timer.d/schedule.conf
    content: |
      [Timer]
      OnCalendar=
      OnCalendar=*-*-* 01:00
      RandomizedDelaySec=60m
  register: dnf_automatic_timer_dropin

- name: dnf-automatic | Enable and start install timer
  ansible.builtin.systemd:
    name: dnf-automatic-install.timer
    enabled: true
    state: "{{ 'restarted' if dnf_automatic_timer_dropin is changed else 'started' }}"
    daemon_reload: true

A drop-in is a file in <unit>.d/ that overrides parts of a unit without touching the packaged file – package updates can replace the original, the drop-in survives. Note the empty OnCalendar= line: for list-like settings, systemd adds each assignment. Without the empty line the timer would fire at 06:00 and 01:00. An empty assignment clears the list first.

The second task computes its own state with an inline Jinja if/else. A running timer keeps its old schedule until it is restarted, so it must be restarted when – and only when – the drop-in changed. register on the copy task and the test is changed express exactly that, without a handler. (A handler would work too, but would only run at the very end of the play; this way the timer is correct immediately.)

Reboot at 03:30, check at 06:00

- name: dnf-automatic | Configure nightly automatic reboot if required
  ansible.builtin.cron:
    name: "automatic reboot after security update"
    hour: "3"
    minute: "30"
    job: >-
      OUT=$(/usr/bin/needs-restarting -r 2>&1);
      [ $? -ne 0 ] && { echo "$OUT" | /usr/bin/mail -s "... Rebooting now ..." {{ dnf_automatic_reboot_mail }};
      /usr/sbin/shutdown -r now; }

The cron module manages one crontab entry, identified by its name – change the job and the same entry is updated instead of a second one appearing. The job asks needs-restarting -r whether a reboot is pending, and only then mails the reason and reboots. >- is YAML for “fold these lines into one” – a crontab entry must be a single line. A second entry at 06:00 runs the same check again: if a reboot is still pending, the automatic one failed, and a warning goes out. The recipient comes from defaults/, where dnf_automatic_reboot_mail simply points to the host’s admin address and can be overridden per host.

One more detail that makes the plan hold: the Nextcloud hosts run on German time, the blog on UTC. Because the timer and the cron jobs both use the host clock, the order is the same everywhere, even if the hours on the wall clock differ.

🗂️ common_logrotate: logs that do not grow forever

logrotate runs daily from a timer that ships with the OS. It reads every file in /etc/logrotate.d/ and rotates the logs described there: rename the current log, start a new one, compress and eventually delete the old ones. Grafana and Prometheus ship without such a file; fail2ban and rkhunter ship with incomplete ones. The role is four copy tasks with content: – the file text is written directly in the task – and no handler: logrotate reads the files on its next run.

The one directive that needs an explanation is postrotate. A running program keeps its log file open. When logrotate renames the file, the program keeps writing into the renamed one – the old log grows, the new one stays empty. postrotate tells the program to reopen its log: Grafana gets a SIGHUP, fail2ban gets fail2ban-client flushlogs. rkhunter needs nothing, because it only writes during its nightly scan and never keeps the file open. || true at the end keeps logrotate from reporting an error when the service happens not to run.

☸️ common_k3s: a Kubernetes cluster on one server

Now the big one. First, what it builds. Kubernetes runs applications in containers and keeps them in the state you describe: “run one WordPress pod with these settings, reachable under this name”. You write that description as objects (YAML), hand them to the API server, and Kubernetes makes reality match. K3s is a compact Kubernetes distribution: one binary that contains the API server, the scheduler, the node agent (kubelet), a container runtime and a simple pod network (Flannel). Here it runs as a single-node cluster – one server is the whole cluster. How I ended up there is an older story.

Why Kubernetes on a single server at all? A blog and a Nextcloud would also run as plain packages on the host. The cluster pays for itself in a few ways that matter for operating them: every application is described as a set of objects in the repository, so it can be rebuilt identically on a new server; updates mean changing an image tag, and going back means changing it back; each application lives in its own namespace with its own network rules and resource limits; and both stacks are deployed the same way, so what you learn on one applies to the other. The price is complexity and about 1.5 GB of memory for the platform – which is why this part spends so long on it.

On top of K3s the role installs three things: Helm, a package manager for Kubernetes; cert-manager, which fetches and renews Let’s Encrypt certificates; and the F5 NGINX Ingress Controller, the web server at the edge that receives every HTTPS request and passes it to the right application. The role is split into five files, included in a fixed order:

- name: common_k3s | K3s install/upgrade
  ansible.builtin.import_tasks: install_k3s.yml
- name: common_k3s | Helm install/upgrade
  ansible.builtin.import_tasks: install_helm.yml
- name: common_k3s | cert-manager + Let's Encrypt ClusterIssuer
  ansible.builtin.import_tasks: cert_manager.yml
- name: common_k3s | F5 nginx-ingress controller
  ansible.builtin.import_tasks: nginx_ingress.yml
- name: common_k3s | Default ServiceAccount hardening (CIS 5.1.5)
  ansible.builtin.import_tasks: default_sa_hardening.yml

import_tasks is static: Ansible reads all five files when the playbook starts, as if they were one long file. That means tags, --list-tasks and syntax checks see every task. Its dynamic cousin, include_tasks, reads a file only when the play reaches it – useful when the file name or the decision to include it depends on something computed during the run. Part 5 shows the one place where that matters. Here, everything is known in advance, so the static form is the right one.

install_k3s.yml, part 1: configuration before installation

The first file writes three configuration files before K3s is installed or updated – so the settings apply from the very first start and on every re-run:

• An audit policy for the API server: who did what in the cluster. Secrets and ConfigMaps are logged with metadata only – who accessed them, never their content. Executing commands in pods (pods/exec) is logged with the request, because it is the most powerful thing one can do to a running workload. Changes to permissions (RBAC) are logged in full. The noisy read-only stream of “events” is dropped.
• An admission configuration that limits how many events per second the API server accepts, so a crash-looping pod cannot flood it.
• config.yaml, K3s’s own settings: which built-in components to switch off, the API server flags for the audit log and the admission plugins, and a limit of processes per pod.

- name: K3s | Deploy /etc/rancher/k3s/config.yaml
  ansible.builtin.copy:
    dest: /etc/rancher/k3s/config.yaml
    mode: "0600"
    content: |
      disable:
        - traefik
        - servicelb
        - local-storage
      flannel-backend: vxlan
      write-kubeconfig-mode: "0600"
      kube-apiserver-arg:
        - "audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log"
        - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
        - "enable-admission-plugins=NodeRestriction,EventRateLimit"
        - "admission-control-config-file=/etc/rancher/k3s/admission-control-config.yaml"
      kubelet-arg:
        - "pod-max-pids=4096"
  notify: Restart k3s service

Three components are disabled. Traefik is K3s’s default ingress; this setup uses NGINX instead. ServiceLB is a small load balancer; the ingress binds directly to the host’s ports, so it is not needed. local-storage creates storage volumes on request – nothing here asks for one, all data lives in fixed directories on the host, so it only ran as one more privileged process. What does not run cannot be attacked.

The admission plugin line carries one of the repository’s better lessons, documented in the file: in vanilla Kubernetes, enable-admission-plugins adds to the default list. In K3s it replaces it. Setting it to just EventRateLimit silently dropped NodeRestriction – a plugin that stops a compromised node agent from modifying anything but its own objects. A re-run of the CIS checker (kube-bench) caught it. Since then, NodeRestriction is listed explicitly, with a comment explaining why. The file also explains a CIS recommendation it deliberately does not follow (forcing a registry pull on every pod start): in a setup without separate tenants it adds almost no security, but turns every short registry outage into failing pods.

All three tasks notify the same handler, Restart k3s service – one restart, no matter how many files changed. And one small task right after them creates the audit log directory, because the API server does not create it itself and refuses to start without it. It has to exist before that restart.

install_k3s.yml, part 2: install or upgrade, pinned

Now the binary. The role compares the installed version with the pin:

- name: K3s | Check installed k3s version
  ansible.builtin.shell: |
    set -o pipefail
    k3s --version 2>/dev/null | awk '{print $3}'
  register: k3s_installed_version
  changed_when: false
  check_mode: false
  failed_when: false

- name: K3s | Download k3s binary (install or upgrade to {{ k3s_version }})
  ansible.builtin.get_url:
    url: "https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s"
    dest: /usr/local/bin/k3s
    mode: "0755"
  when: not k3s_binary.stat.exists or (k3s_installed_version.stdout | trim) != k3s_version
  notify: Restart k3s service
  timeout: 600

This is the pinned binary pattern, and it returns for Helm, the helm-diff plugin and all monitoring programs: read the installed version, download only if it differs, notify a restart. On a fresh server the version is an empty string, which never equals the pin – so the same task covers installation and upgrade. An upgrade in K3s is literally “replace the binary, restart the service”; the cluster’s data survives. set -o pipefail makes the pipe fail if k3s --version fails, not just if awk does. timeout: 600 is a task keyword, not a module option: if the download hangs, Ansible gives up after ten minutes instead of waiting forever.

The installer script that creates the systemd service comes from the K3s repository at the same tag as the binary, not from the usual get.k3s.io. That address always serves the newest script, and it answered with HTTP 500 for a while during the first installation of a new server, while GitHub was fine. Pinned binary plus pinned script means the same installation every time. The tag contains a +, which must be written as %2B in a URL – the urlencode filter does that.

- name: K3s | Run K3s installer (skip binary download, flags from config.yaml)
  ansible.builtin.command: sh /tmp/k3s-install.sh
  environment:
    INSTALL_K3S_SKIP_DOWNLOAD: "true"
  args:
    creates: /etc/systemd/system/k3s.service
  when: not k3s_unit.stat.exists

- name: K3s | Apply pending handlers (restart on config/binary change)
  ansible.builtin.meta: flush_handlers

Three details in two tasks. environment sets a variable for this one command: the installer should not download the binary again. creates makes the command run only if that file does not exist yet – the oldest idempotence trick in Ansible. And the script is run as sh /tmp/k3s-install.sh instead of being executed directly, because the CIS hardening mounts /tmp with noexec: files there cannot be executed, but they can still be read by a shell.

The when condition has a story too. It originally checked whether the binary existed. Then a first installation was interrupted after the binary download but before the installer – and every following run saw the binary, skipped the installer, and failed further down because the service did not exist. Now the decision is made on the systemd unit, and an interrupted installation heals itself on the next run.

Then meta: flush_handlers. Handlers normally run at the end of the play – but every role after this one talks to Kubernetes. If K3s was just upgraded or reconfigured, the restart has to happen now, not after the application was deployed against the old version. flush_handlers runs all pending handlers at this point. The handler file itself is a single entry:

- name: Restart k3s service
  ansible.builtin.service:
    name: k3s
    state: restarted

After that, the role waits until the API server answers – not with a fixed sleep, but by asking:

- name: K3s | Wait for the API server to become available
  ansible.builtin.command: k3s kubectl get nodes
  register: k3s_nodes
  until: k3s_nodes.rc == 0
  retries: 30
  delay: 5
  changed_when: false

until repeats the task until the condition is true, here at most 30 times with 5 seconds between attempts. On a fast restart the play continues after a few seconds; on a slow one it waits up to two and a half minutes; if K3s never comes up, the run fails with a clear message instead of a confusing error in the next role.

Two small things close the file. K3s writes its certificate files readable for everyone, and regenerates some of them on restart. find collects all *.crt files, and a file task with loop sets each one to 0600 – on every run, because the files come back. (loop_control.label prints only the path per item instead of the whole file record.) Finally the cluster’s admin configuration is copied to /root/.kube/config, so kubectl and helm find the cluster in a root shell. remote_src: true means: copy from one place on the server to another, not from my machine.

install_helm.yml: Helm and the helm-diff plugin

Helm installs charts – packages of Kubernetes objects with settings (“values”) – and keeps track of each installation as a release. The role installs it with the same pinned pattern: version check, tarball download only on a difference, unpack, copy to /usr/local/bin. Two lines in the unpack and install tasks are worth a look:

  when:
    - helm_download.changed | default(false)
    - not ansible_check_mode

A list under when means “all of these”. The first condition: only if something was downloaded. The second one is about dry runs. In check mode the download task only pretends – it reports “would change”, but there is no file. Unpacking a file that does not exist would fail, so the role skips that step in check mode. That way a dry run on a server with an outdated Helm reports the upgrade correctly instead of crashing. The same two lines guard every downloaded archive in the repository.

The second half installs helm-diff, a plugin that lets the Helm module compute a real difference between the running release and the requested one. The installed version is read from helm plugin list, which prints a small table, roughly like this:

NAME    VERSION   TYPE   APIVERSION   PROVENANCE   SOURCE
diff    3.x.x     cli/v1 legacy       ...

…and one Jinja expression picks the version out of it:

when: >-
  (helm_plugins.stdout_lines | default([]) | map('split')
   | selectattr('0', 'equalto', 'diff') | map(attribute='1') | first | default('')) != helm_diff_version

Read it as a pipeline, left to right: take the output lines → split each line into words (now each line is a list) → keep only the lines whose first word (index 0) is diff → take the second word (index 1) of each → the first of those, or an empty string if there is none. If that is not the pinned version, the block below runs. A block groups tasks so that one when applies to all of them.

Inside the block, the download carries a checksum: checksum: "sha256:{{ helm_diff_sha256 }}". get_url verifies the file after downloading and fails if it does not match. The plugin is pinned together with its checksum in versions.yml, and the bot that proposes version updates is configured to remind me that both have to change together.

cert_manager.yml: certificates without certbot

cert-manager watches the cluster for objects that say “this host name needs a certificate”, obtains one from Let’s Encrypt, stores it as a Kubernetes Secret and renews it before it expires. Let’s Encrypt checks that you control a domain with an HTTP-01 challenge: it gives cert-manager a token, and then requests http://your-domain/.well-known/acme-challenge/<token>. If the token comes back, the domain is yours.

- name: cert-manager | Install/upgrade cert-manager via Helm
  kubernetes.core.helm:
    kubeconfig: "{{ k3s_kubeconfig }}"
    name: cert-manager
    chart_ref: jetstack/cert-manager
    chart_version: "{{ cert_manager_chart_version }}"
    release_namespace: cert-manager
    create_namespace: true
    update_repo_cache: true
    values:
      crds:
        enabled: true
    wait: true
    timeout: 300s

This used to be a shell task with helm upgrade --install. It worked, but it reported “changed” on every run and created a new Helm revision each time – so the playbook could never say “nothing to do”. The kubernetes.core.helm module compares the requested chart version and values with the running release and only upgrades on a real difference. wait: true waits until the release’s pods are ready.

Then comes a step that confuses many people the first time. cert-manager brings new object types – for example ClusterIssuer, “where and how to get certificates”. Kubernetes only knows such a type after its CustomResourceDefinition (CRD) is registered. Applying a ClusterIssuer a moment too early fails with “no matches for kind”. So the role waits until the webhook is available and the CRD is established, then applies the issuer with the K8s apply pattern (explained in full in the Calico section below).

The issuer template contains the single most hard-won setting of this file. The challenge request has to reach cert-manager’s temporary solver, but there is already an Ingress for the same host name – the application’s own. F5’s controller silently ignores a second Ingress for the same host. The fix is in two annotations: the solver’s Ingress is created as a minion, which the controller merges into the existing master Ingress for that host, and it is excluded from the HTTP-to-HTTPS redirect, because Let’s Encrypt checks over plain HTTP.

nginx_ingress.yml: the front door

The ingress controller is the only thing that answers on ports 80 and 443. It runs as a DaemonSet (one pod per node – here: exactly one) with hostNetwork, so it binds directly to the host’s ports like a normal web server would. The file has four steps.

A fallback certificate. When the controller receives a request for a host name it has no certificate for – wrong DNS, a certificate that is not issued yet – F5’s controller does not present a self-signed one, it simply closes the connection. From a browser, that is indistinguishable from a server that is down. That was found during a DNS mistake, when “wrong DNS” and “server down” looked identical. The role therefore generates a self-signed certificate once (openssl with creates:) and registers it as the default. Real certificates are unaffected: nginx picks the one that matches the host name.

- name: ingress-nginx | Read default TLS certificate and key
  ansible.builtin.slurp:
    src: "/etc/ingress-nginx-defaults/{{ item }}"
  loop: [tls.crt, tls.key]
  register: ingress_default_tls
  no_log: true

- name: ingress-nginx | Apply default TLS certificate as a K8s Secret
  kubernetes.core.k8s:
    definition:
      apiVersion: v1
      kind: Secret
      type: kubernetes.io/tls
      metadata: { name: ingress-default-tls, namespace: ingress-nginx }
      data:
        tls.crt: "{{ ingress_default_tls.results[0].content }}"
        tls.key: "{{ ingress_default_tls.results[1].content }}"
  no_log: true

slurp reads a file from the server and returns it base64-encoded – which happens to be exactly the format Kubernetes expects in a Secret’s data. Because the task has a loop, the registered variable has a results list, one entry per item, in loop order: results[0] is the certificate, results[1] the key. Both tasks hide their output with no_log: one of these values is a private key.

A one-time migration. The setup used to run the community kubernetes/ingress-nginx project, which was retired. A shell task checks whether that chart is still installed and removes it before installing F5’s chart, so Helm never tries to upgrade between two incompatible charts. It prints “uninstalled” or “skipped”, and changed_when looks for the word – a simple way to give a script an honest status.

The release itself. The Helm values come from a template, rendered in memory and parsed as YAML: values: "{{ lookup('ansible.builtin.template', 'ingress-nginx-values.yml.j2') | from_yaml }}". No temporary file on the server. The values file is where the edge is configured: TLS 1.2 and 1.3 only, a fixed cipher list, HTTP/2, connection reuse to the application pods, gzip, no version number in the Server header, and the rate-limit zones that the blog’s and Nextcloud’s login pages use (parts 3 and 4). It also documents what is deliberately not there: Brotli compression (the controller image does not contain the module, so the settings were inert) and OCSP stapling (Let’s Encrypt no longer runs OCSP). The TLS post has the full picture.

Waiting. Finally kubectl wait blocks until the controller pod is ready, so the application roles do not deploy into a cluster without a front door.

default_sa_hardening.yml: no tokens by default

Every pod in Kubernetes runs as a ServiceAccount, and by default it gets a token mounted that lets it talk to the API server. A web application does not need that – and a compromised one should not have it. CIS recommends switching it off for each namespace’s default account:

- name: K3s | Find non-kube-system default ServiceAccounts (CIS 5.1.5)
  kubernetes.core.k8s_info:
    kind: ServiceAccount
    field_selectors:
      - metadata.name=default
  register: k3s_default_sas

- name: K3s | Disable SA token automount on every non-kube-system default ServiceAccount
  kubernetes.core.k8s:
    definition:
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: default
        namespace: "{{ item }}"
      automountServiceAccountToken: false
  loop: >-
    {{ k3s_default_sas.resources | map(attribute='metadata.namespace')
       | reject('equalto', 'kube-system') | list }}

k8s_info is the read-only counterpart of k8s – like kubectl get. field_selectors filters on the server: only accounts named default. The loop builds its list with a filter chain: from all found accounts take the namespace, drop kube-system (the cluster’s own components there need their tokens), and turn it into a list. The k8s task then applies a partial definition – only the one field that should change; everything else about the account stays as it is. The application namespaces get the same patch in their own roles, because at this point in a fresh installation they do not exist yet.

The role declares one input in meta/argument_specs.yml: email, the address Let’s Encrypt sends expiry warnings to – marked no_log like every personal value.

🛡️ common_calico: making NetworkPolicies real

A NetworkPolicy is a Kubernetes object that says which pods may talk to which – for example: “the database pod accepts connections only from the WordPress pod”. The blog and Nextcloud both rely on such policies. There is a catch that is easy to miss: Kubernetes itself does not enforce them. The network plugin does – and Flannel, the plugin built into K3s, does not. A policy that denies everything is accepted without complaint and has no effect at all. That was verified on the test server: a deny-all policy changed nothing until Calico was installed, and worked immediately afterwards.

Calico can do the networking itself, but here it runs in policy-only mode: Flannel keeps moving the packets, and Calico’s agent (Felix) only reads the policies from the Kubernetes API and turns them into packet filter rules on the host. The manifest is derived from Calico’s official “Canal” manifest (Calico for policy, Flannel for networking), with everything removed that K3s already provides: the network plugin installation, the Flannel container, the network configuration. What remains are 34 objects – 23 CRDs, the permissions, a DaemonSet for the agent and a Deployment for the controller.

The K8s apply pattern, completely

The whole role is one task plus two waits, and the task is the pattern that 26 tasks in the repository use to put Kubernetes objects into the cluster. Here it is, to be read from the inside out:

- name: Calico | Apply policy-only manifest
  kubernetes.core.k8s:
    kubeconfig: "{{ k3s_kubeconfig }}"
    state: present
    apply: true
    definition: "{{ item }}"
  loop: "{{ k8s_objects if ansible_diff_mode else [k8s_objects] }}"
  loop_control:
    label: "{{ (item.kind ~ '/' ~ item.metadata.name) if item is mapping
               else ((item | length) ~ ' objects') }}"
  vars:
    k8s_objects: "{{ lookup('ansible.builtin.template', 'calico-policy-only.yml.j2')
                     | ansible.builtin.from_yaml_all | select | list }}"
  register: calico_apply

1. lookup('ansible.builtin.template', ...) renders the Jinja template on my machine into one long string. Nothing is copied to the server. (Lookups always run on the controller.)
2. from_yaml_all parses that string – many YAML documents separated by --- – into a list of objects.
3. select without an argument drops empty entries, for example from a trailing ---.
4. vars: on a task defines k8s_objects for this task only.
5. kubernetes.core.k8s with state: present and apply: true behaves like kubectl apply: create what is missing, update what differs, leave the rest – and report “changed” only if a live object really changed. It talks to the API directly through the Python client installed by common_k3s; no kubectl, no temporary file.
6. The loop is the clever part. In a normal run the list [k8s_objects] has one item – the whole list of 34 objects – so the module runs once. With --diff (ansible_diff_mode is true) the loop runs over the objects themselves, one module call each, because only then does the diff show the change per object. Fast when it does not matter, detailed when I want to see.
7. loop_control.label keeps the output readable: “DaemonSet/calico-node” per object, or “34 objects”.

The two tasks after it wait for the agent and the controller with kubectl rollout status – but only when: calico_apply.changed. An unchanged run does not wait at all.

One more design note from the role’s header comment, because it is a good general lesson: the permissions for the Calico agent are copied from upstream in full, not trimmed to what policy-only mode “should” need. Several internal loops in the agent run regardless of mode, and without their permissions they do not skip themselves gracefully – the pod simply never becomes ready. Trimming permissions is good practice, but not below what the software actually requires.

🛠️ Running it: the platform day to day

Most of this part runs by itself – nightly updates, certificate renewals, log rotation. What remains for the administrator is to know where to look and how to make a planned change.

Looking at it

# The nightly plan: when do the timers fire next?
systemctl list-timers dnf-automatic-install.timer logrotate.timer

# Is a reboot pending? (exit code 1 = yes)
needs-restarting -r

# Does the server send mail? (the address from the inventory)
echo "test from $(hostname -f)" | mail -s "mail test" you@example.org
tail /var/log/msmtp.log

# Kubernetes: the node, all pods, and the Helm releases
k3s kubectl get nodes
k3s kubectl get pods -A
helm list -A

# All certificates: valid, and when they are renewed
k3s kubectl get certificate -A \
  -o custom-columns=NAME:.metadata.name,READY:.status.conditions[0].status,RENEWAL:.status.renewalTime

Mails from this part

Planned changes

• Host packages that are not security fixes (a new Grafana, say) come only with the package role: ansible-playbook blog.yml --limit <host> --tags blog_packages (or next_packages with the Nextcloud playbook).
• K3s, Helm, the charts: change the pin in versions.yml (or merge the bot’s pull request), run scripts/check-drift.sh <playbook> <host> --tags common_k3s to see what would happen, then the real run with --tags common_k3s. K3s upgrades one minor version at a time – read the release notes, and take a snapshot of the server first.
• Calico: the same with --tags common_calico. Afterwards k3s kubectl -n kube-system get ds calico-node must show the pod ready again.
• Kernel parameters: edit the template, run --tags common_sysctl. Never “fix” IP forwarding or reverse-path filtering by the book – the cluster needs them as they are.

🧭 What comes next

At this point the platform is complete: a patched base system with sane kernel settings, mail, nightly updates and rotated logs, and a Kubernetes cluster with Helm, certificates, a front door and working network policies. Part 3 puts the blog on top of it – WordPress, MariaDB and Redis as pods, and everything blog_wordpress does inside the running application. Part 4 does the same for Nextcloud. The roles that were skipped here – SSH, the firewall, hardening and SELinux – are all in part 5, the security part.