Deep Dive: K3s – A Whole Kubernetes in One Binary, What It Takes Out, and Six Things It Did Not Do by Itself

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

K3s is certified Kubernetes in a single binary of about 70 MB. API server, scheduler, controller manager, kubelet, kube-proxy, containerd, Flannel, CoreDNS and a network policy controller all start from one process and one systemd unit. That makes it the obvious choice for a cluster of exactly one node – and it hides a lot. This post explains what actually runs inside that binary, what part 2 of the tour configures, and six things that K3s did not do by itself, which were only found in operation.

🧭 What is inside the binary

One supervisor, many components

k3s server is a supervisor. It starts the Kubernetes control-plane components as goroutines inside its own process – API server, scheduler, controller manager – and the node components next to them: the kubelet, kube-proxy, the Flannel CNI and an embedded network policy controller (code from kube-router). containerd runs as a separate child process with its own socket under /run/k3s/containerd/. Everything shares one log stream in the journal (journalctl -u k3s) and one version number.

The datastore: SQLite instead of etcd

Upstream Kubernetes stores its state in etcd. A single-node K3s uses SQLite through kine, a shim that translates the etcd API into SQL. The whole cluster state – every Deployment, Secret, ConfigMap, event – lives in /var/lib/rancher/k3s/server/db/state.db: 24 MB on the blog, 23 MB on a Nextcloud server, plus a 20 MB write-ahead log. With more than one server node, K3s switches to embedded etcd.

containerd, the CRI and the pause container

The kubelet talks to containerd through the Container Runtime Interface. For every pod, containerd first starts a tiny pause container that holds the pod’s network namespace; the real containers join it. k3s crictl is the CRI client for looking at that level directly – pods, containers, images – without going through the API server. K3s uses cgroup v2 on AlmaLinux 9, so the pod’s resource limits are plain cgroup files under /sys/fs/cgroup/kubepods.slice/.

Networking: Flannel, kube-proxy, the policy controller

• Flannel gives every pod an address from 10.42.0.0/16 (on one node: one /24), creates a bridge cni0 and a veth pair per pod. Backend here: VXLAN – irrelevant on one node, but the default for a second.
• kube-proxy turns Services into iptables rules (through iptables-nft, into nftables tables): ClusterIP DNAT, NodePorts, load balancing over endpoints.
• The network policy controller turns NetworkPolicies into chains (KUBE-ROUTER-*, KUBE-NWPLCY-*, one KUBE-POD-FW-* per pod). It is part of the binary and on by default. Remember this one.

Certificates and tokens

K3s creates its own CA and all certificates on first start (/var/lib/rancher/k3s/server/tls/), valid for a year, and renews them automatically when the service starts within 90 days of expiry. The admin kubeconfig is /etc/rancher/k3s/k3s.yaml; the node token in /var/lib/rancher/k3s/server/token is what a second node would need to join.

🛡️ Why it is here

• Kubernetes semantics on one small server. Deployments with rolling or recreate updates, probes that restart a hung container, NetworkPolicies between pods, resource limits, an admission policy for images – without running a control-plane cluster.
• One binary, one unit, one pinned version. The whole platform is reproducible from Ansible; an upgrade is “replace the binary, restart the unit”.
• Good AlmaLinux and SELinux support: a maintained SELinux policy (k3s-selinux) and containerd with SELinux labelling.

🔧 What this repository does

config.yaml: what is switched off, what is switched on

disable:
  - traefik          # F5 nginx-ingress instead (installed with Helm)
  - servicelb        # no LoadBalancer services; ingress uses hostNetwork
  - local-storage    # no PVCs at all - everything is hostPath (0 PVCs on all hosts)
flannel-backend: vxlan
write-kubeconfig-mode: "0600"
selinux: true
kube-apiserver-arg:
  - "audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log"
  - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
  - "audit-log-maxage=30"
  - "audit-log-maxbackup=10"
  - "audit-log-maxsize=100"
  - "enable-admission-plugins=NodeRestriction,EventRateLimit"
  - "admission-control-config-file=/etc/rancher/k3s/admission-control-config.yaml"
kubelet-arg:
  - "pod-max-pids=4096"

Most of the arguments come from the CIS Kubernetes benchmark: an API audit log with rotation, EventRateLimit so a crash-looping pod cannot flood the API with events, a PID limit per pod against fork bombs. One recommendation is deliberately not followed: AlwaysPullImages would pull every image from the registry on every pod start – including the WordPress cron pod every five minutes – and turn a short registry outage into failing pods everywhere, for a multi-tenant protection that a single-tenant cluster does not need.

Pinned, and installed in two steps

The binary comes from the GitHub release of k3s_version (versions.yml), the install script from the same tag – not from get.k3s.io, which always serves the newest script and answered HTTP 500 for a while on the day the second Nextcloud server was set up. The script runs with INSTALL_K3S_SKIP_DOWNLOAD=true, only to create the unit. Around K3s the role installs Helm, cert-manager with a Let’s Encrypt ClusterIssuer and the F5 nginx-ingress controller, and patches every default ServiceAccount so that pods no longer get an API token they never use (CIS 5.1.5). The application roles add a ValidatingAdmissionPolicy per namespace that rejects pods whose images do not come from an allowed list of registries – built into the API server since Kubernetes 1.30, no extra component.

The upgrade path

1. Read the running version (k3s --version).
2. If it differs from the pin: download the binary with get_url, force: true.
3. restorecon on the binary.
4. Flush the handlers right away – the restart happens here, not at the end of the play.
5. Wait until kubectl get nodes answers, then continue with the roles that deploy into the cluster.
6. At the end of the run, common_version_active compares the kubelet version reported by the API with the pin.

Steps 2, 3 and 6 each exist because of a mistake, see below.

🐛 Six things K3s did not do by itself

1. The upgrade that did nothing

Until 19 October 2026, the download task had no force. Ansible’s get_url then skips the download when the destination file already exists – so every version bump since the first install had been a no-op on the existing hosts, quietly. It surfaced because the drift check for v1.37.1 showed no change for K3s at all. Since then the smoke test at the end of every run (common_version_active) reads the version the kubelet actually reports, not the one in the inventory.

2. Pods that SELinux did not confine

SELinux was enforcing on all hosts and the HostPath directories carried container_file_t. But K3s only starts containerd with SELinux support when its configuration says selinux: true – and it did not. Every pod ran as spc_t or unconfined_service_t, which SELinux does not restrict at all. Since 19 October 2026 the option is set and the k3s-selinux policy is installed; every pod now runs as container_t with its own MCS categories, and pods that existed before the switch had to be recreated to pick it up. The SELinux deep dive tells the whole story.

3. A new binary with the wrong label

get_url writes through a temporary file and renames it. The new binary therefore gets the label of its directory (bin_t) instead of container_runtime_exec_t from the policy, and K3s would start in the wrong domain. A restorecon after every download fixes that.

4. The policy engine that was never needed

For two months Calico ran next to K3s as the supposed NetworkPolicy engine. It enforced nothing – Calico’s Felix only handles interfaces called cali*, Flannel creates veth* – and all enforcement had come from K3s’s own controller the whole time. The proof was unpleasant: with the controller switched off for a test (disable-network-policy), a pod in the default namespace reached MariaDB on port 3306. Calico is gone, and the configuration now carries a comment in capitals: never disable the embedded controller. The nftables deep dive describes the firewall side.

5. An admission flag that replaced instead of added

In upstream Kubernetes, --enable-admission-plugins adds plugins to the default list. In K3s it replaces K3s’s own list. Setting it to just EventRateLimit silently dropped NodeRestriction – the plugin that stops a compromised kubelet from modifying other nodes’ objects. A kube-bench re-run caught it (CIS 1.2.14 went from PASS to FAIL). Both are listed explicitly now. A related trap: the API server does not create the directory of its audit log and refuses to start without it; the role creates it before the restart.

📊 Measured

About a gigabyte of RAM for the whole control plane is the price of “real Kubernetes on one node”. Two numbers stood out. The audit log sat at its cap of ten 100 MB backups – rotation worked, but the “30 days” of the configuration were really four and a half. And on the Nextcloud server the control plane used five times the CPU of the blog with fewer pods.

Seven, not a bug: the same work at five times the cost

The audit log showed the same write load on all three servers, and a 30-second CPU profile – taken without a restart through the kubelet’s /debug/pprof, which profiles the whole K3s process – showed no single culprit: everything was about four and a half times as expensive, mostly the Go scheduler waking goroutines and the system calls behind it. Limiting Go to four threads (GOMAXPROCS=4) changed nothing. The answer was in /sys/devices/system/cpu/vulnerabilities/: the Nextcloud servers run on an older AMD EPYC generation that is affected by Retbleed, and the kernel protects every return from a kernel function with a “return thunk”. For a program that lives on system calls and context switches that costs a lot; the blog’s newer CPU is “not affected”. Switching the mitigation off on a shared virtual machine would be careless – about 2.6 % of an eight-core server is the price.

Six: an audit log that drowned in its own noise

A breakdown of one 100 MB file showed where the volume came from: lease heartbeats of the leader elections and the kubelet (23 %), health, discovery and OpenAPI requests (14 %), the API server talking to itself (12 %), and read-only requests of the platform’s own components, service accounts and nodes (about 40 %). None of it has security value; it only pushed the interesting entries out of the retention window. The policy now drops exactly these – with two rules placed before the drop rule on purpose: reads of secrets stay logged for every reader, and everything an administrator does stays logged. A simulation against the real file predicted that 10 % would remain; after the rollout the log grew by about 28 MB a day instead of 230.

rules:
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version", "/openapi*", "/api*", "/metrics"]
  - level: None
    users: ["system:apiserver"]
  - level: None
    resources: [{group: "coordination.k8s.io", resources: ["leases"]}]
  - level: Metadata                      # before the next rule: secrets for every reader
    resources: [{group: "", resources: ["secrets"]}]
  - level: None
    verbs: ["get", "list", "watch"]
    userGroups: ["system:serviceaccounts", "system:nodes"]
  ...                                    # exec/attach at Request, RBAC at RequestResponse, rest at Metadata

🧰 Day-to-day administration

# the cluster
k3s kubectl get nodes -o wide
k3s kubectl get pods -A
k3s kubectl top pods -A            # metrics-server is part of K3s

# one level down: containerd directly, without the API server
k3s crictl pods
k3s crictl ps
k3s crictl images

# logs of the whole platform (API server, kubelet, Flannel, policy controller)
journalctl -u k3s --since -1h

# certificates: when do they expire? (renewed on a restart within 90 days)
for c in /var/lib/rancher/k3s/server/tls/*.crt; do
  printf '%-45s ' "$(basename "$c")"; openssl x509 -enddate -noout -in "$c" | cut -d= -f2
done

# SELinux: in which domain do the pods run?
ps -eZ | grep container_t | head

Upgrades

One minor version at a time, as Kubernetes requires (1.36 → 1.37, not 1.35 → 1.37). Raise k3s_version in versions.yml, run the drift check (--check --diff) – it must now show the binary download –, run the playbook, and read the version report at the end. A restart of K3s restarts neither the pods nor containerd’s containers; the API is gone for about 20 seconds, the websites are not.

Backups – and what is actually needed

The datastore is not backed up here, on purpose. Every object in it is created by Ansible from the repository and the vault: namespaces, Secrets, Deployments, policies. The data that matters lives outside the cluster in HostPath directories and in the databases, and those are pulled every night by the backup server. Restoring a server therefore means: install AlmaLinux, run the playbook, restore the data directories. The cluster itself is disposable – the strongest argument for keeping everything in code.

🆘 Cheat sheet: symptom → command

The short version of this article for the moment something is wrong. Run on the host as root; <…> are placeholders.

🔭 What comes next

• kube-proxy in nftables mode once K3s makes it the default – then the iptables-nft compatibility layer can go.
• A second node would change three things: etcd instead of SQLite, HostPath volumes would no longer work for pods that can move, and Flannel’s VXLAN would actually carry traffic.

⚖️ Alternatives – and why K3s

K3s keeps its promise: real Kubernetes in one binary. What the last weeks showed is that “one binary” is also “one place where defaults hide”. Four of the six problems above were not bugs in K3s but defaults and assumptions – SELinux off unless asked for, an admission flag that replaces instead of adds, a policy controller that was already there, and our own catch-all audit rule that logged every heartbeat. Each of them is now written down as a comment in the role, next to the line that would bring it back.