Deep Dive: SELinux on a K3s Host – Type Enforcement, and the Two Times It Quietly Did Nothing

🌐 Also available in: Deutsch · Français · Español

Deep Dive, part of the series around the Ansible Tour. Level: advanced Linux administration. The tour explains the roles; the deep dives explain the technology underneath them – how it works, why it is here, what exactly this setup does with it, how to run it day to day, and what else there would have been.

SELinux is the layer most administrators switch off first and understand last. That is a pity, because it is the only layer on these servers that still holds when a process has already been taken over – even one running as root. This post first explains how SELinux decides, with the details that matter in practice: types, transitions, categories, and how a denial is read. Then it looks at the blog and the two Nextcloud servers: what is confined, how, and the two occasions on which SELinux was switched on, enforcing, and still protected nothing – because a single detail elsewhere made sure the right rules never applied.

🧭 How SELinux decides

Discretionary versus mandatory

Classic Unix permissions are discretionary: the owner of a file decides who may read it, and root may do everything. SELinux adds a mandatory layer on top. The kernel asks a central policy on every relevant operation – open a file, connect a socket, send a signal, execute a program – and the answer does not depend on who owns the file or whether the process is root. Both checks have to say yes. A web server running as root still cannot read /etc/shadow if the policy does not allow its type to read the type of that file.

Labels: four fields, one of which does almost all the work

Every process and every object – file, directory, socket, port, even the kernel’s own structures – carries a security context of the form user:role:type:level. On a server with the targeted policy that AlmaLinux ships, three of the four fields barely matter: the user is almost always system_u or unconfined_u, the role system_r or object_r. The decisions are made by the type. For a process the type is called its domain.

$ ps -eZ | grep -E ' (mariadbd|redis-server|php-fpm)$'
system_u:system_r:mysqld_t:s0              ... mariadbd
system_u:system_r:redis_t:s0               ... redis-server
system_u:system_r:container_t:s0:c628,c864 ... php-fpm

$ ls -Zd /var/lib/mysql /var/lib/redis
unconfined_u:object_r:mysqld_db_t:s0       /var/lib/mysql
system_u:object_r:redis_var_lib_t:s0       /var/lib/redis

Labels on files are stored as extended attributes (security.selinux) in the file system. Where a file’s label comes from is the source of most SELinux trouble, so it is worth knowing the three ways: a new file inherits the type of its parent directory; a type transition rule in the policy can say “files that domain X creates in a directory of type Y get type Z”; and restorecon sets labels from the file context database – rules like /var/lib/mysql(/.*)? → mysqld_db_t, which the policy ships and which semanage fcontext extends. Moving a file keeps its label; copying gives it a new one. A config file moved from a home directory into /etc and then unreadable for its service is the classic.

Type enforcement

The policy is essentially a large set of allow rules between types: allow mysqld_t mysqld_db_t:file { read write open … }; – a process in domain mysqld_t may read and write files of type mysqld_db_t. Whatever is not allowed is denied. The targeted policy on these hosts contains well over a hundred thousand such rules, compiled into a binary loaded into the kernel; 439 modules make up the whole, from apache to zabbix. The point of the name “targeted”: only services with their own module are confined. Everything else runs in an unconfined domain that is allowed almost everything – which, as the second case below shows, is exactly where things can slip through unnoticed.

Domain transitions – and why NoNewPrivileges can stop them

A process gets its domain at execve(). The rule type_transition init_t mysqld_exec_t:process mysqld_t says: when something in init_t (systemd) executes a file of type mysqld_exec_t, the new process runs in mysqld_t. That is why the label of an executable matters as much as the label of the data: a mariadbd binary labelled bin_t would never enter its domain.

There is a subtle interaction with a systemd option that is otherwise a pure hardening win. NoNewPrivileges=yes sets the kernel flag PR_SET_NO_NEW_PRIVS: after that, execve() can never gain privileges – no setuid, no file capabilities. SELinux treats a domain transition as a potential privilege change too. Under NNP it only allows a transition if the target domain is a subset of the source (bounded) or if the policy explicitly grants nnp_transition for that pair. If neither applies, the transition is refused – and the program keeps running in the domain of its parent. Several other systemd sandbox options imply NNP without saying so. Remember this; it is the first case below.

MCS: the level field that keeps containers apart

The fourth field, the level, normally reads s0. With Multi-Category Security it can carry categories: s0:c628,c864. A process may only access an object if its categories dominate the object’s: a process with c628,c864 can read a file with exactly those categories, or a file with none (s0), but not a file labelled c53,c213. Container runtimes use this to separate containers that all share one type: every pod gets its own random pair. All 46 container processes on a Nextcloud host run as container_t, and still Collabora (c53,c213) cannot touch the files of the PHP container (c628,c864).

One consequence is easy to overlook: a host directory mounted into pods and labelled container_file_t:s0 – without categories – is reachable by every container. MCS isolates the containers’ own file systems, not shared volumes. On these servers each HostPath directory belongs to exactly one application, so that is fine; on a multi-tenant node it would not be.

Reading a denial

When the policy says no, the kernel writes an AVC message (access vector cache) to the audit log. This one came from the backup check of the file integrity tool, and it contains everything needed to understand it:

type=AVC msg=audit(...): avc:  denied  { getattr } for  pid=... comm="aide"
  name="/" dev="vda2" scontext=system_u:system_r:aide_t:s0
  tcontext=system_u:object_r:dosfs_t:s0 tclass=filesystem permissive=0

Read it as a sentence: the process aide, running in domain aide_t, wanted getattr on an object of class filesystem with type dosfs_t – the vfat EFI partition – and was refused (permissive=0). ausearch -m AVC -ts today finds such messages, audit2why explains which rule is missing or which boolean would allow it. audit2allow -M would write a module that allows exactly what was denied – and that is the most dangerous command in SELinux administration, because it turns every symptom into a permission without asking whether the access was legitimate.

🛡️ Why it is on, everywhere

All three servers run the targeted policy in enforcing mode, and every role assumes it. The reason is a threat model rather than a checkbox. The realistic attack on a blog is a vulnerable WordPress plugin; on Nextcloud, a malicious document that exploits the office suite. In both cases the attacker ends up with code execution inside a PHP or Collabora process. Everything after that – reading the database files directly, stealing SSH keys, planting a binary on the host, attacking the other application – is a matter of what that process may do. Unix permissions do not help much there; inside a container the process is often root anyway. SELinux is the layer that answers “no” independently of user IDs.

• A PHP-FPM worker in container_t can write only container_file_t – its own volume. The policy contains no rule that lets container_t read shadow_t, so /etc/shadow stays out of reach even for a root process that has broken out of its mount namespace.
• The host MariaDB in mysqld_t may read and write its own mysqld_db_t, bind its port and little else. A compromised database server cannot read the Nextcloud user data in /data, which is container_file_t.
• Redis in redis_t likewise: its own redis_var_lib_t, its port, its socket.
• Kernel modules, raw devices, ptrace on other domains: denied to all of these by default.

The cost is small once the labels are right. Almost every SELinux problem on these hosts so far has been a missing or wrong label, not a missing rule – and the policy for databases, Redis and containers comes ready-made with the distribution and container-selinux.

🔧 What this repository does

common_selinux: enforcing and the right labels

The role sets the mode to enforcing with the targeted policy and labels the host directories that the pods mount. Which directories depends on the stack: the playbook sets the variable stack, and a dictionary picks the profile – web root and database directory for the blog, web root and user data for Nextcloud. For each directory it adds a persistent file context rule and applies it:

semanage fcontext -a -t container_file_t "/srv/wordpress-www(/.*)?"
restorecon -Rv /srv/wordpress-www

The rule lives in the policy store, so it survives a full relabel; restorecon applies it to what is already there. The role reads the local rules first (semanage fcontext -l -C) and only adds what is missing – newer semanage versions no longer fail on an existing rule but print a message to stdout and exit 0, which once made the task report “changed” on every run.

Case 1: a database that was enforcing – and unconfined

In September a systemd drop-in hardened the host MariaDB on the Nextcloud servers: no capabilities, read-only file system except its data directory, private /tmp, a system call filter – and, explicitly and implicitly, NoNewPrivileges. MariaDB started, Nextcloud worked, the drop-in looked like a clear improvement. Five weeks later a look at the running processes showed this:

$ ps -eZ | grep mariadbd
system_u:system_r:initrc_t:s0   ... mariadbd

initrc_t instead of mysqld_t. The distribution’s unit does not start mariadbd directly but through /bin/sh -c, to run a Galera recovery helper first. The shell runs in initrc_t; when it executes mariadbd, the transition into mysqld_t has to happen – and under NNP the policy does not allow it from initrc_t. The audit log said so, every start, in a line nobody had read: avc: denied { nnp_transition } … scontext=…initrc_t… tcontext=…mysqld_t… tclass=process2. The database kept running in the shell’s domain, which is far less restricted. From 11 September until 16 October the host MariaDB was effectively not confined by SELinux at all.

The fix is one line in the drop-in: start the binary directly, without the shell (Galera is not used here, so the helper has nothing to do):

ExecStart=
ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER

Executed directly by systemd, the transition from init_t is allowed, and ps -eZ shows mysqld_t again. It was tested first with systemd-run and the same properties, so the change never had to be tried on the production database.

Case 2: pods that never ran as container_t

The second case is bigger. SELinux was enforcing on all three servers, the host directories carried container_file_t, container-selinux was installed – and on 19 October, while this series was being written, ps -eZ showed that not a single pod ran as container_t. On the blog they were spc_t (“super privileged container”, practically unconfined), on the Nextcloud servers unconfined_service_t.

Two things were missing. K3s only starts its embedded containerd with SELinux support when its configuration says selinux: true; otherwise containerd’s enable_selinux is false and it never asks the policy for a container label. And K3s needs its own policy package, k3s-selinux, which labels the binary container_runtime_exec_t so that it runs as container_runtime_t. On the blog the install script had added that package once; on the Nextcloud servers the binary was plain bin_t, so systemd started it as unconfined_service_t, and every child inherited that. All the careful labelling of the volumes had been preparation for a check that never took place.

common_k3s now does four things:

• adds Rancher’s signed package repository and installs k3s-selinux,
• sets selinux: true in /etc/rancher/k3s/config.yaml,
• runs restorecon -v /usr/local/bin/k3s after every binary download – get_url replaces the file through a temporary file in the same directory, which gives the new binary the directory’s bin_t and would quietly undo the policy on the next upgrade,
• and recreates all workloads once, when not a single pause container runs as container_t yet. Pods keep the domain they were started with; a configuration change alone confines nothing that is already running.

Afterwards the picture on a Nextcloud server: 46 processes in container_t, each pod with its own category pair, only privileged pods in spc_t, and not one AVC – neither from Nextcloud, nor from Collabora with its mount namespaces, nor from notify_push. Collabora was the component with the most potential for trouble, because it builds a jail for every document with bind mounts; container_t allows the mount namespace operations it needs.

Case 3, small: one rule for AIDE

The file integrity checker AIDE runs as aide_t and walks the whole file system. The EFI partition is vfat (dosfs_t), and the policy does not let aide_t look at it. Excluding /boot/efi from the check stopped AIDE from reading the files, but not from calling statfs() on the mount point – 13 denials per run, every night. The cleanest fix is a local module with exactly one rule, built on the host and loaded only when its source changes:

module aide_dosfs 1.0;
require { type aide_t; type dosfs_t; class filesystem getattr; }
allow aide_t dosfs_t:filesystem getattr;

checkmodule -M -m -o aide_dosfs.mod aide_dosfs.te
semodule_package -o aide_dosfs.pp -m aide_dosfs.mod
semodule -i aide_dosfs.pp

This is the counter-example to audit2allow: the rule was written by hand, after deciding that reading file system metadata is legitimate and nothing more is needed. Since then the three servers log no AVC at all – which is also the precondition for noticing the next real one.

The monitoring stack: own domains, still learning

Until 19 October, eighteen processes on a Nextcloud server still ran as unconfined_service_t: Prometheus, the exporters, Grafana and its plugins, Monit. They are installed as single binaries in /usr/local/bin (bin_t) or from a vendor repository, and the targeted policy has no module for them. Since that day they have domains of their own – small local modules homelab_prometheus, homelab_node_exporter, homelab_mysqld_exporter and homelab_grafana, built with the policy’s standard macros from selinux-policy-devel, with types for their data and logs. One detail almost repeated case 1: the units run with NoNewPrivileges, so each module also needs init_nnp_daemon_domain(); without it systemd’s transition was refused, Grafana stayed in init_t, could not read its own database and was down for five minutes on the blog. The new domains start in permissive mode: they are real domains, but denials are only logged (permissive=1). After some days of normal operation the logged accesses become reviewed allow rules, written by hand, and the domains switch to enforcing. Monit deliberately gets no domain – it runs arbitrary checks and restarts services as root – but a systemd sandbox instead, which brought its exposure score from 9.6 to 7.1.

🧰 Day-to-day administration

A few habits that have proven themselves:

• After any change to a service unit, check its domain, not only its status. A service that is “active (running)” in the wrong domain looks perfectly healthy.
• Treat every AVC as a finding. On a quiet system it is rare enough to read each one; the moment a background of known denials is accepted, the important one disappears in it.
• New host directory for a pod? Add it to the stack’s profile in common_selinux, never label it by hand. The blog’s backup directory for the update job was missing from the profile until 19 October; it worked only because the pods were not confined yet.
• Permissive mode for a single domain (semanage permissive -a <domain>) is a debugging tool for minutes, not a fix. setenforce 0 on a production host is not an option at all.
• After a policy update, semodule -l still lists the local modules; if a module stops compiling against a new policy, semodule -i fails loudly in the playbook run, not silently at runtime.

🔭 What comes next

• Alerts on new AVCs through Loki, so the next silent denial – like the nnp_transition that was logged at every MariaDB start for five weeks – produces a mail on the first occurrence.
• Switching the monitoring domains from permissive to enforcing once the logged accesses are reviewed – planned for the end of the first week.
• Per-workload policies with udica, which generates a policy from a container’s mounts and ports – tighter than the generic container_t.
• A check in the playbook’s final verification that every application pod runs as container_t. Case 2 lasted months because nothing ever asked.

⚖️ Alternatives – and why SELinux

SELinux fits because it is already there: AlmaLinux ships a mature targeted policy, container-selinux integrates with containerd, K3s brings its own policy package, and administrators of RHEL-family systems are expected to know it. Its real cost is not complexity but silence – as the two cases show, it can be on and do nothing without anyone noticing. The answer to that is not to switch it off, but to check what it is actually doing.