AlmaLinux 10 on Trial: My Nextcloud Playbook, Nine Stumbling Blocks, and an AI That Works While I Am Out
My Nextcloud servers will stay on AlmaLinux 9 for a long time. Still, I wanted to know what my playbook needs for AlmaLinux 10. I handed the work to Claude, was out for part of the day and followed the progress by mail.

🌐 Also available in: Deutsch · Français · Español · Italiano · Português · Nederlands · Polski
In short: My Nextcloud servers run on AlmaLinux 9, and that will stay so for quite a while. Still, I wanted to know what my Ansible playbook needs for AlmaLinux 10. So I had a test server freshly installed with AlmaLinux 10.3 and handed the work to Claude. For part of the day I was out and followed the progress through the mails of the playbook runs. The result: nine stumbling blocks, nearly all of them in the hardening, not in Nextcloud. The two most interesting ones: kernel modules that moved to another package, and SELinux paths where /run and /var/run swapped roles. The code is in the branch el10-support.
Table of Contents
🤔 Why now?
AlmaLinux 9 gets security updates until 2032, so there is no rush. That is exactly why now is a good moment: I can find out on a test machine what has to change, instead of doing it under pressure some day.
For context: Nextcloud, nginx and PHP run in a single-node K3s cluster, MariaDB and Redis directly on the host. On top of that comes a lot of hardening: SELinux in enforcing mode with my own policy modules for Prometheus, Grafana and the exporters, systemd sandboxing for the database, nftables, fail2ban, auditd, AIDE, rkhunter and CIS settings. That hardening is what creaks first when the major version changes.
🤖 I handed the work to Claude
This time I gave the work almost entirely to Claude, the AI by Anthropic, which works for me as Claude Code directly in the repository. My rules were short:
- Only the test server. The production servers are not touched, not even with
--check. - Nothing may change on AlmaLinux 9.
- Push only when everything is done, and nothing secret goes into the repository.
With that, Claude worked on its own: analysed the fresh server, ran the playbook, investigated every error, adapted the roles and started the next run. As a safety net, Claude created its own inventory that contains only the test server. Not even a forgotten -l test could have reached another host.
For part of the day I was out. I still knew what was going on: every playbook run mails its verify report at the end. On my phone I could watch the runs get further each time. When I was back home, the playbook ran through completely on AlmaLinux 10 and the branch was on GitHub.
It did not all go smoothly, by the way. Once, Claude briefly used git stash for a lint comparison while a playbook run was going on in the background. For about a minute the working tree was back at the old state. Claude noticed it, reported it openly, discarded the run as meaningless and repeated it. That is the kind of mistake that could just as well have happened to me.
🧭 The most important decision: no copy of the project
My first thought was to copy the project and turn the copy into an AlmaLinux 10 version. Claude advised against it, and I agreed. Two code bases with more than 30 roles mean every security fix has to be applied twice. I know that already from my blog repository, which I split off from the Nextcloud one. And the copy I might only need in a year would be a year of fixes behind by then.
Instead there is one new file, inventory/group_vars/all/os.yml. It holds everything that differs between 9 and 10, derived from the Ansible facts:
os_el_major: "{{ ansible_facts['distribution_major_version'] | int }}"
os_is_el10: "{{ os_el_major | int >= 10 }}"
nextcloud_kv_name: "{{ 'valkey' if os_is_el10 | bool else 'redis' }}"
nextcloud_mariadb_package: "{{ 'MariaDB-server' if nextcloud_mariadb_community_repo | bool else 'mariadb-server' }}"
os_selinux_run_dir: "{{ '/run' if os_is_el10 | bool else '/var/run' }}"
The roles only use these names now. The rule: On AlmaLinux 9 exactly the same must come out as before, byte for byte. Wherever a configuration file became a template, Claude diffed the EL9 result against the old file. My production servers do not notice any of this.
🧱 The stumbling blocks, in the order they appeared
1. No more DNF modules
AlmaLinux 10 dropped modularity. On 9 my playbook disabled the AppStream module mariadb and moved Redis to version 7 with dnf module switch-to. Neither exists on 10. These steps now run on 9 only.
2. Redis is now Valkey
AlmaLinux 10 no longer ships Redis but its fork Valkey 8.1. Same protocol, same port 6379, the Nextcloud pod notices nothing. Only the package, service and paths differ: valkey.service, /etc/valkey/valkey.conf, /var/lib/valkey.
The good news: the SELinux policy labels Valkey with the Redis types (redis_t, redis_conf_t, …). My small policy module that lets Redis into its domain despite NoNewPrivileges therefore works unchanged. Instead of a new role, the existing next_redis role got variables for names and paths.
3. MariaDB: better take what the distribution ships
On 9 I use MariaDB’s community repo with the 10.11 series. AlmaLinux 10 ships exactly that series itself. So on 10 I take the distribution package: one third-party repo less, and security updates arrive the normal dnf-automatic way.
The distribution package has a different systemd unit, though, and it clashed with my sandboxing. PrivateTmp=true in the unit would silently have overridden my 4 GB tmpfs for /tmp. And ProtectSystem=strict made /run/mariadb read-only, where the database keeps its PID file. A small drop-in for AlmaLinux 10 only fixes both. Checked afterwards with ps -eZ: mariadbd runs in mysqld_t, /tmp is the tmpfs.
4. k3s-selinux for AlmaLinux 10 exists only in the testing channel
Without the k3s-selinux package all pods run unconfined, even though SELinux says “Enforcing”. Rancher’s stable RPM repo does not know AlmaLinux 10 yet. Builds exist only in the testing channel, with its own signing key. On 10 that signed testing repo is used now, on 9 it stays the stable one. That is a deliberately open point: as soon as Rancher catches up, the switch is one line in os.yml.
5. journald.conf is gone
My CIS task changed three lines in /etc/systemd/journald.conf. On AlmaLinux 10 that file no longer exists, the defaults live only under /usr/lib/systemd/. The fix is a drop-in under /etc/systemd/journald.conf.d/, the cleaner way anyway.
6. The pods cannot reach the API server
This was the most interesting error. K3s started, but installing cert-manager ran into its timeout. CoreDNS never became ready, metrics-server kept restarting, and cert-manager reported:
dial tcp 10.43.0.1:443: i/o timeout
So the pods could not reach the API server. Claude found the reason in the K3s journal:
Warning: Extension comment revision 0 not supported, missing kernel module?
Warning: Extension conntrack revision 0 not supported, missing kernel module?
On AlmaLinux 10 the netfilter modules K3s needs (br_netfilter, nft_compat, xt_comment, xt_conntrack, xt_MASQUERADE and more) moved from the kernel-modules package to kernel-modules-extra. Without them K3s cannot create its rules.
A nice side finding: the first look with modinfo said “module not found”. That was my own SSH hardening, though: ProtectKernelModules=yes hides /usr/lib/modules in every SSH session. Only systemd-run outside the sandbox showed the truth. The common_k3s role now installs kernel-modules-extra on AlmaLinux 10, also through systemd-run.
7. auditd no longer loads rules
Destination directory /etc/audit/rules.d does not exist – with audit 4 on AlmaLinux 10, the rules directory, augenrules and the service that loads the rules at boot are in the separate package audit-rules. Without it auditd runs but loads not a single rule. An error that would never have shown up without the playbook stopping.
8. SELinux: /run and /var/run swapped roles
Grafana runs in its own SELinux domain on my servers. On AlmaLinux 10 it no longer started:
failed to write pidfile: open /var/run/grafana/grafana-server.pid: permission denied
/run/grafana had the wrong label. The reason is in file_contexts.subs_dist, the file that defines which paths SELinux treats as equivalent:
| System | file_contexts.subs_dist |
|---|---|
| AlmaLinux 9 | /run /var/run |
| AlmaLinux 10 | /var/run /run |
On 9, /run/… is looked up as /var/run/…. An entry in my policy therefore has to say /var/run/grafana there. I had worked that out the hard way a month ago. On 10 it is exactly the other way round. The same file that is correct on 9 has no effect on 10. The fix: the paths now come from os_selinux_run_dir.
This also uncovered an old hidden bug. The loop that sets the labels ended with [ -e "$p" ] && restorecon …. If the last path of the list is missing, the loop ends with status 1 and the task fails. On 9 the last path always existed through a symlink, on 10 it disappears briefly while Grafana restarts. [ ! -e "$p" ] || restorecon … fixes it.
9. Small stuff and a hanging AIDE
- rkhunter: the path of the dnf plugin was tied to Python 3.9, AlmaLinux 10 has Python 3.12.
- ClamAV: the freshclam package is called
clamav-freshclamin EPEL 10 instead ofclamav-update. - dnf-automatic: the list of vendor repos may only contain repos the host actually has, otherwise dnf aborts.
- AIDE: once the integrity check hung for two and a half hours. One thread spun endlessly inside the regex library, the others waited. Eight repeats afterwards each finished in under 35 seconds. A rare race, then. On AlmaLinux 10, AIDE now gets a time limit of 30 minutes, so a hang can never again block the playbook or the nightly job.
✅ What ran without changes
To be fair: most of the more than 30 roles ran right away. nftables, fail2ban, the SSH hardening including the port change and reboot, Prometheus, Grafana, Loki and the whole Nextcloud deployment in K3s – without a single change. EPEL 10 has every package I need.
Finally, Claude checked the server from the outside, carefully, because my firewall bans port scanners for an hour. Instead of a scan there were 15 single connection attempts with pauses, far below the threshold. Only 80, 443 and the SSH port are open, everything else is closed. testssl.sh found not a single vulnerability, and the server already offers the post-quantum key exchange X25519MLKEM768. My Zabbix now also watches the test server from the outside.
➕ Addendum: /var/tmp mounted 129 times
The final drift check still reported one change, every time in the same place: the hardened mount of /var/tmp. Claude found the reason: /var/tmp was mounted 129 times on top of itself. The Ansible module mounts inside the SSH session, and thanks to my SSH hardening that session lives in its own mount namespace. When systemd later created the same mount on the host, it landed on top. From then on the module read the source as /var/tmp/var/tmp, thought nothing was mounted and mounted again on every run, doubling the stack each time. Worse: on the host itself /tmp was not hardened at all. The hardening only applied inside SSH sessions. Now Ansible only writes fstab, and the mount happens on the host through systemd-run. After a reboot there is one layer, and the drift check shows changed=0. My production servers are not affected, because they have long since rebooted with the fstab entry in place. A fresh install would have ended up exactly like this, though.
➕ Addendum 2: which versions Nextcloud wants, MariaDB 11.8, CIS and a real restore
In the evening a few more questions came up, and Claude worked through them on the test server. First: which versions does Nextcloud 35 actually require, and what does AlmaLinux ship? The AlmaLinux 9 numbers come straight from AlmaLinux’s repository metadata:
| AlmaLinux 9 default | AlmaLinux 9 via module | AlmaLinux 10 | Nextcloud 35 supports | Nextcloud recommends | |
|---|---|---|---|---|---|
| Operating system | – | – | – | RHEL 9 and 10 | RHEL 10 |
| MariaDB | 10.5 ❌ | 10.11, 11.8 | 10.11, 11.8 | 10.11 / 11.4 / 11.8 / 12.3 | 11.8 |
| PHP | 8.0 ❌ | 8.1 to 8.4 | 8.3, 8.4 | 8.3 / 8.4 / 8.5 | 8.5 |
| Redis/Valkey | Redis 6.2 | Redis 7.2 | Valkey 8.1 | not specified | – |
AlmaLinux 9’s default versions are simply too old for Nextcloud 35. That is exactly why PHP runs in a container on my servers (already 8.5.13 there) and MariaDB on the production servers comes from the community repo. On AlmaLinux 10 the playbook now takes mariadb11.8-server, the version Nextcloud recommends, straight from the distribution.
The switch from 10.11 to 11.8 on the test server ran like a real migration: dump, maintenance mode, package swap, mariadb-upgrade. There were two surprises:
- MariaDB 11.8 was killed by my sandboxing (
status=31/SYS). The audit log saidSECCOMP syscall=92, which ischown. Version 11.8 callschown, 10.11 did not, and my systemd hardening forbids it. The fix isSystemCallFilter=@chownin a drop-in that comes after the hardening, because forSystemCallFilterthe later line wins. It is harmless: mariadbd has no capabilities at all and can only change the owner of its own files. - After the package swap,
mariadb.servicewas disabled. The playbook enabled it again. Without that, the database simply would not have come back at the next reboot.
CIS benchmark for AlmaLinux 10: With OpenSCAP and the “CIS AlmaLinux OS 10 Level 1 – Server” profile, the first scan showed 236 passed and 62 failed checks. Real gaps included unused network protocols (SCTP, DCCP, TIPC …), the umask, a session timeout, the permissions of the GRUB environment and a crypto policy without weak SSH MACs. Since then SSH no longer offers SHA1 MACs. After the fixes, which apply to AlmaLinux 9 and 10, the result is 254 passed and 44 failed. The rest is deliberate or has no effect: K3s needs IP forwarding, I use nftables instead of firewalld, root login is key-only for Ansible, and the many password rules do not matter because there are no password logins. I deliberately left out a GRUB password, because on a VPS it would lock out the provider’s console.
Backup and restore, end to end: Claude ran my real scripts against the test server. A file was created before the backup and another one after it. The backup took 45 seconds, the restore 74 seconds. Afterwards the first file was back and the second one gone, in the file system and in the database. Along the way:
- MariaDB 11.8 warns that the names
mysqldumpandmysqlwill be removed. The scripts now usemariadb-dumpandmariadb, falling back to the old names. - The restore script ended without a word when the confirmation came through a pipe. An
sshin the script had swallowed the input. A</dev/nullfixes it.
And one finding that affects AlmaLinux 9 as well: restorecon -F restarted node_exporter on every playbook run, just because files written from the SSH session carried the SELinux user unconfined_u. Only the type matters for access, so -F is gone.
After all that, everything on the test server was green: a complete run without errors, verify report 37/37, drift check changed=0.
What is different on AlmaLinux 10, at a glance: nothing is missing. SELinux protects the pods just the same, and all my own domains, the sandboxing, the firewall and the monitoring work the same way. What is not final yet is k3s-selinux, which only comes from Rancher’s testing channel. As long as that is the case, my production servers stay on AlmaLinux 9. In the long run I will watch the iptables compatibility, which the kernel lists as “unmaintained”.
🔭 What is still open
- k3s-selinux from the testing channel, until Rancher offers AlmaLinux 10 as stable.
- The first night: only at night do all timers run (updates, AIDE, rkhunter, ClamAV, Trivy). That is when SELinux messages show up that a playbook run never triggers. If anything comes up, I will add it here.
- The actual move of the production servers: some day, by reinstalling and restoring, not by an in-place upgrade.
- iptables compatibility is marked “unmaintained”. At boot, the AlmaLinux 10 kernel reports
Unmaintained driver is detected: nft_compat(and the same forip_tablesandip_set). These are exactly the modules K3s needs today: kube-proxy, flannel and the NetworkPolicy controller write their rules through iptables-nft, and on the test server 458 rules depend onnft_compat. It works, but it is no longer maintained. Since Kubernetes 1.33 kube-proxy can use nftables natively, the rest of the K3s network cannot yet. That is my next experiment on the test server. My own firewall is pure nftables and not affected.
🎯 Conclusion: the hardening is the real test
- A major version change hits the hardening first. Nextcloud itself ran at once. The playbook stumbled over SELinux, systemd sandboxing, auditd and kernel modules.
- One code base instead of a copy. All differences are in one place, AlmaLinux 9 stays unchanged, and every future security fix applies to both versions.
- If you maintain your own SELinux modules with
.fcfiles, remember point 8. The swapped/runequivalence only shows when a service fails because of it. - Delegating works when the guard rails are clear. “Only this server, nothing changes on 9, push only at the end” – Claude did not need more. I followed the rest on my phone.
→ Code: branch el10-support on GitHub · all changes compared
📚 Related posts:
