📦 What was supposed to be a quick “let me set up a NAS this weekend” turned into a week of rabbit holes, yak shaving and an unreasonable number of git commits. But hey — the homelab is better for it, and everything is now automated and documented. As usual, the full Ansible codebase lives here:
👉 github.com/aptupgrademe/www_k3s
Before I get to the NAS and OpenZFS — which is the actual main event — let me quickly dump the side-quests that happened along the way. Skip to the ZFS section if you’re impatient. I won’t judge.
🐿️ Side Quests: Everything That Wasn’t the NAS
You know how it goes. You start configuring a NAS and three days later you’re fixing Grafana alert rules and writing a French operations guide. Here’s what happened:
🔒 Security fixes (the “wait, that was open?!” edition)
- rpcbind was listening on port 111 — publicly. NFS is not used anywhere. Fixed. 😬
- rkhunter was sending “infected!” emails every morning — because SourceForge switched to HTTPS redirects and
curlwasn’t following them. Addedcurl -L. Also automatedrkhunter --propupdafterdnf-automaticso nightly security updates don’t look like rootkits. - Grafana alert rules had thresholds baked into the PromQL, causing “condition must not be empty” errors when metrics were below threshold. Moved thresholds into the evaluator. Now actually works.
- systemd hardening drop-ins deployed:
sshdwent from exposure score 9.6 → 7.8,node_exporterandprometheusfrom 9.2 → 4.4 each. - Unnecessary services axed:
rpcbind,gssproxy,sssd— none of them needed, all of them running.
⚡ Performance
- Autoptimize installed on WordPress: JS/CSS tag count dropped from 40 to 6. Page size down ~18%.
- robots.txt was returning 404. Yoast SEO generates it dynamically via PHP, but nginx was treating it as a static file. One
try_filesdirective later — fixed. tcp_max_syn_backlograised from 512 to 4096 to matchsomaxconn.- Transparent Huge Pages switched from
alwaystomadvise— the default causes latency spikes in K3s pods, MariaDB and Redis.
🎨 Design & SEO
- The blog header got a redesign: Superman logo now lives on the right (mirrored, so he faces inward instead of flying off the page), site title on the left, dark background matching the nav bars. Small things, big difference.
- 165 images had no alt text. Fixed with a WP-CLI script that derives descriptions from filenames automatically. That took longer than the NAS setup.
- robots.txt now properly exposes the Yoast-generated sitemap to search engines.
- Added “Home” as the first navigation menu entry. Turns out it wasn’t obvious how to get back to the front page. Oops.
📚 Documentation
Because apparently I enjoy writing docs more than is healthy: the operations guides are now available in German, English and French for WordPress, Nextcloud and the new NAS. Yes, French. The docs/ folder in the repo has all six HTML files.
🗄️ The Main Event: OpenZFS NAS on Debian 13
Right. The thing I actually set out to do.
Why ZFS? A quick history
ZFS was originally developed by Sun Microsystems and first shipped with Solaris 10 in 2005. The name stood for “Zettabyte File System” — Sun wasn’t being modest. It was designed from scratch to solve problems that traditional UNIX filesystems had been papering over for decades: silent corruption, inconsistent state after crashes, the general horror of fsck on large volumes.
When Oracle acquired Sun in 2010, ZFS development on the open-source side forked into OpenZFS — today the actively maintained community project that runs on Linux, FreeBSD, macOS and others. That’s what we’re using.
One thing I assumed for a long time: ZFS is a FreeBSD thing. Turns out that’s not quite right. OpenZFS ships in Debian 13’s repos and installs cleanly. I’m running it for the first time, so I have no real-world numbers yet. My guess is that raw throughput might be slightly lower than a tuned ext4/mdadm setup — but the trade-off of checksumming, silent corruption detection and snapshots makes the data significantly safer. We’ll see how it holds up. 🧪
✅ What makes ZFS great
- End-to-end checksums — every block gets a checksum on write, verified on read. Silent corruption (“bit rot”) is detected and, in a redundant pool, automatically repaired.
- Copy-on-Write (CoW) — data is never overwritten in place. This makes snapshots essentially free and eliminates the RAID-5 write-hole problem.
- Snapshots — point-in-time copies of a dataset, created instantly and stored space-efficiently. Roll back in seconds.
- Built-in RAID (RAIDZ) — no separate RAID controller, no silent metadata corruption. RAIDZ1/2/3 are ZFS-native.
- Compression — transparent, per-dataset.
zstdgives 30–50% space savings on documents and backups at negligible CPU cost. - ARC cache — ZFS manages its own adaptive read cache in RAM. More RAM = faster reads.
- Pooled storage — all datasets share the pool’s space dynamically. No more carving out fixed partitions.
⚠️ The trade-offs
- RAM hungry — the ARC is greedy. A real NAS wants 16–32 GB.
- No easy RAIDZ resizing — you can’t just add a disk to an existing RAIDZ vdev. ZFS 2.2+ added expansion, but it’s still not instant.
- ECC RAM strongly recommended — ZFS trusts RAM. If RAM silently corrupts a bit before writing, ZFS checksums the corruption as valid. ECC prevents this.
- Learning curve — pools, vdevs, datasets, properties. Different mental model from traditional filesystems. Worth it, but there’s a weekend in it.
🏗️ The Actual Setup
Previously, the same three SSDs ran a classic Linux software RAID5 (md) with ext4 on top. It worked, but offered none of ZFS’s integrity guarantees, snapshots or compression. Time to upgrade.
The hardware is an HP MicroServer Gen10 Plus V2 — Intel Xeon E-2314, 31 GB RAM. Storage layout:
sdb(465 GB SSD) → Debian 13 OSsda+sdc+sdd(3× 1.8 TB WD Red SA500 SSD) → ZFS RAIDZ1sde(1.8 TB Toshiba HDD) →/mnt/usbexternal backup
The Ansible role creates the pool like this:
zpool create -f
-o ashift=12 # 4K sector alignment for modern SSDs
-o autotrim=on # TRIM commands for SSD health
-O compression=zstd # transparent compression
-O atime=off # no access-time writes on reads
-O xattr=sa # Samba xattrs in inodes (faster)
-O recordsize=128k # overridden to 1M per dataset
-O mountpoint=/raid5
data raidz /dev/disk/by-id/ata-WD_Red_SA500_...
📁 Samba — Single Source of Truth
The key design choice: Samba share definitions in vars.yml drive everything. The Ansible role loops over them to create ZFS datasets, set ownership and write smb.conf. Add a share in one place — the rest follows automatically:
nas_samba_shares:
- name: shared
path: "{{ nas_zfs_mountpoint }}/shared"
valid_users: raphael
force_user: raphael
- name: angelika
path: "{{ nas_zfs_mountpoint }}/angelika"
valid_users: raphael
force_user: raphael
Samba runs SMB3 with multi-channel support and async I/O. The xattr=sa ZFS property stores Windows extended attributes directly in the inode rather than hidden dot-files, which makes metadata noticeably faster. 🎛️
🤖 Jenkins, GitLab & Pi-hole
The MicroServer does a bit more than just serve files:
Jenkins (port 8090) runs two nightly backup jobs — rsync-based backups of two remote Nextcloud instances (CVJM and Sofie). The SSH key used for these connections is stored encrypted in Ansible Vault and restored to /home/raphael/.ssh/ on every fresh install. A full OS reinstall doesn’t break the backup connectivity.
GitLab CE 19.1 (port 80) serves as a local Git server. Having an on-prem GitLab for homelab projects means no dependency on external services. The Ansible role handles the post-install setup entirely via the GitLab API: sets the root password, creates a standard user, creates a homelab group and an empty www_k3s project — all idempotent.
Pi-hole v6 (port 8000 for the web UI, port 53 for DNS) blocks ads at the network level. It’s on port 8000 instead of the default 80 because GitLab already lives there. DNS upstream is 8.8.8.8/8.8.4.4.
And because it’s an HP server: HPE AMSD (Agentless Management Service) is installed first — before everything else. Without it, the iLO BMC doesn’t get temperature data from the OS and the fans run at full speed. Loud. Very loud.
All of this — NAS and internet servers alike — is one ansible-playbook command away from a fresh rebuild:
👉 github.com/aptupgrademe/www_k3s
Next up: actually reinstalling hp1 and running the playbook for the first time on a clean Debian 13. I’ll report back on whether ZFS on Linux lives up to the hype. 🤞
About the Author
