Building a RAIDZ1 NAS on Debian 13 with OpenZFS: Datasets, Compression, and the Lies dd Told Me

🪟➡️🐧 Context: retiring the Windows box

Some backstory, since “why does this random Debian box suddenly have three drives and strong opinions about compression” deserves an answer. Until recently, a Windows Server handled both Active Directory and file-server/NAS duties around here — and credit where due, it did the job perfectly well. Logins worked, shares worked, nobody complained, nothing was on fire. But every time I context-switched from that box into anything DevSecOps-shaped, I felt the friction: different mental model, different tooling, different muscle memory for “okay, how do I actually check what’s going on here.” The Windows server existed purely for NAS + AD login duty, while a separate box was still quietly doing all the actual Linux work — jump host duties, the odd CI-adjacent task, and lately also the thing generating this very sentence. Two servers, two operating philosophies, and a slowly growing sense that I was curating a small museum of “the way this used to be done” rather than running a coherent setup. 🏛️

Sure, I could’ve reached for Proxmox and virtualized my way around the decision instead of actually making it — spin up a Windows guest, keep AD around out of habit, call it consolidation. But I’ve reached the point where I’d genuinely rather put everything on Linux, on one box, and lean into that: one toolchain, one thing to patch, one thing to monitor, instead of maintaining two entirely different worlds for the sake of a file share and a login prompt. And chasing that consolidation is exactly how I ended up back at OpenZFS — if I’m rebuilding the NAS role from scratch on Linux anyway, I want the storage layer underneath it to actually be good, not just “good enough that Explorer stops complaining.”

So: I reimaged a spare box as a fresh Debian 13 (trixie) install and decided it was time to stop treating storage as an afterthought — you know, the classic “I’ll organize my files properly someday” lie we all tell ourselves. Three 2TB WD Red SA500 SSDs went in, and the goal was simple on paper: a redundant pool, compression that pays for itself, and per-user shares that don’t step on each other’s toes. OpenZFS made this genuinely pleasant — once I got past a packaging quirk, a cosmetic upstream bug that gaslit me for ten minutes, and my own deeply embarrassing first benchmark. Here’s the full walkthrough, warts included. 🐛

🤔 Why ZFS and not NTFS or mdadm + ext4

Short version: I wanted checksummed data, transparent compression, cheap snapshots, and a storage layer that understands “dataset” as a first-class concept rather than “directory I promise to treat specially, pinky swear.” Coming off years of NTFS on that Windows Server, and having lived with ext4 everywhere else, it’s worth actually spelling out what ZFS does differently instead of just asserting it’s better — because on paper NTFS and ext4 both sound fine. They mount, they hold files, they’ve been production-grade for decades. The differences only show up when something goes quietly wrong, which is exactly the scenario neither of them is built to catch.

Checksums — the actual headline feature. ZFS checksums every block of data and every block of metadata, and verifies that checksum on every single read, not just during an occasional scheduled scan. NTFS has no data checksumming at all — its integrity streams feature only ever covers metadata, is opt-in, and was quietly dropped from ReFS in later Windows Server releases for anything but metadata. ext4 is in the same boat: metadata_csum protects the filesystem’s own bookkeeping, but your actual file contents are trusted blindly. If a bit flips somewhere on the physical media — and on any drive large enough and old enough, some bits eventually will — NTFS and ext4 will hand you the corrupted byte with a smile and zero indication anything happened. This is silent bit-rot, and “silent” is the operative word: no error, no log entry, no alert, just a slightly wrong pixel in a photo or a slightly wrong byte in a backup archive that nobody notices until it matters.

Self-healing, not just detection. This is where ZFS pulls ahead of even checksummed alternatives: because it manages the redundancy (RAIDZ, mirrors) and the checksums in the same layer, it can tell not just “this block is wrong” but reconstruct the correct value from parity and rewrite it automatically — during normal reads, and comprehensively during a zpool scrub. Compare that to the classic mdadm + ext4 stack: the RAID layer and the filesystem layer don’t talk to each other. mdadm can tell you a whole disk died. It cannot tell you that one specific block quietly went bad while every disk reports “I’m fine” — there’s no checksum being compared, so there’s nothing to disagree about. The corruption just silently propagates through every parity calculation as if it were correct data, because as far as RAID is concerned, it is.

No “initial resync” at creation, because parity isn’t a separate concept. Anyone who’s built an mdadm RAID5/6 array knows the ritual: create it, then wait — sometimes for hours — while it grinds through an initial resync of the entire array, writing consistent parity across every single block regardless of whether that block holds real data or is still completely empty. It has to: mdadm operates purely at the block-device level, underneath the filesystem, with zero visibility into which blocks are “real” and which are unused space. It can’t skip anything, so it doesn’t. zpool create on this same set of three disks, by contrast, took a few seconds. No resync, no wait, nothing to catch up on — because in ZFS, parity is computed and committed as part of the same atomic write that lays down the data itself, and unallocated space simply has no parity to be inconsistent about. There’s no “initial state” to correct because there’s never a moment where data and parity disagree in the first place.

The closest ZFS equivalent to a RAID rebuild is a resilver — triggered when a failed disk gets replaced or a new one is attached to a mirror — and even that inherits the same filesystem-awareness advantage: ZFS resilvers only the blocks that are actually allocated, not the drive’s full raw capacity. A pool that’s 40% full only has to resilver 40% worth of data; a traditional mdadm rebuild would grind through 100% of the array regardless, because it doesn’t know the difference. The proactive equivalent of an mdadm check/repair pass is zpool scrub — same idea, same efficiency win: it walks and verifies (and, unlike a dumb RAID check, actually corrects) every allocated block against its checksum, and skips everything that was never written in the first place.

Copy-on-write vs. journaling. ext4 and NTFS are both journaling filesystems: they log metadata changes before committing them, so a crash mid-write leaves you with consistent metadata, but an in-progress data write can still leave you with a half-written file. ZFS never overwrites live data in place — every write goes to a new block, and the filesystem atomically switches over once it’s fully committed. There is no window where a crash can leave a file half-old, half-new; you either see the last complete state or the new complete state, never a smear of both.

Snapshots that are actually cheap. NTFS has Volume Shadow Copy, which works but is tied to a separate VSS service, has practical limits on how many you’ll realistically keep, and can get noticeably slow as changes accumulate. ext4 has nothing native at all — you’re reaching for LVM snapshots underneath it, which are their own layer with their own performance cost as they diverge from the origin. ZFS snapshots are near-instant (copy-on-write again — a snapshot just means “stop reclaiming these blocks yet”), cost nothing until data actually changes, and you can reasonably keep dozens sitting around without a second thought.

Compression as a default, not a compromise. NTFS compression exists but is old, single-threaded, and slow enough that turning it on for anything performance-sensitive is a mistake most admins learn once. ext4 has no native compression whatsoever. ZFS’s lz4 compression, as covered below, is fast enough that leaving it on is simply the correct default — not a tradeoff you have to think about.

Pooled storage as the native model. Growing an NTFS volume or an ext4 filesystem means wrestling with Storage Spaces or LVM/mdadm respectively — separate tools bolted underneath, each with its own mental model and failure modes. ZFS pools and datasets are the native abstraction, not an add-on: adding capacity, carving out a new dataset, or setting a per-dataset quota are all just zpool/zfs subcommands, no second toolchain required.

None of this makes NTFS or ext4 bad filesystems — they’re mature, well-understood, and NTFS in particular ran that Windows Server flawlessly for years. But “flawlessly, as far as I could tell” is exactly the phrase that should make you nervous about silent corruption: the whole point is that neither filesystem would have told me if it happened. mdadm + LVM + ext4 can approximate some of ZFS’s feature list with enough duct tape and personal discipline. ZFS just has it, batteries included, and doesn’t ask you to hold three separate tools together with willpower — or trust that nothing ever quietly goes wrong on the one axis nobody’s watching.

📦 Step 1: Getting OpenZFS onto Debian 13

Debian doesn’t ship ZFS in main for licensing reasons (CDDL vs GPL — software licensing’s longest-running feud, older than most of the drives in this pool), so it lives in contrib. That component isn’t enabled by default:

# /etc/apt/sources.list — add contrib
deb http://deb.debian.org/debian trixie main contrib
deb http://deb.debian.org/debian trixie-updates main contrib
deb http://security.debian.org/debian-security trixie-security main contrib

apt-get update
apt-get install -y linux-headers-amd64 zfs-dkms zfsutils-linux

This pulls in DKMS, builds zfs.ko and spl.ko against the running kernel (6.12.107+deb13-amd64 at the time), and — because Secure Boot module signing is a thing now — generates a self-signed MOK (Machine Owner Key) on the fly to sign the freshly built modules. If your box has Secure Boot enabled you’ll need to enroll that MOK on next reboot; this one didn’t, so it was a non-issue, but it’s worth knowing DKMS is quietly forging its own signing credentials in the background like a very polite, very legal counterfeiter. 🔏

🏗️ Step 2: Building the pool

Three disks, RAIDZ1 (one disk worth of parity — the right call for three drives; RAIDZ2 wants more spindles to not waste too much capacity). The one rule I actually care about here: never point ZFS at /dev/sdX. Device letters are assigned at boot and can shuffle after a kernel update, a BIOS change, or just the universe having a bad day. Use the stable /dev/disk/by-id/ paths instead, unless you enjoy the specific horror of your parity disk quietly becoming your data disk:

zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  storage raidz1 \
  /dev/disk/by-id/ata-WD_Red_SA500_2.5_2TB_25273ZD00458 \
  /dev/disk/by-id/ata-WD_Red_SA500_2.5_2TB_25273ZD01387 \
  /dev/disk/by-id/ata-WD_Red_SA500_2.5_2TB_25273ZD02050

zpool set autotrim=on storage
zfs set relatime=on storage

ashift=12 tells ZFS these are 4K-sector devices (SSDs universally are, even when they lie about it in their firmware and report 512-byte logical sectors like it’s still 2009). Get this wrong at pool creation and you cannot fix it later without destroying and rebuilding the pool — this is the one setting in this whole build with genuinely no take-backsies, so it gets the scary bold text. autotrim=on matters specifically because these are SSDs; without it you’re relying on a periodic manual zpool trim and letting write amplification creep up quietly in the background like storage-layer cholesterol.

End result: three 2TB drives, ~5.45TB raw, ~3.6TB usable after RAIDZ1 parity. The missing ~1.85TB didn’t vanish — it’s the toll booth for “one drive can die and I still have my data.”

🗜️ Compression: lz4, and why not zstd

ZFS compression is transparent, per-dataset, and — with lz4 — essentially free, which in storage-nerd terms is the closest thing to a free lunch you’ll ever get offered. lz4 was purpose-built to be so fast that enabling it is never a throughput regression; it also has an early-abort heuristic that bails out immediately on incompressible data (already-compressed video, encrypted blobs) instead of stubbornly trying anyway and burning CPU for nothing. zstd compresses tighter but costs more CPU per byte. For a general-purpose NAS workload — documents, photos, the occasional VM image, code repos — lz4 was the obvious default. If this were a backup target holding mostly plain-text dumps, I’d have reached for zstd-3 instead and happily paid the CPU tax.

Now that there’s real data on it instead of synthetic benchmark files, here’s what it’s actually buying me — and in the spirit of this post’s running theme, the honest answer is “not much,” and that’s exactly what you’d expect once you know what’s actually stored:

$ zfs get compressratio storage storage/raphael storage/angelika storage/kerstin storage/nextcloud
NAME               PROPERTY       VALUE
storage            compressratio  1.01x
storage/raphael    compressratio  1.00x
storage/angelika   compressratio  1.05x
storage/kerstin    compressratio  1.05x
storage/nextcloud  compressratio  1.03x

Barely-there ratios, and the reason isn’t a misconfiguration — it’s the data itself. This pool is mostly photos, videos, Outlook PST files, and already-compressed archives: exactly the content lz4’s early-abort heuristic is designed to recognize and stop wasting CPU on. That’s not a failure of compression, it’s compression working correctly and getting out of the way. The dataset closest to actually benefiting is storage/kerstin and storage/angelika at 1.05x — still modest, but real, free savings on whatever fraction of that data happens to be text-adjacent (documents, config files, the metadata overhead of hundreds of thousands of small files). If this pool were hosting a database dump or a pile of log files instead of a family photo archive, this table would look a lot more impressive — and that’s the actual lesson: compression ratio tells you as much about your data as it does about your filesystem.

👻 The cosmetic bug that made me doubt myself

I set xattr=sa at pool creation (extended attributes stored directly in the dnode instead of as hidden directory entries — faster, and required for decent ACL performance). The command exited 0, no complaints, no drama. Then zfs get xattr storage stubbornly reported on instead of sa, every single time, like it had never heard of the setting I’d just given it.

Turns out this is a known, confirmed display bug in OpenZFS 2.3.2 (see openzfs/zfs discussion #16996 if you want the gory details) — the property is actually set correctly, zfs get just displays the wrong label. Not a misconfiguration, just cosmetic noise that will absolutely gaslight you into re-running the same command five times before you think to Google it. 🕵️

🧠 ARC tuning: leaving RAM for everything else

ZFS’s Adaptive Replacement Cache (ARC) will happily eat most of your RAM if you let it — which is great on a dedicated storage appliance and considerably less great on a box that also runs other services and would like a word. Left unchecked, ARC behaves exactly like a teenager left alone with a fridge: give it space, it will fill it, and it will not proactively give any back. With 31GB total RAM, I capped it:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=17179869184
options zfs zfs_arc_min=2147483648

16GiB max, 2GiB floor. Applied live via /sys/module/zfs/parameters/zfs_arc_max for immediate effect, then baked into the initramfs (update-initramfs -u -k all) so it survives a reboot and doesn’t just go back to raiding the fridge.

🤥 Benchmarking, or: how I lied to myself for five minutes

First pass, I did the thing every ZFS newcomer does and immediately regretted:

dd if=/dev/zero of=/storage/testfile bs=1M count=10000

8.7 GB/s write. 14.1 GB/s read. On three SATA SSDs. On a home network. I briefly considered writing a strongly-worded LinkedIn post about how underrated consumer SSDs are. Then I remembered how compression works: /dev/zero produces an infinite stream of zeroes, lz4 compresses a run of zeroes to almost nothing, and I had just benchmarked my compressor, not my disks. Deeply silly, briefly exciting, ultimately meaningless — the storage equivalent of timing how fast you can read a book by only reading blank pages. 📖💨

Redid it properly with incompressible data, because lying to yourself is only fun once:

dd if=/dev/urandom of=/storage/testfile bs=1M count=4096          # buffered write
dd if=/dev/urandom of=/storage/testfile bs=1M count=4096 oflag=direct   # O_DIRECT write
dd if=/storage/testfile of=/dev/null bs=1M                        # buffered read
dd if=/storage/testfile of=/dev/null bs=1M iflag=direct           # O_DIRECT read

Ran this properly a second time, weeks later, once the pool had real data on it and — importantly — once every other job on the box (a 380GB migration, a 235GB cross-dataset move, an ACL rebuild) had actually finished and load average had settled back near zero. Benchmarking against a pool that’s still busy digesting yesterday’s work is just measuring contention, not the disks.

That write row is not a typo, and it surprised me enough to double- and triple-check it wasn’t contention from something else before believing it: O_DIRECT writes came in roughly 9x slower than buffered writes on this pool, which is the opposite of the intuition most people carry over from other filesystems (“direct I/O skips a layer, so it should be faster”). On ZFS specifically, that intuition inverts. The buffered path isn’t just “cached” — it’s where ZFS does its actual write optimization: incoming writes get batched in memory and committed to the RAIDZ vdev as part of a transaction group, aligned and coalesced into efficient, full-stripe writes. O_DIRECT, by design, skips exactly that batching layer, so each write has to go straight to the vdev on its own terms — which on a parity-based array can mean smaller, less-aligned I/O and the overhead that comes with it. Bypassing the cache doesn’t bypass any actual inefficiency here; it bypasses the thing that was making the writes efficient in the first place.

The read numbers get the same asterisk as before: this is single-threaded, single-file throughput, not a realistic picture of aggregate pool performance under concurrent load, and both read figures may be partially served out of ARC rather than hitting disk — ARC is not the same thing as the Linux page cache, so echo 3 > /proc/sys/vm/drop_caches does nothing to clear it, and OpenZFS’s O_DIRECT read path will still happily hand you ARC-resident blocks instead of forcing a real disk read if the data’s already there. Take single-run dd numbers as a sanity check on “is anything obviously broken,” not a spec sheet you’d put on a sales slide — and if you only remember one thing from this section, let it be “don’t assume O_DIRECT is the fast path without measuring it on your specific filesystem first.”

👨‍👩‍👧 Why every user gets their own dataset

The pool serves three people, each with a Samba share. Each person’s share root is its own ZFS dataset (storage/raphael, storage/angelika, storage/kerstin) rather than three plain directories inside one big dataset. This wasn’t a default I fell into — it’s a deliberate structural choice, and it pays off in ways a shared directory tree simply can’t compete with:

• Quotas. zfs set quota=500G storage/kerstin caps one user without touching the others. A quota is a dataset property; a directory has no such concept on its own, no matter how sternly you ask it.
• Independent snapshots. zfs snapshot storage/kerstin@before-cleanup lets one person roll back a bad “delete everything” moment without dragging the other two datasets along for the ride.
• Instant usage accounting. zfs list reads space usage straight out of dataset metadata. No walking the tree, no du -sh grinding through hundreds of gigabytes of small files for a minute and a half while you stare at a blinking cursor questioning your career choices.
• Clean ownership boundaries. Each dataset’s mountpoint lines up 1:1 with its Samba share, with its own Unix owner. Filesystem permissions and share permissions stay in sync instead of relying on Samba-layer tricks alone.
• Per-dataset tuning. Compression, atime, record size — all overridable per dataset if one user’s workload ever warranted something different from the pool default.
• Granular replication. zfs send/receive operates per dataset. Want to back up just one person’s share to an external disk without touching the others? One-liner, not a selective rsync exclude-list held together by regret.

The price for all this: moving data between datasets is no longer free. 😬 mv within a single dataset is a pure rename — metadata-only, instant regardless of size. mv across datasets is a completely different animal, even though both datasets live on the same pool and the same physical disks: ZFS treats each dataset as its own filesystem, so the kernel has to actually copy every byte to the new location and then unlink the original. I relearned this the fun way while untangling a nested folder mixup — a same-dataset flatten finished instantly, and a cross-dataset move of ~235GB sat there for the better part of an hour doing what was, for all practical purposes, a full copy while I stared at a progress-less terminal wondering if I’d broken something. It’s not you, ZFS, it’s dataset boundaries. Worth knowing before you kick one off expecting three-second magic like the last one.

🔑 One admin, three shares: why plain chmod couldn’t do this

The access model I wanted is simple to say out loud: everyone gets full access to their own folder, and I additionally get full access to everyone’s, on both the Samba share and the raw Linux filesystem underneath it. Simple to say, and then I actually tried to build it with classic chmod and immediately hit the wall that every sysadmin hits eventually: a Unix file has exactly one owner and exactly one group. That’s it. That’s the whole cast. Everyone else falls into the single bucket labeled “other.”

That model can express “owner gets access” and “this one group gets access” — it cannot express “angelika gets access AND raphael also gets access, but kerstin does not,” because there’s no second group slot to put raphael in without also inventing a shared group, deciding who else might join it later, and hoping nobody adds the wrong person to it in six months. You can hack around this by adding raphael as a secondary member of angelika‘s and kerstin‘s private groups and setting the setgid bit so new files inherit the group — and for a lot of setups that’s a perfectly fine, boring, low-effort answer. I didn’t like it here for two reasons: it grants access at the group level (so it silently changes if group membership ever changes for an unrelated reason), and new-file inheritance depends on the setgid bit plus whatever umask the creating process happens to be using — which works, but “works because three separate mechanisms all lined up correctly” is not a sentence that inspires confidence at 2am.

POSIX ACLs solve the actual problem instead of working around it: they let a file or directory carry additional, explicitly named user and group entries beyond the classic owner/group/other triad. “angelika owns this, and raphael specifically also gets rwx, full stop” — no shared group required, no bystanders. First, ACL support has to be turned on per dataset (off by default):

zfs set acltype=posixacl storage/angelika
zfs set acltype=posixacl storage/kerstin

Then the grant itself, in two parts — a recursive grant for everything that already exists, and a default ACL so anything created from now on inherits the same rule automatically, without relying on umask roulette:

setfacl -R  -m u:raphael:rwx /storage/angelika    # apply to everything that exists now
setfacl -R -d -m u:raphael:rwx /storage/angelika  # apply to everything created from now on

Same two lines for /storage/kerstin. Combined with each folder’s own permissions locked to 750 (owner and group only, nothing for “other” — so kerstin and angelika categorically can’t wander into each other’s data even by accident), the end state is exactly the stated goal: each person has full run of their own folder, I have full run of all three, and nobody gets an accidental side door. Worth double-checking the actual effect rather than trusting the command didn’t silently no-op:

$ getfacl /storage/angelika
user::rwx
user:raphael:rwx
group::r-x
mask::rwx
other::---
default:user::rwx
default:user:raphael:rwx
default:group::r-x
default:mask::rwx
default:other::---

And on Samba’s side, this ACL work turned out to matter less than expected — the three shares already use force user, which makes every connection to a share operate as that share’s owner regardless of who actually authenticated. So my access to angelika’s and kerstin’s shares over SMB was already covered by that directive, provided the underlying folder is actually owned by the right person — which, in a fun bit of self-inflicted archaeology, it briefly wasn’t (a leftover artifact from an earlier reorganization had left one dataset owned by me instead of its actual user, silently granting me SMB write access I hadn’t intended and denying the folder’s rightful owner her own default rights until I caught and fixed it). The ACLs specifically close the other gap: plain SSH / local shell access, which doesn’t go through force user at all and answers to raw Unix permissions alone.

😵 “Wait, is this even one pool anymore?” — the df confusion

A few weeks into running this, a quick df -h made it look like something had gone badly wrong:

Filesystem       Size  Used Avail Use% Mounted on
storage          1.9T  256K  1.9T   1% /storage
storage/raphael  3.1T  1.2T  1.9T  40% /storage/raphael
storage/angelika 1.9T   15G  1.9T   1% /storage/angelika
storage/kerstin  2.1T  236G  1.9T  12% /storage/kerstin

Four different “Size” values, on a pool that’s supposed to be one unified block of storage acting like a RAID5. Did it silently fragment into separate pools overnight? Did I misconfigure something? 😱 No — this is df being technically correct in the most unhelpful way possible.

ZFS datasets don’t have a fixed size the way a classic RAID partition does. df‘s “Size” column is computed per-mount as Used + Available, and — this is the part that actually matters — Available is identical across every dataset, because it’s the same shared free space in the same pool. Only “Used” differs, because that’s genuinely different per person. Run zfs list instead and it stops being mysterious:

$ zfs list -r storage
NAME                USED  AVAIL  REFER  MOUNTPOINT
storage            1.67T  1.85T   160K  /storage
storage/angelika   14.3G  1.85T  14.3G  /storage/angelika
storage/kerstin     235G  1.85T   235G  /storage/kerstin
storage/nextcloud   232G  1.85T   232G  /storage/nextcloud
storage/raphael    1.20T  1.85T  1.20T  /storage/raphael

Same AVAIL — 1.85T — on every single line, because it’s one shared pot. raphael‘s df “size” of 3.1T is just 1.20T used + 1.85T avail; angelika‘s 1.9T is 14.3G + 1.85T. Nothing fragmented, nothing lost — the moment anyone writes a gigabyte, that shared AVAIL drops for everyone at once, which is exactly the RAID5-like behavior that was expected in the first place. zpool list settles it for good:

$ zpool list storage
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH
storage  5.45T  2.50T  2.95T        -         -     0%    45%  1.00x    ONLINE

One pool, 5.45T raw, one RAIDZ1 vdev, all three disks in it. df just wasn’t built with “multiple filesystems dynamically sharing one capacity pool” in mind — it’s reporting honestly, it’s just answering a question (“how big is this mount”) that doesn’t really apply the same way here. zfs list / zpool list are the tools that actually answer “how much room do I have,” not df.

📋 Cheatsheet: the commands I actually use on this pool

Half of learning ZFS is realizing the subcommand you want almost certainly already exists, buried under a name that makes perfect sense in hindsight. Here’s the working set for this exact setup — pool storage, datasets storage/raphael, storage/angelika, storage/kerstin, storage/nextcloud.

Pool health

# Is everything online? Any read/write/checksum errors?
zpool status storage

# One-line health summary, good for a cron/monitoring check
zpool status -x

# I/O activity per vdev, refreshed every 2s
zpool iostat -v storage 2

# Manually kick a trim outside the autotrim schedule
zpool trim storage

# Integrity scrub — reads and verifies every block against its checksum
zpool scrub storage
zpool status storage   # shows progress while it's running

Datasets: listing and space usage

# All datasets under the pool, with usage — instant, no tree walk
zfs list -r storage

# Include snapshots in the listing
zfs list -t all -r storage

# Just one dataset
zfs list storage/kerstin

Compression: how much is it actually buying me

# The headline number: ratio of logical (pre-compression) to physical size
zfs get compressratio storage
zfs get compressratio storage/raphael storage/angelika storage/kerstin storage/nextcloud

# Same thing, the manual way — logicalused is what it WOULD take uncompressed,
# used is what it actually takes on disk
zfs get logicalused,used storage

# Which compression algorithm is actually active per dataset
zfs get compression storage/raphael storage/nextcloud

Quotas and reservations

# Cap a user's dataset so they can't eat the whole pool
zfs set quota=500G storage/kerstin

# Guarantee a dataset a minimum amount of space, even if others fill up
zfs set reservation=100G storage/nextcloud

# See what's currently set
zfs get quota,reservation -r storage

Snapshots

# Take one
zfs snapshot storage/kerstin@2026-09-01-before-cleanup

# List them
zfs list -t snapshot -r storage

# Roll back to one (destroys anything written after it — says so loudly for a reason)
zfs rollback storage/kerstin@2026-09-01-before-cleanup

# Recover a single file without a full rollback — snapshots are browsable here
ls /storage/kerstin/.zfs/snapshot/2026-09-01-before-cleanup/

# Done with it
zfs destroy storage/kerstin@2026-09-01-before-cleanup

Properties: get/set, the general form

# Everything set on a dataset, and where each value comes from
# (default / inherited / local override)
zfs get all storage/raphael | grep -v default

# Set anything per-dataset — overrides pool default for that subtree only
zfs set atime=off storage/nextcloud
zfs set recordsize=1M storage/nextcloud   # bigger records suit large sequential files

ARC (the RAM cache)

# Live stats: hit rate, size, current bounds
cat /proc/spl/kstat/zfs/arcstats | grep -E '^(size|c_max|c_min|hits|misses) '

# Or the friendlier summary, if arcstat is installed
arcstat 2

Replication

# Full send of a dataset to another pool/host, piped over SSH
zfs snapshot storage/kerstin@backup-2026-09-01
zfs send storage/kerstin@backup-2026-09-01 | ssh backup-host zfs receive backuppool/kerstin

# Incremental send — only what changed since the last snapshot
zfs send -i storage/kerstin@backup-2026-08-01 storage/kerstin@backup-2026-09-01 | ssh backup-host zfs receive backuppool/kerstin

🚑 Troubleshooting: what I’ve actually needed so far

Most of this pool’s life has been boring, which is the entire point of running ZFS — boring is a feature. But a few things came up while building it that are worth having on file for next time, filed here so future-me doesn’t have to relearn any of this the hard way twice.

Pool is DEGRADED or a disk shows FAULTED

# First stop: what exactly is wrong, and with which vdev/disk
zpool status -v storage

# Confirm the disk is still physically present and matches the pool's idea of it
ls -la /dev/disk/by-id/ | grep WD_Red

# Once the disk is physically replaced, tell ZFS to resilver the new one in
zpool replace storage ata-WD_Red_SA500_2.5_2TB_OLDSERIAL ata-WD_Red_SA500_2.5_2TB_NEWSERIAL

# Watch the resilver progress
zpool status storage

# If a disk had a transient error but is actually fine (loose cable, one-off glitch),
# clear the error counters instead of replacing anything
zpool clear storage

Resilver or scrub seems stuck / is taking forever

# Check it's actually making progress, not hung
zpool status storage   # look at the % done and time-remaining estimate, run twice a few minutes apart

# See if something else is hammering the pool's I/O concurrently
zpool iostat -v storage 2

# Scrub/resilver speed is throttled by design so it doesn't starve normal I/O;
# these bump the priority if you genuinely need it to finish faster
cat /sys/module/zfs/parameters/zfs_scan_vdev_limit
echo 33554432 > /sys/module/zfs/parameters/zfs_scan_vdev_limit   # example: raise the per-vdev scan limit

Dataset won’t mount / “filesystem already mounted” / “dataset is busy”

# What does ZFS think is mounted where
zfs mount

# Force ZFS to (re)mount everything it thinks should be mounted
zfs mount -a

# "Device or resource busy" on unmount usually means an open file handle —
# find who's holding it
lsof +D /storage/kerstin
fuser -vm /storage/kerstin

# Last resort: force-unmount (drops any open handles, use with care)
umount -l /storage/kerstin

“Permission denied” moving/writing into another user’s dataset

Ran into this directly moving kerstin‘s files into her dataset as the raphael Unix user — the dataset’s owner and mode don’t grant write access, by design, and ZFS was not interested in my excuses. Either do the operation as root, or fix ownership afterward:

# Check who actually owns the mountpoint and what the mode is
ls -ld /storage/kerstin

# Do the privileged operation, then hand ownership back if root touched it
chown -R kerstin:kerstin /storage/kerstin

ZFS kernel module missing after a kernel update

Since this is DKMS-built (out-of-tree), a new kernel means DKMS has to rebuild zfs.ko/spl.ko against it before the module will load — this normally happens automatically as part of the kernel package’s postinst, but it’s the first thing to check if a pool refuses to import after a reboot and you’re briefly convinced you’ve lost 3.6TB of data to the void:

# Is the module actually loaded for the running kernel?
lsmod | grep zfs

# Does DKMS have a built module for this exact kernel version?
dkms status

# If not, force a rebuild against the current kernel
dkms install zfs/$(dkms status zfs | head -1 | cut -d, -f2 | tr -d ' ')  -k $(uname -r)

# Then try importing again
zpool import storage

A dependent service (GitLab, a container, anything) starts before the pool is mounted

Symptom: the service comes up fine but writes into an empty directory on the root filesystem instead of the real ZFS-backed data, because its mountpoint wasn’t ready yet at boot — quietly, with no error, which is the worst kind of bug. Check the actual systemd ordering rather than assuming it’s fine:

# Does local-fs.target really wait for this dataset's mount unit?
systemctl show local-fs.target -p After | tr ' ' '\n' | grep storage

# Is the mount unit itself enabled and correctly ordered before local-fs.target?
systemctl show storage-kerstin.mount -p Before,After,WantedBy

# And does the dependent service start late enough (after multi-user.target,
# which itself sits after local-fs.target)?
systemctl show gitlab-runsvdir.service -p After

Free space “disappeared” and du doesn’t explain it

# Snapshots hold on to space for blocks that changed since they were taken —
# a forgotten snapshot is the single most common cause of "where did my space go"
zfs list -t snapshot -r storage -o name,used,creation

# See exactly how much of a dataset's usage is snapshots vs. live data
zfs get usedbydataset,usedbysnapshots storage/kerstin

ARC is eating all the RAM

# Confirm it's actually ARC and not something else
free -h
cat /proc/spl/kstat/zfs/arcstats | grep '^size '

# Confirm the cap is actually applied (compare against zfs_arc_max in modprobe.d)
cat /sys/module/zfs/parameters/zfs_arc_max

# If the modprobe.d change hasn't taken effect yet, push it live without a reboot
echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max

🏁 Final layout

$ zpool status storage
  pool: storage
 state: ONLINE
config:
        NAME                                       STATE
        storage                                    ONLINE
          raidz1-0                                 ONLINE
            ata-WD_Red_SA500_2.5_2TB_25273ZD00458   ONLINE
            ata-WD_Red_SA500_2.5_2TB_25273ZD01387   ONLINE
            ata-WD_Red_SA500_2.5_2TB_25273ZD02050   ONLINE
errors: No known data errors

$ zfs list -r storage
NAME               USED  AVAIL  REFER  MOUNTPOINT
storage           ...    3.5T   ...    /storage
storage/raphael    ...    3.5T   ...    /storage/raphael
storage/angelika   ...    3.5T   ...    /storage/angelika
storage/kerstin    ...    3.5T   ...    /storage/kerstin

Three SSDs, one parity drive’s worth of redundancy, transparent compression that costs nothing, and per-user datasets that make quotas, snapshots, and backups a one-liner instead of a project. Not bad for an afternoon — even counting the ten minutes I spent fighting a bug that wasn’t real and the five minutes I spent impressed by a benchmark that was, in fact, entirely fictional. 🎉