Short one, and a direct consequence of the snapshot post. Adding daily snapshots introduced something that hadn’t existed on this NAS before: a thing that grows on its own, quietly, forever, without anybody asking it to. So I went looking for whatever would tell me when the pool started filling up. The answer was mildly embarrassing.
🔍 ZED has opinions, but not about this
ZED is running. It’s configured. I fixed exactly that in the last addendum, and the fix holds:$ systemctl is-active zed
active
$ grep ZED_EMAIL_ADDR /etc/zfs/zed.d/zed.rc
ZED_EMAIL_ADDR="you@example.com"🧊 Why “just delete something” isn’t the escape hatch it used to be
On ext4, a full disk is annoying. On ZFS it’s worse, for two reasons that compound each other. First, write performance degrades meaningfully once you’re past roughly 80% — the allocator has to work harder to find contiguous space, and it shows. Second, and this is the one that gets people: deleting files can fail on a completely full pool, because deleting is itself a write. Removing a file means updating metadata, and copy-on-write means updating metadata means allocating a block. No free blocks, no delete. Now layer snapshots on top. As covered in the snapshot post, deleting a file that a snapshot still references frees exactly nothing:storage/demo USED: 200M REFER: 128KREFER is what’s in the live directory: essentially empty. USED is what the dataset actually occupies, because a snapshot is holding the blocks. So the instinctive recovery move — free up space by deleting stuff — gets you nowhere on a snapshotted dataset, and may not even execute on a truly full pool. That’s a combination worth not discovering live.🛠️ Five lines and a timer
The check itself is unremarkable, which is the point:#!/bin/bash
# /usr/local/sbin/zfs-capacity-check.sh
set -euo pipefail
WARN="${1:-80}"
RECIPIENT="$(grep -hE '^ZED_EMAIL_ADDR=' /etc/zfs/zed.d/zed.rc 2>/dev/null \
| head -1 | cut -d= -f2- | tr -d '"' | xargs)"
[ -n "$RECIPIENT" ] || exit 0
problems=""
while read -r pool cap; do
num="${cap%\%}"
[ "$num" -ge "$WARN" ] || continue
problems="${problems}Pool '${pool}' is ${cap} full (threshold: ${WARN}%).
"
done < <(zpool list -H -o name,capacity)
[ -n "$problems" ] || exit 0
# ... assemble report, pipe into mail ...zed.rc rather than hardcoded. There’s already exactly one place on this box where “who gets told when storage misbehaves” is configured, and adding a second one is how those two quietly drift apart six months from now. And the mail doesn’t just say “pool is full.” It lists the largest snapshots, because on a ZFS box with automatic snapshots, that is overwhelmingly the answer to “where did my space go” — and the message that arrives at 3 a.m. should contain the answer, not just the question. The timer is the same shape as the snapshot ones: daily, Persistent=true, randomized delay.✅ Testing the thing that’s supposed to warn me
The last addendum ended on a component whose entire job was raising an alarm, failing silently. I’m not walking into that twice. So rather than trusting a check that correctly reports “nothing wrong” on a pool at 49%, I lowered the threshold until it had to fire:$ /usr/local/sbin/zfs-capacity-check.sh 40
$ tail -2 /var/log/msmtp.log
Sep 02 21:19:09 host=smtp.strato.de ... recipients=you@example.com
smtpstatus=250 exitcode=EX_OK
Sep 11 14:47:14 host=smtp.strato.de ... recipients=you@example.com
mailsize=861 smtpstatus=250 exitcode=EX_OK250, EX_OK. The whole chain works: script, mail, msmtp, Strato, delivered. And look at the line above it. September 2nd — that’s the ZED test message from the previous addendum, sitting right there in the same log. Two alerting paths, nine days apart, both verified the same way, for the same reason.
