Confession time. A while back I discovered that fail2ban had been dead
on my server for two days and I had no idea. A config file pointed at
a log path that no longer existed, fail2ban refused to start, systemd shrugged,
and the whole thing just… sat there. Not banning anyone. Silently. 🙈
The scary part isn’t that a service died — services die. The scary part
is that nothing told me. A security control can fail open and you’d
never know until you go looking. So I did the sensible thing and added a
watchdog whose entire job is to notice when something is wrong and yell at me
about it.
🤔 Why Monit (and not a hand-rolled script)
My first instinct was to write a little bash script on a cron timer. Then I
remembered: this is a solved problem, and reinventing monitoring is how you end
up with a watchdog that itself fails silently. There are established tools:
Prometheus + Alertmanager (great if you already live in that
world), Icinga/Zabbix (excellent, and overkill for a hobby
box), and Monit — a tiny, purpose-built daemon that does
exactly one thing well: watch services and resources, and alert (or
auto-restart) when they misbehave.
Monit won for a homelab-grade blog for one reason above all: it doesn’t just
alert, it can self-heal. If fail2ban dies, Monit restarts it
and emails me. The two-day outage would have been a two-minute blip
plus a heads-up.
📦 Requirements & Install
- AlmaLinux 9 with EPEL enabled (Monit lives there):
dnf install monit— you get Monit 6.0. - A working mail relay. I already had
msmtppointed at my provider for rkhunter/ClamAV alerts, so Monit could reuse the same credentials. - Root, because it checks system services, the firewall and disk.
The whole thing is an Ansible role now, so it’s reproducible across hosts, but
the moving parts are just two files: the main /etc/monitrc (daemon
settings + mail relay) and a drop-in of checks in /etc/monit.d/.
⚙️ What It Actually Checks
The philosophy: silent while healthy. Monit only emails on a
state change — a healthy thing starts failing, or a failing thing
recovers. A quiet inbox means a healthy box, so a mail from Monit is never
noise.
- 🥷 fail2ban — the guest of honour. Auto-restart via systemd; if it refuses to stay up, Monit stops flapping and sends a “give up” alert instead. The exact failure that started all this would now be caught.
- 🔌 sshd, k3s, node_exporter, Prometheus, Grafana — alert if the systemd unit isn’t active.
- 🧱 Firewall — not just “is the unit running” but “is the nftables input chain actually default-drop”. A firewall that’s loaded but flushed is worse than useless, so I check the real thing.
- 📦 K3s pods — anything stuck in CrashLoopBackOff / ImagePullBackOff / OOMKilled / Error.
- 💽 Disk, load, memory — the boring resource stuff that silently kills servers at 3am.
A stripped-down check looks pleasingly readable:
check process fail2ban with pidfile /run/fail2ban/fail2ban.pid
start program = "/usr/bin/systemctl start fail2ban"
if does not exist then restart
if 3 restarts within 5 cycles then timeout
check program firewall with path "/usr/local/sbin/monit-check-firewall.sh"
if status != 0 then alert📧 The Mail Gotcha (a good hour of my life)
Here’s the part that’ll save you time. My working msmtp setup used
STARTTLS on port 587. So naturally I configured Monit the same
way. Monit promptly failed with:
Mail: Authentication failed -- no supported authentication methods foundWhich is a lie, sort of. The relay absolutely supports AUTH. The real
problem: Monit 6 has no STARTTLS keyword. Its using means implicit SSL, whose standard port is 465,
ssl
not 587. On 587 Monit was talking plaintext, the server offered no AUTH before
STARTTLS (which never happened), and Monit concluded there were “no supported
authentication methods”. 🤦
To prove the relay itself was fine (and not chase a phantom), I tested the
credentials directly with a few lines of Python smtplib —
both ports logged in cleanly and the server advertised AUTH PLAIN. So: relay good, Monit config wrong. The fix was one word and one
LOGIN
number:
set mailserver smtp.example.com port 465
username "..." password "..."
using sslLesson: msmtp/STARTTLS on 587 ≠ Monit/implicit-SSL on 465.
Same relay, same credentials, different port and TLS mode. Don’t copy the 587
setting across.
🧪 How I Tested It (the honest way)
A watchdog you haven’t seen bark is just a decoration. So I made it bark. I
stopped a safe, non-critical service (node_exporter), forced an
immediate check with monit validate instead of waiting for the
next cycle, and watched. Then I started the service again to confirm the
recovery path. Here’s what actually landed in my inbox — a failure, a
recovery, and (from a second pass) another failure:
Monit on www.apt-upgrade.me reported:
Status failed: svc_node_exporter
at Tue, 28 Jul 2026 08:33:52
Action: alert
status failed (1) -- systemd service 'node_exporter' is inactive (expected: active)
-- Monit watchdogMonit on www.apt-upgrade.me reported:
Status succeeded: svc_node_exporter
at Tue, 28 Jul 2026 08:35:53
Action: alert
status succeeded (0) -- no output
-- Monit watchdogMonit on www.apt-upgrade.me reported:
Status failed: svc_node_exporter
at Tue, 28 Jul 2026 08:37:04
Action: alert
status failed (1) -- systemd service 'node_exporter' is inactive (expected: active)
-- Monit watchdogDetection ✔️, alert email ✔️, recovery email ✔️. That “status succeeded” in the
middle is the recovery notice — the reassuring “never mind, it’s back”
that tells you the loop actually closes. 🎯
🎬 The Takeaway
Every security control you run should have something checking that it’s still
running. A firewall that’s been flushed, a fail2ban that won’t start, a disk
quietly filling to 100% — these don’t announce themselves. Now, on this
box, they do: a small daemon that stays quiet when all is well and pings my
inbox the moment it isn’t. The best monitoring is the kind you forget about,
right up until the one day it saves you. 🛎️
As always, the whole thing — the Monit role, the checks, the mail config,
all of it — is public and version-controlled in my Ansible repo:
github.com/aptupgrademe/www_k3s.
Steal it, improve it, tell me what I got wrong. 🧑💻
