A Monit Watchdog After fail2ban Died Silently

🌐 Also available in: Deutsch · Français · Español

Confession time. A while back I discovered that fail2ban had been dead on my server for two days and I had no idea. A config file pointed at a log path that no longer existed, fail2ban refused to start, systemd shrugged, and the whole thing just… sat there. Not banning anyone. Silently. 🙈 The scary part isn’t that a service died — services die. The scary part is that nothing told me. A security control can fail open and you’d never know until you go looking. So I did the sensible thing and added a watchdog whose entire job is to notice when something is wrong and yell at me about it.

Table of Contents

🤔 Why Monit (and not a hand-rolled script)

My first instinct was to write a little bash script on a cron timer. Then I remembered: this is a solved problem, and reinventing monitoring is how you end up with a watchdog that itself fails silently. There are established tools: Prometheus + Alertmanager (great if you already live in that world), Icinga/Zabbix (excellent, and overkill for a hobby box), and Monit — a tiny, purpose-built daemon that does exactly one thing well: watch services and resources, and alert (or auto-restart) when they misbehave. Monit won for a homelab-grade blog for one reason above all: it doesn’t just alert, it can self-heal. If fail2ban dies, Monit restarts it and emails me. The two-day outage would have been a two-minute blip plus a heads-up.

📦 Requirements & Install

AlmaLinux 9 with EPEL enabled (Monit lives there): dnf install monit — you get Monit 6.0.
A working mail relay. I already had msmtp pointed at my provider for rkhunter/ClamAV alerts, so Monit could reuse the same credentials.
Root, because it checks system services, the firewall and disk.

The whole thing is an Ansible role now, so it’s reproducible across hosts, but the moving parts are just two files: the main /etc/monitrc (daemon settings + mail relay) and a drop-in of checks in /etc/monit.d/.

⚙️ What It Actually Checks

The philosophy: silent while healthy. Monit only emails on a state change — a healthy thing starts failing, or a failing thing recovers. A quiet inbox means a healthy box, so a mail from Monit is never noise.

🥷 fail2ban — the guest of honour. Auto-restart via systemd; if it refuses to stay up, Monit stops flapping and sends a “give up” alert instead. The exact failure that started all this would now be caught.
🔌 sshd, k3s, node_exporter, Prometheus, Grafana — alert if the systemd unit isn’t active.
🧱 Firewall — not just “is the unit running” but “is the nftables input chain actually default-drop”. A firewall that’s loaded but flushed is worse than useless, so I check the real thing.
📦 K3s pods — anything stuck in CrashLoopBackOff / ImagePullBackOff / OOMKilled / Error.
💽 Disk, load, memory — the boring resource stuff that silently kills servers at 3am.

A stripped-down check looks pleasingly readable:

check process fail2ban with pidfile /run/fail2ban/fail2ban.pid
    start program = "/usr/bin/systemctl start fail2ban"
    if does not exist then restart
    if 3 restarts within 5 cycles then timeout

check program firewall with path "/usr/local/sbin/monit-check-firewall.sh"
    if status != 0 then alert

📧 The Mail Gotcha (a good hour of my life)

Here’s the part that’ll save you time. My working msmtp setup used STARTTLS on port 587. So naturally I configured Monit the same way. Monit promptly failed with:

Mail: Authentication failed -- no supported authentication methods found

Which is a lie, sort of. The relay absolutely supports AUTH. The real problem: Monit 6 has no STARTTLS keyword. Its

using
ssl

means implicit SSL, whose standard port is 465, not 587. On 587 Monit was talking plaintext, the server offered no AUTH before STARTTLS (which never happened), and Monit concluded there were “no supported authentication methods”. 🤦 To prove the relay itself was fine (and not chase a phantom), I tested the credentials directly with a few lines of Python smtplib — both ports logged in cleanly and the server advertised

AUTH PLAIN
LOGIN

. So: relay good, Monit config wrong. The fix was one word and one number:

set mailserver smtp.example.com port 465
    username "..." password "..."
    using ssl

Lesson: msmtp/STARTTLS on 587 ≠ Monit/implicit-SSL on 465. Same relay, same credentials, different port and TLS mode. Don’t copy the 587 setting across.

🧪 How I Tested It (the honest way)

A watchdog you haven’t seen bark is just a decoration. So I made it bark. I stopped a safe, non-critical service (node_exporter), forced an immediate check with monit validate instead of waiting for the next cycle, and watched. Then I started the service again to confirm the recovery path. Here’s what actually landed in my inbox — a failure, a recovery, and (from a second pass) another failure:

Monit on www.apt-upgrade.me reported:

    Status failed: svc_node_exporter
    at Tue, 28 Jul 2026 08:33:52
    Action: alert

    status failed (1) -- systemd service 'node_exporter' is inactive (expected: active)

    -- Monit watchdog

Monit on www.apt-upgrade.me reported:

    Status succeeded: svc_node_exporter
    at Tue, 28 Jul 2026 08:35:53
    Action: alert

    status succeeded (0) -- no output

    -- Monit watchdog

Monit on www.apt-upgrade.me reported:

    Status failed: svc_node_exporter
    at Tue, 28 Jul 2026 08:37:04
    Action: alert

    status failed (1) -- systemd service 'node_exporter' is inactive (expected: active)

    -- Monit watchdog

Detection ✔️, alert email ✔️, recovery email ✔️. That “status succeeded” in the middle is the recovery notice — the reassuring “never mind, it’s back” that tells you the loop actually closes. 🎯

🎬 The Takeaway

Every security control you run should have something checking that it’s still running. A firewall that’s been flushed, a fail2ban that won’t start, a disk quietly filling to 100% — these don’t announce themselves. Now, on this box, they do: a small daemon that stays quiet when all is well and pings my inbox the moment it isn’t. The best monitoring is the kind you forget about, right up until the one day it saves you. 🛎️ As always, the whole thing — the Monit role, the checks, the mail config, all of it — is public and version-controlled in my Ansible repo: github.com/aptupgrademe/www_k3s. Steal it, improve it, tell me what I got wrong. 🧑‍💻