Monitoring Till the Doctor Comes!

🌐 Also available in: Deutsch · Français · Español

From 5 pm, nature switches to dark mode – system-wide, no toggle, no opt-out. Unlike here on the blog, you cannot click it away. 🌑 Single digits by morning, and on grey days here in southern Germany the solar panels produce just enough for one coffee – espresso, not cappuccino. ☕ My thoughts drift back to the last Mallorca holiday: sun, sea and Playa de Palma. At home the party is quieter: servers instead of sangria, dashboards instead of a dance floor. 🍹

Seriously now.

Servers, and web applications in particular, fall ill just like real patients. They run a fever (CPU at the limit), get short of breath (disk full), suffer memory loss (RAM running out) or simply have a bad day and answer more slowly than usual. Some symptoms go away by themselves, others are the first signs of something serious and have to be looked into. That is exactly what monitoring is: regularly taking pulse, temperature and blood pressure, spotting anomalies before they turn into an emergency, and knowing where to look when it gets serious.

And as with a doctor: if you only take your own pulse, you will not notice your own cardiac arrest. That was exactly the gap left in the monitoring here.

🩺 The check-up: who checks what?

For orientation: there are two worlds here. On the internet there are three rented servers – the blog and two Nextcloud instances. At home there is a small rack of my own hardware, including the NAS and backup server, which runs around the clock anyway and spends most of its time idling. Over the months, monitoring layers have grown in both worlds. Each one answers a different question.

On the three servers on the internet – each one watches itself:

From the NAS server at home – the view from outside and onto my own hardware:

The takeaway: each layer exists for one kind of failure. Monit sees whether something runs, Prometheus whether it is enough, Loki why it is stuck, the heartbeats whether anything happened at all.

🚑 The gap: who calls the doctor when the patient collapses?

The first table gives it away: everything in it runs on the server it watches. If a server crashes completely, the hoster has an outage or the network is gone, Monit, Grafana and Loki go down with it. Nobody sends a mail – and in the inbox silence looks exactly like “all good”. An unmonitored server is Schrödinger’s server: up and down at the same time – until someone opens the box, usually a user with a phone in their hand. 🐈📦

Until now you would have noticed through the weekly test mail that does not arrive, or the Monday check from outside. In the worst case after days. There is a second case, too: every service runs, but the site only shows “Error establishing a database connection”, or Nextcloud is stuck in maintenance mode. For Monit everything is green. The question, by the way, is almost two thousand years old – Juvenal asked it in his Satires: Quis custodiet ipsos custodes? Who watches the watchmen?

This is about more than technology: I want to notice an outage myself before anyone contacts me about it. That is the difference between reactive and proactive. Reactive means acting after a problem has surfaced – usually because someone calls or writes “the cloud is not working”. By then the damage is done, and the user saw it before I did. Proactive means making sure in advance that you spot problems yourself and early – ideally before anyone notices, at the very least so that you can answer the first message with “I know, I am on it.” Every layer in the tables above is a piece of proactivity; without a view from outside, the biggest outage of all stayed reactive. There are two kinds of admins: those who learn about an outage from their monitoring – and those who learn about it from their users.

The takeaway: monitoring that runs on the same machine as the patient cannot report its total failure. That takes someone outside.

🏠 The house call: an uptime check from home

The fix is deliberately small and needs no new hardware – the NAS server at home is awake anyway. A Python script there calls each of the three sites on the internet every 5 minutes the way a visitor would, just without a browser:

• WordPress: the home page answers 200 and is a real page, not a database error page or a maintenance notice.
• Nextcloud: status.php reports “installed”, “not in maintenance” and “no pending database upgrade”.
• Collabora: /hosting/discovery returns the WOPI XML that Nextcloud needs to open documents.
• Certificate: once a day – invalid or valid for less than 14 days is a finding.

Mail goes out only on a change: “down” after two failed runs in a row (a single hiccup is not an alarm), “back” with the length of the outage, and a reminder every 12 hours while something stays down. If all targets fail at once, the script first checks two neutral sites on the internet – if they are gone too, it is the own line, and there is no false alarm. In miniature this is computer science’s two generals problem: when no answer comes back, you cannot tell whether the receiver fell or the messenger – so you ask a third party.

The user agent contains “uptime” so that the visitor statistics do not count the 288 daily visits from my own living room. And a nod to Heisenberg: a measurement should change what it measures as little as possible – twelve requests an hour disappear in the background noise.

Its own limit: if the NAS server is off or the line at home is down, nobody checks. Still a big step – before, the only outside check ran once a week.

The takeaway: check from outside what the user sees, not whether a process runs. And report changes of state, not states – otherwise the mails soon end up unread in the archive. An alarm that rings all the time is a smoke detector in a sauna. 🔥

💓 Taking the blood pressure: what “pressure” means

Also new on the dashboard is a value that stayed empty for a long time: pressure (Pressure Stall Information, PSI). It does not measure how full something is, but how long programs had to wait because a resource was short. A CPU can be 100 % busy without anyone waiting – only the waiting makes a server feel slow. After all, at the supermarket checkout nobody cares how busy the cashier is – only how long the queue is.

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=31
full avg10=0.00 avg60=0.00 avg300=0.00 total=31

• cpu / memory / io: waited for CPU time, for free memory (the kernel had to reclaim first – the earliest sign before OOM kills) or for the disk.
• some: share of time in which at least one process was waiting.
• full: share of time in which all were waiting and nothing made progress – the more serious value.
• avg10 / avg60 / avg300: percent of time, averaged over 10 seconds, 1 and 5 minutes.

Rule of thumb: a few percent are normal. some above 10–20 % for a long time is noticeable; full on memory or disk clearly above 0 for a while is a finding.

A mistake of my own on the side: on these servers PSI is off in the kernel by default and is switched on with the boot parameter psi=1. Setting it was quick – but it only takes effect at the next boot. The blog rebooted the following night; the two Nextcloud servers ran for another day and a half with an empty panel until they were rebooted by hand.

The takeaway: a boot parameter only counts after the boot. A change is done only when the value arrives where you want to see it.

🔍 A finding during the check-up

Going through everything turned up one more thing: the temperature script for the rack at home – it only measures my own hardware, not the servers on the internet – has no mail configured. The first worry was that nobody would report overheating. A look into the iLO showed that its own AlertMail had been set up long ago and reports critical temperatures by itself. The script only adds the early, quiet hints in its report.

The takeaway: before building a new alert, look at what already alerts. Duplicate alarms are no gain.

🔭 What comes next: sharper, not more

Monitoring like this is never finished. Developing it further does not mean adding new tools here, though – it means making the existing tests sharper and more specific. A few considerations that are realistic and need no new infrastructure – ideas, not promises:

• Function instead of reachability: a test account logs in to Nextcloud, uploads a small file via WebDAV, fetches it again and compares the checksum. That checks login, storage, database and Redis in one go.
• Content instead of status code: on the blog a test opens a specific post and expects its heading – a page can answer 200 and still be empty or come from a stale cache.
• Slow before broken: record the response time and warn when it is clearly above its own median of the last days. Most outages announce themselves as “sluggish” first.
• Mutual checks: the three servers on the internet check each other – so someone stays awake even when the line at home is down.
• Who checks the house call? The check itself needs a heartbeat, for example a timestamp that a monitor on the internet misses when it stops arriving. Juvenal sends his regards.

On top of that comes fine-tuning: thresholds that fire too often or too rarely get adjusted, and every real incident that no test noticed turns into a question: which test would have found it?

The takeaway: monitoring matures with every incident. The more specific a test, the closer it is to what a user actually does.

📜 The script

Python standard library only, mail through the local mail command (msmtp here). The configuration with addresses and the mail recipient sits next to it, shown below with example values.

#!/usr/bin/env python3
"""
uptime_check.py - is every site still reachable and working, seen from outside?

Monit, Prometheus, Grafana and Loki run on the servers they watch. If a whole
server goes down (crash, hoster outage, network), they go down with it and
nobody sends a mail. This script runs on a machine at home every 5 minutes,
checks each site the way a visitor would and mails only when the state
changes: DOWN after <fail_after> failed runs in a row, UP again with the
downtime, a reminder every <remind_hours> while it stays down.

  uptime_check.py            run all checks (cron)
  uptime_check.py --dry-run  run all checks, print mails instead of sending
  uptime_check.py --status   show the stored state

Kinds of checks (uptime_check.conf, [targets]):
  wordpress  page answers 200 and is a real page, not a database error
  nextcloud  status.php: installed, not in maintenance, no pending DB upgrade
  collabora  /hosting/discovery answers with the WOPI discovery XML
Every target also gets a TLS certificate check (warning below cert_warn_days).

Standard library only. Mail goes through the local "mail" command (msmtp).
"""
import configparser, datetime as dt, json, os, socket, ssl, subprocess, sys, time
import urllib.error, urllib.request
from urllib.parse import urlsplit

HERE = os.path.dirname(os.path.abspath(__file__))
CONF = os.path.join(HERE, "uptime_check.conf")
STATE = os.path.join(HERE, "state.json")
LOG = os.path.join(HERE, "uptime_check.log")
UA = "uptime-check/1.0 (+homelab)"   # "uptime" keeps it out of the visitor statistics


def log(msg):
    with open(LOG, "a") as f:
        f.write(f"{dt.datetime.now():%Y-%m-%d %H:%M:%S} {msg}\n")


def fetch(url, timeout):
    """GET url -> (status, body text, seconds). Raises on network errors."""
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = r.read(2_000_000).decode("utf-8", "replace")
            return r.status, body, time.monotonic() - t0
    except urllib.error.HTTPError as e:
        return e.code, "", time.monotonic() - t0


# Each check returns None when everything is fine, otherwise a short reason.
def check_wordpress(base, timeout):
    status, body, secs = fetch(base, timeout)
    if status != 200:
        return f"HTTP {status}"
    if "Error establishing a database connection" in body:
        return "WordPress meldet: keine Datenbankverbindung"
    if "Briefly unavailable for scheduled maintenance" in body:
        return "WordPress steckt im Wartungsmodus"
    if "wp-content" not in body or "</html>" not in body:
        return "Antwort ist keine vollständige WordPress-Seite"
    return None


def check_nextcloud(base, timeout):
    status, body, secs = fetch(base.rstrip("/") + "/status.php", timeout)
    if status != 200:
        return f"status.php: HTTP {status}"
    try:
        s = json.loads(body)
    except ValueError:
        return "status.php liefert kein JSON"
    if not s.get("installed"):
        return "status.php: installed = false"
    if s.get("maintenance"):
        return "Nextcloud ist im Wartungsmodus"
    if s.get("needsDbUpgrade"):
        return "Nextcloud wartet auf ein Datenbank-Upgrade (occ upgrade)"
    return None


def check_collabora(base, timeout):
    status, body, secs = fetch(base.rstrip("/") + "/hosting/discovery", timeout)
    if status != 200:
        return f"discovery: HTTP {status}"
    if "<wopi-discovery" not in body:
        return "discovery liefert kein WOPI-XML"
    return None


CHECKS = {"wordpress": check_wordpress, "nextcloud": check_nextcloud, "collabora": check_collabora}


def cert_days_left(base, timeout):
    host = urlsplit(base).hostname
    ctx = ssl.create_default_context()        # also fails on an invalid or wrong-name certificate
    with socket.create_connection((host, 443), timeout=timeout) as sock:
        with ctx.wrap_socket(sock, server_hostname=host) as tls:
            not_after = ssl.cert_time_to_seconds(tls.getpeercert()["notAfter"])
    return int((not_after - time.time()) // 86400)


def run_check(kind, base, timeout):
    try:
        return CHECKS[kind](base, timeout)
    except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError, ssl.SSLError, OSError) as e:
        reason = getattr(e, "reason", e)
        return f"nicht erreichbar ({reason})"


def online(refs, timeout):
    """Is this machine itself online? True if any reference URL answers at all."""
    for url in refs:
        try:
            fetch(url, timeout)
            return True
        except Exception:
            continue
    return False


def fmt_duration(seconds):
    m = int(seconds // 60)
    return f"{m // 60} h {m % 60} min" if m >= 60 else f"{m} min"


def send_mail(to, subject, text, dry_run):
    if dry_run:
        print(f"--- MAIL an {to}: {subject}\n{text}\n---")
        return
    r = subprocess.run(["mail", "-s", subject, to], input=text, text=True, capture_output=True)
    if r.returncode != 0:
        log(f"MAIL FEHLER rc={r.returncode}: {r.stderr.strip()}")


def main():
    dry_run = "--dry-run" in sys.argv
    cfg = configparser.ConfigParser()
    if not cfg.read(CONF):
        sys.exit(f"config missing: {CONF}")
    g = cfg["general"]
    timeout = g.getint("timeout", 20)
    fail_after = g.getint("fail_after", 2)
    remind = g.getint("remind_hours", 12) * 3600
    cert_warn = g.getint("cert_warn_days", 14)
    refs = g.get("reference_urls", "").split()
    mail_to = g["mail_to"]

    try:
        state = json.load(open(STATE))
    except (OSError, ValueError):
        state = {}
    if "--status" in sys.argv:
        print(json.dumps(state, indent=2, ensure_ascii=False))
        return

    now = int(time.time())
    targets = {name: val.split(None, 1) for name, val in cfg["targets"].items()}
    results = {name: run_check(kind, url, timeout) for name, (kind, url) in targets.items()}

    # Everything failed at once? Then it is more likely this machine's own
    # line than all servers - check a neutral reference before alarming.
    if all(results.values()) and not online(refs, timeout):
        log("alle Ziele fehlgeschlagen und Referenz nicht erreichbar - eigene Leitung offline, nichts gezählt")
        return

    mails = []
    for name, (kind, url) in targets.items():
        st = state.setdefault(name, {"status": "up", "fails": 0, "since": now, "last_mail": 0})
        reason = results[name]
        if reason is None:
            if st["status"] == "down":
                mails.append(f"WIEDER DA  {name} ({url}) nach {fmt_duration(now - st['since'])}")
                log(f"UP {name} nach {fmt_duration(now - st['since'])}")
                st.update(status="up", since=now)
            st["fails"] = 0
        else:
            st["fails"] += 1
            log(f"FAIL {name} ({st['fails']}x): {reason}")
            if st["status"] == "up" and st["fails"] >= fail_after:
                st.update(status="down", since=now, last_mail=now, reason=reason)
                mails.append(f"AUSFALL    {name} ({url}): {reason}")
            elif st["status"] == "down" and now - st["last_mail"] >= remind:
                st["last_mail"] = now
                mails.append(f"NOCH DOWN  {name} ({url}) seit {fmt_duration(now - st['since'])}: {reason}")

        # Certificate: checked once per target and day, so at most one warning a day.
        if reason is None and now - st.get("cert_checked", 0) >= 86400:
            st["cert_checked"] = now
            try:
                days = cert_days_left(url, timeout)
                if days < cert_warn:
                    mails.append(f"ZERTIFIKAT {name}: läuft in {days} Tagen ab (Erneuerung hängt?)")
            except Exception as e:
                mails.append(f"ZERTIFIKAT {name}: Prüfung fehlgeschlagen ({e})")

    if mails:
        down = sum(1 for s in state.values() if s["status"] == "down")
        subject = f"Erreichbarkeit: {down} Ziel(e) down" if down else "Erreichbarkeit: alles wieder erreichbar"
        text = "\n".join(mails) + f"\n\nGeprüft {dt.datetime.now():%Y-%m-%d %H:%M} von {socket.gethostname()}.\n"
        send_mail(mail_to, subject, text, dry_run)

    if not dry_run:
        tmp = STATE + ".tmp"
        with open(tmp, "w") as f:
            json.dump(state, f, indent=1)
        os.replace(tmp, STATE)


if __name__ == "__main__":
    main()

The configuration uptime_check.conf (example values):

# uptime_check.conf - chmod 600 (contains the mail address)
[general]
mail_to = you@example.org
# seconds per request
timeout = 20
# failed runs in a row before a DOWN mail (cron every 5 min -> 10 min)
fail_after = 2
remind_hours = 12
cert_warn_days = 14
# neutral sites: if ALL targets fail, these tell "server down" from "own line down"
reference_urls = https://www.google.com/generate_204 https://one.one.one.one/

[targets]
# name = kind base-url
blog = wordpress https://blog.example.org/
cloud = nextcloud https://cloud.example.org/
office = collabora https://office.example.org/

And the crontab entry:

*/5 * * * * cd $HOME/uptime-check && python3 -I uptime_check.py >> cron.log 2>&1

To try it: --dry-run only prints the mails, --status shows the stored state. The easiest way to test an outage is an extra target pointing at an address that does not exist (.invalid) – not wrong paths on your own server, or fail2ban bans your own line.

🎯 The lessons

• Monitoring on the server itself cannot report its total failure – someone has to look from outside.
• Check what the user sees: a real page, an “installed: true” – not just a running process.
• Report changes of state, not states; two failures before the alarm filter out hiccups.
• Pressure shows waiting, not utilisation – the thing users experience as “slow”.
• A boot parameter works only after the boot; it is done only when the value arrives.

And now: heating on, kettle on, one more look at the Mallorca photos. The servers will call when the doctor is needed. There’s no place like 127.0.0.1 – but 127.0.0.1 is exactly the one that cannot tell anyone it is down itself. That is what the house call is for now. 😉