My Evening Off: A Leaky Thumbnail, a Noisy Cluster and a Header That Wouldn't Behave
I only wanted to update Hugo. By midnight I had found an EXIF thumbnail that showed more than the photo, silenced a chatty Kubernetes exporter, stopped my firewall from logging scanners, rebuilt my header five times and explained to myself why GitHub is not my backup.

🌐 Also available in: Deutsch · Français · Español · Italiano · Português · Nederlands · Polski
In short: The plan for the evening was one line: update Hugo. What I got instead were seven small construction sites, one after the other. An image that gave away more than I wanted, a cluster that complained 137 times a day, a firewall log that only ever recorded the same noise, a header that took five attempts, a ventilation unit that set off my alarms, and two questions I should have asked myself much earlier. None of it was dramatic. All of it was worth knowing. I worked through the evening together with Claude Code, which did the digging while I did the deciding.
Before you start: The sections below cover different topics that do not necessarily belong together. What connects them is simply that they all went through my head on the same evening. Feel free to jump straight to the one that interests you.
Table of Contents
🔄 It started harmlessly: Hugo 0.170.1
Hugo does not come from a distribution package on my machine. It is a single binary in a tools folder next to the site, pinned to one version. A weekly report tells me when there is a newer one. This time it said: 0.169.0 → 0.170.1.
My rule for build tools: build before, build after, compare every file. The result was reassuring. Apart from the generator meta tag on the pages that carry it, the two builds were identical. The search index differed by a few hundred bytes, but it does that between two runs of the same version as well. One thing worth knowing: since 0.170.0 there is no separate “extended” edition any more, only one Hugo binary. That only matters if you use Sass, which I do not.
Lesson: A version jump of a build tool is a test, not an act of faith. Comparing two builds file by file takes a minute and replaces a lot of hoping.
🕵️ The thumbnail that showed more than the photo
While I was at it, I asked a question I had never asked: where exactly does Hugo remove image metadata? The honest answer: nowhere. Hugo copies original images unchanged. Only the resized variants it generates itself come out clean. At some point I had stripped all old images by hand, and new ones were cleaned whenever they were converted. That is a habit, not a guarantee.
So I checked all 1,418 images. No GPS, no camera serial numbers. But two of the preview images for social media contained an embedded EXIF thumbnail of the uncropped original photo. The visible image showed a cropped rack. The tiny thumbnail inside the file showed more of the room around it. Nobody sees that in a browser. Anyone who looks at the file with an EXIF viewer does.
The fix has two parts. The two images were cleaned and redeployed straight away. And the build now runs a small script before Hugo, every time: it removes EXIF, GPS, XMP, IPTC and comments from every image in the source folders. It does this losslessly and keeps the colour profiles, and any file it cleans shows up as a change in Git, so nothing happens silently.
Lesson: Cropping a photo does not crop its metadata. If your pipeline does not strip it, assume it is still there.
🎢 A header that wouldn’t behave
The blog had just got a new roller coaster banner in the top right: a coaster whose track is a glowing blue network cable, ending in a network plug. I liked the idea. The execution took the rest of the evening. And honestly, I enjoyed every minute of it: being creative and trying to think like a graphic designer is my kind of fun. When I am working on something visual, with images and web design, I easily get into a real flow and lose track of time.
The first versions were AI images full of detail: the domain as lettering, binary digits, a train with several cars and me in it. At full size they looked great. At 112 pixels high, next to the big site title, they were a mess. My face was 15 pixels tall, the digits were orange crumbs, and the domain stood there twice: once in the title, once in the image.
What helped in the end was not taste but a calculator. The title was 1,047 pixels wide, which left the banner 330 pixels on a laptop. So the title got shorter. Only the domain is large now, and “Homelab, Linux & DevSecOps” moved into the line below. That freed up 700 to 870 pixels, enough for a banner in a 6:1 format. The image lost the person and the lettering. Text belongs in the HTML, where it stays sharp and readable for screen readers.
Two technical details I enjoyed:
- Transparency from a black background. AI images of glowing objects come on pure black. Treating the brightest colour channel of each pixel as its opacity and dividing the colour by it turns black into transparency while the glow survives. Laid back on black, the result differs from the original by less than one percent.
- A favicon from the site’s own font. The old favicon was a 32-pixel image scaled up to 512. The new one is “RM” in Oswald Bold, the heading font of the site, with the letters converted to vector paths. The SVG needs no font and stays sharp at any size.
And the last piece came a little later. The wide banner was right, but something was missing. The train was empty. A roller coaster without passengers is just a track. Now Tux cheers in the front car, and two parcels labelled .deb ride behind him into the plug. That is apt upgrade, literally.
Lesson: Measure the space before you design for it. And a picture needs a joke, not just a theme.
🔊 A noisy cluster and a firewall that only logged noise
Next I went through the logs of my three servers, looking for anything that produces too many lines or hints at a configuration error. Two findings stood out.
The cluster. On my two Nextcloud servers, which run on K3s, kube-state-metrics wrote the same warning 137 times a day: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice. The exporter was still told to watch the old Endpoints resource. No alert rule and no dashboard used those metrics, so the fix was one word in the configuration: endpointslices instead of endpoints. Rolled out on both hosts, and the warning count dropped to zero.
The firewall. The drop log of nftables is rate-limited to two lines per minute, which makes 2,880 per day at most. It sat at that limit every single day. My first reading was “late ACK packets of closed connections”. That was wrong. I had mistaken the ACK= field of the log line, which is the acknowledgement number, for the ACK flag. Counted properly, 2,577 of 2,629 lines in 24 hours were plain SYN packets, so new connection attempts from the internet to closed ports. Background noise from scanners that stay below my port-scan limit.
The firewall did everything right. But the log was useless: always full of the same noise, and the rare interesting packet was sampled away. The fix is one rule before the log rule:
tcp flags & (fin|syn|rst|ack) == syn counter drop comment "scan noise: counted, not logged"
The same packets are dropped as before. Port-scan bans and fail2ban are untouched, since none of my jails reads this log. But the noise now ends up in a counter instead of the journal, and the remaining unusual drops are logged in full.
Lesson: Count first, then interpret. My first explanation sounded plausible, and it was wrong.
📮 The mailbox key
Between all this I asked a question I had never fully thought through: how exactly does the deploy key work that uploads the site to the server? The answer turned out to be a nice piece of design, so here it is.
The deploy key is a normal SSH key pair, used only for this one job. On the server, its line in authorized_keys starts with restrict,command="/usr/local/sbin/blog-deploy". No matter what the client asks for, the server runs only that one script. The script checks the requested action and allows exactly three things: rsync into the upload folder, activate a new version, and roll back. restrict forbids terminals and every kind of forwarding. Ansible manages the key with exclusive, so any other key someone adds for that user disappears at the next playbook run.
The deploy itself follows the classic release pattern. The upload lands in a separate folder, the server turns it into a new release directory with hard links, so unchanged files take no extra space, and then switches a current symlink in one atomic step. Visitors only ever see the old or the new version, never a half-uploaded one. The last five releases stay, and a rollback takes one second.
Lesson: A stolen deploy key should be a mailbox key. It can drop something in, but it cannot open the door.
💾 GitHub is not my backup
The last question of the evening: if my machine at home died tonight, would GitHub be enough? No, and on purpose. The code, the playbooks, the posts and the theme are on GitHub. The vault passwords that decrypt the secrets are not. Neither are the real inventories with addresses, the SSH keys or the internal documentation. All of that lives only on my home server, and that is where it belongs.
It is backed up anyway: by ZFS snapshots and by a nightly backup of my home directory to a separate backup PC. What is neither on GitHub nor in a backup can be regenerated: the built site and the downloaded build tools.
Lesson: GitHub is a place where code lives, not a backup. The things that matter most in a recovery are exactly the ones you must never push.
🌬️ A ventilation unit tests my monitoring
The new filters for our ventilation unit had arrived. Changing them was not on the plan for the evening, but after 8 pm I did it anyway, quicker than expected. It is a Helios unit with a network connection: through the LAN I can set, for example, which fan level it runs at what times of day and how much air it exchanges with the outside.
(Why a ventilation unit at all? Three reasons, in ascending order of importance. One: I rarely have to open a window to air the rooms. Two: the unit swaps only the air, not the heat. The built-in heat exchanger pulls the warmth out of the stale air before it leaves and hands it to the fresh air coming in. So especially now in winter, the warmth stays in the house. Some of our neighbours, on the other hand, keep the heating in the basement running diligently, just so the warmth can then escape through the tilted window for what feels like hours. Three: we live right next to a field, so far fewer crawly things find their way into the house. The spiders have filed a complaint.)
A few days earlier I had added the unit to my Zabbix monitoring, which pings every device in the home network. What I had not thought about: as soon as I open the unit, it switches itself off for safety, and the electronics go down with it, network stack included. My inbox noticed before I did:
Problem started at 20:28:37
Problem name: ICMP Ping: Unavailable by ICMP ping
Host: Helios easyControls
Severity: High
Resolved in 7m 0s at 20:35:37
Seven minutes, exactly the time the unit stood open for the filter change. Nobody planned a test alarm, but there it was, and it proved that the alerting works end to end, from the missing ping to the mail.
Lesson: The best alarm test is the one you trigger by accident. Next time I will set a maintenance window in Zabbix first. 😉
🔜 What comes next: goodbye Alexa? Not yet
One more construction site turned up along the way, and it is too big to fit here. Our Echo Dot kept complaining about poor connection quality, although an access point is close by. On closer inspection:
- it lives in the guest network, which is why my server could not see it at all
- its signal arrives at only −74 dBm
- the Wi-Fi time switch turns the network off at night, which an always-online device does not like
- the router’s DNS filter, a recently added built-in ad blocker, also applies to the guest network and may block some of the addresses the Echo calls home to
There is a wall in between, but as the crow flies the access point is close. Which of these is the real cause is not settled yet.
That raised the bigger question: do we need an Alexa at all? We mostly want music, and the occasional “turn it down”. I would have liked to do this with open source. My must-haves, though, are clear: a network port, voice control and no DIY hardware. Here is what I looked at:
- Raspberry Pi with a HiFiBerry board and moOde audio. Fully open source, network cable, music from my own NAS. But it is exactly the tinkering box I do not want in the living room, and it has no voice control.
- A Bluesound Pulse Flex 2i. A proper speaker with the network port built in, which an open-source music server at home such as Music Assistant can drive. But it has no microphone.
- A Sonos One or Era 100. Network port built in (the Era via an adapter), microphones built in, great sound. The catch: Sonos’ own local voice control only speaks English and French. In German, a Sonos listens via Alexa, and I would be back where I started.
- Home Assistant Voice Preview Edition as an open-source voice device next to a network speaker. That works in German and can recognise speech locally on my server. But it means two devices, the voice part only has Wi-Fi, and song titles are noticeably harder for it than for Alexa.
To be fair, all of this was only a quick side trip between other things, and so far I have only scratched the surface. Right now no single device does all three things without Amazon or Google. So the open-source speaker stays a project in the back of my mind, and it will get its own post once the market or my patience moves.
Until then I am taking the pragmatic route: I am working on moving our Echo Dot (5th generation) from the classic Alexa to Alexa+. It is included with Prime, the device should be compatible, and in Germany it still comes by invitation. Fixing the night-time Wi-Fi switch for that one device is next on the list.
🎯 The lessons of one evening
- Compare builds file by file before trusting a new tool version.
- Cropping does not crop metadata. Strip it in the pipeline, not by habit.
- Measure the space before you design for it.
- Count first, then interpret. Log what is unusual, count what is noise.
- Give a deploy key exactly one job.
- Keep secrets out of Git, and back them up somewhere else.
- The best alarm test is the one you trigger by accident.
None of these took more than an hour. Together they were a whole evening off. That is homelab life for you.
