<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Observability – apt-upgrade.me – Homelab, Linux &amp; DevSecOps</title><link>https://www.apt-upgrade.me/tag/observability/</link><description>One nerd's homelab: Linux, Kubernetes, security, and the occasional disaster.</description><language>en-GB</language><lastBuildDate>Mon, 21 Sep 2026 20:00:00 +0200</lastBuildDate><atom:link href="https://www.apt-upgrade.me/tag/observability/index.xml" rel="self" type="application/rss+xml"/><item><title>Bare Metal to K3s: A Nextcloud Migration Battle Log</title><link>https://www.apt-upgrade.me/2026/09/21/bare-metal-to-k3s-a-nextcloud-migration-battle-log/</link><pubDate>Mon, 21 Sep 2026 20:00:00 +0200</pubDate><guid>https://www.apt-upgrade.me/2026/09/21/bare-metal-to-k3s-a-nextcloud-migration-battle-log/</guid><description>&lt;p&gt;&lt;strong&gt;In short:&lt;/strong&gt; A production Nextcloud moved from a native bare-metal installation to K3s. On paper: back up, restore, switch DNS. In practice: major version jumps, an antivirus socket, Let&amp;rsquo;s Encrypt rate limits, a checkbox that lied and a database spring clean. Here is the war diary – including the honest comparison between native and container.&lt;/p&gt;&#10;&lt;p&gt;Today I moved a production &lt;a href="https://nextcloud.com/"&gt;Nextcloud&lt;/a&gt; from a cosy &lt;strong&gt;native bare-metal install&lt;/strong&gt; to a shiny &lt;strong&gt;K3s/container&lt;/strong&gt; stack. On paper: pull a backup, restore it, flip DNS, done. In practice: a series of small landmines, each invisible until you step on it. Here&amp;rsquo;s the battle log. 🪖&lt;/p&gt;</description></item><item><title>My 5-Year IT Learning Roadmap — From Sysadmin to Infrastructure Engineer</title><link>https://www.apt-upgrade.me/2026/08/27/my-5-year-it-learning-roadmap-from-sysadmin-to-infrastructure-engineer/</link><pubDate>Thu, 27 Aug 2026 20:00:00 +0200</pubDate><guid>https://www.apt-upgrade.me/2026/08/27/my-5-year-it-learning-roadmap-from-sysadmin-to-infrastructure-engineer/</guid><description>&lt;p&gt;&lt;strong&gt;In short:&lt;/strong&gt; Where do I want to go in IT over the next five years? My plan: from sysadmin to infrastructure engineer, someone who understands systems instead of just operating them. This post describes how I want to learn – with a 20/30/30/20 rule and a learning cycle from understanding to explaining –, my realistic time budget and the roadmap year by year, from Linux and &lt;a href="https://graylog.org/"&gt;Graylog&lt;/a&gt; to Kubernetes and AI.&lt;/p&gt;</description></item><item><title>Staring Into the Void: A Love Letter to Log Files</title><link>https://www.apt-upgrade.me/2025/12/19/staring-into-the-void-a-love-letter-to-log-files/</link><pubDate>Fri, 19 Dec 2025 20:00:00 +0100</pubDate><guid>https://www.apt-upgrade.me/2025/12/19/staring-into-the-void-a-love-letter-to-log-files/</guid><description>&lt;p&gt;&lt;strong&gt;In short:&lt;/strong&gt; Logs are where systems tell the truth – loud, sometimes ugly, but indispensable when something goes wrong. Today I installed &lt;a href="https://graylog.org/"&gt;Graylog&lt;/a&gt; and &lt;a href="https://fluentbit.io/"&gt;Fluent Bit&lt;/a&gt; on my server, as a learning environment close to what you would run in production. The post explains why logging matters so much and shows my Fluent Bit configuration.&lt;/p&gt;&#10;&lt;article class="wp-log-nerd-post"&gt;If you’ve ever stared at a terminal full of log messages, you know: logs are where systems&#10;tell the truth. They are noisy, sometimes ugly, but absolutely essential when things go wrong. Today I installed &lt;strong&gt;Graylog&lt;/strong&gt; and &lt;strong&gt;Fluent Bit&lt;/strong&gt; on my server. Not only&#10;because I work with these tools professionally, but also because I will increasingly use them in the future, as my homelab is meant for&#10;experimenting, learning, and keeping my brain busy.&lt;h2&gt;Why Logging Matters&lt;/h2&gt;&lt;p&gt;Logging is one of the most fundamental building blocks of reliable systems. Metrics tell you&#10;&lt;em&gt;that&lt;/em&gt; something is wrong, logs tell you &lt;em&gt;why&lt;/em&gt;. Centralised logging becomes crucial&#10;once you operate more than a single machine.&lt;/p&gt;&lt;h2&gt;Fluent Bit, Graylog &amp;amp; Sidecar&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Fluent Bit&lt;/strong&gt; acts as the client component on the server. Its job is simple:&#10;collect logs locally and ship them to Graylog.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Graylog&lt;/strong&gt; provides the GUI and backend where logs are stored, processed, searched,&#10;and used for alerting. It can easily be used to collect log messages from several thousand&#10;servers and analyse them centrally.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Sidecar&lt;/strong&gt; is an additional service that allows centralised configuration of the&#10;Fluent Bit agent directly from Graylog, making rollouts and changes much more convenient.&lt;/p&gt;&lt;h2&gt;Memory Tuning &amp;amp; First Lessons Learned&lt;/h2&gt;&lt;p&gt;Functionally everything worked quite well, but I had to significantly reduce the RAM&#10;requirements. Interestingly, Graylog immediately grabbed swap space — even though the server&#10;has a total of 32 GB RAM available.&lt;/p&gt;&lt;p&gt;These Java-related memory topics are still new to me, so that’s something I’ll have to dig&#10;deeper into. For the beginning, having a &lt;strong&gt;Grafana dashboard&lt;/strong&gt; was extremely&#10;helpful to monitor CPU and RAM usage in detail.&lt;/p&gt;&lt;p&gt;To limit heap usage, I added the following entry to&#10;&lt;code&gt;/etc/graylog/datanode/datanode.conf&lt;/code&gt;:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;opensearch_heap = 2g&lt;/code&gt;&lt;/pre&gt;&lt;h2&gt;Fluent Bit Configuration Example&lt;/h2&gt;&lt;p&gt;The following is the current &lt;strong&gt;Fluent Bit&lt;/strong&gt; configuration running on my Ubuntu/Debian system. It collects logs from multiple local sources (systemd journal, syslog, auth logs, and kernel logs) and forwards everything to Graylog using the &lt;em&gt;GELF&lt;/em&gt; output over TCP.&lt;/p&gt;&lt;p&gt;This setup works well for a homelab environment while still being close to what you would run in production.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;&amp;#10;############################################&amp;#10;# Fluent Bit – Ubuntu / Debian&amp;#10;############################################&amp;#10;&amp;#10;[SERVICE]&amp;#10; Flush 30&amp;#10; Daemon Off&amp;#10; Log_Level info&amp;#10; storage.type memory&amp;#10;&amp;#10;&amp;#10;###################################################################&amp;#10;# INPUTS&amp;#10;###################################################################&amp;#10;&amp;#10;# Systemd Journal (primary input source)&amp;#10;[INPUT]&amp;#10; Name systemd&amp;#10; Tag journal.*&amp;#10; Read_From_Tail On&amp;#10;&amp;#10;&amp;#10;# /var/log/syslog&amp;#10;[INPUT]&amp;#10; Name tail&amp;#10; Path /var/log/syslog&amp;#10; Tag syslog.*&amp;#10; DB /var/log/fluent-bit/db_syslog.db&amp;#10; Mem_Buf_Limit 5MB&amp;#10; Skip_Long_Lines On&amp;#10;&amp;#10;&amp;#10;# Authentication Logs&amp;#10;[INPUT]&amp;#10; Name tail&amp;#10; Path /var/log/auth.log&amp;#10; Tag auth.*&amp;#10;&amp;#10;&amp;#10;# Kernel Log&amp;#10;[INPUT]&amp;#10; Name tail&amp;#10; Path /var/log/kern.log&amp;#10; Tag kernel.*&amp;#10;&amp;#10;&amp;#10;###################################################################&amp;#10;# OUTPUT&amp;#10;###################################################################&amp;#10;&amp;#10;[OUTPUT]&amp;#10; Name gelf&amp;#10; Match *&amp;#10; Host 192.168.10.x&amp;#10; Port 12201&amp;#10; Mode udp&amp;#10; Gelf_Short_Message_Key log&amp;#10; Gelf_Host_Key host&amp;#10;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;/p&gt;&lt;p&gt;With this configuration, Fluent Bit reliably ships all relevant system logs to Graylog,&#10;where they can be indexed, searched, and used for alerting.&lt;/p&gt;&lt;h2&gt;Final Thoughts&lt;/h2&gt;&lt;p&gt;Centralised logging is a powerful tool — even (or especially) in a homelab. It helps build&#10;intuition for production systems and makes troubleshooting far less painful when things&#10;eventually break.&lt;/p&gt;&lt;/article&gt;</description></item><item><title>How to Summon an Observability Beast: Installing Icinga 2 + IcingaDB + Web + Director on Debian</title><link>https://www.apt-upgrade.me/2025/12/09/how-to-summon-an-observability-beast-installing-icinga-2-icingadb-web-director-on-debian/</link><pubDate>Tue, 09 Dec 2025 20:00:00 +0100</pubDate><guid>https://www.apt-upgrade.me/2025/12/09/how-to-summon-an-observability-beast-installing-icinga-2-icingadb-web-director-on-debian/</guid><description>&lt;p&gt;&lt;em&gt;A nerdy homelab story featuring Redis, too many databases, and not enough sleep.&lt;/em&gt;&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;p&gt;&lt;strong&gt;In short:&lt;/strong&gt; Congratulations, traveller: anyone who wants to monitor their homelab like a responsible adult ends up with the full Icinga stack sooner or later. This post shows step by step how to put together Icinga 2, Redis, the database, IcingaDB, Icinga Web 2, the API and the Director on &lt;a href="https://www.debian.org/"&gt;Debian&lt;/a&gt; – with a pinch of sarcasm and too little sleep.&lt;/p&gt;</description></item></channel></rss>