Staring Into the Void: A Love Letter to Log Files

If you’ve ever stared at a terminal full of log messages, you know: logs are where systems
tell the truth. They are noisy, sometimes ugly, but absolutely essential when things go wrong.Today I installed Graylog and Fluent Bit on my server. Not only
because I work with these tools professionally, but also because I will increasingly use them in the future, as my homelab is meant for
experimenting, learning, and keeping my brain busy.

Why Logging Matters

Logging is one of the most fundamental building blocks of reliable systems. Metrics tell you
that something is wrong, logs tell you why. Centralized logging becomes crucial
once you operate more than a single machine.

Fluent Bit, Graylog & Sidecar

Fluent Bit acts as the client component on the server. Its job is simple:
collect logs locally and ship them to Graylog.

Graylog provides the GUI and backend where logs are stored, processed, searched,
and used for alerting. It can easily be used to collect log messages from several thousand
servers and analyze them centrally.

Sidecar is an additional service that allows centralized configuration of the
Fluent Bit agent directly from Graylog, making rollouts and changes much more convenient.

Memory Tuning & First Lessons Learned

Functionally everything worked quite well, but I had to significantly reduce the RAM
requirements. Interestingly, Graylog immediately grabbed swap space — even though the server
has a total of 32 GB RAM available.

These Java-related memory topics are still new to me, so that’s something I’ll have to dig
deeper into. For the beginning, having a Grafana dashboard was extremely
helpful to monitor CPU and RAM usage in detail.

To limit heap usage, I added the following entry to
/etc/graylog/datanode/datanode.conf:

opensearch_heap = 2g

Fluent Bit Configuration Example

The following is the current Fluent Bit configuration running on my Ubuntu/Debian system. It collects logs from multiple local sources (systemd journal, syslog, auth logs, and kernel logs) and forwards everything to Graylog using the GELF output over TCP.

This setup works well for a homelab environment while still being close to what you would run in production.


############################################
# Fluent Bit – Ubuntu / Debian
############################################

[SERVICE]
    Flush        30
    Daemon       Off
    Log_Level    info
    storage.type memory


###################################################################
# INPUTS
###################################################################

# Systemd Journal (primary input source)
[INPUT]
    Name              systemd
    Tag               journal.*
    Read_From_Tail    On


# /var/log/syslog
[INPUT]
    Name              tail
    Path              /var/log/syslog
    Tag               syslog.*
    DB                /var/log/fluent-bit/db_syslog.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On


# Authentication Logs
[INPUT]
    Name              tail
    Path              /var/log/auth.log
    Tag               auth.*


# Kernel Log
[INPUT]
    Name              tail
    Path              /var/log/kern.log
    Tag               kernel.*


###################################################################
# OUTPUT
###################################################################

[OUTPUT]
    Name                    gelf
    Match                   *
    Host                    192.168.10.50
    Port                    12201
    Mode                    udp
    Gelf_Short_Message_Key  log
    Gelf_Host_Key           host

With this configuration, Fluent Bit reliably ships all relevant system logs to Graylog,
where they can be indexed, searched, and used for alerting.

Final Thoughts

Centralized logging is a powerful tool — even (or especially) in a homelab. It helps build
intuition for production systems and makes troubleshooting far less painful when things
eventually break.