One Year Later: I Finally Open-Sourced My Homelab K3s Stack

In a few of my recent posts I mentioned that I was trying to figure out how to publish my homelab automation code *without accidentally open-sourcing my entire infrastructure along with it* 😅
Turns out that’s a surprisingly non-trivial problem when your Ansible inventory looks like a cyberpunk treasure chest full of vaults, secrets and host-specific variables. 🔐🐉

After a bit of trial, error, caffeine and questionable life choices… I finally found a setup that works beautifully:

👉 two separate git remotes.

A **private self-hosted Gitea instance** keeps the *real* repository — including encrypted secrets, vault files and all the spicy internal bits.
Meanwhile a **public GitHub repo** only receives the clean automation code, with sensitive files excluded via `.gitignore` and replaced by friendly `.example` templates. 🧼✨

Anything secret is encrypted using **Ansible Vault (AES-256)** anyway — so even if something somehow slips through, all an attacker gets is encrypted gobbledygook. 🧠💾

📦 Repo:
👉 github.com/aptupgrademe/www_k3s

The project provisions either **Nextcloud** or **WordPress** on a fresh **single-node K3s cluster** running on **AlmaLinux 9**.

One command.
Zero click-ops.
Maximum nerd energy. ⚡🤓

🧠 The Idea

Run:

ansible-playbook blog.yml --limit <host> --ask-vault-pass

…against a completely fresh VPS and watch it transform itself into a fully hardened, TLS-enabled, monitored WordPress or Nextcloud server like some kind of infrastructure Pokémon evolution. 🐛➡️🐉

No manual SSH fiddling.
No mystery setup steps.
No “TODO: document later” graveyard. 💀

Everything is idempotent — re-run it anytime without nuking your data or summoning configuration drift demons.

⚙️ Stack at a Glance

• 🏗️ K3s — Kubernetes without the enterprise trauma
• 🔀 F5 nginx-ingress — running as DaemonSet with hostNetwork: true
• 🔒 cert-manager — automatic Let’s Encrypt certs because browsers yearn for green locks
• 📝 WordPress 6.9-fpm + nginx sidecar + MariaDB 10.11
• ☁️ Nextcloud 33-fpm + Redis + Collabora CODE for nerd-grade self-hosted office vibes
• 📊 Prometheus + Grafana — because if you can’t graph it, does it even exist?
• 🛡️ iptables — default DROP, portscan detection, ICMP rate limiting, tiny digital moat 🏰
• 🔐 Fail2Ban, auditd, rkhunter, dnf-automatic — paranoia as code™

🚀 What One Playbook Run Actually Does

1. 🔑 Hardens SSH — custom port, key-only auth, tighter configs
2. 🔥 Builds IPv4 + IPv6 firewall rules with portscan detection via xt_recent
3. 🧱 Enables SELinux enforcing with correct container contexts
4. ☸️ Installs K3s, Helm, cert-manager and nginx-ingress
5. 📦 Deploys the full Kubernetes workload stack automatically
6. 📝 Configures WordPress via WP-CLI like a civilized automation goblin
7. ☁️ Configures Nextcloud via occ including Redis, SMTP and trusted domains
8. 📈 Sets up monitoring exporters + Prometheus scraping + Grafana dashboards
9. 🛡️ Deploys Fail2Ban, auditd, rkhunter scans and unattended security updates
10. ☕ Probably consumes less coffee than I did while debugging it

🔐 Secrets & Security

All credentials are encrypted using Ansible Vault (AES-256).
Sensitive files like:

• vault.yml
• vars.yml
• hosts.yml

…never touch the public repository. 🚫🌍

The GitHub repo only contains reusable automation code and `.example` templates:

vars.yml.example
vault.yml.example
hosts.yml.example

Clone → fill in values → encrypt → deploy. ✨

🙃 Things That Were Way Harder Than Expected

• • Let’s Encrypt HTTP-01 challenge returning 301
    Turns out redirecting everything to HTTPS also redirects the ACME challenge. Who could have guessed? 🙃
    Fixed using F5 Mergeable Ingresses (master/minion pattern) so /.well-known/ bypasses redirects.

“`

• • Infinite HTTPS redirect loop
    My first nginx logic basically DDoSed itself philosophically.
    Fixed with a cleaner three-variable redirect pattern.
    “`

“`

• • mysqld_exporter v0.19.0 killed DATA_SOURCE_NAME
    Credentials now need a mounted .my.cnf file.
    Also discovered that 0600 permissions don’t help if the container runs as nobody. 🤡
    “`

“`

• • iptables OUTPUT DROP broke K3s internals
    Because apparently Kubernetes enjoys talking to itself constantly.
    Fixed with explicit ACCEPT rules for pod/service CIDRs.
    “`

“`

• Grafana dashboard 1860 showed “No data”
  Modern Grafana dashboards no longer contain __inputs.
  Solution: tiny Python patcher script talking to the Grafana API like a little automation goblin.
  “`

🧪 Try It Yourself

git clone https://github.com/aptupgrademe/www_k3s.git
cd www_k3s

cp inventory/hosts.yml.example inventory/hosts.yml
cp inventory/host_vars/test/vars.yml.example inventory/host_vars/myhost/vars.yml

# fill in your values, then:
ansible-vault encrypt inventory/host_vars/myhost/vault.yml

ansible-playbook blog.yml --limit myhost --ask-vault-pass

The full quickstart, role overview and variable documentation live in the GitHub README.

PRs, feedback and fellow infrastructure nerds welcome. 🚀🐧☸️