🌐 Also available in: Deutsch · Français · Español
A blog and two Nextcloud servers for local organisations: nothing on them is worth a targeted attack, and that is exactly why they get attacked. Not by someone who wants my data, but by scanners that try every IP address on the internet, every hour, with the same handful of tricks. Most of them give up after the first locked door. The question is what happens with the ones that don’t.
The answer has a name: Defense in Depth. No single measure has to be perfect, because behind every layer there is another one. This post collects all the layers my servers have by now – and then does something slightly uncomfortable: I asked my AI co-admin (Claude) to read everything I have published about this setup and think like an attacker. Where would it start?
🧅 Why one wall is not enough
Every security measure has a failure mode. A firewall rule gets a typo. A service that is supposed to ban attackers silently stops – mine did, for two days, and nobody noticed. A plugin ships a vulnerability on a Tuesday and the fix arrives on Thursday. If one of these is the only thing between the internet and your data, you are one bad day away from a breach.
Defense in Depth turns that around: assume each layer will fail at some point, and make sure the next one still holds. It also has a second, less obvious benefit – an attacker who gets through one layer has to make noise to get through the next, and noise is something you can detect.
🛡️ The layers
1. The network edge
The firewall (nftables) drops everything that is not explicitly allowed: web traffic on 80/443 and SSH on a non-standard port. Monitoring, databases and internal ports answer only on localhost or inside the cluster. SSH additionally accepts new connections only from IP ranges of my own country – which does not stop anyone determined (a rented server in the right country is cheap), but it removes about 98 % of the background noise, so the remaining attempts actually stand out.
2. SSH
Keys only, no passwords – pinned in a config snippet that sorts before anything a cloud image drops in, after I found a provider image quietly re-enabling password logins. On top, sshd runs in a systemd sandbox that every session inherits: no loading of kernel modules, no changing kernel parameters. Root can still do that deliberately via systemd-run, so this is a hurdle, not a wall – but the classic “log in, load a rootkit module” sequence fails, and module loading is recorded by the audit log either way.
3. Automatic blocking
fail2ban reads the SSH log and the web server’s access log and bans addresses that guess passwords, scan for well-known files or hammer the login pages. Repeat offenders get banned much longer. The bans go straight into the firewall, and a watchdog restarts fail2ban if it ever dies again.
4. The web server
TLS only, with strict security headers. Login URLs are rate-limited. Paths that attackers love – configuration files, .env, .git, WordPress’ XML-RPC – are blocked before they ever reach PHP, and WordPress no longer lists its user names through the REST API.
5. Containers
The applications run in Kubernetes (K3s). NetworkPolicies decide which pod may talk to which – a compromised WordPress cannot simply connect to anything it likes. An admission policy only allows container images from approved registries, service account tokens are not mounted where they are not needed, and SELinux is enforcing, so a container can only write to the directories explicitly labelled for it.
6. The applications themselves
WordPress updates core, plugins, themes and translations every night – with a database backup first and a check afterwards that the site still answers. Unused plugins and themes are deleted, not just deactivated, because deactivated code is still reachable. Nextcloud enforces a password policy and slows down brute-force attempts; administrator accounts use two-factor authentication.
7. Detection
Prevention fails eventually; detection is how you find out. AIDE checks every night whether system files changed, rkhunter looks for rootkits, ClamAV scans uploaded files, auditd watches identity files, sudo and the cluster’s secrets, and a watchdog (Monit) mails me when a service, the firewall, a certificate or the backup is not in the state it should be. All of them are silent while everything is fine – so every mail means something.
8. Recovery
The last layer is the one you hope never to need. Backups are pulled by a machine at home, not pushed by the servers – a compromised server has no credentials to reach them – and kept with 30 days of history, so even something that went unnoticed for a week can be rolled back.
9. The process
Everything above is configured by Ansible from a Git repository – which makes the repository part of the attack surface. Secrets live only in an encrypted vault that never touches Git. Every commit runs through a linter and a secret scanner. Versions are pinned, updates arrive as reviewed pull requests, and every change runs as a dry run against the real servers before it runs for real.
🕵️ Thinking like an attacker
Here is the uncomfortable part. My Ansible code is public, and so are several posts about how these servers are built. So I asked Claude: you know all of this – where would you start?
The answer was not “the firewall” or “SSH”. It was much more mundane:
- The applications. A plugin or app with a fresh vulnerability is by far the most likely way in. Nightly updates shrink that window; they cannot close it.
- The people. One of the Nextcloud servers has more than a hundred accounts. One reused password or one convincing phishing mail beats every technical layer above – no exploit needed.
- The supply chain. Whoever controls my Git account or my admin machine does not need to attack the servers at all: I would deploy their code myself.
- Forgotten things. Accounts nobody uses any more, shared links without an expiry date, old settings that still lie dormant in a database.
The lesson: once the technical basics are in place, the weak points move to where people and processes are.
🔑 Public design, secret keys
Which raises the obvious question: isn’t it careless to publish all of this? Cryptographers answered that in 1883. Kerckhoffs’s principle says that a system must stay secure even if everything about it is public – except the key. That is how every encryption you use today works: the algorithms are published, studied and attacked by experts all over the world, and they stay secure because the key stays secret.
My servers follow the same idea. Knowing that SSH accepts only keys, on which port, from which country, behind which firewall rules does not get anyone a single step closer without the private key. Knowing that the configuration lives in an Ansible vault does not help without the vault password. Knowing that administrators use two-factor authentication does not help without the phone. The design is public; the keys never leave the places they belong to.
The opposite – security that only works as long as nobody knows how it works – is fragile: one leaked document, one curious insider, and it is gone. Hiding details can still cost an attacker time, but only as an extra layer, never as the foundation. The one thing that must not be published is the list of gaps that are not closed yet.
And one more thing, for anyone reading this with the wrong intentions: not every layer is described in this post. A few of the protections and tripwires on these servers are deliberately left out – you will only find out about them by setting them off.
🔧 What I changed after that conversation
- Two-factor authentication for every administrator account on both Nextcloud servers and on the blog – with backup codes printed, and the knowledge that SSH access can always switch it off again if a phone gets lost.
- A password policy on both Nextcloud servers: at least 12 characters, no very common passwords, and no passwords that appear in known data leaks. That last check uses k-anonymity: only the first five characters of the password’s hash leave the server.
- Inactive accounts disabled: everyone who had not logged in for more than a year, plus accounts that never logged in at all – thirty in total, including an unused second administrator. Their data stays; the accounts can be switched back on in seconds.
- Public upload links expire instead of staying open forever.
- Three places where a password could have ended up in a log – only with a diff or an error message, but still – are now hidden.
And one honest mistake. Enabling the password policy on the bigger Nextcloud woke up a setting left over from the old server: passwords expire after 300 days. For a few minutes every older password counted as expired – including my own. The setting is now pinned explicitly, and I learned to read an app’s existing configuration before switching it on.
🤝 The organisational layer
Not everything can be solved with a config file:
- Tell the users. A short note on password managers, two-factor authentication and phishing (“Nextcloud will never ask for your password by e-mail”) protects more than any server setting.
- Plan for being unavailable. Who can step in if I can’t? Where are the recovery codes and the vault password kept – on paper, in a sealed envelope, not only in my head?
- Look regularly. A short monthly check: new admin accounts, open share links, anything odd in the audit and ban logs.
🎯 Conclusion
None of these layers is impressive on its own, and every one of them has a way around it. Together they turn “one mistake and it’s over” into “several mistakes, each of them noticed”. That is the whole idea of Defense in Depth – and the attacker’s view showed me that the next layers are not more firewall rules, but accounts, habits and the machines I administer from.
