Eight hours for a backup: how an SMR disk, rdiff-backup and Hugo slowed each other down
My backup PC recently ran for more than eight hours, although hardly any data had changed. The trail led to a hard disk with roof shingles, a tool that commits every file to disk on its own, and a blog that rewrites 70,000 files on every build.

🌐 Also available in: Deutsch · Français · Español · Italiano · Português · Nederlands · Polski
In short: My backup PC recently needed eight hours and 18 minutes for one run – for data that had hardly changed. There was not one cause but three that amplified each other: an SMR hard disk that collapses under many small writes, rdiff-backup, which forces every file onto the disk individually, and my Hugo blog, which rewrites all of its roughly 70,000 files on every build. Now I back up the blog as a single tar.gz archive (38 seconds instead of up to four hours), rdiff-backup runs with --no-fsync, and anything that can be regenerated stays out.
Table of Contents
🐢 The symptom: eight hours for almost nothing
My backup PC is an old machine that I switch on when it is time to back up: it fetches everything and then shuts itself down again. It backs up four things one after the other: Nextcloud, the blog, the shares on the NAS and my home directory on the workstation. The tool for all of it: rdiff-backup. It keeps a current mirror and, for every changed file, puts an “increment” file with the old state next to it – that way you can get back any earlier version.
For a long time this was fast. Then the statistics looked like this:
| Run | Nextcloud | Blog | NAS | Home | Total |
|---|---|---|---|---|---|
| 12 Nov | 15 min | 0 min | 1 min | 39 min | 56 min |
| 17 Nov | 55 min | 169 min | 6 min | 167 min | 6 h 37 min |
| 19 Nov | 4 min | 110 min | 186 min | 198 min | 8 h 18 min |
Same data, same hardware – sometimes just under an hour, sometimes more than eight. None of the runs needed noticeably more space. So it was not the amount of data, but something else.
🏠 Finding 1: the hard disk has a shingled roof
First look at the hardware: the backup disk is an 8 TB Seagate Barracuda. This model uses SMR (Shingled Magnetic Recording): the tracks overlap like roof shingles. That fits more data on the platter, but there is a catch. Changing one track also overwrites the neighbouring track lying half on top of it – so the disk has to read and rewrite whole areas (bands).
So that you do not notice this in everyday use, the disk has a small, conventionally written buffer area. New data lands there first, and the disk moves it into the shingled bands during quiet periods. Large files in one piece are no problem. Many small writes, on the other hand, fill the buffer quickly – and then write performance drops from over 100 MB/s to a fraction of that, because the disk is accepting and reshuffling at the same time.
On a disk for films or photos you never notice this. On a backup made of hundreds of thousands of small files you notice it very clearly.
💾 Finding 2: rdiff-backup commits every file individually
Second look: how rdiff-backup writes. For every file it creates or replaces, it calls fsync – that forces the operating system to write the file physically to the disk at once instead of collecting it in memory. It is well meant: if the machine crashes in the middle of a run, every file already backed up really is on the disk.
On an SMR disk, however, it is the worst case: every file is a single small synchronous write. In the runs I measured around 10 to 15 files per second. For a few hundred changed files that does not matter. For 200,000 it is four hours.
🔁 Finding 3: Hugo rewrites everything on every build
And where do 200,000 changed files come from when I change just one post? From the switch to Hugo. Since then my blog has been a collection of static files: around 150 posts in 8 languages, plus tag pages, categories, pagination, feeds, the search index and images in several sizes – about 70,000 files in total.
Hugo rebuilds the whole site on every deploy and puts it in place completely anew, including the pages that have not changed at all. For rdiff-backup, after every deploy almost every file has changed: a new mirror copy plus an increment file with the old state. Depending on how many deploys had happened since the last run, that meant 59,000 to 214,000 new files per run. At 10 to 15 files per second: 110 to 234 minutes for the blog alone.
Under WordPress this was no problem – everything lived in a database, and the backup was a single dump.
🧩 The same pattern in two more places
Once the principle was clear, it showed up elsewhere too:
- NAS (186 minutes on 19 Nov): the day before, I had tidied up my file share and moved many folders. For rdiff-backup a moved file is a deleted one plus a new one – two writes per file. A one-off effect; on 20 Nov it was back to 54 seconds.
- Home directory (up to 198 minutes): of roughly 420,000 files, about 360,000 are things that can be regenerated at any time: container images, caches, Python environments,
node_modules, the built website, work files from CD ripping. They change constantly and are worthless in an emergency.
🛠️ What I changed
1. The blog is backed up as one archive. Instead of mirroring 70,000 individual files with rdiff-backup, the source machine packs the current state with tar, compresses it with gzip -1 and sends it to the backup PC as one data stream. Exactly one file lands there. That is the ideal case for any hard disk, and especially for an SMR disk.
ssh source "tar -C /storage --numeric-owner -cf - blog | gzip -1" > blog_2026-11-20_0814.tar.gz.part
tar -tzf blog_2026-11-20_0814.tar.gz.part | wc -l # check first, only then rename
Beforehand the script checks that the last backup on the source machine succeeded, and afterwards that the archive is readable and contains the home page. Only then does it get its final name. gzip -1 is enough: the images are already compressed, a higher level only costs time.
Retention: all archives of the last 30 days, plus the first archive of each month for one year. That is coarser than rdiff-backup’s version history, but more than enough for a blog whose source lives in Git anyway. The old rdiff history stays in place for the time before.
2. rdiff-backup without fsync for every file. For Nextcloud, the NAS and the home directory, rdiff-backup now runs with --no-fsync. At the very end of the whole run there is a single sync that writes everything to the disk. The risk is manageable: if the backup PC crashes mid-run, that one run is incomplete – but that happens with fsync too, and rdiff-backup cleans up an aborted run by itself at the next start (regress).
3. What can be regenerated stays out. The home directory now has an exclusion list: container storage, .cache, .venvs, node_modules, .npm, the built website and work folders. All of it can be restored with one command.
📊 The result
The first run with all changes on 20 Nov:
| Step | before (19 Nov) | after (20 Nov) |
|---|---|---|
| Nextcloud (810,000 files) | 4 min 26 s | 2 min 29 s |
| Blog | 110 min | 38 s (one archive, 888 MB, 113,002 entries) |
| NAS | 186 min | 54 s |
| Home | 198 min | 101 min 22 s |
| Total | 8 h 18 min | 1 h 45 min |
For the home directory the first run is a special case: rdiff-backup records the newly excluded files once as “deleted” – hundreds of thousands of increment files one last time. From the next run on they are gone entirely.
💡 Would an SSD have solved this too?
I asked myself the same question. An SSD (or a hard disk with classic CMR) would be much faster with many small files, no doubt. But it would only have eased the symptom: 200,000 increment files per run for a website that can be rebuilt from Git at any time are a waste on an SSD too. First reduce the number of files, then think about hardware. Switching to a CMR disk, or to Borg, which packs small files into large chunks, remains an option for later.
🤝 How the analysis went
I did the troubleshooting with Claude Code: I just said “the backup takes eight hours, see whether you can optimise something” and added a few ideas (a packed archive, buffering more in RAM, replacing rdiff-backup, checking the disk). Claude evaluated the run times from the logs of the last weeks, identified the disk model as SMR, counted the increment files per run and made the connection to the Hugo build. It wrote the scripts; when I said “let’s do it again”, it aborted the running backup and restarted it with the new version.
🎯 Conclusion: not the amount, the count
- Backup time depends on the number of files, not on the gigabytes. 888 MB in one piece: 38 seconds. The same data as 200,000 individual files: four hours.
- SMR disks are a poor choice for backups made of many small files. Check the data sheet before buying – “SMR” is often only in the small print.
- Back up static websites as an archive, not file by file. The source is in Git anyway.
- What can be regenerated does not belong in the backup. Caches and container images cost time and do not help in an emergency.
- Measure first, then buy. A new disk would only have made the problem smaller.
📚 Related posts:
