OrienteeringI once watched a room full of very capable engineers discover that the backups were there, in the cloud, large enough to look convincing, copied exactly where everyone expected them to be… and completely useless.
They had data in them. Gigabytes of it. They looked like backups.
They just wouldn’t restore.
That’s the bit people skip when they talk about backups. They talk about the file existing. They talk about the schedule running. They talk about the bucket, the retention policy, the dashboard with a comforting amount of green on it.
Then the day comes when someone actually has to restore the thing, and the backup turns out to be more of a strongly held opinion than an operational fact.
So let’s start with the unpleasant sentence.
A backup you’ve never restored is not a backup.
It’s hope, wearing a cron job.
The tool is not the point
I’m not particularly religious about backup tooling.
That might sound odd, because this is the internet and therefore I’m supposed to pick one thing, defend it with unnecessary heat, and then spend the comments discovering how many people have made a different tool part of their personality. But I’ve been around long enough to fall back to rsync more times than any modern tool author would enjoy hearing.
Not because rsync is glamorous. It’s about as glamorous as a spare kettle in the office kitchen. But it’s simple, well known, boring in the right ways, and very good at resumable transfer. When you are shifting data over a link that might wobble, boring starts to look quite attractive.
The point isn’t that everyone should use rsync.
The point is that the features matter more than the name on the binary.
Separate two jobs in your head:
- Backup: turn live data into a restorable artefact.
- Transfer: get that artefact somewhere else without lying about success.
Some tools do both. Plenty only do the first job and leave the second to you. That’s fine, as long as you know which job you are asking each tool to do. If your database dump command produces a file, and rclone, rsync, object-storage sync, SFTP, or a human with a USB stick moves it off the machine, your backup design includes both halves.
Ignore one half and the whole thing quietly becomes theatre.
What the backup half needs to do
The backup job has one main responsibility: capture data in a state you can restore.
That sounds obvious, which usually means it is where the nasty surprises live.
For a static directory, that might be a tarball. For a database, it might be a dump, a native backup command, a filesystem snapshot taken at the right point, or point-in-time recovery logs paired with a base backup. For an application, it may be a bundle of data, config, secrets references, object storage, and enough metadata to put the thing back together.
Whatever you choose, the backup half needs a few properties:
- It must capture a consistent view of the data, or explain exactly why that is impossible.
- It must fail loudly when the capture fails.
- It must produce something you can verify before you ship it away.
- It must be restorable without needing the original machine to still be alive.
- It must support a retention policy you understand.
- It must have a restore command or process that a tired person can follow.
That last one isn’t decoration. The restore path is part of the backup.
If you need the same server, the same magic directory, the same snowflake package version, and Dave’s laptop because Dave was the only one who knew where the key lived, you don’t have a backup. You have a hostage situation!
What the transfer half needs to do
The transfer job looks simpler, which is how it gets away with murder.
Copy the file somewhere else. Done.
Except data moves over networks, and networks are where optimism goes for its character-building exercises. Home broadband drops. Wi-Fi sulks. VPNs expire. Object storage APIs throttle. A route halfway across a continent decides today is interpretive dance day. If the transfer tool can’t handle interruption, resume cleanly, verify what landed, and report failure in a way you will actually notice, it isn’t good enough for anything you care about.
The transfer half needs its own checklist:
- Resumable transfer, especially for large artefacts.
- Verification of the copied object, not merely “the command exited zero”.
- Retries with limits, so transient failure doesn’t become silent success.
- Bandwidth control, because your backup job is not the only thing using the network.
- Atomic or clearly staged writes, so a half-uploaded object doesn’t masquerade as complete.
- Logs you can read without becoming an archaeologist.
This is where simple tools often win. Not because they are clever, but because you can reason about them at three in the morning.
And yes, I said bandwidth control.
Because backups are not free just because the CPU looks bored.
The part people forget: the blast radius
Backups cost resources.
On a database, the obvious cost is compute and memory, but that’s only the start. A backup can hammer disk IO. It can create locks or pressure transaction logs. It can make a replica fall behind. It can fill a volume you thought had plenty of room. Then, once the artefact exists, the transfer can saturate the network link and make everything else on the box feel haunted.
Commercial systems have versions of this problem with bigger numbers and more meetings. Self-hosters have it with worse broadband and fewer spare machines.
My current home setup is deliberately light on databases. Anything persistent that I can’t rebuild from static sources lives somewhere else, because I don’t have enough hardware to give it the uptime guarantees I would want, and my home broadband is not a business circuit. It is shared with a family who may, at any given moment, be streaming 4K video, playing games, updating something enormous, or doing whatever it is children do on social media (I don’t want to know).
Build a backup process that assumes that link is always available and uncontended, and you have built a process for a house you don’t live in.
I have also experienced the proverbial shovel through a cable. Twice.
That does focus the mind a bit.
So the practical question is not “what is the best backup tool?”.
It’s “what can this system afford to do, regularly, without hurting the thing it is meant to protect?”.
Move the pain away
The usual mitigation is to stop making the live thing pay the whole bill.
If your database supports it, back up from a dedicated read replica rather than the primary. That replica still has a job, and you can still hurt it, but at least the blast radius is away from the write path your users are actually depending on. If the backup makes the replica lag, that is a signal you can observe and react to, not a lock or IO spike on the main database while someone is trying to check out.
And if your database has a proper write-ahead log, binlog, transaction-log, point-in-time-recovery story, use it. A full backup plus WAL/archive logs is a different beast from a full backup alone. The full backup gives you the base; the logs give you the route back to a point in time. That’s the difference between “we can restore last night” and “we can restore to 10:43, before someone did the stupid thing”.
Storage matters too. Put the backup workspace on a disk that can take the punishment. A separate volume with decent IOPS can keep the backup process from fighting the database for the same tired spindle, or the same cloud volume queue, or whatever the modern equivalent of a tiny overloaded disk sounds like when it is quietly ruining your afternoon.
The shorter the backup runs, the less time it has to interfere. Long-running backups make replicas drift further from the real data, keep snapshots alive longer, stretch transaction logs, and generally increase the chance that some other maintenance job wanders in wearing big boots.
Network is the same story. If you have a backup machine, and the data path justifies it, give it a separate network interface for transfers. Better still, give it a good one. Hardware offload and compression are not magic, and you should measure them rather than assume, but a dedicated path for backup traffic can stop “copy yesterday’s world to object storage” from turning the rest of the network into soup.
The self-hosted version may be smaller. It might be as simple as “do the heavy copy overnight”, “limit transfer bandwidth”, “don’t back up while the family are all streaming”, or “keep the database somewhere that is not my living-room broadband”. That’s still engineering. It’s just engineering with a smaller budget and more people asking why the telly is buffering.
The restore test is the process
The process I trust is boring, which is usually a good sign.
The important thing in that diagram is that the restore is not a side quest. It’s in the middle of the process, and it has a gate. Did the verification work? If yes, carry on. If no, fix the gap and try again. You don’t know whether the previous boxes worked until you have restored into a clean target and run enough checks to prove the result is useful.
For a self-hosted service, that clean target might be a temporary VM, a spare Docker volume, a disposable directory, or a completely separate machine if you have one. It doesn’t need to be fancy. It needs to be clean enough that you are not accidentally relying on leftovers from the original system.
A small restore test might look like this:
- Pick one service.
- Restore its data somewhere disposable.
- Start the service against the restored copy.
- Log in, browse, query, or run the thing that proves the data is alive.
- Write down what was missing.
- Fix the backup process.
- Do it again later, on purpose.
That last bit matters. A restore test you did once, eighteen months ago, against a version of the app you no longer run, is useful history. It is not current evidence.
Replication is not a time machine
Clustered systems still need backups.
This is one of those sentences that feels too obvious until you watch someone argue the opposite with diagrams. Replication protects availability. It keeps another copy warm so a machine can die without taking the service with it.
It also faithfully replicates bad writes, accidental deletes, corrupt application behaviour, and a surprising amount of human stupidity.
If someone drops the wrong table, your cluster may helpfully make sure every node knows about it.
Very modern. Very fast. Very dead.
A backup gives you a time dimension. It lets you say “take me back to before the mistake”. Replication, by itself, says “good news, the mistake is highly available”.
That’s why “we run a cluster” is not a backup plan. It may be part of the resilience story, but it doesn’t replace the boring artefact you can restore when the whole cluster has agreed on the wrong answer.
Eventually consistent means more homework
There is another elephant in the room, and it is the one wearing a distributed-systems hat.
If your architecture is eventually consistent, a snapshot is not automatically a backup. It is a snapshot. Useful, perhaps necessary, but not the whole answer.
The question becomes: a snapshot of what, at which point in logical time, and with which events already applied?
If the database says one thing, the queue says another, the object store has a third version, and the search index is ten minutes behind all of them, “restore the database” may not restore the system. It may restore a very expensive argument between components.
That doesn’t mean backup is impossible. It means the restore plan has to match the architecture. You may need point-in-time recovery, event logs, replay rules, idempotent consumers, rebuildable projections, or a written answer for which parts of the system are authoritative and which parts can be regenerated.
The self-hosted version is usually smaller, but the shape is the same. If your photo library stores metadata in one place and files in another, back both up in a way that can be put back together. If your app rebuilds a search index from the database, fine, document that. If the index contains the only copy of something, it is not an index. It is data wearing a funny hat.
The backup that looked fine
Back to those cloud backups that would not restore.
The context was a large commercial migration. New architecture, new cloud, database upgrade, zero downtime as the target. The team needed to build and rebuild a complete test environment over and over, and restoring from backup should have turned a two-day sync into a couple of hours.
Except every backup failed on restore.
Not one. Not the latest. Every one we tried.
That’s the cold-sweat version of this lesson. The files existed. They were big. They had crossed the network. They lived in the place backups were meant to live. From the outside, the process looked healthy.
It was not healthy.
The backup files had been mangled in the transfer path because of a peculiar race in the way the database setup and file spanning interacted. The details are less important than the shape: the system was confidently producing artefacts that only revealed their uselessness when someone performed the restore.
And because this was a real migration, not a blog post with a tidy lab, the fix was not “install my favourite backup tool”. It involved fixing the backup path, doing a lot more validation, and eventually writing a transfer utility that matched the network complexity and cost profile we actually had.
That’s the lesson I trust: your environment gets a vote.
Where I would start
For a budget self-hoster, I would start by classifying data.
Not all services deserve the same backup treatment. Some things are mostly config and a Git repository. Some are disposable. Some are irreplaceable family photos, documents, source files, or private records you would be furious to lose.
Give the irreplaceable things the most boring, most tested path first.
Then write down, for each service:
- What data matters?
- Where does it live?
- Can it be rebuilt from source, config, or another system?
- How long can it be offline?
- How much data changes each day?
- How expensive is transfer, in money and bandwidth?
- What does “restored” actually mean?
That last question is the whole game. A restored database that the application can’t boot from is not restored. A restored folder with wrong ownership is not restored. A restored service with missing secrets is not restored. A restored photo library with the originals but no metadata might be acceptable, or it might be a disaster, depending on what you thought you were protecting.
You only find that out by testing.
So yes, pick tools. Pick boring tools. Pick tools with encryption, verification, retention, resumable transfer, clear failure modes, and restore documentation you can follow without a priest and a full moon.
But don’t mistake the tool for the system.
The system is the loop: capture, verify, transfer, verify, restore, check, learn.
Until that loop has run, you don’t have a backup.
You have a file.
And maybe a very comforting lie.