A backup strategy for root servers that holds up when it matters

The checklist for the first 30 minutes ends with a tar archive of /etc and the observation that this is not a backup yet, only a copy on the same disk. That is where this article picks up: how to turn it into a strategy that survives a total loss.

One sentence holds all of it together: a backup you have never restored anything from is not a backup, it is a hope. Everything that follows serves one purpose, turning that hope into a tested fact.

All commands run as root. As a regular user, put sudo in front of every command. The reference systems are Debian 13 (trixie), Debian 12 (bookworm), Ubuntu 24.04 LTS and Ubuntu 22.04 LTS. Wherever the four differ, it is noted on the spot.

Before you change anything: the way back

The backup itself rarely breaks. What is dangerous is everything around it: a restore that writes over the running system, a repository that fills up the system disk, key files that replace your own access. Four points before you start.

1. Know your console access before you need it

On KernelHost KVM root servers and dedicated servers you reach the VNC console in the customer panel. It does not sit on the network stack of the guest system and still answers when SSH has gone quiet. That is your rescue path once a restore has written an old sshd_config or authorized_keys over the current one. Log in through it once beforehand and check that you know the root password.

2. Never restore straight to / on the first attempt

The first restore always goes into an empty directory, for example /var/tmp/restore-test. From there you compare and copy back exactly what you need. Restoring directly to / also overwrites the files that have changed since the backup for good reasons.

3. Keep a second session open

As long as you are working on SSH keys or on the authorized_keys of the target, the same rule applies as when building a firewall: keep a second terminal with a live connection open, and close it only once a fresh connection works.

4. Check free space before the repository grows

df -h /
df -i /

Most people forget the second line: a file system also runs full while gigabytes are still free, namely when it runs out of inodes. A local repository that fills the system disk is one of the most common self-inflicted outages. That is why the target here sits outside the server from the very start.

The 3-2-1 rule applied to a single server

• Three copies. The production data is the first copy. So you need two backups, not one.
• Two different storage locations. Two directories on the same disk are one location, and so are two disks in the same RAID: an accidental rm hits both. RAID protects against the failure of one drive and against nothing else.
• One copy off site. Not the same server, not the same management account, ideally not the same location.

Two additions are needed. First, one copy should sit somewhere the server itself cannot delete, because whoever takes over your server finds the credentials for the target on it. Second, a copy only counts once it has been verified. A system image in the same customer panel is convenient, but it does not count as an off-site copy.

Fix two numbers as well: how many hours of data loss you can accept (this sets the interval between two runs), and how long the restore may take (this decides whether you also need a system image).

What belongs in the backup, and what does not

The most common mistake is not backing up too little, it is backing up everything. If you write away / without any exclusions, you take package caches, temporary files and swap along with it.

apt-mark showmanual > /root/package-list.txt
dpkg --get-selections > /root/package-selections.txt
wc -l /root/package-list.txt /root/package-selections.txt

apt-mark showmanual lists only the packages you installed on purpose, without the dependencies that were pulled in with them: the short list you need for a rebuild.

File, database, image: three methods that do not replace each other

What follows from this is an order, not a choice. First the database server writes a dump, then the file backup runs, and the data directory stays excluded. A file copy of /var/lib/mysql taken while the server is running captures different tables at different points in time. Whether it can be restored is something you find out at the worst possible moment.

The credentials file, the permissions and the pitfalls of the dump itself are covered in Back up MySQL and MariaDB databases automatically. What matters here is the handover point: the dumps land in /var/backups/db, stay there for a day or two, and the repository takes care of retention. For PostgreSQL you use pg_dumpall as the postgres user.

restic or Borg

Both split files into chunks, store identical chunks only once, encrypt, and keep versioned snapshots. Both are packaged on all four distributions. The difference that decides it is in the last row.

On storage where you cannot install anything, Borg is out. With a second Linux server under your own management, append-only mode speaks for Borg.

Setting up restic

apt update
apt install -y restic
restic version

Check the output before you create a repository: the four distributions ship very different versions, and a repository written by a newer version cannot necessarily be opened by an older one.

Access to the target

The target here is a second server reached over SSH. The key belongs to root, because only root is allowed to read every file that has to be backed up:

ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519_backup -C 'backup srv01'
ssh-copy-id -i /root/.ssh/id_ed25519_backup.pub backup@203.0.113.50

cat > /root/.ssh/config <<'EOF'
Host backuptarget
    HostName 203.0.113.50
    User backup
    IdentityFile /root/.ssh/id_ed25519_backup
    BatchMode yes
EOF
chmod 600 /root/.ssh/config

Success check: ssh backuptarget true completes with no output and no prompt. The very first connection asks about the host key: answer that now, and not later inside a service that cannot wait for anyone.

Password and repository

install -d -m 700 /etc/restic
head -c 32 /dev/urandom | base64 | tr -d '\n' > /etc/restic/repo.pass
chmod 600 /etc/restic/repo.pass

cat > /etc/restic/env <<'EOF'
RESTIC_REPOSITORY=sftp:backuptarget:/srv/backup/srv01
RESTIC_PASSWORD_FILE=/etc/restic/repo.pass
EOF
chmod 600 /etc/restic/env

This file works for the shell and for systemd alike. In the shell:

set -a; . /etc/restic/env; set +a
restic init

Success check: restic cat config prints a short JSON structure with the ID of the repository. If you get the question Is there a repository at the following location?, either the path is wrong or restic init never ran.

The first run

cat > /etc/restic/excludes.txt <<'EOF'
/proc
/sys
/dev
/run
/tmp
/var/tmp
/var/cache
/var/lib/apt/lists
/var/lib/mysql
/swapfile
EOF

restic backup / --one-file-system --exclude-file=/etc/restic/excludes.txt --exclude-caches --tag system

--one-file-system keeps the backup on the root file system and leaves mounted network shares out. --exclude-caches skips directories that have marked themselves as caches. Excluding /var/lib/mysql is the previous section put into practice.

Success check: the run ends with a line such as snapshot 0a1b2c3d saved. After that:

restic snapshots
restic stats latest

The list shows the timestamp, the host name and the paths. If restic stats latest reports an unexpectedly small size, one of your exclusions reaches too far.

The same thing with Borg

apt install -y borgbackup
borg --version
export BORG_REPO='ssh://backup@203.0.113.50/./srv01'
borg init --encryption=repokey-blake2

borg create --stats --compression zstd ::'system-{now}' /etc /var/www /home /var/backups
borg list
borg info

All four distributions ship a version from the 1.x series. Borg has to be present on both sides, and the version on the target should not be older than the one on the server. Run one repository per server: that saves you from having to restrict the pruning to the archives of exactly that server, and it lets you use separate access keys.

Success check: borg list shows the archive with its timestamp, and borg info shows the size before and after deduplication.

Retention: longer than most people plan for

Retention time is not decided by how long you would like to keep data, but by how long it takes for damage to be noticed. A deleted directory is spotted the same day, a corrupted table often only weeks later. Keep seven days, and all you have is seven days of backups that contain the damage.

restic forget --dry-run --keep-daily 7 --keep-weekly 4 --keep-monthly 6
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

The dry run shows which snapshots would disappear. Without --prune only the references go, and the space stays occupied. With Borg this has been a two-step process since 1.2, and the forgotten second command explains most repositories that refuse to shrink:

borg prune --list --dry-run --keep-daily=7 --keep-weekly=4 --keep-monthly=6
borg prune --list --keep-daily=7 --keep-weekly=4 --keep-monthly=6
borg compact

Success check: restic snapshots or borg list shows the expected spread, and the space used on the target goes down.

Encryption, and the key nobody can find afterwards

Both tools encrypt before the data leaves the server. The target sees only unreadable chunks, which is what makes storage owned by someone else acceptable in the first place. The price is plain: without the password or the key, the data is lost for good. There is no back door.

Two rules follow from that. First, the password belongs in a second place, usually a password manager. A password that exists only in /etc/restic/repo.pass is gone when the server is gone, and the backup goes with it. Second, with Borg and repokey you export the key and store it elsewhere, because in that mode it lives inside the repository itself:

borg key export ::

Success check: take the password out of the password manager and run restic snapshots with it on a different machine. A password you have never used from somewhere else is unconfirmed.

Protecting the target against deletion

An attacker with root rights also has access to the stored password and to the key for the target, and can therefore delete the backup before encrypting the production data. So you need storage that accepts new snapshots but allows no deletion. With Borg you set that up in the authorized_keys of the backup user on the target:

command="borg serve --append-only --restrict-to-path /srv/backup/srv01",restrict ssh-ed25519 AAAA... backup srv01

From then on the key only accepts backups, and only into that one path. Expect a prune to run through here without freeing any space, because the deletion is never carried out on the target. You do the cleanup where the server has no access. With restic, the REST server in append-only mode or object storage with versioning takes on that role. Plain SFTP access does not.

Automating it with a systemd timer

A timer is preferable to a cron job because it writes its output to the journal, catches up on missed runs and is allowed to spread out the start time. For how units are built: Create your own systemd service.

cat > /etc/systemd/system/backup.service <<'EOF'
[Unit]
Description=Daily backup with restic
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
Nice=10
IOSchedulingClass=idle
ExecStart=/usr/bin/restic backup / --one-file-system --exclude-file=/etc/restic/excludes.txt --exclude-caches --tag system
ExecStart=/usr/bin/restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
EOF

cat > /etc/systemd/system/backup.timer <<'EOF'
[Unit]
Description=Starts the daily backup

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now backup.timer
systemd-analyze calendar '*-*-* 02:30:00'

Several ExecStart lines in a oneshot unit run one after the other, and a failure breaks the chain. So the cleanup only runs after a backup that actually worked. If the target is a mounted file system, a guard belongs in front of it, otherwise the run quietly writes into the empty mountpoint:

ExecStartPre=/usr/bin/mountpoint -q /mnt/backup

Success check:

systemctl start backup.service
systemctl list-timers backup.timer
journalctl -u backup.service -n 50 --no-pager

systemctl list-timers has to show a next start time. If there is nothing there, the timer is not enabled. The most important point comes last: a backup that quietly stops running is the normal way backups fail. Check systemctl is-failed backup.service, or add a call to an external monitoring service as the final ExecStart line, one that raises an alarm when the daily sign of life fails to arrive.

Testing the restore

The small test, monthly, five minutes

mkdir -p /var/tmp/restore-test
restic restore latest --target /var/tmp/restore-test --include /etc/ssh/sshd_config
diff /etc/ssh/sshd_config /var/tmp/restore-test/etc/ssh/sshd_config && echo "identical"

The same with Borg, which stores paths without a leading slash:

cd /var/tmp/restore-test
borg extract --list ::system-2026-09-03T02:30:00 etc/ssh/sshd_config

Check the integrity of the repository as well. In each case the second line reads the data back and recomputes the checksums, with restic only for a fraction of it so the run does not take hours:

restic check
restic check --read-data-subset=1/7

borg check
borg check --verify-data

Success check: restic check ends with no errors were found, and borg check ends without an error message. A repository that is only checked once a year may have been broken for eleven months.

The big test, once a year

The small test proves that files can be read. It does not prove that you can get the service running again. For that you need the full run on a second, empty server: install the base system, apply the package list, connect the repository, pull the data back, import the dump, start the services. Time it. That number is your real restore time, and experience says it is a multiple of the one you estimated.

Write down what was missing. It is almost always the same things: a file outside the backed-up paths, a service with its configuration under /opt, a certificate without its account key, a database without its users and permissions. That list is what the test is worth.

Common errors and how to fix them

Host key verification failed. The service runs as root, and root has never confirmed the host key of the target. Run ssh backuptarget true once by hand, or ssh-keyscan -H 203.0.113.50 >> /root/.ssh/known_hosts. It works on the console but not in the timer: almost always this error.

Permission denied (publickey). Wrong user, wrong key or wrong permissions on the target: .ssh needs 700, authorized_keys needs 600, and both have to belong to the user on the target.

Is there a repository at the following location? restic finds no repository structure: wrong path, restic init never ran, or the target is unreachable at the moment.

Fatal: wrong password or no key found The password file does not match the repository, usually because it was generated again after the fact. Use cat -A /etc/restic/repo.pass to check whether a stray space has crept in.

repository is already locked exclusively by An aborted run left its lock behind. First make sure nothing is still running, then use restic unlock. With Borg the message is Failed to create/acquire the lock with the addition (timeout), and the command is borg break-lock. Both are risky as long as a run really is still active.

Warning: The repository at location ... was previously located at ... The address of the repository has changed, and Borg asks about it interactively. Inside a unit the command then waits for an answer that never comes. Once you have confirmed that it is the same repository, set BORG_RELOCATED_REPO_ACCESS_IS_OK=yes in the environment file.

No space left on device on the target. Either the cleanup does not run at all, or it runs without --prune or without borg compact. If it is the other way round and a dump has filled the root file system, Disk full on Linux: how to clean up takes it from there.

The run reports success but backs up almost nothing. An exclusion reaches too far, or a path is misspelled. Compare restic stats latest with the value from the day before. A backup that is suddenly orders of magnitude smaller is an alarm, not a success.

Differences between the distributions

• Package names are the same on all four systems: restic and borgbackup. The shipped versions are not.
• Logging: Debian 13 and many Debian 12 installations do not ship rsyslog. There you will find the output of the run in the journal only, that is through journalctl -u backup.service. On Ubuntu 22.04 and 24.04 there is also the copy under /var/log.
• Mounting snapshots: restic mount and borg mount need fuse3. If the package is missing, the command aborts with a note about a missing fusermount3. The route without FUSE is restic restore or borg extract.
• Database tool: Debian ships MariaDB only, where the command is mariadb-dump with mysqldump as a link to it. On Ubuntu, MySQL 8 can be in use as well, and there only mysqldump exists.

The final check

The strategy is finished once you can back up these seven points with a command and not with an assumption:

1. restic snapshots or borg list shows a snapshot from last night.
2. systemctl list-timers backup.timer shows a next start time.
3. restic check or borg check reports no error.
4. A single file could be restored this month and was identical afterwards.
5. The spread matches the retention you planned, and the repository is not growing without limit.
6. The password is stored in a second place, and you have already used it from there once.
7. At least one storage location accepts backups without the server being able to delete them.

If point four is missing, what you have is an assumption. If point six is missing, encrypted garbage. If point seven is missing, a backup that does not survive exactly the attack it is needed for most.