Setting up automatic security updates with unattended-upgrades

On most servers there is a gap between the day a security update is published and the day somebody installs it. It rarely comes from carelessness. It comes from someone deciding to take care of it "next week". Scanners work through a freshly published vulnerability within hours. The package unattended-upgrades closes that gap.

The short version is step 7 of the checklist for a new root server. This article is about everything that comes after it: security updates versus all updates, reboot behavior on a game server, mail notification, excluded packages, the dry run and the proof in the log.

Tested on Debian 13 (trixie), Debian 12 (bookworm), Ubuntu 24.04 LTS and Ubuntu 22.04 LTS. Wherever the four differ, the difference is noted. The commands assume root; as a regular user, put a sudo in front of each one.

What is actually running here

Troubleshooting takes a long time here because it is unclear which of the three parts involved is not doing its job:

1. Two systemd timers: apt-daily.timer for package lists and downloads, apt-daily-upgrade.timer for the installation.
2. The script /usr/lib/apt/apt.systemd.daily, which evaluates the switches under APT::Periodic::.
3. The program unattended-upgrade, which evaluates the switches under Unattended-Upgrade:: and does the actual work.

Watch the singular: the package is unattended-upgrades, the program is unattended-upgrade.

systemctl list-timers 'apt-daily*' --all
systemctl cat apt-daily-upgrade.timer

The LAST and PASSED columns show whether a run has ever taken place. The unit contains OnCalendar=*-*-* 6:00 together with RandomizedDelaySec=60m: the run happens somewhere between six and seven, and with apt-daily.timer the spread is twelve hours. If you check at 06:05, you will wrongly conclude that the mechanism is broken.

Before the first change: the way back

Automatic updates act while you are not sitting in front of the machine. Three things can go wrong: a reboot takes the server out of service, a package operation aborts and leaves dpkg in an incomplete state, or a service no longer starts afterwards.

The access route that still works in all three cases is not SSH. KVM root servers and dedicated servers from KernelHost have no IPMI and no iDRAC, so emergency access runs through the VNC console in the customer panel. It sits on the virtualization layer, or on the uplink itself, which means a fault inside the guest system does not reach it. Log in through it once beforehand and check the root password. A rescue path that you only try out in an emergency is not a rescue path.

Then back up the initial state:

mkdir -p /root/pre-unattended
cp -a /etc/apt/apt.conf.d/50unattended-upgrades /root/pre-unattended/
dpkg --get-selections > /root/pre-unattended/packages.txt
apt-mark showhold > /root/pre-unattended/holds.txt

There are two kill switches. The hard one goes through the timers:

systemctl disable --now apt-daily-upgrade.timer apt-daily.timer

The soft way sets the value APT::Periodic::Unattended-Upgrade "0"; in /etc/apt/apt.conf.d/20auto-upgrades. The package lists then stay current, but nothing gets installed.

Success check:

systemctl is-enabled apt-daily-upgrade.timer
apt-config dump APT::Periodic

For rolling back: /var/log/apt/history.log names the old and the new version number. However, apt install package=version only succeeds as long as the old version is still sitting on a mirror, and the archives usually keep the current state only. Plan for fixing forward, and for backups.

Installation and actually switching it on

apt update
apt install -y unattended-upgrades

Difference between the distributions: on Ubuntu 22.04 and 24.04 the package is installed and active on server images, so check there first. On Debian it is missing from minimal images, and the installation asks whether stable updates should be applied automatically. If it does not run interactively, during an image build for example, the default applies and the file that matters never gets created.

The reason is that the package on its own switches nothing on. That is the job of this file:

cat /etc/apt/apt.conf.d/20auto-upgrades

If it is missing, the command reports cat: /etc/apt/apt.conf.d/20auto-upgrades: No such file or directory. In that case, bring the question back with dpkg-reconfigure -plow unattended-upgrades or write the file yourself:

cat > /etc/apt/apt.conf.d/20auto-upgrades <<'EOF'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";
EOF

The values are not yes/no switches but intervals in days: "1" daily, "7" at most weekly, "0" off. AutocleanInterval clears out package files that no longer exist on the mirror, which is noticeable on small system disks (disk full on Linux).

Success check: apt-config dump APT::Periodic prints the lines that are set. If nothing comes back at all, no file was read and the nightly run does nothing.

Your own settings in the right place

The /etc/apt/apt.conf.d/50unattended-upgrades that ships with the package belongs to the package. If you edit it, the next package update produces a conffile conflict, and unattended-upgrades then skips the package. The tool for automatic updates would be excluded from them itself.

Your own values belong in a separate file with a higher number. APT reads the directory in alphabetical order, and for simple values the last one read wins:

cat > /etc/apt/apt.conf.d/52unattended-upgrades-local <<'EOF'
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::MinimalSteps "true";
EOF

MinimalSteps is on by default and splits the run into small steps that can be aborted cleanly during shutdown. Remove-Unused-Kernel-Packages clears out old kernels that would otherwise fill the /boot partition. Both cleanup switches are set by default on Ubuntu, but not on Debian.

The file name is not a detail. APT only reads files here that have no extension or end in .conf, and only names made of letters, digits, hyphen, underscore and dot. A backup copy called 52unattended-upgrades-local.bak is passed over silently, which is convenient; a file called 52-my-settings.txt is passed over just as silently, which is annoying.

Success check, immediately after every change:

apt-config dump > /dev/null && echo "Syntax OK"
apt-config dump | grep "^Unattended-Upgrade::"

A missing semicolon paralyzes every apt call, the nightly one included.

Only security updates, or all updates

A list of allowed origins decides which packages are eligible. The distributions already differ in the key name: Debian uses Unattended-Upgrade::Origins-Pattern, Ubuntu uses Unattended-Upgrade::Allowed-Origins.

sed -n '/Allowed-Origins\|Origins-Pattern/,/};/p' /etc/apt/apt.conf.d/50unattended-upgrades

Active by default are the patterns for the security origin and for the base archive of your own release. The -updates, -proposed and -backports pockets are commented out. On Ubuntu there are two additional entries for the extended maintenance service; without a subscription they deliver nothing. The placeholders ${distro_id} and ${distro_codename} are resolved by the program at runtime, to Debian and trixie for example.

Where the parts of a pattern come from is shown by apt-cache policy. For every source there is a line starting at release with the fields o= (origin), a= (archive), n= (codename), l= (label) and c= (component). The Debian security source carries the label Debian-Security, and that is exactly what the shipped pattern targets.

If you want the ongoing fixes from the distribution to arrive automatically as well, add the pattern in your own file. In the apt configuration, lists are appended to and not replaced:

Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,codename=${distro_codename}-updates";
};

On Ubuntu the equivalent is "${distro_id}:${distro_codename}-updates"; inside an Allowed-Origins block. The only way to replace a list completely is to empty it first with #clear Unattended-Upgrade::Origins-Pattern;.

The same procedure applies to third-party repositories. They often deliver new feature versions rather than pure security fixes, though, and applying that unattended at night is a different class of risk. For production systems the rule is therefore: stay with the security sources.

The dry run

apt update
unattended-upgrade --dry-run --debug

The apt update in front of it is not decoration: the dry run does not refresh the package lists itself, so without that call you are judging the state of yesterday. Nothing gets installed. Four places in the output matter:

• Allowed origins are: with the list that is actually in effect. That is the proof that your change has arrived, not a look into the file.
• Initial blacklist: with your exclusions. If nothing appears there although you entered some, your file is not being read.
• Lines starting at Checking:, one per package, with its origin and whether it is allowed.
• At the end either Packages that will be upgraded: with a list, or No packages found that can be upgraded unattended and no pending auto-removals.

The last message also appears when apt list --upgradable does list packages. That is not an error, it is the filter doing its job: every package that shows up there and is missing from the dry run either comes from an origin that is not allowed, is on your exclusion list, is pinned with apt-mark hold, or triggers a configuration prompt.

Excluding packages from the automation

Some things should not be updated at night: a database whose restart disconnects an application, or the kernel of a system that must not reboot unannounced. The first of two ways affects the automation only:

Unattended-Upgrade::Package-Blacklist {
        "mariadb-server$";
        "nginx$";
        "linux-image-";
};

The entries are regular expressions, anchored at the start of the package name. "nginx" without a dollar sign therefore also matches nginx-common and nginx-full; the $ limits the entry to exactly that one name. Conversely, "linux-image-" without a $ is the right form when you mean all kernel packages.

The second way affects every apt call, including your manual apt upgrade:

apt-mark hold mariadb-server
apt-mark showhold

You undo that with apt-mark unhold. If a package should merely not be updated unattended, use the exclusion list; if it should not be updated at all, use hold.

Success check: apt-config dump | grep -i "Package-Blacklist" and apt-mark showhold. The uncomfortable side of this: an excluded package is one you have to update yourself. Put a date in the calendar for it, otherwise in six months the exclusion is a forgotten vulnerability.

Automatic reboot

After the update a new kernel sits on the disk, but it only runs after a reboot; replaced libraries only take effect once the service restarts. Three switches control this:

Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Automatic-Reboot is off by default. Automatic-Reboot-WithUsers is on by default, and that surprises a lot of people: an open SSH session does not prevent the reboot, and if you want it to, you set the value to "false". Automatic-Reboot-Time is mandatory as soon as the reboot is enabled. Without it, now applies, so the server reboots straight after the run, which means between six and seven in the morning.

It is not triggered by the kernel package but by a marker file:

ls -l /var/run/reboot-required
cat /var/run/reboot-required.pkgs

The difference in this article with the biggest consequences: Ubuntu creates this file (it comes from the package update-notifier-common), and the second file names the packages responsible. Debian does not create it by default. There, Automatic-Reboot "true" can stay permanently ineffective without anything reporting an error. If you rely on it, you run for months on an old kernel and a good feeling. On Debian this is what helps instead:

apt install -y needrestart
needrestart -b

The output names the running kernel under NEEDRESTART-KCUR and the expected one under NEEDRESTART-KEXP; if they differ, a reboot is due. The NEEDRESTART-SVC lines list the services that are still running with replaced libraries.

Why the setting looks different on a game server

On a web server a reboot at 02:00 is a matter of seconds that nobody notices. On a game server players are connected, the world is held in RAM, and the save state is only written in full during an orderly shutdown. If the process is killed hard, you lose the progress made since the last autosave, and in the worse case the world file ends up corrupted.

There is a systemd detail on top of that: during shutdown every service gets its stop signal and then a limited grace period, after which it is killed hard. A game server that saves first when it shuts down often needs more time than the default allows, plus a stop command that sends a stop to the server console instead of just a signal to a launcher. For game servers this combination is therefore the sensible one:

• Automatic-Reboot "false". The security updates keep running, the reboot stays your decision.
• A maintenance window with few players online, announced rather than sprung on them.
• A systemd unit with a saving stop command and a sufficient TimeoutStopSec, see create a systemd service.
• If it does have to be automatic, then set Automatic-Reboot-Time to an hour with a low player count.

You can test this without a reboot: stop the service with systemctl stop, look at the save state, start it again. If that runs cleanly, it will also survive a reboot at 02:00.

Moving the time of the run

systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
OnCalendar=*-*-* 03:30
RandomizedDelaySec=30m

The empty OnCalendar= line is mandatory, it clears the shipped value. Without it your time is added on top and the run happens twice a day.

systemctl daemon-reload
systemctl restart apt-daily-upgrade.timer
systemctl list-timers 'apt-daily*' --all

Success check: the NEXT column shows the new time.

Notification by email

A mechanism nobody ever hears from is not being monitored, it is being forgotten. Two lines change that:

Unattended-Upgrade::Mail "admin@example.com";
Unattended-Upgrade::MailReport "on-change";

MailReport knows three values: always after every run, on-change only when something was installed or went wrong, only-on-error only in case of failure. Replace the older MailOnlyOnError switch if you come across it.

What this needs is a way to send mail. unattended-upgrades sends through /usr/bin/mail or /usr/sbin/sendmail. If both are missing, nothing goes out and you never notice. On a fresh server that is the normal case:

apt install -y bsd-mailx
echo "Test message" | mail -s "Test from the server" admin@example.com

This pulls in a mail transport service along with it. Afterwards check with ss -lntp | grep ':25' that it listens on 127.0.0.1 only. Messages sent straight from a server IP address also land in the spam folder frequently; if the report is supposed to arrive, route the delivery through a relay server with a properly configured sender domain.

Practical advice: set always for the first two weeks. A daily message reporting that there was nothing to do is the simplest proof that the mechanism is running. After that, go back to on-change, otherwise you will soon be deleting it unread.

Checking whether anything actually happened

ls -l /var/log/unattended-upgrades/
tail -n 40 /var/log/unattended-upgrades/unattended-upgrades.log

There are up to three files in there. unattended-upgrades.log records what the program decided. unattended-upgrades-dpkg.log contains the raw output of the installation, and that is where you look when a package failed while being configured. unattended-upgrades-shutdown.log appears after a run that ended during shutdown.

An uneventful run ends with No packages found that can be upgraded unattended and no pending auto-removals. A run that did something contains Packages that will be upgraded: and, further down, All upgrades installed. If the file is empty or the directory is missing, no run has ever taken place.

Independently of that, the package manager logs every action with the old and the new version number:

grep -E "^(Start-Date|Commandline|Upgrade|End-Date):" /var/log/apt/history.log | tail -n 20
zgrep -h "^Upgrade:" /var/log/apt/history.log*.gz | tail -n 10

The second call reaches the rotated files. If you look at the current one only, you find nothing and draw the wrong conclusion from that. The third source is systemd:

journalctl -u apt-daily-upgrade.service --since "-7 days" --no-pager

The trap almost everyone walks into once: systemctl status unattended-upgrades.service permanently shows active (running). That is not an update in progress. This unit finishes a run that has already started when the system shuts down, and waits the rest of the time. The installing is done by apt-daily-upgrade.service.

Common errors and fixes

cat: /etc/apt/apt.conf.d/20auto-upgrades: No such file or directory
The package is installed but nothing is switched on; on Debian this is the normal state after a non-interactive installation. The fix: dpkg-reconfigure -plow unattended-upgrades.

No packages found that can be upgraded unattended and no pending auto-removals, although apt list --upgradable shows packages
Not an error, just the filter. Check in this order: the allowed origins in the dry run, the exclusion list, apt-mark showhold, configuration prompts.

E: Syntax error /etc/apt/apt.conf.d/52unattended-upgrades-local:2: Extra junk at end of file
A missing semicolon at the end of a line, or a curly brace that was never closed. As long as the error is in there, every apt call fails, not just the automation.

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1234 (unattended-upgr) together with E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
The automatic run is busy, and the process name truncated to fifteen characters gives it away. Wait a few minutes. Killing the process produces the next error in this list. In detail: apt could not get lock.

E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
Run exactly that command. Unattended-Upgrade::AutoFixInterruptedDpkg, which is set by default, lets the next run repair this by itself, but only if it gets that far.

Package nginx-common has conffile prompt and needs to be upgraded manually in the log
You have changed a configuration file that belongs to the package, and the new version brings a changed copy with it. The program then skips that package permanently. Do the update by hand. Open cases are shown by:

find /etc -type f \( -name "*.dpkg-dist" -o -name "*.dpkg-new" -o -name "*.ucf-dist" \)

Cache has broken packages, exiting in the log
An incomplete state is left over from an earlier operation. Run apt --fix-broken install, then dpkg --configure -a. Until that is done, the automation does nothing every night.

Your settings have no effect and there is no error message.
That is the signature of a file name that was passed over. Check with apt-config dump | grep -i unattended whether the values were read.

Automatic-Reboot "true" does nothing on Debian.
The marker file /var/run/reboot-required is missing there.

The four systems compared

What the automation does not do

• No upgrade to a new major release and no third-party repositories, as long as no pattern for them is configured.
• Nothing outside the package manager. Programs unpacked by hand, extensions of a content management system, plugins of a game server and containers all keep running at the version they are on.
• No replacement for backups and monitoring. An update can bring a service to a halt, and if nobody is watching, it stays down until morning.
• No protection against network attacks. Current packages close known vulnerabilities, but volumetric attacks are only stopped by filtering in the network upstream, at KernelHost in the maincubes datacenter in Frankfurt am Main.

The final check

1. apt-config dump APT::Periodic outputs Update-Package-Lists "1" and Unattended-Upgrade "1".
2. apt-config dump | grep "^Unattended-Upgrade::" shows your own values.
3. systemctl list-timers 'apt-daily*' --all shows both timers with a plausible NEXT column.
4. unattended-upgrade --dry-run --debug runs through without errors and names the expected origins.
5. tail -n 40 /var/log/unattended-upgrades/unattended-upgrades.log contains an entry with a date after the first run.
6. ls -l /var/run/reboot-required on Ubuntu, needrestart -b on Debian.

Point five is the one that really counts, and it needs a day of patience: everything before it is preparation, and only the entry written tonight is the proof.