Updating a Linux server safely with apt

On a fresh server an upgrade is a formality: there is nothing that could break. As soon as the server carries live services, the details decide whether an upgrade passes unnoticed or whether afterwards a configuration file has been overwritten, a service still holds the old library in memory, or the running kernel is a different one from the kernel on disk. This article works through them in order. The short version for a newly provisioned system is in the checklist for a new root server.

Everything here applies to Debian 13 (trixie), Debian 12 (bookworm), Ubuntu 24.04 LTS and Ubuntu 22.04 LTS. The commands are written for running as root. As a normal user, prefix every command with sudo.

Before you run the first command: the way back

An upgrade can get expensive in three ways: a configuration prompt overwrites your SSH configuration, a new kernel fails to boot, or the connection drops in the middle of a dpkg transaction. All three are manageable, but only if you prepare in advance.

A session that survives a dropped connection

If the SSH connection drops while packages are being unpacked, the running process receives a SIGHUP and dpkg stops in the middle of a transaction. The result is a half-configured package database. Run longer upgrades in a session that keeps going on the server even when your terminal disappears:

apt install -y tmux
tmux new -s upgrade

If the connection fails, log in again and pull the session back with tmux attach -t upgrade. The run carried on in the meantime.

Getting onto the server when SSH no longer answers

On KVM root servers and dedicated servers from KernelHost you reach the VNC console in the customer panel. It does not hang off the network stack of the guest system and works even when no service is listening any more. Log in through it once beforehand and make sure you know the root password.

In case a kernel refuses to boot, you also need a visible boot menu. On servers it is often hidden:

grep -E '^GRUB_TIMEOUT|^GRUB_TIMEOUT_STYLE' /etc/default/grub

If it shows GRUB_TIMEOUT_STYLE=hidden or GRUB_TIMEOUT=0, set GRUB_TIMEOUT_STYLE=menu and GRUB_TIMEOUT=5 and run update-grub. In an emergency, pick the previous kernel in the VNC console under "Advanced options".

Check the state and record it

Three checks up front, which turn something up more often than you would expect:

df -h / /boot /var
dpkg --audit
apt-get check

Success check: dpkg --audit prints nothing, apt-get check completes without an error line, and /boot has at least 300 MB free. A full /boot is the most common cause of an aborted kernel upgrade, and there is more on that in Disk full on Linux. If dpkg --audit reports something, repair that first with dpkg --configure -a.

Then write the current state to disk so that you can compare later:

dpkg --get-selections > /root/packages-before-upgrade.txt
apt-mark showhold > /root/holds-before-upgrade.txt
cp -a /etc/apt /root/apt-config-$(date +%F)

update, upgrade, full-upgrade and dist-upgrade

These four words get mixed up constantly. The difference is not cosmetic: it decides whether a new kernel gets installed and whether packages are allowed to disappear.

Two things follow from this. First: dist-upgrade has nothing to do with moving to a new distribution release. The name is historical, and the command stays inside your current release.

Second, the difference between apt upgrade and apt-get upgrade is exactly why some servers end up without a new kernel. Debian pulls the kernel in through the metapackage linux-image-amd64, Ubuntu through linux-image-generic or linux-image-virtual. On every ABI change that metapackage points at a new package whose version number is part of the name. apt-get upgrade installs no new packages as a matter of principle and therefore leaves the kernel where it is, while apt upgrade installs it because a dependency requires it. If you use apt-get upgrade in a maintenance script, you need --with-new-pkgs there as well.

On the choice of tool: called from a script, apt prints the line WARNING: apt does not have a stable CLI interface. Use with caution in scripts. That is not an error message, but it is a fair warning. In scripts and Ansible roles use apt-get, while apt is more convenient interactively.

The procedure with a check after every step

Step 1: fetch the package lists.

apt update

Success check: the output contains no line that begins with Err: or W:. At the end you get either All packages are up to date. or a number followed by packages can be upgraded. Every error line here means you would be working on from an incomplete picture.

Step 2: look at what would come. This step is missing from most guides and it is the most important one in the whole procedure.

apt list --upgradable
apt full-upgrade -s

The -s switch simulates and changes nothing. The lines under The following packages will be REMOVED: are the only ones you really have to check. If nothing is listed there, full-upgrade is just as harmless as upgrade. If a package you need is listed there, use apt upgrade and sort out the cause separately.

On Debian, apt install apt-listchanges is worth adding. Before anything is applied, the package shows you the changelogs and, more importantly, the NEWS files written by the package maintainers. That is where the parts requiring manual work are described.

Step 3: apply the updates.

apt upgrade

Stay with it. The run asks questions, and a -y answers only apt's own question, not the prompts about configuration files. Those come from dpkg and wait patiently until somebody answers.

Success check:

apt list --upgradable
dpkg --audit
systemctl --failed
journalctl -p 3 -b --no-pager | tail -n 20

What you expect: apt list --upgradable prints nothing except Listing..., dpkg --audit stays silent, and systemctl --failed reports 0 loaded units listed. What actually happened is recorded permanently in /var/log/apt/history.log, and the complete dpkg output in /var/log/apt/term.log.

Packages kept back: two different causes

When apt leaves packages out, there are two fundamentally different reasons for it, and they get confused regularly.

First, a genuine hold. Somebody pinned the package down explicitly:

apt-mark showhold

If the command prints something, that was a deliberate decision, usually for databases or kernel modules. Release it with apt-mark unhold PACKAGENAME, set it with apt-mark hold PACKAGENAME. A hold at the dpkg level shows up with dpkg --get-selections | grep -w hold.

Second, the message The following packages have been kept back:. That is not a hold. It means apt would have to install an additional package or remove one for this upgrade, and the command you used is not allowed to do that. The proof in a single line, without changing anything:

apt full-upgrade -s | head -n 20

If the package turns up there, you have found the explanation and apt full-upgrade resolves it. On Debian 13 the newer apt generation formats this output differently, but nothing changes in substance.

The special case of Ubuntu. Ubuntu rolls updates out in phases, so not every server gets them on the same day. A package can therefore be kept back even though no hold is set and no dependency is in the way. The diagnosis works by elimination: apt-mark showhold is empty, apt full-upgrade -s shows no removal, and apt-cache policy PACKAGENAME still names a newer candidate. In that case wait a few days, or pull the update in early:

apt -o APT::Get::Always-Include-Phased-Updates=true upgrade

Debian 13 and Debian 12 have no phased rollout, so this cause does not apply there in the first place.

When apt asks about a configuration file

This prompt appears only when two conditions are true at the same time: the file has been changed locally since installation, and the package ships a new version of it. It looks like this:

Configuration file '/etc/ssh/sshd_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sshd_config (Y/I/N/O/D/Z) [default=N] ?

The default is N, which keeps your version. That is the safe answer, though not the right one in every case.

With /etc/ssh/sshd_config, particular care is in order: a Y can reset PermitRootLogin and PasswordAuthentication to the package defaults, and your access depends on exactly those. That is why your own SSH settings belong in a separate file under /etc/ssh/sshd_config.d/, as described in Securing SSH and setting up key login. A file that does not exist in the package at all never triggers a prompt.

However you answer, dpkg throws nothing away. With N the new version lands next to it as .dpkg-dist, with Y your old one as .dpkg-old. Going through these files is the real follow-up work:

find /etc -name '*.dpkg-dist' -o -name '*.dpkg-old' -o -name '*.dpkg-new' -o -name '*.ucf-dist'
diff -u /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-dist

For unattended runs, in a maintenance script for example, fix the behavior in advance. This combination keeps your version and asks nothing:

DEBIAN_FRONTEND=noninteractive apt-get -y \
  -o Dpkg::Options::="--force-confdef" \
  -o Dpkg::Options::="--force-confold" \
  upgrade

--force-confnew would be the opposite and always takes the package version. On a server with its own configuration that is rarely what you want.

Kernel updates and the question of the reboot

A new kernel lands in /boot and in the boot menu when it is installed. The running kernel stays in memory unchanged until the reboot. A server can therefore be fully up to date and vulnerable at the same time. The simplest proof is a comparison:

uname -r
ls -1 /boot/vmlinuz-*

If /boot holds a higher version than uname -r reports, a reboot is due. On Ubuntu 24.04 and 22.04 there is also a marker file that takes library updates into account:

test -f /var/run/reboot-required && cat /var/run/reboot-required.pkgs

The second file names the packages that requested the reboot. On Debian 13 and Debian 12 this marker is not created reliably, so needrestart is the right route there.

needrestart: which services are still using the old library

An update to OpenSSL or glibc swaps out the file on disk. Every process that has already loaded it carries on with the old version. needrestart finds exactly those processes. On Ubuntu 24.04 and 22.04 it is preinstalled and reports in with a full-screen prompt after every upgrade, on Debian you install it afterwards:

apt install -y needrestart
needrestart -b

The -b mode produces machine-readable output. Two entries in it are decisive: NEEDRESTART-KSTA with the value 1 means the running kernel is the expected one, and any other value means a newer one is waiting. Every NEEDRESTART-SVC line names a service that should be restarted.

You control the behavior through the environment variable NEEDRESTART_MODE: a restarts services automatically, l only lists them, i asks. The same thing goes into /etc/needrestart/needrestart.conf to make it permanent. For scripts the variable is the better choice, because it changes nothing in the configuration:

NEEDRESTART_MODE=a DEBIAN_FRONTEND=noninteractive apt-get -y upgrade

One limit is worth knowing: needrestart looks at the processes of the host system. It does not renew services inside containers, and there you need new images.

The release upgrade is a category of its own

Moving from Debian 12 to Debian 13, or from Ubuntu 22.04 to 24.04, is not an upgrade in the sense described above. It swaps the package sources and replaces practically every package. Three rules always apply here: one release per run, no skipped intermediate releases, and a backup beforehand that the server can genuinely be restored from.

On Debian you switch the sources over first. Debian 12 uses /etc/apt/sources.list for that, Debian 13 the file /etc/apt/sources.list.d/debian.sources in the newer format. After that comes a deliberately two-stage procedure, exactly as the release notes prescribe:

apt update
apt-get upgrade --without-new-pkgs
apt full-upgrade

The middle step first updates the packages that need no restructuring. That keeps the number of packages in motion at any one time small and makes an abort repairable. Do not forget the non-free-firmware component: it has stood on its own since Debian 12 and is missing from old source files. If your apt knows the command apt modernize-sources (check with apt --version), it converts the old source file into the new format.

On Ubuntu there is a dedicated tool for this, and you should use nothing else:

apt install -y ubuntu-release-upgrader-core
do-release-upgrade -c
do-release-upgrade

The -c switch only checks and changes nothing. Whether an upgrade is offered depends on Prompt in /etc/update-manager/release-upgrades: with lts only the jump to the next LTS release appears, and only after its first point release. The path from 20.04 to 24.04 necessarily runs through 22.04.

One detail surprises many people: if do-release-upgrade runs over SSH, it starts an additional SSH service on port 1022 as a fallback and points out that you may have to open that port in the firewall. Use it, but do not rely on it. The tmux session and access through the VNC console are more dependable.

Cleaning up after the upgrade

After a larger run, three kinds of leftovers are lying around: downloaded package files, orphaned packages, and configuration remnants of removed packages.

apt autoremove --purge
apt autoclean

autoclean deletes from /var/cache/apt/archives only the files that are no longer offered anyway. apt clean empties the cache completely, so it frees more space, but it turns every reinstall into a fresh download.

You recognize configuration remnants by the dpkg status rc, meaning removed but configuration still present:

dpkg -l | awk '/^rc/ {print $2}'

Whatever is listed there you remove for good with apt purge PACKAGENAME. With old kernels more care is called for, because one wrong move leaves the server unbootable:

uname -r
dpkg -l 'linux-image-*' | awk '/^ii/ {print $2}'

Never remove the kernel that the first line names, and keep one older working kernel as well so the boot menu has a fallback. In most cases apt autoremove --purge gets this right on its own anyway.

Common errors and their fixes

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1234 (unattended-upgr): a second package operation is already running, usually the automatic security update. Do not abort it, let it finish. The complete procedure including the repair is in apt could not get lock.

E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.: a previous run was interrupted, typically by a dropped connection. Run exactly that command and repeat the upgrade afterwards.

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).: a package is installed but its dependencies are missing. apt --fix-broken install fetches them. If the error comes back, it is almost always a third-party repository that offers packages for a different distribution release.

E: Release file for http://deb.debian.org/debian/dists/trixie/InRelease is not valid yet (invalid for another 5h 3min 2s).: the server clock is running slow. Check with timedatectl status whether System clock synchronized: yes is reported, correct the time and repeat apt update. The repository is not the problem.

W: GPG error: ... The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ..., followed by E: The repository '...' is not signed.: the signing key of a third-party repository is missing or has been replaced. These days the key belongs in /etc/apt/keyrings/ and is referenced from the source entry through signed-by or through the field Signed-By:. apt-key is deprecated and should no longer be used.

E: Repository '... InRelease' changed its 'Suite' value from 'stable' to 'oldstable': this happens as a matter of course when a new Debian release comes out and your sources point at stable instead of at the codename. Confirm it once with apt update --allow-releaseinfo-change, then switch the sources to the codename. Otherwise a server that follows stable will change distribution release unintentionally at some point.

E: The repository 'http://... Release' does not have a Release file.: the repository offers nothing for your release, usually because a third-party repository does not support the codename yet, or because the distribution has reached its end of life. Disable the affected line and check the repository.

No space left on device in the middle of unpacking: /boot or /var is full. Tidy up with dpkg --configure -a, free some space, repeat. Before every kernel upgrade a look at df -h /boot pays off.

debconf: unable to initialize frontend: Dialog: only a notice, not a fault. It appears without a fully equipped terminal, inside a script for example. With DEBIAN_FRONTEND=noninteractive it disappears.

The four systems compared

The final check

The fact that a command came back without an error does not mean the system is in good shape. These six checks actually tell you something:

1. apt list --upgradable prints nothing except Listing....
2. apt-mark showhold contains only what you deliberately held back.
3. dpkg --audit stays silent.
4. needrestart -b reports NEEDRESTART-KSTA: 1 and no outstanding services.
5. systemctl --failed lists nothing.
6. find /etc -name '*.dpkg-dist' finds nothing any more, because you have worked through every difference.

If point four calls for a reboot, schedule it and carry it out. A server that waits for months on a pending reboot collects exactly the holes you were updating against in the first place. Afterwards check uname -r and systemctl --failed one last time. The reboot is the moment that shows whether everything comes back up.