Disk full: finding and freeing up space on a Linux server

A full disk rarely announces itself politely. Usually a service fails first: MariaDB writes OS error 28 to the log, nginx answers with a 500, a backup aborts with write error: No space left on device, and if things go really badly, nobody reaches the machine over SSH any more because sshd cannot create session files. This article covers the whole path: measure, locate, free up, verify. Including the two cases where most guides stop, namely when df reports full and du finds nothing.

Measure first: which partition is actually full?

Before you delete anything, you need to know which file system is affected. A full /boot has completely different causes than a full /var.

df -h
df -hT -x tmpfs -x devtmpfs -x squashfs

The second line hides the pseudo file systems. On Ubuntu that is particularly useful, because every installed snap application shows up as its own squashfs loop mount and clutters the output. Those loop mounts are always 100% full, by the way, which is normal and not a problem.

The column that matters is Mounted on. Typical setups on servers:

One more detail that often causes confusion: by default, ext4 reserves five percent of the capacity for the root user. A service running as www-data or mysql therefore hits No space left on device while root can still write without any trouble at that very moment. On pure data partitions (so not on the root file system) you can lower that reserve safely:

tune2fs -m 1 /dev/sdb1

On / the reserve should stay. It is exactly the cushion that lets you repair a system that has run full in the first place.

Using du properly instead of getting lost

The classic approach is one level at a time, always with -x:

du -xh --max-depth=1 / | sort -h
du -xh --max-depth=1 /var | sort -h

The -x is the most important switch in this whole article. It keeps du inside a single file system and stops the command from descending into /proc, /sys, mounted network shares or backup disks. Without -x the search runs for minutes and returns numbers that have nothing to do with the full partition. sort -h sorts the human-readable sizes correctly, so the biggest chunk ends up at the bottom.

If you prefer to click through interactively, install ncdu and start it with -x as well:

apt-get install -y ncdu
ncdu -x /

Individual large files are found faster directly:

find /var -xdev -type f -size +100M -exec ls -lh {} +

Two pitfalls regularly lead to wrong conclusions here. First: du counts allocated blocks, not the logical file size. With sparse files (database tablespaces, virtual disk images) the two values differ widely. A comparison makes that visible:

du -sh /var/log
du --apparent-size -sh /var/log

Second: du counts hard links only once. If you work without root, you also get lines like du: cannot read directory '/var/lib/private': Permission denied and therefore systematically undersized totals. So run every analysis as root or with sudo.

Limiting the systemd journal

On servers, the journal is the most common silent storage hog. By default it may use ten percent of the file system, capped at four gigabytes, and on top of that it keeps 15 percent of the file system free. On a 500 gigabyte disk that means up to four gigabytes of pure log.

journalctl --disk-usage

There is a real difference between the distributions here, and many guides leave it out. Whether the journal ends up on the disk at all depends, with the default setting Storage=auto, purely on whether the directory /var/log/journal exists:

ls -d /var/log/journal
ls -d /run/log/journal

On minimal Debian installations and on many cloud images of Debian 12 and Debian 13, /var/log/journal does not exist. The journal then lives under /run/log/journal, so in RAM, disappears on every reboot and puts no load on the disk at all. It loads the RAM instead. Ubuntu Server 22.04 and 24.04, by contrast, usually create the directory and log persistently to disk. Check instead of guessing.

To free up space immediately:

journalctl --rotate
journalctl --vacuum-size=200M
journalctl --vacuum-time=7d

The --rotate call in front of it is not decoration: the vacuum options only delete journal files that have already been archived, never the currently active one. If the active file makes up most of the size, nothing appears to happen without rotating first, and that is exactly where readers stumble who copy the command out of a forum post.

For a permanent limit, use a file of your own so that later package updates do not overwrite anything:

mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nSystemMaxUse=200M\nRuntimeMaxUse=50M\n' > /etc/systemd/journald.conf.d/00-size.conf
systemctl restart systemd-journald

To confirm that it really took effect: journalctl --disk-usage must now report a smaller value, and df -h must show more free space. If the journal shrinks while df stays unchanged, some process is still holding deleted files open. More on that below.

Second distribution difference: on classic server installations of Debian and Ubuntu, rsyslog often runs alongside and writes the same messages a second time to /var/log/syslog. On minimal Ubuntu images (cloud, container) rsyslog is missing. Check with ls -l /var/log/syslog. If the file exists and is huge, the journal is not the problem: a missing or broken logrotate rule under /etc/logrotate.d/ is.

apt cache and old kernels

Downloaded packages stay behind after the installation. On a long-running server that quickly adds up to several gigabytes.

du -sh /var/cache/apt
apt-get clean
du -sh /var/lib/apt/lists

apt-get clean empties /var/cache/apt/archives completely, apt-get autoclean only removes the packages that no longer exist in the sources. What clean does not touch are the package lists under /var/lib/apt/lists. With many repositories configured they can reach several hundred megabytes, and they can be rebuilt safely:

rm -rf /var/lib/apt/lists/*
apt-get update

The second line is not optional here, it is mandatory, and it has to run right away. Between deleting the lists and the next apt-get update, apt does not know a single package: every apt-get install in that state aborts with E: Unable to locate package ... and exit code 100, even though the package obviously does exist in the sources.

The second classic is old kernels. Ubuntu keeps installing new kernels through unattended-upgrades, but only clears out the old ones if you explicitly allow it. Each kernel plus its initramfs takes up roughly 100 to 150 megabytes in a /boot that is often only 512 megabytes to one gigabyte in size.

uname -r
dpkg -l 'linux-image-*'
apt-get autoremove --purge

The running kernel from uname -r is never removed, and neither is the newest one. On Ubuntu you can prevent the problem by setting the line Unattended-Upgrade::Remove-Unused-Kernel-Packages "true"; in /etc/apt/apt.conf.d/50unattended-upgrades. On Debian 12 and 13, unattended-upgrades is not active by default, so /boot only grows there if somebody updates by hand regularly and never cleans up.

When /boot is already full and apt no longer completes

This is the case that really hurts. Typical messages, word for word:

update-initramfs: failed for /boot/initrd.img-6.8.0-60-generic with 1.
dpkg: error processing package linux-image-6.8.0-60-generic (--configure):
 installed linux-image-6.8.0-60-generic package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)

The package database is now in a half-finished state, and every further apt call fails at the same point. The way out, in this order:

1. Note down uname -r. That version is not touched under any circumstances.
2. Print ls -lh /boot and pick out the oldest version that is not running.
3. Delete only its initrd.img-*, not the vmlinuz-*. The initramfs file is by far the largest one and can be regenerated at any time.
4. Run apt-get -f install so that dpkg can finish the interrupted configuration.
5. Only then apt-get autoremove --purge, so that the old packages disappear cleanly together with their GRUB entry.
6. Run update-grub and read the output.

What you should not do: delete kernel files from /boot at random with rm and ignore the rest. dpkg then still believes the packages are installed, GRUB offers entries that point nowhere, and the next reboot ends up in the GRUB rescue prompt. If you have already deleted something, bring the consistency back with apt-get install --reinstall of the affected package or with dpkg --purge, followed in every case by update-grub.

Verification: df -h /boot shows free space again, dpkg -l 'linux-image-*' lists only two or three entries with status ii, and the output of update-grub names exactly the kernels that are actually present in /boot.

Docker, Snap and container logs

On Docker hosts the answer is almost always in /var/lib/docker. Do not guess, ask:

docker system df
docker system df -v

The verbose variant separates images, containers, volumes and build cache cleanly. After that, clean up in a targeted way:

docker image prune -a
docker builder prune
docker system prune -a

A word of caution about --volumes: this switch also deletes volumes that no running container is attached to. If you run your database in a named volume and have just stopped the container, you lose the data. Without a current backup, docker system prune -a --volumes has no place on a production server.

The underestimated item is the container logs under /var/lib/docker/containers/*/*-json.log. The default json-file driver does not rotate at all unless you tell it to. Over a few months, a talkative container writes double-digit gigabytes into a single file. The remedy goes into /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "3" }
}

Then systemctl restart docker. Careful, and hardly any guide mentions this: the setting only applies to newly created containers. Existing ones keep their old configuration until they are created again once, which with Compose means docker compose up -d --force-recreate.

The Docker versions differ noticeably here. Debian 12 ships docker.io 20.10, Debian 13 brings 26.1, Ubuntu 22.04 and 24.04 are now at 29.1. The newer the version, the more space the BuildKit cache takes up, and the more important docker builder prune becomes, because docker system prune on its own does not always clear it completely.

On Ubuntu, Snap comes on top of that. Old revisions stay behind in a disabled state and keep occupying space:

snap list --all
snap set system refresh.retain=2

The value 2 is the minimum that snapd accepts, smaller values are rejected with an error message. Debian does not ship Snap, so this section simply does not apply there.

df reports full, du finds nothing

Now for the interesting case. df -h shows 100%, but the total from du only adds up to half of that. There are exactly four plausible causes.

Deleted files that are still open

This is by far the most common reason. Somebody ran rm /var/log/riesig.log while a service still had the file open. The directory entry is gone, which is why du no longer sees anything. The blocks stay allocated until the last file descriptor is closed, which is why df still sees them.

apt-get install -y lsof
lsof +L1

The NLINK column then shows a 0, and (deleted) appears after the path. If lsof finds nothing, there is no output and the return value is 1, which is not an error. Without lsof you can go through the process file system directly:

ls -l /proc/*/fd 2>/dev/null | grep deleted

Important: run this as root, without exception. As a normal user you only see your own descriptors and therefore miss exactly those system services that are the culprit in nine out of ten cases.

The clean way to release the space is a restart of the service, for example systemctl restart rsyslog. If a restart is out of the question, the file can be truncated to zero length through its descriptor. Process ID and descriptor number come from the lsof output:

truncate -s 0 /proc/1234/fd/7

That frees the blocks immediately, and the process keeps writing afterwards. This method is meant for plain log files that are opened in append mode. Never apply it to database files, virtual machine images or anything else with random access, because there it leads to data loss.

How you can tell that it worked: df -h shows more free space right away, and lsof +L1 no longer lists the entry. A second du run, on the other hand, does not change at all, because the file was already invisible there before. That is also exactly why rebooting the server seems to solve the problem by magic.

Files hidden underneath a mountpoint

A classic: somebody wrote data to /mnt/backup before the actual disk was mounted there. The data still sits on the root file system, but it is covered by the mount on top of it. It only becomes visible through a second view of the same file system:

mkdir -p /mnt/rootview
mount --bind / /mnt/rootview
du -xh --max-depth=2 /mnt/rootview | sort -h
umount /mnt/rootview

The bind mount is harmless, it does not move anything around and is removed again with umount.

The root reserve and the permission mistake

The five percent ext4 reserve mentioned earlier explains the gap between "not quite full yet" and "services are already failing". And finally: whoever starts du without root does not see entire directory trees. The apparent difference is then simply a permission problem. On btrfs and ZFS, snapshots come into play as an additional explanation, and du never shows those either.

When it is not the space running out but the inodes

There is a second kind of "full" that produces exactly the same error message. Every file and every directory occupies an inode, and on ext4 their number is fixed from the moment of formatting.

df -i
stat -f /

If the IUse% column in df -i reads 100 while df -h reports plenty of free space, the diagnosis is clear: the server does not have a space problem, it has a count problem. Millions of tiny files have used up every inode. The error message is still No space left on device, and that is exactly why most people look in the wrong place.

You find the culprits by counting files instead of bytes:

find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

Frequent candidates are the mail queue under /var/spool/postfix or /var/spool/exim4, PHP sessions under /var/lib/php/sessions, cache directories of web applications, sprawling Node dependency trees and a /tmp that is never cleared.

The deletion itself brings the next hurdle. rm /var/lib/php/sessions/* fails with very many files, reporting bash: /usr/bin/rm: Argument list too long, because the command line exceeds the size limit. The approach that always works:

find /var/lib/php/sessions -type f -mtime +7 -delete

What you need to know: the inode count of an existing ext4 file system cannot be raised afterwards. The only options are growing the file system (the inodes grow proportionally with it) or creating it again with a denser allocation, for example with mkfs.ext4 -i 8192 /dev/sdb1. If you can foresee working with very many small files, on mail servers or image archives for instance, you are better served with XFS, because XFS allocates inodes dynamically and practically only runs out when the space runs out as well. Debian and Ubuntu format with ext4 by default, XFS has to be chosen deliberately.

Verifying the result and keeping it from coming back

A cleanup only counts as successful once three things line up: df -h shows more free space, df -i shows a falling inode usage, and the service that failed in the first place is running again. A command that ran through without an error message is no proof on its own. journalctl --vacuum-size and docker system prune in particular are fond of doing precisely nothing without saying a word.

For day-to-day operation, a short list has proven itself: limit the journal hard with SystemMaxUse, set Docker log rotation in the daemon.json, enable Remove-Unused-Kernel-Packages on Ubuntu, cover your own application logs through /etc/logrotate.d/ and set up a simple cron job that sends a mail once a threshold is crossed. On KVM and dedicated servers it is also worth putting /var, or at least /var/log, on a separate partition. A log running wild then takes down a service but not the operating system, and you can still reach the machine over SSH in any case.