Setting up swap and preventing out of memory crashes

The service is gone. No crash report, no stack trace, the log file ends in the middle of a line. MariaDB stops answering, nginx returns 502, the Minecraft server is offline, and the application itself shows nothing unusual. It is almost always the same pattern: the kernel ran out of free memory and terminated a process to keep the system alive. This article shows how to prove that beyond doubt, how to create a swap file cleanly and make the entry permanent, and in which cases swap only postpones the problem by a few minutes.

Proving it: dmesg and journalctl

Before you set anything up, you need the evidence. On every OOM kill (out of memory) the kernel writes a detailed block into the ring buffer:

dmesg -T | grep -iE 'out of memory|oom-kill'

If no line comes back and the command exits with code 1, there has been no kernel OOM since the last boot. That exit code is not a fault, it is simply the usual answer from grep when there is no match. If something does come back, it looks like this on all four systems covered here:

mariadbd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=1043,uid=107
Out of memory: Killed process 1043 (mariadbd) total-vm:2894760kB, anon-rss:1583204kB, file-rss:0kB, shmem-rss:0kB, UID:107 pgtables:3820kB oom_score_adj:0

Three things in there matter. First: invoked oom-killer names the process that requested the memory, which is not necessarily the culprit. Second: the process that was killed appears on the Killed process line. Third: what counts is anon-rss, the anonymous memory actually in use. total-vm is reserved address space and with Java or Go it is regularly a multiple of that, so it says nothing.

If dmesg runs without root, Debian 12, Debian 13, Ubuntu 22.04 and Ubuntu 24.04 all answer with dmesg: read kernel buffer failed: Operation not permitted, because kernel.dmesg_restrict is set to 1 everywhere. So work with sudo or as root. If the message stays even as root, you are sitting in an LXC or OpenVZ container that has no access to the host system's ring buffer. In that case only the second route through journalctl -k gets you further.

The ring buffer is empty after a reboot. Hence the second route through the journal, and this is where the trap sits that most guides walk straight past:

journalctl -k --grep "Out of memory"

-k implies -b and therefore shows the current boot only. If the box was rebooted after the incident, this command returns nothing even though the event was logged. For the previous boot, or for a time range:

journalctl -k -b -1 --grep "Out of memory"

journalctl --since "7 days ago" --grep "Out of memory|oom-kill"

If the first of the two commands answers with No journal boot entry found for the specified boot (-1), then there is simply no earlier boot stored in the journal. That is not a fault either, it is the normal case on a system whose journal has only been recording since the current boot.

All of this only works if the journal is persistent in the first place. Check that:

ls -d /var/log/journal

On Debian and Ubuntu this directory exists out of the box, so the test almost always returns a hit and the mkdir below is then a no-op that breaks nothing. If the directory really is missing, the journal lives in /run only and is lost on every reboot. Two commands make up for it:

mkdir -p /var/log/journal

systemctl restart systemd-journald

Kernel OOM or systemd limit? Two different causes

This distinction decides whether swap helps at all. Watch the constraint field in the kernel message.

CONSTRAINT_NONE means the whole system ran out of memory. Swap helps here.

CONSTRAINT_MEMCG means a single control group hit its limit while the rest of the system had plenty of headroom. Swap does not help here, either the limit has to go up or the application has to become more frugal. You can spot such kills on the service itself as well:

systemctl status mariadb

If it says Main process exited, code=killed, status=9/KILL and further down Failed with result 'oom-kill', it was an OOM kill. Whether a cgroup limit was involved is revealed by a counter file. It requires cgroup v2, which is the default on all four distributions; under the older cgroup v1 the path does not exist:

cat /sys/fs/cgroup/system.slice/mariadb.service/memory.events

systemctl show mariadb -p MemoryMax -p MemoryHigh

A value greater than zero for oom_kill together with a configured MemoryMax is the proof of a local limit.

There is a third candidate that often gets overlooked because it writes nothing to dmesg at all: systemd-oomd. The service works in userspace, evaluates the PSI pressure indicator and terminates entire control groups before the kernel steps in. Its message in the journal reads roughly like Killed /system.slice/... due to memory pressure for /system.slice being 60.00% > 50.00% for > 20s with reclaim activity. Check whether it is running:

systemctl is-active systemd-oomd

On Ubuntu, systemd-oomd has been installed and active out of the box since 22.04, on Debian it is not part of the standard set. An inactive on a Debian server is therefore expected and no indication of a fault.

Important for later: systemd-oomd can also trigger because the swap is filling up. With oomd active, more swap can therefore cause the kills to happen earlier rather than later.

Before the swap: how much memory is really missing

Two minutes of measuring save you one wrong decision.

free -h

The only interesting column is available, not free. Cache counts as available and is released when needed. Anyone reading the free column will consider every healthy system overloaded.

ps -eo pid,comm,rss,%mem --sort=-rss | head -n 11

systemd-cgtop --order=memory -b -n 1

The first line shows the largest individual processes, the second groups them by service. In practice the same three suspects turn up almost every time: MariaDB with an oversized innodb_buffer_pool_size, PHP-FPM with too high a pm.max_children and a JVM with an overly generous -Xmx.

As a starting size for the swap file this table is enough. Bigger is not better on a server, because swap that you genuinely use in full makes the machine unusable.

Creating the swap file

First check whether swap already exists. Ubuntu servers from the ISO installer often ship with /swap.img in place, Debian from the installer usually with a real swap partition. Cloud images of both distributions normally have nothing at all.

swapon --show

If the output stays empty, there is no swap. Next check the file system, because the procedure depends on it:

findmnt -no FSTYPE -T /

With ext4 or xfs you can carry straight on. Create the file with dd, not with fallocate. fallocate is faster, but depending on the file system and kernel it produces a file with unwritten regions, and swapon then rejects it. dd writes real zeros and works everywhere:

dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress

chmod 600 /swapfile

Before the next step, one check is worth doing that many people skip. If an active swap already sits under /swapfile, from an earlier attempt or from the image default, mkswap refuses to work with mkswap: error: /swapfile is mounted; will not make swapspace. The only thing that counts is whether the path is listed as active in /proc/swaps. So look, and switch it off if needed:

swapon --show

swapoff /swapfile

If swapon --show returns no line with /swapfile, you can skip the swapoff. After that, write the swap area:

mkswap /swapfile

mkswap answers with Setting up swapspace version 1, size = 2 GiB (2147479552 bytes) and a new UUID. Only then switch it on:

swapon /swapfile

The order is not negotiable. chmod before mkswap, mkswap before swapon, and any swapoff before all of it.

How you can tell that it really worked

The fact that swapon runs through without an error message is no proof yet. These two outputs are the proof:

swapon --show

free -h

swapon --show has to print a line with /swapfile, file, the size and a priority. In free -h the Swap: line must have jumped from 0B to the new size. If either of the two outputs stays unchanged, the swap is not active, no matter what the command reported beforehand.

Making it permanent without risking the boot

A swapon does not survive a reboot. The entry belongs in /etc/fstab, and that is exactly where servers get wrecked. Back it up first:

cp /etc/fstab /etc/fstab.bak

echo '/swapfile none swap sw 0 0' >> /etc/fstab

Watch the two greater-than signs. A single > overwrites the entire file, and then the machine no longer starts cleanly. After that, check the syntax before you reboot:

findmnt --verify

One warning is completely normal and no reason to remove the line again: [W] non-bind mount source /swapfile is a directory or regular file. With a swap file that is unavoidable, and the command still exits with return code 0. If the file is also missing a valid swap header, [W] cannot detect on-disk filesystem type is added, and then you really have forgotten a mkswap.

If you already switched the swap on by hand with swapon /swapfile above, it is active now. If not, swapon -a activates all entries from the fstab without a reboot. The real test, though, is a different one. systemd generates its own unit from every fstab line, and for /swapfile it is called swapfile.swap. If that unit appears and is active, the swap is guaranteed to be mounted at the next boot:

systemctl daemon-reload

systemctl list-units --type swap

Expect a line reading swapfile.swap loaded active active Swap. If it is missing, the fstab line is wrong and a reboot would end without swap. Only once that fits is the reboot worth doing, followed by a check through swapon --show.

Setting swappiness correctly

The kernel parameter vm.swappiness controls how readily anonymous pages are swapped out instead of file cache being dropped. The default is identical on Debian 12, Debian 13, Ubuntu 22.04 and Ubuntu 24.04, namely 60:

cat /proc/sys/vm/swappiness

Since kernel 5.8 the value range runs from 0 to 200, and all four distributions are above that version. Two widespread misconceptions: vm.swappiness=0 does not switch swap off, it only prevents precautionary swapping and still lets the kernel swap before it kills. And a low value does not make a system faster when the memory is simply not there.

Reasonable values: 10 to 20 on database servers, 60 on mixed web servers, 100 and above if you use zram. To make that permanent it belongs in a file of its own under /etc/sysctl.d/, not in /etc/sysctl.conf, which gets in the way during package updates:

echo 'vm.swappiness = 10' > /etc/sysctl.d/99-swappiness.conf

sysctl --system

cat /proc/sys/vm/swappiness

The third command is the success check. sysctl --system reads all directories in a fixed order, and a file that already exists with a higher number can override your value.

When it goes wrong: the error messages verbatim

swapon: /swapfile: insecure permissions 0644, 0600 suggested. Only a warning, the swap runs anyway. Fix it regardless, otherwise any user can read the contents of swapped-out processes: chmod 600 /swapfile.

swapon: /swapfile: swapon failed: Invalid argument The most common error. Either mkswap was forgotten, or the file contains holes. The kernel log then additionally shows swapon: swapfile has holes. Solution: delete the file and create it again with dd instead of fallocate.

On btrfs the same error applies, plus BTRFS warning: swapfile must not be copy-on-write. Here the order is a different one: the file has to be created empty and marked as not copy-on-write before it is filled. One note up front that can decide the fate of a running server: truncate -s 0 and rm also run through when an active swap is still mounted under that path. The kernel then points at blocks that no longer exist. If swapon --show lists the path as active, a swapoff /swapfile is therefore mandatory beforehand.

truncate -s 0 /swapfile

chattr +C /swapfile

After that, fill it as usual with dd, chmod 600, mkswap, swapon. On compressed subvolumes and inside snapshots, swap still does not work.

swapon: /swapfile: swapon failed: Operation not permitted You are sitting in a container. LXC, OpenVZ and Docker share the host's kernel and are not allowed to switch on a swap of their own. Check with:

systemd-detect-virt

If the command reports kvm, qemu or none, a kernel of its own is running and swap is possible. If it reports lxc, openvz or docker, only more RAM or a product with full virtualization helps. The KVM root servers and dedicated servers from KernelHost come with their own kernel, so swap can be set up there without restrictions.

dd: error writing '/swapfile': No space left on device The disk is too full. First df -h, then remove the half-written file with rm /swapfile, otherwise it keeps occupying space. The same applies here: if swapon --show shows the path as active, a swapoff /swapfile belongs before the deletion.

The system no longer boots and the emergency console appears. Almost always a typo in /etc/fstab. Log in through the console in the customer panel, then mount -o remount,rw /, remove the faulty line or copy /etc/fstab.bak back, and reboot. That is exactly what the backup copy was made for.

Differences between Debian 13, Debian 12, Ubuntu 24.04 and 22.04

The commands are identical on all four systems, the starting state is not.

• Existing swap: Ubuntu Server from the ISO installer often creates /swap.img, Debian from the installer a swap partition. Cloud images of both distributions come without swap. Always run swapon --show first.
• systemd-oomd: installed and active out of the box on Ubuntu since 22.04, on both Debian versions not part of the standard set. That explains why identical software can die in different ways on two apparently identical systems.
• Database: Debian 12 and Debian 13 ship no mysql-server, so MariaDB always runs there (10.11 on Debian 12, 11.8 on Debian 13). Ubuntu 22.04 and 24.04 have both. The defaults for innodb_buffer_pool_size differ accordingly, and on small servers that very value is the most common cause of OOM.
• Java: Debian 12 only knows OpenJDK 17, Debian 13 only OpenJDK 21, Ubuntu 22.04 and 24.04 cover 8 through 21. A JVM without a configured -Xmx takes a quarter of the RAM by default, and with several instances the OOM kill is a certainty.
• Kernel messages: format and wording of the OOM message are the same on all four systems, the examples above fit everywhere.
• swappiness: 60 as the default everywhere.

When swap helps and when it only postpones the problem

Swap reliably helps with short peaks, during a backup, a package upgrade or a nightly import for example. It helps with services that occupy a lot of memory and then do nothing for days, because those pages can happily sit on disk. And in an emergency it buys you minutes in which the SSH session still responds and you can intervene, instead of standing in front of a dead server.

Swap does not help when the permanent demand is simply higher than the RAM. The system then starts to thrash: it swaps out and back in without a break, the load climbs into double digits, CPU usage stays low, everything waits on I/O. In practice that is worse than a clean OOM kill, because even the login no longer gets through. Two commands show whether you are in that state:

vmstat 1 5

test -e /proc/pressure/memory && cat /proc/pressure/memory

With vmstat the columns si and so are what count. Permanently three-digit values mean active swapping in and out. In /proc/pressure/memory, full avg10 is decisive: values above 10 mean the entire system spends ten percent of its time waiting for memory. Above 40 the machine is practically dead. The file only exists, however, if Pressure Stall Information is active in the kernel. The standard kernels from Debian and Ubuntu bring it along, other kernels not necessarily. Hence the safeguard with test -e: if the file is missing, the output stays empty instead of you mistaking a No such file or directory for a defect. PSI can be retrofitted through the kernel boot parameter psi=1.

Which processes actually sit in swap is answered by this line:

grep VmSwap /proc/*/status | sort -k2 -rn | head

Swap is equally useless against a cgroup limit (CONSTRAINT_MEMCG), against memory pinned with mlock and against a JVM whose heap is configured larger than the available RAM. The garbage collector regularly walks the whole heap and pulls every swapped-out page straight back in.

The three tools for the case that swap is not enough

zram creates a compressed swap area in the RAM itself. It is orders of magnitude faster than a file on disk and, depending on the data, gives you 20 to 40 percent more effective memory:

apt update

apt install -y zram-tools

Configuration happens in /etc/default/zramswap through ALGO=zstd and PERCENT=50, followed by systemctl restart zramswap. Check with zramctl and once again swapon --show, where /dev/zram0 then shows up. With zram, vm.swappiness belongs up at 100 to 180, because swapping is cheap here.

earlyoom steps in before the kernel freezes the system, and it kills selectively instead of by heuristic:

apt install -y earlyoom

In /etc/default/earlyoom you set the thresholds through EARLYOOM_ARGS and protect important processes, for example with -m 5 -s 5 --avoid '(^|/)(sshd|systemd)$'. That keeps the SSH login reachable while the memory hog goes down.

Fixed limits per service are the cleanest solution when one particular service breaks out regularly. Through systemctl edit mariadb you enter MemoryHigh=1200M and MemoryMax=1500M. The service is then throttled and, if it comes to that, terminated on its own instead of dragging the whole server down with it. Check through systemctl show mariadb -p MemoryMax.

In most cases, though, the actual fix remains the application configuration: innodb_buffer_pool_size at a realistic size, pm.max_children based on the real memory demand per worker, a configured -Xmx for every JVM. Swap is the safety net, not the solution.

Rolling it back if it does not fit

The way back is short and should go in this order:

swapoff /swapfile

With a heavily used swap the command can run for several minutes, because all pages have to go back into RAM. If it runs into swapoff: /swapfile: swapoff failed: Cannot allocate memory, there is no room in RAM for the transfer back, and services have to be stopped first. Only after a successful swapoff should you remove the fstab line and delete the file:

rm /swapfile

swapon --show

Deleting the file while it is still mounted leads to a system whose memory management points at an inode that no longer exists. That ends badly. To finish, run free -h once more and reboot as a cross-check.