journalctl: analyzing systemd logs and keeping them permanently

A server that is behaving strangely has usually written down what happened long ago. The problem is not a lack of information, it is the sheer amount of it. Run journalctl without arguments and you land at the very beginning of the journal, paging through weeks of old messages. This guide shows you how to cut out exactly the section that belongs to the incident, in three or four commands.

The article creating a systemd service covers the basic forms -u, -b and -f. This one is about everything that comes after that: time windows, filters, priorities, output formats, keeping the journal permanently, size limits, kernel messages and ready-made search patterns.

Everything here applies to Debian 13 (trixie), Debian 12 (bookworm), Ubuntu 24.04 LTS and Ubuntu 22.04 LTS. The commands are written for working as root. If you work as a normal user, prefix every command with sudo.

Before you change anything: your way back

Reading the journal is harmless, and not a single search command in this article changes anything on the system. Exactly three things carry risk, and all of them come later.

A full file system. Without a limit of your own, the journal takes up to ten percent of the file system, capped at four gigabytes. On a small disk, a full /var is the short road to a server that no longer accepts a login.

A typo in the configuration. systemd ignores unknown keys instead of stopping the service. Your limit then sits in the file without taking effect, and nobody tells you unless you ask.

Cleaning up too eagerly. journalctl --vacuum-time=1d deletes irreversibly, including the very evidence you are looking for.

The rescue route for the worst case: KVM root servers and dedicated servers from KernelHost have no IPMI and no iDRAC. The access path that works even without a network is the VNC console in the customer panel. It hangs off the virtualization layer, or off the port itself, not off the network stack of the guest system. Log in there once beforehand, and make sure you know the root password.

Taking stock in five commands

systemctl --version | head -n 1

systemctl is-active systemd-journald

journalctl --disk-usage

ls -d /var/log/journal /run/log/journal

df -h /var

The fourth command is the most important one: it answers whether your journal survives a reboot, and it reports No such file or directory for whichever directory does not exist. Every change in this article later goes into a file of its own under /etc/systemd/journald.conf.d/. The shipped /etc/systemd/journald.conf stays untouched, and rolling it all back takes two commands: delete the file, restart the service.

Narrow the time window instead of scrolling

Almost every incident has a timestamp. That is where the analysis starts, not with the service name.

journalctl --since "-30min"

journalctl --since "2026-09-02 03:10" --until "2026-09-02 03:20"

journalctl --since yesterday --until today

--since and --until have the short forms -S and -U. As a time value, journalctl understands absolute values in the form 2026-09-02 03:10:00, dates without a time (midnight applies then), the keywords yesterday, today, tomorrow and now, plus relative values such as -1h, -30min or 2 days ago.

There is a trap here that costs a lot of time: journalctl works in the local time of the server, while applications often log in UTC. Drop a timestamp from an application log straight into --since and in summer you are searching two hours away from the event. Check the zone, or have everything displayed in UTC right away:

timedatectl

journalctl --since "2026-09-02 01:10" --until "2026-09-02 01:20" --utc --no-pager

Success check: if -- No entries -- comes back, either the window is too narrow or the zone is wrong. Widen the window to an hour as a test before you start doubting the filters.

Boots as a time window

Often the right section is not a range of clock times but a single boot. -b without a value means the current one, -b -1 the one before it.

journalctl --list-boots --no-pager

journalctl -b -1 -p err --no-pager

The list shows one line per boot with an index, and the current one carries the 0. If only that single line appears, the journal is probably not stored permanently at all. -b -1 then answers on Debian 13 with No journal boot entry found for the specified boot (-1) and on Ubuntu 24.04 with No journal boot entry found from the specified boot offset (-1). That is not a defect, it is the statement that there is nothing to fetch there.

Filtering by service, process and priority

The unit, and why the name has to match exactly

journalctl -u nginx.service --since "-2h" --no-pager

-u compares for equality, not for similarity. That produces a particularly unpleasant kind of error, because it never generates an error message: on a Debian system with MariaDB, journalctl -u mysql returns -- No entries -- even though the journal is full of database messages. mysql.service is only an alias for mariadb.service there. systemctl resolves such aliases, the journal stores entries under the real name. So you get a wrong answer that does not look like an error.

You get the real name out of the journal itself. -F lists every value a field has ever taken there:

journalctl -F _SYSTEMD_UNIT | sort | grep -i sql

-u also accepts patterns, which takes the sting out of the question:

journalctl -u "mysql*" -u "mariadb*" -n 60 --no-pager

The same check is worth doing for SSH on Ubuntu 24.04, because the service starts through socket activation there and the logins do not necessarily sit under the name you expect:

journalctl -F _SYSTEMD_UNIT | grep -i ssh

Identifier instead of unit

Besides the unit there is the sender identifier, which is what classic syslog carries as the program name. The filter for it is -t:

journalctl -t sshd -t sshd-session -n 50 --no-pager

Why two? Since version 9.8, OpenSSH moves the sessions into a separate process named sshd-session. On Debian 13 (OpenSSH 10.0), all that is left under -t sshd is the fact that the service is listening, while every login sits under -t sshd-session. Debian 12 (9.2), Ubuntu 24.04 (9.6) and Ubuntu 22.04 (8.9) do not have this split. Filter only on -t sshd on Debian 13 and you will consider a server quiet while every single attempt against it is being logged. The route through the unit is more robust, because the child processes belong to the same one:

journalctl -u ssh --since "-24h" --no-pager

Arbitrary fields, and how they are combined

Every entry carries fields that you can write directly as a filter, for example _COMM (program name), _PID, _UID, _SYSTEMD_UNIT or _TRANSPORT. The rules for combining them are frequently assumed wrong:

• Two filters on different fields are combined with AND.
• Two filters on the same field are combined with OR.
• A single plus sign between two groups combines those groups with OR.

journalctl _SYSTEMD_UNIT=ssh.service _UID=0 --since "-1h" --no-pager

journalctl _SYSTEMD_UNIT=ssh.service + _SYSTEMD_UNIT=nginx.service --since "-1h" --no-pager

Field names are written in capital letters, and the comparison runs against the complete value. So journalctl unit=nginx is not a filter on a substring, it is a syntax error that journalctl answers with Failed to add match.

Priorities

Every entry carries an urgency level from 0 to 7. -p filters on it, always including every more urgent level: -p err also shows crit, alert and emerg.

journalctl -b -p err --no-pager

journalctl -u nginx.service -p 2..4 --since "-24h" --no-pager

And now the pitfall that priority filtering regularly runs into: whatever a service writes to standard output ends up in the journal as info by default, even when it went to standard error. A program that prints an exception together with a stack trace therefore shows up as a harmless operational message, and -p err hides it. Only programs that write directly against the journal interface, or that put a syslog prefix such as <3> in front of their lines, get a matching level. For your own services you therefore filter by unit and text, while for system services and for the kernel -p is very much usable.

Text search

journalctl -u nginx.service --since "-24h" --grep "upstream timed out" --no-pager

--grep (short form -g) searches the message text only, not the remaining fields. Case is ignored automatically as long as the pattern is written entirely in lower case; as soon as a capital letter appears, the comparison becomes exact. You can force either variant with --case-sensitive=yes or --case-sensitive=no. Regular expressions are allowed, so the vertical bar acts as OR.

Following the log live

-f attaches to the end of the journal and shows new lines as soon as they arrive. That is only really useful with a filter, otherwise half the system rushes past. -n decides how many past lines appear first; without a value it is ten. You stop it with Ctrl+C.

journalctl -u nginx.service -f -n 100

journalctl -f -u nginx.service -u php8.2-fpm.service

journalctl -f -p warning

The usual routine for a reproducible error: watch along in the first session, trigger the error in a second one. With very wide lines, -o cat helps, because only the message text appears then, without timestamp and sender.

Success check: if nothing at all arrives when you trigger it, the service is not logging to the journal but to a file of its own. With web servers and databases that is the normal case. The way forward then runs through the configuration of the application, not through more journalctl options.

Output formats

The default format is designed for humans at a terminal. For lining logs up against each other, for pipes and for analysis there are better fits. You switch with -o:

journalctl -u ssh.service -n 1 -o verbose --no-pager

This one command is worth more than any list of fields in a guide: you see which fields your entries actually carry, and you can then use any of them as a filter. Four options round that out:

• --no-pager switches the pager off. In scripts and in front of every pipe it belongs there, otherwise the call waits for a key that nobody presses.
• -r reverses the order, so the newest lines are at the top.
• -e jumps straight to the end inside the pager.
• -x adds an explanatory catalog text to systemd's own messages.

With --output-fields= you can boil the output down to the fields that interest you. It only takes effect with verbose, json and related formats:

journalctl -u ssh.service --since "-1h" -o json --output-fields=MESSAGE,_PID --no-pager

Keeping the journal across reboots

Whether the journal survives a reboot is decided by the default setting Storage=auto, following one simple rule: if /var/log/journal exists, everything is written there and kept. If it does not exist, everything ends up in memory (RAM) under /run/log/journal and is gone after the next reboot. Do not rely on the distribution, both states occur across installation variants and cloud images. A journal in RAM is the most common explanation for why nobody can say what happened before a crash.

If you switch this over, set the limit in the same step. Experience shows it never gets done afterwards:

mkdir -p /etc/systemd/journald.conf.d

cat > /etc/systemd/journald.conf.d/10-kh-journal.conf <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=1month
EOF

systemctl restart systemd-journald

journalctl --flush

Storage=persistent creates the directory itself, so no mkdir is needed for that. journalctl --flush moves whatever is still sitting in /run over to /var/log/journal.

Success checks, in this order:

systemd-analyze cat-config systemd/journald.conf | grep -v '^#'

ls -d /var/log/journal

journalctl -u systemd-journald -b -n 20 --no-pager

The first command shows the effective configuration built from the main file and all drop-in files, so what the service really read. The third one is the typo test: if there is a line containing Unknown key, your setting has no effect. Depending on the systemd version it reads Unknown key name 'SystemMaxUsage' in section 'Journal', ignoring or Unknown key 'SystemMaxUsage' in section [Journal], ignoring.

The final proof is a real reboot. After it, journalctl --list-boots has to show at least two lines and journalctl -b -1 -n 20 has to return entries. If you prefer to create the directory by hand, take the following route. The call to systemd-tmpfiles sets the owner, the permissions and the access control lists that let the relevant groups read along:

mkdir -p /var/log/journal

systemd-tmpfiles --create --prefix /var/log/journal

systemctl restart systemd-journald

Reading without root

A normal user sees only their own messages and gets the note Hint: You are currently not seeing messages from other users and the system. along with a pointer to the groups that are allowed to read everything. On Debian and Ubuntu that is usually adm:

usermod -aG adm username

Success check: group membership only takes effect in a new login. So log out, log in again, then run id -nG and journalctl -n 5. If the hint stays away and system messages appear, it worked.

Limit the size before the disk does it for you

The shipped /etc/systemd/journald.conf lists every key as a commented-out line together with its default value, which makes it the most reliable source for what currently applies on your system. After every change, systemctl restart systemd-journald is required, and journalctl --disk-usage then shows the result.

For the immediate fix on a full disk there is an order of operations that many people trip over: the cleanup options only touch closed journal files, never the one that is currently active. Without rotating first, apparently nothing happens, and the output reads roughly Vacuuming done, freed 0B of archived journals.

journalctl --rotate

journalctl --vacuum-time=7d

More on that, together with the other space hogs, is in the article Disk full on Linux.

A second reason for missing lines is the built-in rate limiting through RateLimitIntervalSec and RateLimitBurst: if a service writes a great many messages in a short time, journald discards the excess and notes that with a line containing the word Suppressed. When a log has gaps even though the service demonstrably was running, look for that first:

journalctl -b --grep "Suppressed" --no-pager

Kernel messages: journalctl -k and dmesg

-k shows kernel messages only and implies -b, so it automatically restricts the output to the current boot. If you are looking for the cause after a reboot, you have to ask for the previous one explicitly:

journalctl -k -p err --no-pager

journalctl -k -b -1 --no-pager

The difference to dmesg is where the data is stored. dmesg reads the ring buffer of the kernel: limited in size, it overflows, and it is empty after a reboot. The journal keeps the same messages, provided it is persistent. For an incident from yesterday, journalctl -k is therefore the only reliable source. Two practical notes on top: dmesg -T converts the seconds since boot into clock times and drifts slightly off after long uptimes, while the journal always carries the real time. And kernel.dmesg_restrict is set to 1 on all four distributions, so a call without root fails with Operation not permitted.

Why many servers no longer have a /var/log/syslog

The journal is not an addition to the classic text files, it is their replacement. /var/log/syslog and /var/log/auth.log are not created by systemd but by an additional syslog service, usually rsyslog. That is a package of its own, and on minimal installations of Debian 12 and Debian 13 it is no longer part of the delivery. On the server images of Ubuntu 22.04 and 24.04 it is present.

ls -l /var/log/syslog /var/log/auth.log

systemctl status rsyslog --no-pager

If no syslog service is installed, the second command answers with Unit rsyslog.service could not be found. That explains three observations in one go: guides using tail -f /var/log/syslog fail with tail: cannot open '/var/log/syslog' for reading: No such file or directory, the search for login attempts in /var/log/auth.log comes up empty, and fail2ban has to take its events from the journal. How that is configured is in the article setting up fail2ban.

Installing rsyslog afterwards just because you are used to those files is rarely worth it: you then store everything twice and additionally need a working logrotate rule. It does make sense when logs are supposed to go to a central system.

Ready-made search patterns for an incident

A service has died or keeps restarting

systemctl list-units --type=service --state=failed --no-pager

journalctl -b -p err --no-pager

journalctl -b --grep "Main process exited|Failed with result|Start request repeated" --no-pager

The third line finds the messages with which systemd reports crashes and its built-in start rate limiter. If a core dump becomes interesting, this file settles first who is responsible for it:

cat /proc/sys/kernel/core_pattern

If it holds a call to systemd-coredump, then coredumpctl list lists the dumps that exist. If it holds something else, or just core, a different handler is in charge and coredumpctl stays empty.

Memory shortage

journalctl -k -b --grep "Out of memory|oom-kill" --no-pager

journalctl --since "-7d" --grep "Out of memory|oom-kill" --no-pager

journalctl -u systemd-oomd --since "-7d" --no-pager

The second command deliberately leaves out -k and therefore searches across boots. The third one applies to Ubuntu 22.04 and 24.04, where systemd-oomd runs out of the box and terminates whole control groups before the kernel steps in; on Debian this service is not part of the standard installation. How to tell the messages apart is in the article Setting up swap and preventing out of memory.

Failed logins

journalctl -u ssh --since "-24h" --grep "Failed password|Invalid user" --no-pager

More interesting than the individual lines is the distribution. The line below counts the source addresses and sorts them in descending order:

journalctl -u ssh --since "-24h" -g "Failed password" -o cat --no-pager | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

The trick is counting from the end: the message always ends in from ADDRESS port NUMBER ssh2, so the fourth field from the back is the address, regardless of whether a valid or an invented username stood in front of it and regardless of IPv4 or IPv6. -o cat is mandatory here, because otherwise the timestamp and the sender shift the field count. The counter-check for successful logins:

journalctl -u ssh --since "-7d" -g "Accepted (publickey|password)" -o cat --no-pager

What happened at 03:14, and what came before it?

journalctl --since "2026-09-02 03:10" --until "2026-09-02 03:20" -o short-iso --no-pager

journalctl -b -1 -n 100 --no-pager

journalctl _PID=1234 --since "-1h" --no-pager

The second command shows the last hundred lines before the previous end. You recognize an orderly shutdown by systemd stopping one service after another there. If the output breaks off in the middle of normal operation, it was a crash, a hard reset or a power loss. For the third command: process numbers get reused, so without a time window you may be mixing two different programs into one output.

Common errors and how to fix them

No journal files were found.: there are no readable journal files. Either systemd-journald is not running, or you have no access as a normal user. Check systemctl is-active systemd-journald first, then repeat as root.

-- No entries --: the filter found nothing. That is not an error message, it is a correct answer to a possibly wrong question. The three most common causes: a unit name that is only an alias, a time window that is too narrow, and a -k even though the message you are after did not come from the kernel at all.

Hint: You are currently not seeing messages from other users and the system.: you are reading as a normal user. Join the adm group and log in again, or work with sudo right away.

No journal boot entry found for the specified boot (-1) or No journal boot entry found from the specified boot offset (-1): there is no earlier boot in the journal. That is the normal case as long as the journal lives in RAM only.

Failed to add match: the expression is not a valid field filter. Field names are in capital letters, and the comparison runs against the complete value. For partial matches inside the message text, --grep is the right tool.

Unknown key name 'SystemMaxUsage' in section 'Journal', ignoring: a typo in the drop-in file. systemd reads past the line and carries on with the default. You find it through journalctl -u systemd-journald -b.

Vacuuming done, freed 0B of archived journals: there was nothing archived to delete, because the active file is holding the space. Run journalctl --rotate first, then clean up again.

Operation not permitted with dmesg: kernel.dmesg_restrict is set to 1. Repeat as root, or switch to journalctl -k.

File /var/log/journal/.../system.journal corrupted or uncleanly shut down, renaming and replacing.: the server went down hard while a journal file was open. journald appends a tilde to the old name and starts a new file. journalctl --verify checks the state of all files and prints one line with PASS per file; complaints almost always concern exactly those old files with the tilde.

Differences between Debian 13, Debian 12, Ubuntu 24.04 and 22.04

• Debian 13 (trixie): systemd 257. OpenSSH 10.0, logins sit under the identifier sshd-session. rsyslog not present in minimal installations, so /var/log/syslog is missing there. systemd-oomd not part of the standard installation.
• Debian 12 (bookworm): systemd 252. OpenSSH 9.2, everything under sshd. rsyslog present depending on the installation variant, so /var/log/syslog is not guaranteed. systemd-oomd not part of the standard installation.
• Ubuntu 24.04 LTS: systemd 255. OpenSSH 9.6, everything under sshd. rsyslog present. systemd-oomd active out of the box. SSH runs through socket activation, so look up the actual unit name in the journal.
• Ubuntu 22.04 LTS: systemd 249. OpenSSH 8.9, everything under sshd. rsyslog present. systemd-oomd active out of the box. The oldest of the four systemd versions, so a few of the newer output formats are missing there.

What matters is the same on all four systems: the same filters, the same priorities, the same configuration file and the same rule that /var/log/journal decides how long anything is kept.

The order that proves itself in daily work: the time window first, then the unit, then the priority, and a search pattern last. Go about it that way and most incidents take fewer than three commands. And once you have made sure that the journal survives reboots and still has an upper limit, you can answer the question of why even when the server has long been running again.