Set up simple server monitoring with built-in tools

A server does not speak up on its own when something goes wrong. It keeps running until it stops running, and the first feedback comes from a customer, or from you when you happen to look. This guide builds the smallest monitoring setup that puts an end to that: one check script, one systemd timer, one notification channel. No time series database, no dashboard, no additional open port.

The reference systems are Debian 13 (trixie), Debian 12 (bookworm), Ubuntu 24.04 LTS and Ubuntu 22.04 LTS. Wherever the four differ, it is stated explicitly. All commands are written for use as root; as a regular user, put a sudo in front of every command. This guide expands on step 8 of the checklist for a new root server.

What you should actually monitor

The most common mistake in a first monitoring setup is not measuring too little, it is measuring too much. Collect 40 metrics and you will look at none of them. The only thing worth watching is what can bring the server down and what you are able to act on. That leaves seven items.

CPU utilization in percent is not on the list: a server pulling 100 percent because a video encoder is running is doing exactly what it is meant to do. Network throughput and process count are missing for the same reason. Both help when you are looking for a cause, but neither works as an alert, because there is no value above which you would be forced to act.

The way back, before the first file exists

Monitoring is a read-only activity and normally cannot break anything. Three things can break something anyway.

A script that repairs instead of reporting. The idea is tempting: if nginx is dead, the script should simply restart it. What you get is a service that starts every ten minutes, runs for half a second and hides the cause. And a script that deletes on its own when the disk fills up will eventually delete something that was needed. The first version reads only and calls neither systemctl restart nor rm nor kill.

Open network interfaces. Metrics exporters that listen on every address are one of the most common accidental data exposures on single servers. The approach used here opens no port and needs no firewall rule.

The alert path itself. Monitoring whose notification has never been tested is not monitoring, it is a good feeling. The test for it is further down and it is not optional.

How to reach the server without SSH

KVM root servers and dedicated servers have neither IPMI nor iDRAC. When SSH stops answering, your way in is the VNC console in the customer panel. It does not hang off the network stack of the guest system, so a firewall rule or an overloaded SSH service cannot block it. Log in there once beforehand and make sure you know the root password.

The off switch

If the monitoring itself becomes the problem, for example because it fires alerts every minute, you need two commands. Memorize them before you start:

systemctl disable --now kh-monitor.timer
systemctl mask kh-monitor.service

The first one stops the timer immediately and keeps it from coming back at the next boot. The second is the emergency brake: a masked service cannot be started by hand by accident either, and you undo that with systemctl unmask kh-monitor.service. What makes the approach low risk above all is that only new files are created; taking it apart again means deleting those files. Even so, keep a second SSH session open while you work on the system.

Check the metrics by hand first

Before a script evaluates anything, you should have seen every value yourself at least once. Otherwise you will not be able to tell later whether an alert is justified or whether your threshold is nonsense.

Disk space and inodes

df -h
df --output=pcent,target -x tmpfs -x devtmpfs -x squashfs -x overlay
df -i

The exclusions are necessary. On Ubuntu 22.04 and 24.04, snap mounts its packages as read-only squashfs images, and those sit at 100 percent permanently. Without -x squashfs your monitoring reports a full disk from the very first run, every day, forever. The same applies to the overlay mount points of Docker.

There are two quirks you should know about. df does not accept -P and --output together and aborts with a message about mutually exclusive options. And on ext4, five percent are reserved for root out of the box, which is why df already reports 100 percent while root can still write. What to do after the alert is covered in Disk full on Linux.

The inode check is not a side issue. A directory holding millions of tiny session or cache files can use up every inode while df -h still shows plenty of free space. Writes then fail with No space left on device, and the obvious explanation is the wrong one.

Memory

free -m
awk '/^MemAvailable:/ { printf "%d MB\n", $2 / 1024 }' /proc/meminfo

On a healthy Linux system the free column is almost always small, because the kernel uses unused memory as a file cache. The only reliable number is available, or MemAvailable: the amount of memory a new application can get without anything being swapped out. Alert on that value, never on free.

The kernel log tells you whether things were already tight in the past:

journalctl -k -b --grep "Out of memory"

Every hit is a process the kernel killed because memory ran out. How to react without blindly adding swap is covered in Out of memory and setting up swap correctly.

System load

nproc
cat /proc/loadavg
uptime

The first three fields in /proc/loadavg are the averages over one, five and fifteen minutes. Two things about them are regularly misunderstood. First, load on Linux is not a pure CPU figure: processes waiting for disk access count towards it. A load of 20 on four cores can mean the CPU is on fire, or that a drive is stuck. Second, the one minute value is useless for alerts, because every backup run pushes it up briefly. Take the 15 minute average and set the threshold relative to the core count.

If your kernel ships the pressure statistics, they are more informative, because they report CPU, input and output, and memory separately. They are not available everywhere:

test -d /proc/pressure && cat /proc/pressure/io || echo "no pressure statistics in this kernel"

Services

systemctl is-system-running
systemctl --failed --no-pager
systemctl is-active nginx

systemctl is-system-running is the shortest useful overall check. It prints running when not a single unit is in a failed state, and degraded as soon as one is. The exit code is 0 or non-zero accordingly.

Two pitfalls. The degraded state persists until you clear it with systemctl reset-failed after the repair; otherwise a single job that failed once keeps the message alive for weeks. And when it comes to checking individual services, the reference systems differ: on Ubuntu 24.04, SSH is started through socket activation, so ssh.service is inactive while idle even though SSH is perfectly reachable. Monitor ssh.service there and you get a permanent false alarm. On Ubuntu 24.04 the unit to monitor is ssh.socket, while on Debian 12, Debian 13 and Ubuntu 22.04 it is ssh.service.

Certificate expiry

openssl x509 -enddate -noout -in /etc/letsencrypt/live/example.com/fullchain.pem
openssl x509 -checkend 1814400 -noout -in /etc/letsencrypt/live/example.com/fullchain.pem

The second command is the interesting one. -checkend expects a number of seconds, and 1814400 is 21 days. If the certificate expires within that window, the command prints Certificate will expire and returns exit code 1, otherwise Certificate will not expire and 0. /etc/letsencrypt/live and /etc/letsencrypt/archive are readable by root only.

The check has a gap that many guides leave out: it checks the file on disk, not the certificate your web server actually hands out. If the renewal goes through but the reload of the web server fails, the file is new and the key being served is old. The file check reports nothing while visitors are already seeing a certificate warning. Only a look from outside catches this case:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -enddate

-servername is not optional as soon as several certificates share one IP address. Without it you get the default certificate of the server and check the wrong domain.

The check script

All the checks together make up a script that does nothing but read, compare and report when something is wrong. It needs curl and, for the webhook path, jq. On minimal Debian installations both are missing:

apt update
apt install -y curl jq

cat > /usr/local/sbin/kh-monitor <<'EOF'
#!/bin/bash
set -u

DISK_WARN="${DISK_WARN:-85}"
INODE_WARN="${INODE_WARN:-85}"
MEM_MIN_MB="${MEM_MIN_MB:-200}"
LOAD_FACTOR="${LOAD_FACTOR:-2}"
CERT_DAYS="${CERT_DAYS:-21}"
UNITS="${UNITS:-ssh nginx}"
STATE_DIR="${STATE_DIRECTORY:-/var/lib/kh-monitor}"

problems=""
add() { problems+="- ${1}"$'\n'; }

notify() {
  printf '%s | %s\n' "$1" "$(printf '%s' "$2" | tr '\n' ' ')"
  if [ -n "${WEBHOOK_URL:-}" ]; then
    printf '%s\n%s' "$1" "$2" | jq -Rs '{text: .}' \
      | curl -fsS -m 10 -o /dev/null -H 'Content-Type: application/json' \
             --data-binary @- "$WEBHOOK_URL"
  fi
  if [ -n "${MAILTO:-}" ]; then
    printf '%s\n' "$2" | mail -s "$1" "$MAILTO"
  fi
}

while read -r pcent target; do
  pcent="${pcent%\%}"
  case "$pcent" in ''|*[!0-9]*) continue ;; esac
  [ "$pcent" -ge "$DISK_WARN" ] && add "Disk ${target} is ${pcent} percent full"
done < <(df --output=pcent,target -x tmpfs -x devtmpfs -x squashfs -x overlay | tail -n +2)

while read -r ipcent target; do
  ipcent="${ipcent%\%}"
  case "$ipcent" in ''|*[!0-9]*) continue ;; esac
  [ "$ipcent" -ge "$INODE_WARN" ] && add "Inodes on ${target} are ${ipcent} percent used"
done < <(df --output=ipcent,target -x tmpfs -x devtmpfs -x squashfs -x overlay | tail -n +2)

mem_avail=$(awk '/^MemAvailable:/ { printf "%d", $2 / 1024 }' /proc/meminfo)
[ "${mem_avail:-0}" -lt "$MEM_MIN_MB" ] && add "only ${mem_avail} MB of memory available"

cores=$(nproc)
load15=$(awk '{ print $3 }' /proc/loadavg)
awk -v l="$load15" -v c="$cores" -v f="$LOAD_FACTOR" 'BEGIN { exit !(l > c * f) }' \
  && add "15 minute load average ${load15} on ${cores} cores"

sysstate=$(systemctl is-system-running)
[ "$sysstate" = "running" ] || add "systemd reports state ${sysstate}"

for unit in $UNITS; do
  systemctl is-active --quiet "$unit" || add "Service ${unit} is $(systemctl is-active "$unit")"
done

for cert in /etc/letsencrypt/live/*/fullchain.pem; do
  [ -r "$cert" ] || continue
  openssl x509 -checkend $(( CERT_DAYS * 86400 )) -noout -in "$cert" >/dev/null 2>&1 \
    || add "Certificate ${cert} expires in less than ${CERT_DAYS} days"
done

mkdir -p "$STATE_DIR"
now=$(printf '%s' "$problems" | sha256sum | cut -d' ' -f1)
before=$(cat "${STATE_DIR}/last" 2>/dev/null || true)
printf '%s' "$now" > "${STATE_DIR}/last"

if [ -z "$problems" ]; then
  [ -n "${HEARTBEAT_URL:-}" ] && curl -fsS -m 10 -o /dev/null "$HEARTBEAT_URL"
  [ -n "$before" ] && [ "$now" != "$before" ] \
    && notify "All clear $(hostname -s)" "All checks are back to normal."
  exit 0
fi

[ "$now" = "$before" ] && exit 0
notify "Warning $(hostname -s)" "$problems"
EOF

Four spots deserve an explanation.

• The loops read from < <( ... ), not from a pipe. A pipe moves the loop into a subshell, and the messages collected in there would be gone after the done. The bug is a nasty one, because the script runs through without an error and simply never reports anything.
• The thresholds are read from the environment. You can override every limit for a single invocation. The alert test further down relies on exactly that.
• The state is stored as a checksum. A report only goes out when the list of problems has changed. Otherwise a full disk sends you the same message every ten minutes, and you switch the monitoring off after two days.
• The certificate loop comes up empty when no Let's Encrypt directory exists. The pattern stays unexpanded, [ -r "$cert" ] fails, and the iteration is skipped.

Check it now, before systemd comes into play:

chmod 700 /usr/local/sbin/kh-monitor
bash -n /usr/local/sbin/kh-monitor && echo "Syntax ok"
/usr/local/sbin/kh-monitor; echo "Exit code $?"

On a healthy server the third command prints nothing except Exit code 0. If you see a message, either something really is wrong, or a threshold does not fit, for example because UNITS names a service that does not exist here.

The systemd timer

A cron job would work as well. A timer, though, has four concrete advantages: it does not start a second instance while the first one is still running, it catches up on a run missed during a reboot, its output ends up in the journal, and it can be switched off with a single command. The service unit needs no [Install] section, because it is activated by the timer and not at system start.

cat > /etc/systemd/system/kh-monitor.service <<'EOF'
[Unit]
Description=Short health check of the server
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/kh-monitor
EnvironmentFile=-/etc/default/kh-monitor
StateDirectory=kh-monitor
SyslogIdentifier=kh-monitor
Nice=10
IOSchedulingClass=idle
EOF

cat > /etc/systemd/system/kh-monitor.timer <<'EOF'
[Unit]
Description=Runs kh-monitor regularly

[Timer]
OnCalendar=*:0/10
RandomizedDelaySec=60
Persistent=true

[Install]
WantedBy=timers.target
EOF

StateDirectory=kh-monitor creates /var/lib/kh-monitor with suitable permissions and sets the STATE_DIRECTORY variable that the script falls back on. The leading minus sign in EnvironmentFile=-/etc/default/kh-monitor means that a missing file is not an error. RandomizedDelaySec=60 spreads out the start time, and Persistent=true catches up at boot on a run that was missed while the machine was off.

The timer activates the service of the same name automatically, so a Unit= line is not needed:

systemd-analyze verify /etc/systemd/system/kh-monitor.service
systemd-analyze calendar '*:0/10'
systemctl daemon-reload
systemctl start kh-monitor.service
systemctl enable --now kh-monitor.timer

Verifying it worked:

systemctl list-timers kh-monitor.timer --no-pager
journalctl -u kh-monitor.service -n 20 --no-pager

The list has to show a line with the next execution time. If it stays empty, the timer is not active. systemd-analyze calendar finds typos in the time expression: it converts the expression into a normalized form and names the next occurrence. Details on unit files and the ways they fail are covered in Create a systemd service.

The notification channel

This is where most home-grown monitoring setups fall apart. The reporter runs on the server it watches, so it goes down together with it. That means it reliably reports everything except the one case that really counts.

The answer is a dead man's switch, better known as a heartbeat: the server checks in with a remote endpoint after every successful run, and when the check-in fails to arrive, the endpoint raises the alarm. The script above does that when HEARTBEAT_URL is set. A network outage, a hung file system and a crashed system all look the same to the remote endpoint, and you want to know about all three.

The credentials belong in a file of their own, not in the script. A webhook URL is a secret, and whoever has it can send messages in your name:

cat > /etc/default/kh-monitor <<'EOF'
WEBHOOK_URL=https://example.example/hooks/xxxxxxxx
HEARTBEAT_URL=https://example.example/heartbeat/xxxxxxxx
UNITS="ssh nginx"
EOF
chmod 600 /etc/default/kh-monitor

The quotation marks around UNITS matter: systemd would cope without them, but in the test run further down the file is read by the shell, and there nginx without quotation marks would be taken for a command. Only enter units that exist on this system, so ssh.socket instead of ssh on Ubuntu 24.04.

If you prefer email, you need a way to send it. A full mail server is overkill, a plain relay is enough:

apt install -y msmtp msmtp-mta bsd-mailx

You configure that in /etc/msmtprc with the credentials of an existing mailbox. The file contains a password and belongs at chmod 600, otherwise msmtp refuses to work and points at the file permissions. Two honest limitations: email from a freshly assigned server IP address often lands in the spam folder or is rejected, and a message you only see the next time you look at your inbox is too slow during an outage. For alerts, push delivery is the more practical choice.

Trigger the alert once

This step is not optional. Force a report by setting one threshold to a nonsensical value for a single invocation:

set -a; . /etc/default/kh-monitor; set +a
DISK_WARN=0 /usr/local/sbin/kh-monitor
rm -f /var/lib/kh-monitor/last

The message now has to actually reach you, not just sit in the journal. The third line deletes the stored state so that the test alert does not suppress the next real run. One side effect is intentional: if delivery fails, kh-monitor.service ends with an error and shows up in systemctl --failed. A silent notification channel would be the worst conceivable fault in a monitoring setup.

The view from outside

The heartbeat tells you that the server is alive. It does not tell you that your website answers. For that you need a check from another location, following the same pattern of script and timer, only on a second host:

curl -fsS -m 10 -o /dev/null -w '%{http_code} %{time_total}\n' https://example.com/

Four points decide how much this check is worth. First: test the real service, not just ICMP. A server that answers a ping while the web server hangs in an endless loop counts as healthy in a ping check. Second: -f makes curl fail on HTTP error codes, and without that switch an error page counts as a success too. Third: only alert after two failures in a row, otherwise every short network hiccup reports an outage. Fourth: a request every minute from the same address can run into rate limiting or be treated as an attack by blocking software; enter the address of the checking host as an exception.

Without a second server, a hosted monitoring service is what is left. Free offerings usually check every five minutes, so you learn about an outage with a matching delay. For a single server that is enough, and it beats the alternative of not learning about it at all.

Common errors and fixes

Failed to start kh-monitor.service: Unit kh-monitor.service not found.: systemctl daemon-reload is missing after creating or changing a unit. The second most common cause is the wrong directory; your own units belong in /etc/systemd/system/.

The unit files have no installation config: you called systemctl enable kh-monitor.service instead of kh-monitor.timer. The service deliberately has no [Install] section, the timer is what you enable.

code=exited, status=203/EXEC: systemd could not execute the file. Either the path in ExecStart is wrong, or chmod 700 is missing, or the file was edited on Windows and carries line endings with a carriage return. sed -i 's/\r$//' /usr/local/sbin/kh-monitor takes care of that.

Syntax error: redirection unexpected: the script was started with sh instead of bash. On Debian and Ubuntu, /bin/sh is the dash shell, and it knows neither < <( ... ) nor += on strings. The first line has to read #!/bin/bash.

bash: mail: command not found: no mail program is installed. Depending on the system, the mail command comes from bsd-mailx or mailutils, and the two differ in their switches. Stick to -s for the subject.

curl: (22) The requested URL returned error: 404: the webhook URL is wrong or has been deleted on the remote side. Without -f, curl would have swallowed the error silently.

curl: (60) SSL certificate problem: certificate has expired: in the check from outside this is not a tool error, it is the finding you were looking for. Against your own webhook, the same message points to a wrong clock on the checking server.

Certificate will expire: regular output of openssl x509 -checkend with exit code 1. Check whether the renewal still runs and whether the web server is reloaded afterwards.

Failed to parse calendar specification: the expression after OnCalendar= is invalid. Test it on its own with systemd-analyze calendar before it moves into the unit.

Warning: Stopping kh-monitor.service, but it can still be activated by: kh-monitor.timer: you stopped the service instead of the timer. The service only runs for seconds anyway, the timer is the thing to switch off.

An alert on every run although nothing has changed: the state is not being saved. Check whether StateDirectory=kh-monitor is in the unit and whether /var/lib/kh-monitor/last exists and is writable.

No alert although something is obviously broken: trigger the forced report from above. If that one fails to arrive as well, the problem is the delivery path, not the checks.

Differences between the four systems

What is identical on all four systems: systemd-analyze, StateDirectory= and RandomizedDelaySec= are available, and the script and the unit files run unchanged.

When the big toolbox pays off

What you have now has clear limits. It stores no history, so you cannot look up whether memory usage has been climbing for three weeks. It knows nothing about correlation across several hosts. And it has neither escalation nor on-call rules nor a way to mute a known alert for two hours.

Those very points answer the question of when to move on. A metrics and dashboard setup pays off as soon as one of them applies:

• You run more than a handful of servers and want to see them side by side.
• You need history and trends, for capacity planning for instance, or to answer a complaint about slowness with numbers.
• Several people share the on-call duty, so you need escalation levels and muting.
• You have to prove availability to a third party.

If none of that applies, the setup is usually a bad deal for a single server. Collector, database, dashboard and exporter together easily take up several hundred megabytes of RAM on exactly the machine whose free memory they are supposed to watch. On top of that come another open port and a second piece of software that wants updating. The decisive point stays the same: if the collector runs on the same server, it reports that server going down no better than a script does. When you do move on, the collector belongs on a different host, and the exporter binds to 127.0.0.1 or is limited to the address of the collector by the firewall.

The middle ground works well: the timer and the script stay because they cover the alerting case, and the metrics setup joins them once you need history. The two do not rule each other out.

Removing it again

If you want to get rid of the setup again, it takes five lines:

systemctl disable --now kh-monitor.timer
rm -f /etc/systemd/system/kh-monitor.timer /etc/systemd/system/kh-monitor.service
rm -f /usr/local/sbin/kh-monitor /etc/default/kh-monitor
rm -rf /var/lib/kh-monitor
systemctl daemon-reload

The final check

Six checks that show the actual state and not the hoped-for one:

1. systemctl list-timers kh-monitor.timer --no-pager names a next execution time.
2. journalctl -u kh-monitor.service --since "1 hour ago" --no-pager shows runs at the expected interval.
3. A forced alert through DISK_WARN=0 reaches you, not just the journal.
4. After a reboot the timer keeps running without you doing anything.
5. The heartbeat proves itself: stop the timer and wait to see whether the remote endpoint raises the alarm.
6. systemctl --failed --no-pager lists nothing, and kh-monitor.service least of all.

Point five is the most inconvenient and the most important one. Monitoring whose alert case has never occurred is a guess. Only once you have caused the outage deliberately and received the message is the chain from the server to your phone provably complete.