Create your own systemd service and run it automatically at boot

A program that does not come back on its own after a reboot is not a service on a server, it is a risk. nohup, screen and tmux keep a process alive for as long as nothing happens. This article writes a unit file of its own line by line, shows the error messages verbatim and describes how to get out again when the service is stuck in a restart loop.

All commands run as root. Tested on Debian 13 (systemd 257), Debian 12 (252), Ubuntu 24.04 LTS (255) and Ubuntu 22.04 LTS (249). Wherever the four differ, it is noted.

What a systemd service does that nohup and screen do not

The difference is not convenience, it is responsibility. systemd starts the process at boot in a defined order, puts it into a cgroup of its own, collects stdout and stderr in the journal, restarts it after a crash and shuts it down cleanly with SIGTERM. The cgroup is what you miss most painfully: a script started with nohup leaves orphaned child processes behind, while a unit tears down its whole group.

Preparation: program, user and directory

Before the unit exists, whatever it is supposed to start has to run. The example is a script that writes to stdout. That is exactly the point: a service under systemd does not log to a file itself, it writes to stdout and systemd puts that into the journal.

mkdir -p /opt/kh-demo

cat > /opt/kh-demo/run.sh <<'EOF'
#!/bin/bash
set -euo pipefail
while true; do
  echo "kh-demo alive, $(date --iso-8601=seconds), PWD=$PWD, GREETING=${GREETING:-not set}"
  sleep 10
done
EOF

chmod +x /opt/kh-demo/run.sh

Then a system user of its own: --system assigns a UID below 1000, and /usr/sbin/nologin prevents interactive logins.

useradd --system --no-create-home --home-dir /opt/kh-demo --shell /usr/sbin/nologin khdemo

id khdemo

chown -R root:khdemo /opt/kh-demo

chmod 750 /opt/kh-demo

The decisive test before the unit file: does the program run as this user? Skip it and you will be debugging systemd later, even though the problem sits in the program.

timeout 3 runuser -u khdemo -- /opt/kh-demo/run.sh; echo "Exit code $?"

Exit code 124 is the desired result here: timeout aborted a program that was still running. Any other value means the script died on its own, and in that case the fault is not with systemd.

The unit file line by line

Your own units belong in /etc/systemd/system/, not in /lib/systemd/system/ or /usr/lib/systemd/system/: those two belong to the package manager and get overwritten by the next apt upgrade.

cat > /etc/systemd/system/kh-demo.service <<'EOF'
[Unit]
Description=KernelHost Demo Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=khdemo
Group=khdemo
WorkingDirectory=/opt/kh-demo
EnvironmentFile=-/etc/kh-demo.env
ExecStart=/opt/kh-demo/run.sh
Restart=on-failure
RestartSec=5s
TimeoutStopSec=20s
SyslogIdentifier=kh-demo
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full

[Install]
WantedBy=multi-user.target
EOF

Section [Unit]

Description= is the text you see in systemctl status. It has no technical effect, but it decides whether you understand at three in the morning what exactly has failed. Write a sentence, not a single word.

After= only defines the ordering, not the dependency. After=network-online.target means: if this target is started during this boot anyway, wait for it. It does not mean that it will be started.

Wants= actually pulls it in, and it does so as a weak dependency. If the target fails, the service still starts. Its counterpart Requires= would drag the service down along with it. For most applications Wants= is the right choice, because a service that refuses to start at all because of a network timeout is worse than one that briefly runs into nothing and comes back thanks to Restart=.

Section [Service]

User= and Group= set the identity of the process. Without these lines the service runs as root. On Debian and Ubuntu, useradd creates a group of the same name by default, which is why Group=khdemo fits.

WorkingDirectory= is the working directory of the process. Without this setting the service starts in /. Any program that looks for configuration files, templates or plugins relative to the current directory goes down immediately. If the directory you specified does not exist, the start ends with 200/CHDIR.

EnvironmentFile= loads variables from a file. The leading minus sign in front of the path makes the file optional. Without the minus, the start fails when the file is missing. The content consists of plain KEY=VALUE lines, not shell. An export in front of them is wrong, and PORT=$BASE_PORT is not expanded.

ExecStart= needs an absolute path to the executable. This is the rule most people trip over, so it gets a section of its own further down.

TimeoutStopSec= limits how long systemd waits after the SIGTERM before SIGKILL follows. The default is 90 seconds.

SyslogIdentifier= sets the name that the lines appear under in the journal. Without this line the sender is called run.sh, which is useless for filtering.

NoNewPrivileges=true forbids the process and all of its children to gain privileges through setuid binaries. PrivateTmp=true gives the service a /tmp of its own. ProtectSystem=full mounts /usr, /boot and /etc read-only. The strict level goes further and then requires StateDirectory= or ReadWritePaths= for everything writable.

Section [Install]

WantedBy=multi-user.target answers the question of when the service should start automatically. multi-user.target is normal multi-user operation without a graphical interface, which is exactly what a server reaches. Only systemctl enable evaluates this section and creates the symlink. If [Install] is missing, enable aborts with The unit files have no installation config (WantedBy=, RequiredBy=, Also=, Alias= settings in the [Install] section, and DefaultInstance= for template units). The service can still be started by hand, but it will never come back after a reboot.

Type=simple, Type=exec and Type=forking

The Type= answers exactly one question: how does systemd know that the service has started?

With Type=simple the service counts as started as soon as the process has been created, so right after fork() and before the program is even executed. This is the default and it is right for almost every modern program that stays in the foreground.

With Type=exec systemd additionally waits until execve() has succeeded. The practical difference is considerable: with Type=simple, systemctl start reports success even when the binary does not exist at all, and the error only turns up in the journal afterwards. With Type=exec the start fails immediately in exactly that case. Available since systemd 240 and therefore on all four distributions.

With Type=forking systemd assumes that the program moves itself into the background: the process that was started exits, a child keeps running, and only that exit counts as the start signal. This is classic Unix daemon behavior. Anyone using this type almost always needs PIDFile= with an absolute path as well, otherwise systemd has to guess which of the remaining processes is the main one.

The recommendation is clear: use Type=forking only when the program absolutely refuses to be talked out of it. Nearly every piece of software has a switch for this, often --foreground, -D FOREGROUND, --no-daemon or daemon off; in the configuration. Foreground plus Type=simple keeps the unit shorter, the logging lands in the journal and Restart= works reliably.

The typical wrong combination: a program that puts itself into the background, running under Type=simple. systemd sees the starting process exit, considers the service finished and kills the children along with it. The symptom is Active: inactive (dead) right after a systemctl start that did not report any error. The reverse case, a foreground program under Type=forking, leaves systemd waiting for an exit that never comes: Job for foo.service failed because a timeout was exceeded, after 90 seconds.

Wiring up After=network-online.target correctly

This is where almost every guide stops too early. network.target only means that network management has been started, not that an IP address is configured. If you need a reachable address, for instance because the program binds to a fixed IP, you want network-online.target.

That target, however, is not reached by itself. It requires a matching wait service to be enabled, and which one that is depends on the network management in use:

systemctl list-unit-files 'systemd-networkd-wait-online.service' 'NetworkManager-wait-online.service' 'ifupdown-wait-online.service' --no-pager

On Ubuntu Server 22.04 and 24.04 the network runs through netplan with systemd-networkd, so systemd-networkd-wait-online.service is active and network-online.target carries real meaning. On a classic Debian installation with ifupdown, ifupdown-wait-online.service does exist but is not enabled. The target then counts as reached immediately, and the waiting period you are relying on never happens.

On Debian 12 and 13 with ifupdown you enable the wait service once, if you need it:

systemctl enable ifupdown-wait-online.service

Enabling two wait services at the same time is not a good idea, because both of them then wait for their respective network management and one inevitably runs into the 90-second timeout. This is the most common reason for a server that suddenly takes a minute and a half longer to boot. More robust than any ordering is a program that copes with a missing connection at startup, combined with Restart=on-failure.

Restart, RestartSec and the start rate limit

Restart=on-failure restarts on a non-zero exit code, on a signal such as SIGSEGV and on a watchdog timeout, but not after exit 0 and not after a manual systemctl stop. Restart=always additionally restarts after a clean exit, and that is the quickest way to build yourself an endless loop.

systemd has a built-in brake against this: by default, five start attempts within ten seconds are allowed (StartLimitBurst=5, StartLimitIntervalSec=10s). After that systemd gives up and reports:

kh-demo.service: Start request repeated too quickly.
kh-demo.service: Failed with result 'exit-code'.
Failed to start kh-demo.service - KernelHost Demo Worker.

The service then stays in state failed and stops responding to systemctl start as well, until the counter is reset:

systemctl reset-failed kh-demo.service

Two details regularly cost time here. First, StartLimitBurst= and StartLimitIntervalSec= belong in the [Unit] section, not in [Service]. In the wrong section they are ignored, with a warning in the journal but without an error. Second, manual systemctl restart calls count too: restart five times while debugging and you trigger the brake yourself.

Rule of thumb: StartLimitIntervalSec larger than RestartSec times StartLimitBurst.

On Debian 13 and Ubuntu 24.04 there are additionally RestartSteps= and RestartMaxDelaySec= (from systemd 254) for exponentially growing wait times. On Debian 12 and Ubuntu 22.04 these directives do not exist and are ignored.

Enabling, starting and proving that it runs

A syntax check pays off before the first start. It finds typos in directive names, misspelled sections and missing programs without starting anything:

systemd-analyze verify /etc/systemd/system/kh-demo.service

No output means everything is in order. A typo such as WorkingDirectiry= produces either Unknown key name 'WorkingDirectiry' in section 'Service', ignoring or Unknown key 'WorkingDirectiry' in section [Service], ignoring, depending on the systemd version. This is where the silent failures come from: systemd ignores unknown keys, the service starts, but it behaves differently than expected.

systemctl daemon-reload

daemon-reload re-reads the unit files, but it does not restart anything. Forget the reload and the next systemctl status gives you: Warning: The unit file, source configuration file or drop-ins of kh-demo.service changed on disk. Run 'systemctl daemon-reload' to reload units. Remember the order: first daemon-reload, then restart.

systemctl enable kh-demo.service

systemctl start kh-demo.service

Both at once works with systemctl enable --now kh-demo.service.

Now the part that most guides leave out: the proof that it really worked, and not merely that no error appeared. Four independent pieces of evidence.

First, the symlink exists. It is the entire mechanism behind enable. If it is missing, the service will not start after a reboot, no matter what status says right now.

ls -l /etc/systemd/system/multi-user.target.wants/kh-demo.service

systemctl is-enabled kh-demo.service

Second, the state. What matters is the part in brackets on the Active: line. active (running) means a process is running. active (exited) means the program has finished and nobody is left. For a long-running service that is a failure, even if it is shown in green.

systemctl status kh-demo.service --no-pager

systemctl is-active kh-demo.service

Third, the journal. Not just whether lines arrive, but whether the right ones arrive.

journalctl -u kh-demo.service -n 20 --no-pager

Follow along in real time: journalctl -u kh-demo.service -f. Only the last boot: -b. A time range: --since "-1h". More on this in the article journalctl: analyzing logs under systemd.

Fourth, the effective configuration. Not the file, but what systemd made of it. The difference counts as soon as drop-ins are involved.

systemctl show kh-demo.service -p ExecStart -p User -p WorkingDirectory -p Restart -p MainPID

systemctl cat kh-demo.service

The final proof is still a real reboot. Afterwards this command shows whether anything fell by the wayside:

systemctl list-units --type=service --state=failed --no-pager

The most common errors, verbatim

systemd reports startup failures through exit codes of its own above 200. The number in status already tells you where to look.

203/EXEC: the wrong ExecStart path

The journal shows a line of the form kh-demo.service: Failed at step EXEC spawning /opt/kh-demo/run.sh: No such file or directory, followed by Main process exited, code=exited, status=203/EXEC.

No such file or directory is misleading, because the message has three causes. First: the file really does not exist, usually because of a typo or because the binary sits in /usr/local/bin instead of /usr/bin. Second: the executable bit is missing, in which case the journal reports Permission denied. Third, the nastiest case of all: the file is executable, but its shebang line points nowhere. A script with #!/usr/bin/python fails on Debian 12 and newer for exactly that reason, because only /usr/bin/python3 exists there. The kernel reports the missing interpreter, and systemd passes it on as a missing script.

The check takes three seconds:

ls -l /opt/kh-demo/run.sh

head -n 1 /opt/kh-demo/run.sh

Relative paths and PATH

ExecStart=node server.js does not work, not even with WorkingDirectory= set. The error while loading reads Neither a valid executable name nor an absolute path. Only the first entry has to be absolute, the arguments after it may stay relative. The correct form is ExecStart=/usr/bin/node server.js, where server.js is resolved relative to the WorkingDirectory. command -v gives you the correct path:

command -v bash

Careful with version managers: under nvm this returns a path such as /root/.nvm/versions/node/v22.14.0/bin/node, which does not exist for the service user. Runtimes for services are installed system-wide, for example through NodeSource or Adoptium Temurin.

Missing environment variables

The classic: the program runs in an interactive login, but not as a service. The reason is that systemd starts no login shell. Neither /etc/profile nor ~/.bashrc nor ~/.profile is read, so everything set there with export is missing inside the service. The PATH of a system service is a fixed minimal path, usually /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Variables therefore go into the unit or into a file of their own:

cat > /etc/kh-demo.env <<'EOF'
GREETING=servus
LANG=de_DE.UTF-8
EOF

chmod 640 /etc/kh-demo.env

chown root:khdemo /etc/kh-demo.env

systemctl restart kh-demo.service

journalctl -u kh-demo.service -n 5 --no-pager

If the output now reads GREETING=servus instead of GREETING=not set, the file has arrived. You can also check this directly:

systemctl show kh-demo.service -p Environment -p EnvironmentFiles

Three traps in files like these: command substitution with $(...) does not happen, a literal $ has to be written as $$, and quotation marks end up inside the value where they do not belong.

Unit not found

Failed to start kh-demo.service: Unit kh-demo.service not found. almost always means one of three things: the file sits in the wrong directory, it has the wrong extension (.services instead of .service), or daemon-reload is missing. ls -l /etc/systemd/system/ clears up the first two cases.

Differences between Debian 13, Debian 12, Ubuntu 24.04 and 22.04

The basics, meaning [Unit], [Service], [Install], Type=simple, Restart=, User= and WorkingDirectory=, are identical on all four systems. The differences sit in the edge cases.

Available directives. Ubuntu 22.04 ships systemd 249, Debian 12 systemd 252, Ubuntu 24.04 systemd 255 and Debian 13 systemd 257. Everything from 253 onwards is missing on the two older systems: Type=notify-reload (253) as well as RestartSteps=, RestartMaxDelaySec= and RestartMode=direct (254). They are not flagged as errors, they are simply ignored. That is how units come about that do what you wanted on one system and something else entirely, apparently for no reason, on the other.

systemctl --version

Networking. Ubuntu Server uses netplan with systemd-networkd, Debian in its default installation uses ifupdown. That makes network-online.target meaningful on Ubuntu without any extra work, and on Debian only after enabling ifupdown-wait-online.service.

Interpreter and runtime paths. This is where most 203/EXEC errors come from when a unit is moved between systems. Node.js is at 20.19 on Debian 13, 18.20 on Debian 12, 18.19 on Ubuntu 24.04 and 12.22 on Ubuntu 22.04. PHP is 8.4 on Debian 13, 8.2 on Debian 12, 8.3 on Ubuntu 24.04 and 8.1 on Ubuntu 22.04. With Java the jump is largest: Debian 13 only provides openjdk-21-jre-headless, Debian 12 only openjdk-17-jre-headless, whereas Ubuntu 24.04 and 22.04 offer 8, 11, 17 and 21. Copy ExecStart=/usr/lib/jvm/java-17-openjdk-amd64/bin/java from Debian 12 to Debian 13 and you reliably get 203/EXEC.

Databases. Debian ships no mysql-server, MariaDB always runs there instead. A unit with After=mysql.service consequently waits on Debian for a service that does not exist, and starts without any delay at all, silently and without a warning. The correct setting there is After=mariadb.service.

Changing, reverting, cleaning up

Your own units are changed directly in the file plus daemon-reload. For units that come from a package you use drop-ins instead, so that the next update does not sweep your adjustment away:

systemctl edit kh-demo.service

That creates /etc/systemd/system/kh-demo.service.d/override.conf, which contains only the directives you changed. Special case: list directives such as ExecStart= are not replaced, they are appended to. If you want to override them, you have to empty the list first:

ExecStart=
ExecStart=/opt/kh-demo/run.sh --verbose

systemctl revert kh-demo.service removes all drop-ins again. Complete teardown of the example, in exactly this order:

systemctl disable --now kh-demo.service

rm -f /etc/systemd/system/kh-demo.service

rm -rf /etc/systemd/system/kh-demo.service.d

systemctl daemon-reload

systemctl reset-failed

Careful with the last step: systemctl reset-failed without an argument resets the failed state of all units, not only the one of the example service. On a system where other services are still in state failed, their messages disappear from systemctl list-units --state=failed as well. If you only want to clean up here, use the targeted form systemctl reset-failed kh-demo.service from the section further above.

Anyone who deletes the file without calling disable first leaves a dead symlink behind in /etc/systemd/system/multi-user.target.wants/, which shows up as a warning on every daemon-reload. It gets cleaned up with find /etc/systemd/system -xtype l -delete. That call removes every dead symlink below /etc/systemd/system though, including leftovers from other services that you may still need. It is safer to look first without -delete, or to go straight for the targeted variant find /etc/systemd/system -xtype l -name 'kh-demo*' -delete. Optionally the user and the files as well:

userdel khdemo

rm -rf /opt/kh-demo /etc/kh-demo.env

An absolute path in ExecStart, the matching Type, a user of its own, a configured WorkingDirectory, variables through EnvironmentFile=: get that together and you have a service that survives a reboot and tells you what went on when something breaks.