Detecting a DDoS attack: how to tell for certain

The service stops responding, the load average is pinned at the top, and someone in the chat asks the obvious question: are we under attack? That question can be measured. It can even be measured quite quickly, provided you know which four numbers to look at, in which order, and which cross-check turns a suspicion into a finding.

This article covers diagnosis only. It is not about fending off an attack, but about telling one apart, beyond doubt, from a load problem, a software bug or plain success. If you want to know what a DDoS attack is on a technical level, see What is a DDoS attack. If you want to act on the finding, continue with Protecting your server against DDoS attacks.

Four suspects, one symptom

"The server is slow" is not a finding, it is a symptom with at least four plausible causes. Before you type a single command, you should know which patterns you actually want to tell apart.

The fourth suspect is missing from this table because it does not need a column of its own: scheduled work. Backups, reindexing runs, log rotations and package updates happen at predictable times. A glance at systemctl list-timers and the crontabs costs ten seconds and closes a surprising number of suspected attacks.

The first 60 seconds: four numbers

Collect four values, in this order. The combination is meaningful, no single value on its own is.

cat /proc/loadavg
ss -s
cat /proc/net/dev
curl -o /dev/null -s -w '%{time_total}\n' http://127.0.0.1/

How to read them:

• High load, high network load, many half-open connections, but 127.0.0.1 answers in milliseconds: the problem sits in front of the application, on the network layer. That is the classic picture of an attack.
• High load, normal network load, 127.0.0.1 answers slowly: application or database. Not an attack, but work for the developer.
• Low load, high network load: highly suspicious. A packet storm that never reaches the application burns little CPU and a lot of line capacity.
• Everything low, yet the service is unreachable from outside: see further below, the section on cases without a measurable trace.

One caveat about the load figure itself: /proc/loadavg also counts processes that are waiting on input and output. A value of 40 on four cores can be an attack just as easily as a saturated disk. vmstat 1 5 separates the two cleanly: column r shows runnable processes, b blocked ones and wa the share of time spent waiting.

Counting connections: ss instead of netstat

Almost every guide out there starts with netstat. On a current system that ends like this:

Command 'netstat' not found, but can be installed with:
apt install net-tools

netstat belongs to the net-tools package, which default installations of Debian 12, Debian 13, Ubuntu 22.04 and Ubuntu 24.04 no longer ship. The same goes for the Red Hat family (AlmaLinux, Rocky, RHEL, Oracle Linux). You can install it afterwards, but ss from iproute2 is the better choice: it is present on practically every server, it is much faster when there are many connections, and it gives you exactly the same information.

ss -s

To run all the commands below you need a handful of packages, and their names differ between distribution families. This is the most common point at which a copied guide gets stuck on AlmaLinux right on the first line:

So on Debian and Ubuntu that is apt-get install -y iproute2 net-tools procps sysstat dnsutils tcpdump, on the Red Hat family dnf -y install iproute net-tools procps-ng sysstat bind-utils tcpdump. Better leave curl off that second list: it is already installed as curl-minimal, and the full package conflicts with it (curl-minimal ... conflicts with curl). If you really do need it, dnf -y --allowerasing install curl gets you there.

The first line gives the total number of sockets, the TCP line breaks it down: estab, closed, orphaned, timewait. A high timewait value on its own is not evidence of an attack, it is the normal consequence of many short HTTP connections.

The distribution across the states is more interesting:

ss -Htan | awk '{print $1}' | sort | uniq -c | sort -rn

What stands out is a large block in SYN-RECV. Those connections were started but never acknowledged, which is exactly the pattern of a SYN flood. You can count them directly:

ss -Htn state syn-recv | wc -l

The column trap that breaks most one-liners

As soon as you hand ss a state filter, the state column disappears from the output. The peer address then sits in column 4 instead of column 5. That is precisely why one-liners copied from forums regularly return nonsense: they count ports instead of IP addresses, or they print empty lines. Rule of thumb: without a filter the peer is $5, with a filter it is $4. The -H additionally suppresses the header line, so wc -l is correct without any adjustment.

Connections per source IP, IPv6-safe:

ss -Htn state established | awk '{print $4}' | sed 's/:[^:]*$//' | sort | uniq -c | sort -rn | head -20

The sed only cuts off the last colon together with the port. The widespread approach with cut -d: -f1 works for IPv4, but it splits IPv6 addresses after the first block and makes the whole evaluation worthless. With IPv6 the square brackets remain in place, which does no harm when counting.

Interpretation needs a sense of proportion. A hundred connections from a single IP can be an attack, but they can just as easily be a corporate NAT, a mobile carrier NAT or a reverse proxy in front of your server. If a content delivery network or a load balancer sits in front, you only ever see its addresses and have to fall back to X-Forwarded-For in the web server log instead.

For UDP the same structure applies with -u. And to find out which service is listening on which port in the first place:

ss -tulnp

Measuring network load and packet rate properly

Bandwidth in megabits is the number everybody quotes. The more meaningful one is the packet rate. An attack with 200,000 tiny packets per second brings a server to a standstill even though the bandwidth looks harmless.

First the interface name, because eth0 is far from being the right answer everywhere. Common names are ens3, enp1s0 or eth0:

ip -br link

Then two measurements one second apart, with no extra packages at all. The interface name is taken from the default route instead of being written out by hand:

IF=$(ip -o route get 1.1.1.1 | awk '{print $5}')
A=$(cat /sys/class/net/$IF/statistics/rx_packets) || exit 1; sleep 1; B=$(cat /sys/class/net/$IF/statistics/rx_packets); echo "$((B-A)) packets/s incoming on $IF"

There is a solid reason for that awkwardness. Hard-code eth0 here while the interface is really called ens3 or enp1s0, as it is on most KVM servers, and the command does print cat: /sys/class/net/eth0/statistics/rx_packets: No such file or directory twice, but then calmly reports 0 packets/s incoming and exits with return code 0. In an article about attack detection that is the most dangerous variant of all: you read zero packets, you sound the all-clear, and the attack keeps running. The || exit 1 is there to prevent exactly that.

The same with rx_bytes gives you the bytes per second. From both values you calculate the average packet size, and that reveals the type of attack:

• under 100 bytes on average at a very high packet rate: SYN, ACK or UDP flood. The target is packet processing, not the line.
• 1,200 to 1,500 bytes at high bandwidth, source ports 53, 123, 389 or 11211: reflection and amplification attack. The source IPs belong to third-party servers, not to the attackers.
• normal size distribution, clean HTTP requests: application layer. In that case the web server log decides, not the packet counter.

It gets more comfortable with sar from the sysstat package:

sar -n DEV 1 3

One pitfall: the live measurement works immediately after installation, the historical evaluation with sar -f does not. On Debian and Ubuntu the data collection is switched off out of the box, /etc/default/sysstat contains ENABLED="false". If you only notice that once an attack is under way, you have no baseline from the day before yesterday. That is the reason to enable this package in advance rather than in an emergency.

And the single most important limitation of all: on the server you only measure what got through. If filtering sits in front of it, you see a fraction of the actual volume, or nothing at all. The reliable number is in the traffic graph in the customer panel, not in /proc/net/dev.

Reading kernel messages

The kernel logs overload situations that stay invisible at the application layer. On Ubuntu and current Debian versions the ring buffer is locked for ordinary users, and without sudo you get:

dmesg: read kernel buffer failed: Operation not permitted

That is not a malfunction, it is kernel.dmesg_restrict=1. With root privileges and timestamps:

dmesg -T | grep -Ei 'syn flood|conntrack|neighbour|drop'

The message most people are looking for reads, word for word:

TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.

There are two variants and one formatting difference worth knowing about:

• Sending cookies means that SYN cookies are active and the connections are still being served. Dropping request appears instead when net.ipv4.tcp_syncookies is set to 0. Requests are then discarded, genuine visitors included.
• Older kernels only name the port number, newer ones also print the listening address in the form 0.0.0.0:443. On Debian 12 you therefore see the short form, on Debian 13 and Ubuntu 24.04 the long one. If you grep for the exact old wording, you will find nothing on newer systems.

Important for an honest diagnosis: this message does not prove an attack. It also shows up when an application runs with a listen backlog that is too small and a legitimate burst of load overruns it. It is an indication, and it has to be backed up by other measurements.

The matching counters come from nstat:

nstat -az TcpExtSyncookiesSent
nstat -az TcpExtListenDrops
nstat -az TcpExtListenOverflows

Another pitfall here: nstat stores an intermediate state when it is called and shows only the difference the next time round. That is intentional and even convenient for measuring, but it surprises everybody who suddenly sees zeros on the second call. -a forces absolute values, -s stops the state from being written forward.

Two more messages that show up regularly during an attack, both of which cause packet loss for legitimate visitors:

nf_conntrack: table full, dropping packet
neighbour: arp_cache: neighbor table overflow!

You check the fill level of connection tracking with cat /proc/sys/net/netfilter/nf_conntrack_count compared against nf_conntrack_max. Both files only exist when the module is loaded, that is, when a firewall is active. On a system without rules they are missing, and that is normal.

Web server logs: patterns instead of gut feeling

On Debian and Ubuntu the logs live under /var/log/nginx/access.log and /var/log/apache2/access.log respectively. On the Red Hat family the Apache path is /var/log/httpd/access_log, without a dot before the suffix. Three evaluations are enough to begin with:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

Telling an attack apart from a genuine surge comes out of the relationship between those three lists:

• Ratio of requests to unique IPs. Ten thousand requests from 6,000 addresses are an audience. Ten thousand requests from 30 addresses are not.
• Follow-up requests. A real visitor loads stylesheets, scripts and images after the HTML page. If the path list contains nothing but a single URL and not one static file, no browser was involved.
• Origin. With a marketing success the referrer field names a source: a news site, a social network, a search engine. With an attack the field is typically empty or filled with an invented address.
• Program identifier. One single, exactly identical user agent across tens of thousands of requests is a tool. Identifiers of very old browser versions are suspicious too.
• Status codes. Mostly 200 speaks for an audience, a wall of 404 or 499 speaks for automated probing or for clients that abort before the response arrives.

Careful with search engines: an IP is happy to claim in its user agent that it is Googlebot. The only way to check that is a reverse lookup followed by a forward lookup. Only when the reverse name points to a domain of the provider and that name resolves back to the same IP is the claim true.

dig -x 66.249.66.1 +short

For a genuine Googlebot address you get back a name such as crawl-66-249-66-1.googlebot.com., which you then resolve back to the same IP with dig +short crawl-66-249-66-1.googlebot.com. If nothing comes back at all, that may simply be a server without outbound DNS resolution, and then it proves nothing.

And the most important sentence about logs of all: an attack on the network layer does not appear in the web server log. A SYN flood never reaches the application and leaves not a single line there. An empty log therefore disproves nothing, it only narrows down the layer.

The cross-check: turning a suspicion into a finding

Up to this point you have indications. They become solid through cross-checks, each of which can disprove exactly one hypothesis.

1. Stop the service. Stop the web server for 30 seconds. If the inbound packet rate stays just as high, the attack sits below the application. If it collapses, it was requests to your application, malicious or not.
2. Inside against outside. If curl answers via 127.0.0.1 in milliseconds while a request from outside runs into a timeout, the application is healthy and the line is the problem.
3. A second measurement. Repeat the packet measurement two minutes later. Attacks persist or come back in waves. A one-off spike was a spike.
4. A view from outside. An external reachability test from several locations separates "unreachable for everyone" from "unreachable only for you". The second case is usually a routing or provider problem on the observer's side, not an attack.
5. Check the direction. Compare rx_packets with tx_packets. If the outbound rate is the conspicuous one, you are not being attacked, your server is doing the attacking. It has then been compromised or is being abused as a reflector, and the case changes its urgency immediately.

Here is how you know the diagnosis holds: you can say in one sentence which protocol hits which port at which rate, inbound or outbound, and you have two independent measurements that show the same thing. Anything less than that is a guess.

When there is no measurable trace

Four situations that regularly cause confusion:

• You cannot get onto the server at all any more. When the line is saturated, SSH does not get through either. Access then runs through the console in the customer panel, which works independently of the system's own connectivity. That is exactly what it is there for.
• The service was down, and there is nothing on the server. With filtering in front of the system that is the rule, not the exception. If the attack is caught in the network, the system only sees a short dip. The evidence is then in the traffic graph in the customer panel.
• The log ends in the middle of the incident. Check whether a log rotation ran in the meantime, the older material sits next to it as access.log.1 or access.log.2.gz. If access_log is switched off or buffered in the configuration, the last lines are simply missing.
• The kernel stays silent. On container-based systems the ring buffer belongs to the host system, so dmesg shows nothing of your own there. On a KVM root server with its own kernel this is a non-issue.

What to document before you contact your provider

A report saying "the server was slow this afternoon" adds several rounds to the handling time. A report with measurements gets worked on straight away. Collect the data while the incident is still running, because the counters in /proc are reset on reboot.

mkdir -p /root/incident && cd /root/incident
date -u > 01-time.txt
ss -s > 02-sockets.txt
ss -Htan | awk '{print $1}' | sort | uniq -c | sort -rn > 03-states.txt
ss -Htn state established | awk '{print $4}' | sed 's/:[^:]*$//' | sort | uniq -c | sort -rn | head -50 > 04-top-ips.txt
ip -s link > 05-interfaces.txt
sar -n DEV 1 10 > 06-packet-rate.txt
dmesg -T | tail -100 > 07-kernel.txt
nstat -az > 08-counters.txt

If the line allows it, a short packet capture belongs in there as well. Limit it, because an unlimited capture on a saturated line fills the disk within minutes and makes the problem worse:

tcpdump -D
tcpdump -ni "$IF" -s 96 -c 2000 -w /root/incident/capture.pcap

tcpdump -D lists the available interfaces, in case the $IF variable from the measurement section further up is no longer set. Both calls need root privileges.

The support ticket should then contain:

1. Start and end time, with the time zone. date -u avoids any discussion about it.
2. The affected IP address and the affected port.
3. The measured packet rate and bandwidth, explicitly with the direction.
4. The protocol distribution and, where visible, the source ports of the peers.
5. A handful of sample source addresses, with a note that senders can be spoofed.
6. An extract from the web server log showing the recurring request pattern, three to five lines are enough.
7. What was changed shortly beforehand: a deployment, a DNS change, an ad campaign, a newly opened port.
8. The result of the cross-check: was the service reachable via 127.0.0.1 while it was down from outside?

Common false conclusions

• Taking many TIME-WAIT connections as proof of an attack. They are the normal consequence of short HTTP connections and disappear on their own.
• Taking the SYN flooding message as proof. A listen backlog configured too small produces it during a legitimate burst of load as well.
• Blocking an IP with many connections without checking first. Behind a corporate NAT, a mobile NAT or an upstream proxy you lock out entire groups of customers that way.
• Concluding an attack from high bandwidth. Check the direction first. Outbound peaks are usually a backup or a popular download.
• Concluding "no attack" from empty logs. Attacks on the network layer never reach the application.
• Rebooting during the incident. The reboot wipes every counter you would have needed for the report, and afterwards the attack simply carries on unchanged.

Once you have run through this sequence a couple of times, a solid finding takes less than five minutes. And if you supply the measurements with the ticket, you skip the follow-up questions and go straight to the solution. Fitting reading: the checklist for new root servers and setting up Fail2ban for the application layer follow-up.