Build your own CDN server: fetch once, deliver a thousand times

Every file a visitor loads from your site is fetched by your origin server from the file system, wrapped into a response and sent off. With ten visitors a day nobody notices. With ten thousand simultaneous visitors, that is exactly the reason the site gets slow even though the actual application has nothing to do.

A CDN server sits in front of it. It fetches every file from the origin once and then delivers it itself, as often as it is needed. The origin therefore sees only a fraction of the requests, and visitors get their files from a server that does nothing but deliver. This article shows how much that really achieves, how you set up such a server with nginx, and what matters when it comes to traffic and location.

What a CDN server actually does

The sequence is told in three sentences:

• A visitor requests a file from the CDN server.
• If it is already there, it goes out immediately. That is a hit.
• If it is not there, the CDN server fetches it from the origin, stores it and delivers it. That is a miss, and it happens only once per file per lifetime.

The principle is the same as with a streaming server: fetch once, deliver many times. The only difference is that a streaming server distributes a running data stream and a CDN server distributes finished files.

The math: how much load a cache takes off

What matters is the hit rate, meaning the share of requests the cache answers itself. With a million requests a day, the relief looks like this:

For static files, 95 percent and more is the normal case, because images, scripts and fonts rarely change. The jump from 95 to 99 percent is more valuable than the numbers suggest: it cuts the remaining load to a fifth once more.

At the same time the traffic shifts. The origin barely transfers anything any more, the CDN server all the more. At 500,000 image requests a day at 400 KB each, that is 200 GB daily, so around 6 TB a month, for images alone. That is why the important number for a CDN node is not the CPU but the uplink and the volume.

When your own CDN server pays off

Commercial CDN providers bill per transferred gigabyte. That is convenient and cheap at low volume. Above a certain amount the ratio flips:

On top of that comes a point that is hard to put a number on: with a node of your own you decide yourself what gets cached, for how long, and when it disappears. You are not bound to the rules of an outside provider, and you can also deliver things that do not fit into other people's schemes.

Setup with nginx as the cache

Install

apt update
apt install -y nginx

Nothing more is needed, the cache is built into nginx. Create the directory the files are supposed to live in:

mkdir -p /var/cache/cdn
chown www-data:www-data /var/cache/cdn

Configure

The cache is described once in the http block and then used in the individual locations:

proxy_cache_path /var/cache/cdn
                 levels=1:2
                 keys_zone=cdn:100m
                 max_size=200g
                 inactive=30d
                 use_temp_path=off;

proxy_cache_key "$scheme$request_method$host$request_uri";

What the values mean one by one: keys_zone=cdn:100m is enough for around 800,000 entries, max_size limits the space on disk, inactive=30d throws away files that nobody has requested for 30 days, and use_temp_path=off avoids needless moving between directories.

Below that, the actual server:

server {
    listen 443 ssl;
    http2 on;
    server_name cdn.example.com;

    ssl_certificate     /etc/letsencrypt/live/cdn.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/cdn.example.com/privkey.pem;

    location / {
        proxy_pass https://origin.example.com;
        proxy_set_header Host origin.example.com;
        proxy_http_version 1.1;

        proxy_cache cdn;
        proxy_cache_valid 200 301 302 24h;
        proxy_cache_valid 404 1m;

        # bundle several simultaneous misses into one request
        proxy_cache_lock on;
        proxy_cache_lock_timeout 10s;

        # keep delivering when the origin has trouble
        proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_background_update on;

        # session cookies must not blow up the cache
        proxy_ignore_headers Set-Cookie;
        proxy_hide_header Set-Cookie;

        add_header X-Cache-Status $upstream_cache_status always;
    }
}

Three lines out of that deserve particular attention. proxy_cache_lock makes sure that with a new file a hundred visitors do not query the origin all at once, but only one does and the rest wait. proxy_cache_use_stale keeps delivery going while the origin restarts or answers slowly. And hiding Set-Cookie prevents a single response with a session cookie from making the whole cache useless.

Checking that it takes effect

nginx -t && systemctl reload nginx
curl -sI https://cdn.example.com/bild.jpg | grep -i x-cache-status
curl -sI https://cdn.example.com/bild.jpg | grep -i x-cache-status

On the first call it says MISS there, on the second HIT. If it still says MISS the second time, it is almost always down to one of the three usual suspects: a Set-Cookie in the response, a Cache-Control: no-store from the origin, or a missing length header.

What gets cached and what does not

The clean solution is for the origin to say so itself. It knows its content, the cache does not. For that it sets the right headers:

If you want to decide in the cache itself, that works through separate locations per file type:

location ~* \.(jpg|jpeg|png|gif|webp|avif|svg|ico|css|js|woff2|mp4|zip)$ {
    proxy_pass https://origin.example.com;
    proxy_cache cdn;
    proxy_cache_valid 200 7d;
    add_header Cache-Control "public, max-age=604800";
    add_header X-Cache-Status $upstream_cache_status always;
}

Getting rid of content on purpose

The simplest way is not having to delete at all: anyone who gives every changed image a new file name, say logo.a1b2c3.png, does not have the problem. The old file drops out of the cache by itself eventually.

Where that is not possible, the free module ngx_cache_purge helps with nginx. Without an extra module, the path can be computed from the cache key, because nginx uses its MD5 sum as the file name:

KEY="httpsGETcdn.example.com/bild.jpg"
H=$(printf '%s' "$KEY" | md5sum | cut -d' ' -f1)
rm -f "/var/cache/cdn/${H: -1}/${H: -3:2}/$H"

For everyday use, a small script that runs these lines for a list of paths is usually enough.

Large files and videos

With files over a few hundred megabytes you need the split into blocks, otherwise the cache reloads the whole file on every seek in the video:

location /video/ {
    proxy_pass https://origin.example.com;
    proxy_cache cdn;
    proxy_cache_valid 200 206 7d;

    slice 1m;
    proxy_set_header Range $slice_range;
    proxy_cache_key "$uri$slice_range";
}

With that, nginx fetches only the blocks that are actually requested. If a viewer jumps into the middle of a film, a few megabytes travel over the line instead of several gigabytes.

TLS, HTTP/2 and HTTP/3

A CDN node delivers many small files to many connections, and that is exactly where the newer protocols pay off. HTTP/2 is done with http2 on;. HTTP/3 mainly helps on mobile connections, because a network change does not tear the connection down:

listen 443 quic reuseport;
listen 443 ssl;
http2 on;
add_header Alt-Svc 'h3=":443"; ma=86400';

While you are at it, do not forget to open UDP 443 in the firewall. That is the most common reason why HTTP/3 is set up and nobody uses it anyway.

Several locations

A single node already brings the largest part of the gain, namely taking the load off the origin. The second gain, short paths to the visitor, only comes with several locations.

The distribution runs over DNS: the same name points to a different node depending on where the visitor comes from. For that you need a DNS provider with geographic assignment. All nodes get the same configuration and the same origin, they know nothing about each other, and that is exactly what makes the setup so robust: if a node fails, you take it out of DNS and the rest take over.

KernelHost offers locations in Germany, on the US West Coast, on the US East Coast, and in Singapore and South Korea. You will find an overview with latencies under server locations. As a rule of thumb: one node per continent on which a meaningful share of your audience sits.

Measuring instead of guessing

Write the cache status into the log, then you can count the hit rate at any time:

log_format cdn '$remote_addr $status $body_bytes_sent '
               '$upstream_cache_status "$request"';
access_log /var/log/nginx/cdn.log cdn;

That can be evaluated with a one-liner:

awk '{print $4}' /var/log/nginx/cdn.log | sort | uniq -c | sort -rn

If the share of HIT for static files is below 90 percent, looking for the cause almost always pays off: usually it is cookies in the responses, lifetimes that are too short, or requests with changing parameters at the end of the address, which look like a new file to the cache every time.

Which server fits

A CDN node is a delivery machine. The order of the important properties therefore looks different than for an application server:

1. Traffic and uplink. This is where all of the load arises. In operation, a node regularly transfers several terabytes a month.
2. Disk. NVMe is clearly ahead, because the cache consists of many small random accesses. The size depends on how much of your content is hot, not on how large the content is in total.
3. RAM. The operating system uses free memory as a second cache in front of the disk. More RAM means more speed here directly.
4. CPU. Least critical, it is only needed for TLS, and modern processors handle that on the side.

Our Unlimited Traffic root servers are made for exactly this case: a real 2 × 1 Gbit/s per server without throttling, unlimited volume at a fixed monthly price, datacenter NVMe in a RAID and 3.2 Tbps of flood protection out of the box. Provisioning takes under 30 seconds, with full root access, IPv4 and a /64 IPv6 subnet. Because billing is prepaid and you can cancel monthly, you can start with one node and add a second location at any time, once your audience spreads out.