Setting up nginx as a reverse proxy

Almost every modern application listens on some high port: Node on 3000, a Docker container on 8080, Gunicorn on 8000, a Java application server on 8443. You do not put that straight on the internet. A reverse proxy belongs in front of it, and in practice that means nginx.

The basic configuration for this fits in five lines. That is exactly the problem: those five lines appear to work, and three weeks later you notice that every visitor shows up in the application log as 127.0.0.1, that the rate limit on the login locks everybody out at once, and that the password reset mail contains a link to http://127.0.0.1:3000. This article deals with precisely those places.

This assumes nginx is already installed. If it is not, see installing nginx on Debian and Ubuntu.

What a reverse proxy actually does

nginx accepts the visitor's connection and then opens its own, second connection to the application. That is the key point from which all the other problems follow.

From the application's point of view, the client is not the visitor but nginx. The source IP is 127.0.0.1. The protocol is http, even when HTTPS was in use on the outside. The Host header is 127.0.0.1:3000 by default and not app.example.com. And it is HTTP/1.0 instead of HTTP/1.1, which is why WebSockets always fail without additional configuration.

Everything the application is supposed to know about the real visitor has to be handed to it by nginx as an HTTP header. It does not happen by itself.

The basic configuration, and where it belongs

The location of the file differs between systems, and this gets mixed up regularly.

Debian and Ubuntu: the configuration goes into /etc/nginx/sites-available/app.conf and is activated with a symlink in /etc/nginx/sites-enabled/. Otherwise the default server block catches every request.

AlmaLinux, Rocky, RHEL and Oracle Linux: here sites-available does not exist at all. The file goes straight into /etc/nginx/conf.d/app.conf and is active immediately.

server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;

    location / {
        proxy_pass http://127.0.0.1:3000;
    }
}

Activating it on Debian and Ubuntu:

ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/
rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx

nginx -t before every reload is not politeness, it is mandatory. A systemctl reload with a broken configuration does leave the old process running, but a later reboot of the server will not bring nginx up at all.

All commands in this section require root privileges, otherwise prefix them with sudo. That applies to the pure check commands as well: nginx -t and nginx -T run as a normal user do not fail because of the configuration, but because of a file they are not allowed to write. So the message [emerg] open() "/run/nginx.pid" failed (13: Permission denied) followed by configuration file /etc/nginx/nginx.conf test failed does not mean your configuration is broken.

This configuration does forward traffic. It is broken anyway, in a way that only shows up later.

The four headers, without which the application is blind

These four lines belong in every location block with proxy_pass:

location / {
    proxy_pass http://127.0.0.1:3000;

    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Host

Without this line, nginx sets Host: 127.0.0.1:3000. The symptoms: Django answers with Invalid HTTP_HOST header and returns 400. Laravel and WordPress generate absolute URLs pointing at 127.0.0.1, visible in redirects and in every mail they send out. A backend that serves several domains always delivers the wrong tenant.

$host is the right choice here and not $http_host: $host contains the hostname without the port and falls back to server_name if a client sends no Host header at all. $http_host passes through whatever arrives, port included.

X-Real-IP

$remote_addr is the IP address nginx is actually talking to. Exactly one address, no comma, no parsing needed. For applications that only have a single field for the client IP, this is the simplest route.

X-Forwarded-For

$proxy_add_x_forwarded_for takes an X-Forwarded-For that may already be present and appends $remote_addr on the right. With several proxies in the path, that builds a chain.

And here is a security hole that hardly any guide mentions: a client can send an X-Forwarded-For of its own. If your nginx sits directly on the internet, $proxy_add_x_forwarded_for merges a value freely invented by the attacker and your real client IP into one shared list. If your application then reads the first entry as the client IP, it will believe any address at all. Rate limits, IP bans and geo logic can all be defeated this way.

Two clean consequences:

• The chain is always read from the right. The last entry is the only one your own proxy wrote.
• If nginx is the only instance in front of the application, overwrite the chain completely instead of extending it:

proxy_set_header X-Forwarded-For $remote_addr;

That way any forged history disappears. $proxy_add_x_forwarded_for is only correct when there is a load balancer or a CDN in front of it that you genuinely trust. In that case the realip module has to be configured as well, so that nginx itself knows the real IP and does not log the address of its predecessor:

set_real_ip_from 10.0.0.0/8;
real_ip_header   X-Forwarded-For;
real_ip_recursive on;

The set_real_ip_from entry is a whitelist. Without it the module has no effect, and with an overly broad entry such as 0.0.0.0/0 it becomes an open door.

X-Forwarded-Proto

The application sees a plain HTTP connection, no matter what happens outside. If this header is missing, the typical result is the following: the application notices "no HTTPS" and redirects to HTTPS, nginx accepts the request, decrypts it, passes it on as HTTP again, and the application redirects once more. The browser reports ERR_TOO_MANY_REDIRECTS. Just as common: cookies with the Secure flag are not set, logins fail without any error message, and pages load images and scripts over http://, which the browser blocks as mixed content.

Use $scheme and not the fixed value "https". Otherwise the port 80 block also claims that the connection was encrypted.

Proof that the headers really arrive

Instead of guessing, build yourself a mirror. This additional server block answers every request with the headers it received, in plain text:

server {
    listen 127.0.0.1:9999;
    default_type text/plain;

    location / {
        return 200 "Host: $host\nX-Real-IP: $http_x_real_ip\nX-Forwarded-For: $http_x_forwarded_for\nX-Forwarded-Proto: $http_x_forwarded_proto\nProtokoll: $server_protocol\n";
    }
}

Point proxy_pass at http://127.0.0.1:9999 for a test, reload nginx and open the page. What you see there is exactly what your application would otherwise have received. If all four lines are filled in and the IP matches your real connection, the configuration is correct. Remove the test block afterwards.

The application does have to evaluate the headers, though. Express needs app.set('trust proxy', 1), Symfony the trusted_proxies setting, Django USE_X_FORWARDED_HOST and SECURE_PROXY_SSL_HEADER. Without that switch the frameworks ignore the headers deliberately, for exactly the spoofing reason described above.

proxy_pass and the slash that changes everything

The most common silent mistake in the whole of nginx configuration. A single character decides which path arrives at the backend.

The rule: as soon as any path follows the host and port, even if it is only a slash, nginx replaces the part of the URL that matches the location prefix. Without a path, the complete URL is passed through unchanged.

The symptom is characteristic: the start page works, but everything below a prefix returns 404, and the backend log shows paths with a doubled prefix such as /api/api/users. A look at the application log clears that up in seconds, while guessing at nginx costs hours.

Two special cases that produce error messages of their own. In a location with a regular expression, a path is not allowed, and nginx aborts at startup with nginx: [emerg] "proxy_pass" cannot have URI part in location given by regular expression, inside named location, or inside "if" statement. And as soon as you use a variable in proxy_pass, for example proxy_pass http://$backend;, nginx no longer resolves the name at startup but at runtime. Without a resolver line in the server block, that ends in no resolver defined to resolve ... and a 502.

Proxying WebSockets

By default nginx speaks HTTP/1.0 to the backend. HTTP/1.0 does not know the upgrade mechanism. That is why every WebSocket fails on the basic configuration, no matter how correct everything else is.

Typical symptoms: the browser console reports WebSocket connection to 'wss://app.example.com/ws' failed, often with the addition Error during WebSocket handshake: Unexpected response code: 400. Socket.io quietly falls back to long polling, the application merely feels sluggish, and the access log fills up with endless lines containing /socket.io/?EIO=4&transport=polling.

First the map that decides whether a connection wants an upgrade at all. It belongs in the http context, so the best place for it is a file of its own, /etc/nginx/conf.d/websocket.conf:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

If this block ends up in a server or location block by accident, nginx will not start any more: nginx: [emerg] "map" directive is not allowed here.

Then in the location block:

location / {
    proxy_pass http://127.0.0.1:3000;

    proxy_http_version 1.1;
    proxy_set_header Upgrade    $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

Why the map and not simply proxy_set_header Connection "upgrade";? Because then perfectly ordinary HTTP requests ask for an upgrade too. Some backends answer that with a 400, and keepalive reuse is lost. The map only sends upgrade when the client actually requested one, and close in every other case.

The test: in the browser developer tools, the WebSocket request has to show the status 101 Switching Protocols. Anything else, in particular 200 or 400, means the upgrade did not get through. It also works on the command line without a browser:

curl -sSi -o /dev/null -w '%{http_code}\n' \
  -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  http://127.0.0.1/ws

Setting timeouts correctly

If a connection drops reproducibly after exactly 60 seconds, that is no coincidence but the default value of proxy_read_timeout. Important to understand: the value does not limit the total duration of the request, it limits the pause between two read operations. A download that takes ten minutes but keeps delivering data runs through without trouble. A WebSocket on which nothing happens for 61 seconds gets dropped.

proxy_connect_timeout 10s;
proxy_send_timeout    60s;
proxy_read_timeout    60s;

proxy_connect_timeout only applies to establishing the connection and is capped at 75 seconds. Setting it higher achieves nothing. For a local backend, 10 seconds is generous, and a low value lets you notice more quickly that the service is not running at all.

For WebSockets the better approach is not proxy_read_timeout 86400s; but a heartbeat in the application that sends a ping every 30 seconds. That keeps the timeout in place as protection against hung connections instead of effectively switching it off.

Two more defaults that bite regularly. client_max_body_size is set to 1 MB, and every larger upload ends in 413 Request Entity Too Large plus the log line client intended to send too large body. And with server-sent events or streaming responses, nothing reaches the visitor for a long time because nginx buffers. In that case proxy_buffering off; in the relevant location block helps, applied selectively and not globally.

Putting HTTPS in front

The most convenient route is certbot with the nginx plugin. It reads the existing server block, adds the TLS part and sets up the renewal:

apt-get install -y certbot python3-certbot-nginx
certbot --nginx -d app.example.com

These package names apply to Debian and Ubuntu. On AlmaLinux, Rocky Linux and Oracle Linux, certbot is not in the base repositories, and a plain dnf install -y certbot python3-certbot-nginx ends there with Error: Unable to find a match. EPEL is mandatory:

dnf install -y epel-release
dnf install -y certbot python3-certbot-nginx

On Oracle Linux 9 the EPEL package is called oracle-epel-release-el9, and the repository may first have to be enabled with dnf config-manager --enable ol9_developer_EPEL.

For several subdomains it is worth taking a look at Let's Encrypt wildcard certificates.

One version difference that produces warnings when you copy other people's configurations: up to nginx 1.24 you enable HTTP/2 in the listen line, and from nginx 1.25.1 onwards there is a separate directive for it. Debian 13 ships nginx 1.26.3 and wants the new notation, while Debian 12 (1.22.1), Ubuntu 24.04 (1.24.0) and Ubuntu 22.04 (1.18.0) want the old one. On the Red Hat side the boundary runs in exactly the same place: AlmaLinux 10 comes with 1.26.3 and therefore the new form, while AlmaLinux 9, Rocky Linux 9 and Oracle Linux 9 come with 1.20.1 and need the old one.

# nginx 1.25.1 and newer, which includes Debian 13
listen 443 ssl;
http2 on;

# nginx up to 1.24, so Debian 12, Ubuntu 24.04 and 22.04
listen 443 ssl http2;

If you use the old form on a recent nginx, nginx -t reports: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead. That is only a warning, so it keeps working. The new form on an old nginx, by contrast, is a hard startup error: unknown directive "http2".

To close, the most important point of the whole exercise: the backend must not be exposed to the internet itself. A reverse proxy is worthless if http://server-ip:3000 is still reachable directly, because then anyone can set the headers themselves however they like. Bind the service to 127.0.0.1.

With Docker this is a particularly nasty trap: -p 3000:3000 publishes the port on all addresses and adds rules that simply bypass a ufw firewall. The correct form is -p 127.0.0.1:3000:3000. Details on the setup are in installing Docker on Debian and Ubuntu.

If you do proxy to an HTTPS backend as an exception, nginx needs one additional line, otherwise it sends no SNI name and the other side delivers the wrong certificate:

proxy_pass https://backend.intern:8443;
proxy_ssl_server_name on;

When it goes wrong: the messages verbatim

The first place to look is always tail -f /var/log/nginx/error.log. The browser error page says nothing, the log says everything.

One small thing up front, so the very first line does not throw you: on AlmaLinux, Rocky Linux and Oracle Linux, /var/log/nginx/error.log does not exist at all right after installation, it is only created when nginx starts for the first time. tail then answers with cannot open ... No such file or directory. On Debian and Ubuntu the package creates access.log and error.log during installation already. Robust across both worlds:

tail -n 50 /var/log/nginx/error.log 2>/dev/null || journalctl -u nginx -n 50 --no-pager

For the special cases around 502 there is a guide of its own: fixing nginx 502 Bad Gateway. If your backend is not yet a cleanly starting service unit, it is worth creating a systemd service first.

How you can tell that it really works

Five checks that are meaningful when taken together:

1. nginx -t reports syntax is ok and test is successful.
2. nginx -T shows the complete assembled configuration. Search it for proxy_set_header and count: all four headers have to be present in every relevant location block. A single proxy_set_header in an inner block cancels all inherited headers of the outer block, and that is the most common cause of "but I did set that".
3. The application log shows the real visitor IP and not 127.0.0.1.
4. The mirror server block from above returns all four values filled in, with X-Forwarded-Proto: https when called over HTTPS.
5. For WebSockets: status 101 in the developer tools, and the connection survives more than 60 seconds of silence.

A log format that records the forwarded IP is helpful as well. That way you see immediately whether nginx and the application mean the same address:

log_format proxied '$remote_addr xff="$http_x_forwarded_for" '
                   'host=$host "$request" $status $body_bytes_sent '
                   'upstream=$upstream_addr rt=$request_time urt=$upstream_response_time';

access_log /var/log/nginx/app.access.log proxied;

The two timing values at the end are worth their weight in gold: $request_time is the total duration from the visitor's point of view, and $upstream_response_time is the duration of the backend response. If the two are close together, the backend is slow. If there is a gap between them, the cause is the line to the client or the buffering.

That gives you a reverse proxy that does not just make the application reachable, but also passes on everything it needs to know about its visitors. If you are setting this configuration up on a fresh system, the checklist for new root servers is a good starting point for everything that comes before it.