All posts

Caddy vs Traefik vs Nginx Proxy Manager on one VPS

Caddy idled at 10.9 MiB and did 4,018 req/s. Traefik did 5,324 req/s at 15.0 MiB. Nginx Proxy Manager used 83.9 MiB and returned 502s.

Written by NodeRings Inc.

NodeRings Inc. writes these notes. The company runs the marketplace and the agent that stays on the provider’s side of the hypervisor.

The same cap, 2 GiB and one CPU, sat in front of twenty containers three times. Traefik 3.7.17 was the fast one: 15.0 MiB idle, and the middle run did 5,324 requests per second with a 99th percentile of 22 ms. Caddy 2.11.7 held 10.9 MiB and did 4,018 requests per second, 99th percentile 37 ms. Nginx Proxy Manager 2.16.0 held 83.9 MiB, did 977, and 1,224 of the responses in that run were 502.

Twenty personal apps will not get near 977 requests a second. The number that changes a 2 GB VPS is the idle RAM, the file you can put back, and whether the certificate is still on disk after a restart. Put Caddy on the box unless the stack is already Traefik. Use Nginx Proxy Manager if you want the buttons and you will copy its data volume on purpose.

Caddy 2.11.7Traefik 3.7.17Nginx Proxy Manager 2.16.0
Idle RAM10.9 MiB15.0 MiB83.9 MiB
Median requests/sec4,0185,324977
99th percentile, that run37 ms22 ms176 ms
Lines for five hostnames1543, plus 12 of entrypoints68 generated per host
HTTP/3 in this buildOn for HTTPSOn, once you set itModule present, config off
After a restart3 seconds, certificate unchanged3 seconds, certificate unchanged5 seconds, database still there
Bar chart of median requests per second and idle RAM. Traefik 5,324 req/s and 15.0 MiB. Caddy 4,018 req/s and 10.9 MiB. Nginx Proxy Manager 977 req/s and 83.9 MiB.
Median of three 30 second runs at 50 connections. Idle RAM is the proxy container after ten minutes of quiet. NPM is one worker.

The other two Caddy runs were 4,147 and 2,314 requests per second, every response a 200. Traefik's other two were 5,964 and 5,296, also all 200s. NPM's other two were 1,018, with 2,470 responses of 502, and 289, with every response a 502. The host shares that one CPU, and the low Caddy run is the same command with the same status codes.

How the three were measured

The cap was 2 GiB of RAM, no swap, and one CPU. Caddy logged Updating GOMAXPROCS=1: determined from CPU quota. Inside the NPM container, nproc printed 1. worker_processes auto started eight workers anyway. That setting counted the machine's CPUs. The quota was one. The NPM column above is worker_processes 1, which is what auto does on a one-CPU VPS. With the eight workers left in place, idle RAM was 102.3 MiB. The 502s were already in the log before that change.

Twenty traefik/whoami containers, tag v1.10.3, and one proxy at a time. The proxy and the backends were removed before the next proxy started. Idle RAM is memory.current on the proxy's cgroup: one request, ten minutes quiet, then a sample every 30 seconds, twenty times. Caddy's twenty samples were identical, 11,468,800 bytes. Traefik's median was 15,761,408 bytes. NPM's median, with one worker, was 87,982,080 bytes. docker stats matched that counter. One whoami container was 1.1 MiB beside Caddy and Traefik, and 3.0 MiB beside NPM, so the row above is the proxy.

The load was the same URL on all three, plain HTTP, so the three URLs matched:

hey -z 30s -c 50 -host whoami01.bench.test http://127.0.0.1:18080/

hey was v0.1.5. Three runs, 30 seconds, 50 connections. The table is the middle run by requests per second, and the 99th percentile from that same run. HTTPS, and HTTP/3, were checked on a separate request. Mixing them into hey would have made the three URLs differ.

hey summaries for the middle run. Caddy 4018.1456 req/s, 99% in 0.0373 secs, all 200. Traefik 5323.7446 req/s, 99% in 0.0217 secs, all 200. Nginx Proxy Manager 976.7505 req/s, 99% in 0.1757 secs, 28231 of 200 and 1224 of 502.
The middle of three runs. NPM's 977 requests per second includes the 502s.

Images, pinned by the digest they were loaded as:

caddy:2.11.7@sha256:092d945299ee6bb9ba2274aa3fa3ba6cdfa6df5ea230906c398525afe0f59b99
traefik:v3.7.17@sha256:9291a9f693983f297aa8bbba80e77009cb0ab2b34e878dde82d4f900a6fb1cd7
jc21/nginx-proxy-manager:2.16.0@sha256:d836846ac9db9c71ddc9ea312189af6eb7d64d89e8f8fabf181aa50bdd449d9e
traefik/whoami:v1.10.3@sha256:a55bd4c3925a4f2ab722aba6f07f367e17ce673ba8d0df68772799f71928f276

Traefik's own version line says 3.7.17. GitHub showed 78,586 stars on Caddy, 65,420 on Traefik, and 34,592 on Nginx Proxy Manager.

Where the 502s come from

NPM's error log, during the runs that look slow, is one line repeated:

connect() to 172.20.0.2:80 failed (99: Cannot assign requested address)

Errno 99 is the container running out of local ports. The ephemeral range inside it was 32,768 to 60,999, which is 28,232 ports, and tcp_fin_timeout was 60 seconds. Two of the one-worker runs returned exactly 28,231 responses with status 200, and the rest were 502. The run in between, started while those ports were still in TIME_WAIT, returned 8,740 responses and every one of them was a 502.

The included proxy config is a proxy_pass with no upstream keepalive:

proxy_pass $forward_scheme://$server:$port$request_uri;

Each request opens a new connection to the backend. Caddy and Traefik returned only 200s on that same command.

After worker_processes 1, idle RAM dropped from 102.3 MiB to 83.9 MiB. The port error was still in the log. If you keep NPM, the fix is an upstream with keepalive, or fewer concurrent clients than the port range can hold for a minute. The 977 in the table is what you get from the config the image writes.

How much RAM each proxy keeps

10.9 MiB, 15.0 MiB, and 83.9 MiB are what the 2 GiB cap charges. smem prints a larger PSS, because it counts the mapped binary: 46.8 MiB for Caddy, 91.5 MiB for Traefik, and 150 MiB for NPM across the processes in that cgroup. Use the smaller number when you are asking whether the proxy fits. Use the larger one when you are asking what the process has mapped.

On a 2 GB VPS the gap that matters is NPM's. Caddy plus twenty backends at 1.1 MiB each is about 33 MiB before your apps do anything. NPM at 83.9 MiB, plus those backends, is already past 100 MiB while the only app is whoami. The other 1.9 GB is for the apps. 84 MiB idle is a slice of a 2 GB plan. How that plan is sold is How NodeRings works.

Five hostnames, and the file you restore

Five names, four ordinary sites and one websocket route. Caddy is 15 lines. The websocket site is the same shape as the others, because reverse_proxy passes Upgrade when the client sends it.

app1.example.com {
	reverse_proxy app1:80
}
app2.example.com {
	reverse_proxy app2:80
}
app3.example.com {
	reverse_proxy app3:80
}
app4.example.com {
	reverse_proxy app4:80
}
ws.example.com {
	reverse_proxy appws:80
}

Traefik's routes file for the same five names is 43 lines. The entrypoints are another 12, and HTTP/3 is one of them: http3: {} under the TLS entrypoint. Both files belong in git. If Traefik is the one requesting certificates, acme.json belongs on the volume with them. This run mounted a certificate file, and the hash of that file was the same after the restart.

NPM's source of truth is the UI. It writes database.sqlite, and nginx generates 68 lines per host, 29 of them with text. Five hosts is 340 lines of generated config. The lines that matter for a websocket are these, and they were in the generated file:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;

A request with those headers reached whoami. whoami printed Upgrade: websocket and answered 200. The header arrived. This backend answered as a normal GET. The backup is still the volume. Copy /data. The generated files come back from the database, and if you only keep the files, the next click in the UI overwrites them.

Three configs for five hostnames. A 15 line Caddyfile, a Traefik file with http3 enabled and five routers, and one generated Nginx Proxy Manager server block with http2 off.
Five names. Caddy is the short file. Traefik is the file plus the entrypoints. NPM is generated, 68 lines on disk per host.
Nginx Proxy Manager 2.16.0 proxy hosts list in the dark theme. Twenty whoami hostnames, HTTP only, each marked online.
The list is the source of truth. Twenty hosts, HTTP only, created during the run.

The names in the test were a Host header. On the VPS the name has to be a record that points at the guest, which is the DNS page.

Does this version speak HTTP/3

Caddy's default for a server is h1 h2 h3. A one-site file with tls internal, on 2.11.7, answered:

HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Server: Caddy

The process log said enabling HTTP/3 listener on :443, and the leaf certificate was written under the data directory. After docker restart the sha256 of that leaf was unchanged, and the next response still carried Alt-Svc.

Traefik leaves HTTP/3 off until the entrypoint sets http3: {}, and the router needs TLS. The entrypoint page says the UDP listener comes with that key, on the same port. On 3.7.17 the HTTPS response was HTTP/2 200 with alt-svc: h3=":443"; ma=2592000, before the restart and after it.

NPM 2.16.0 is OpenResty 1.31.1.1, and nginx -V includes --with-http_v3_module. The server block the UI wrote for these hosts listens on port 80, with http2 off. The module is in the binary. This version's templates leave it unused. The long-running request is issue 1550, and the pull request that would have added the switch in the UI was closed without merging.

What is still there after a restart

The volume was still mounted when the container started again. Caddy's HTTP container was answering again in 3 seconds. Traefik was answering again in 3 seconds, and the mounted certificate had the same hash. NPM was answering again in 5 seconds, database.sqlite was still under /data, and the same hostname returned 200.

docker ps after Nginx Proxy Manager was restarted. npm has been up for 2 seconds and is publishing ports 80, 81, and 443. Twenty whoami containers are still up.
Two seconds after the restart. The proxy is up. The twenty backends were not restarted with it.

If the volume is on the guest's disk, a reboot of the VPS brings it back with the guest. If the volume is anonymous, or you rebuild the guest from a template and forget the copy, NPM comes up empty and Caddy comes up without the certificates it had issued. The Caddyfile in git is the routes. The data directory is the local CA and the leaf. Keep both.

Ports 80 and 443

The proxy has to listen on 80 and 443, and on UDP 443 if you want the HTTP/3 it advertised. Caddy, in this run, had all three:

ss output showing Caddy listening on UDP 443, TCP 443, and TCP 80.
TCP 80, TCP 443, and UDP 443. The UDP line is the HTTP/3 listener.

On the guest, those are the ports. The firewall in front of the guest has to allow them, or the certificate HTTP-01 check and the browser both stop at the hypervisor. UDP 443 is easy to forget, because the site still loads over TCP and the Alt-Svc header is the only sign that HTTP/3 was offered and then dropped.

Questions people ask before they pick

Is plain nginx faster than these three?

Plain nginx is a different program from Nginx Proxy Manager. In the 2022 hot-dog benchmark, the 10-client synthetic window was 1,062,277 responses for nginx and 937,547 for Caddy. On the default reverse-proxy rows in that same test, Caddy served 506,388 and nginx served 395,210.

Does Cloudflare in front change the numbers?

A CDN or a tunnel in front of the VPS is a different path. These numbers are the proxy on the machine, with the client talking to it directly. Whatever you put in front still needs a proxy, or a single app, behind it.

Which reverse proxy uses the least RAM?

Caddy, at 10.9 MiB idle. Traefik was 15.0 MiB and did more requests per second. NPM was 83.9 MiB with one worker, and 102.3 MiB when worker_processes auto had started eight.

How do you back up Nginx Proxy Manager?

Copy the /data volume, database.sqlite included. The UI is the source of truth. The generated files under proxy_host are what the database produced last time, and a later edit in the UI writes them again.

Will a websocket work through Caddy?

The five-line shape above is the websocket route. reverse_proxy forwards Upgrade. Through NPM, the generated headers reached whoami and whoami printed them.