RustDesk server on a VPS

RustDesk Server 1.1.16 ran two containers on this lab host. hbbs held 2.2 MiB idle. hbbr held 1.0 MiB. Those are cgroup memory.current medians of twenty samples taken 15 seconds apart after the key files existed. docker stats printed 2.1 MiB and 1.0 MiB. smem put the same processes at 8.3 MiB and 2.5 MiB PSS. UDP 21116 was listening. Clients that only get TCP through the firewall never finish registration.

RustDesk is an open-source remote desktop stack. Self-hosting means running hbbs (ID/rendezvous) and hbbr (relay) so clients register to your key instead of the public network. You get TeamViewer-like remote control with traffic and the identity key on hardware you control. TCP 21115–21117 and UDP 21116 are part of the design; UDP especially matters for hole punching. The Docker docs push host networking on Linux so the processes see real client addresses. The image is pinned at 1.1.16 by digest.

What do clients need after the server is up?

Install the RustDesk client on each machine, open Network settings, and set ID Server to the VPS hostname or IP (hbbs.example.com or hbbs.example.com:21116). Paste the public key from id_ed25519.pub on both sides; without it, clients refuse or fall back incorrectly. Relay can stay blank so the client derives hbbr unless you split hosts (client configuration). Allow TCP 21115, TCP+UDP 21116, and TCP 21117 on the provider firewall; that trio is the OSS minimum (RustDesk self-host ports). Leave 21118/21119 closed unless you run the web client; the Docker docs warn those WebSocket ports trust X-Forwarded-For without validation if exposed directly. Host networking on Linux matches upstream so peers see real addresses; on bridge networking map 21116/udp explicitly or registration fails (rottenwifi’s Compose notes).

Always-on unattended access needs a fixed password or official permanent-password features on the client. Keep the whole data/ directory backed up; never publish id_ed25519 (private). Losing that identity means every client must be re-keyed. Check A and AAAA for the ID hostname: a broken IPv6 path looks like intermittent reachability until you fix the record or the firewall (rottenwifi). RAM is tiny; the operational cost is firewall rules and key distribution.

How do you install hbbs and hbbr on Docker?

The image is rustdesk/rustdesk-server:1.1.16, digest sha256:8ecdab65deb7c84652a626380e31d11a8f1fbafd97916d57f95c20628f943c00 (about 5.9 MB compressed on this pull). One image supplies both hbbs and hbbr. Skip floating latest in the file you keep, even when Hub points it at the same digest today.

mkdir -p ~/rustdesk/data
chmod 700 ~/rustdesk/data
cd ~/rustdesk

services:
  hbbs:
    container_name: hbbs
    image: rustdesk/rustdesk-server:1.1.16@sha256:8ecdab65deb7c84652a626380e31d11a8f1fbafd97916d57f95c20628f943c00
    command: hbbs
    volumes:
      - ./data:/root
    network_mode: host
    depends_on:
      - hbbr
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
  hbbr:
    container_name: hbbr
    image: rustdesk/rustdesk-server:1.1.16@sha256:8ecdab65deb7c84652a626380e31d11a8f1fbafd97916d57f95c20628f943c00
    command: hbbr -k _
    volumes:
      - ./data:/root
    network_mode: host
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Host networking is a Linux path. It matches the official Compose.

docker compose up -d

On this run, hbbs wrote data/id_ed25519 and data/id_ed25519.pub on first start. Restart hbbr after those files exist so -k _ loads the same pair, because depends_on has already returned. Log rotation matches the Compose checklist.

Which ports must be public?

ss on this host showed listeners on 21115 through 21119, because the binaries bind the web ports even when you never advertise them. Leave 21118 and 21119 closed unless you are serving the web client. The self-host port list says 21115 to 21117 are the minimum for native clients.

ufw allow OpenSSH
ufw allow 21115:21117/tcp
ufw allow 21116/udp

Put the same three allows on the panel firewall in front of the guest. ss can show LISTEN while a cloud rule still filters the probe. That mismatch is covered in provider-side firewall.

Where does the RustDesk key come from?

First start of hbbs prints a line like Private/public key written to id_ed25519/id_ed25519.pub and logs the public key. The client Key field gets the single line in id_ed25519.pub. Tighten the private file:

chmod 600 data/id_ed25519

On each RustDesk client, open Network settings:

• ID Server: the VPS IP or a hostname that points at it
• Relay Server: leave empty when hbbr is on that same host at 21117
• Key: the .pub line
• API Server: leave empty on OSS

Keep hbbr -k _ on a public VPS so the relay loads the key pair from the shared volume. An empty key accepts clients that never present yours.

When does a session use the relay?

Clients register with hbbs on UDP and TCP 21116, and hole punch tries a direct path. When that fails (symmetric NAT is the usual case), the session moves to hbbr on TCP 21117. The self-host docs say most sessions never need the relay, so leave ALWAYS_USE_RELAY unset. Forcing relay makes the VPS carry every frame in both directions.

Why does the server listen and still stay unreachable?

1. UDP 21116 missing while TCP 21115 to 21117 is allowed. ss shows the UDP socket, and an outside UDP probe does not. Clients never finish ID registration.
2. Bridge publish with ports: under the default Docker network. Host networking removes that path so UFW rules match what ss shows.
3. Provider or panel firewall in front of the guest. Open the three allows on that firewall as well as inside the VM.
4. Key mismatch. Both clients need the same .pub line. A regenerated key without updating clients looks like a network failure.

ss -H -lntup | grep -E '2111[5-9]'

When do you need Caddy?

Native clients use the raw ports and the public key. A hostname in the ID Server field is an A record. For the browser client, put Caddy in front of localhost 21118 and 21119, terminate WSS there, and keep those ports off the public firewall. That TLS setup lives in the Caddy HTTPS post. RustDesk also documents a TCP 443-only mode where every session is WSS relay and hole punch is gone. The Compose file above keeps the raw ports, and hole punch stays available.

FAQ

Is 1 GB enough? Idle on this run was a few MiB for both processes. A busy screen share is limited by relay bandwidth. Start on 1 GB and watch the interface counters during a session.

OSS or Pro? This run is OSS (hbbs / hbbr). Pro adds the web console and TCP 21114, on a different image and license.

RustDesk or TeamViewer? TeamViewer keeps the connection on the vendor cloud. With this server, both clients carry the public key from id_ed25519.pub.

Can I skip UDP? No. Native registration needs UDP 21116. If that port stays blocked, fix the firewall before debugging clients.

Does host networking bypass UFW? On Linux, host-network containers use the host stack, so UFW rules apply the way they do to a binary on the host. Bridge publishes are where a UFW rule and the socket stop matching. Prefer host networking for this stack, then open only 21115 to 21117 and UDP 21116.