AgentGateway on a $5 VPS: switch models yourself

AgentGateway 1.7.0 listened on 127.0.0.1:4000 and answered two model names from two local mocks. model: fast came back from fast. The same URL with model: strong came back from strong. A prompt carrying 4111 1111 1111 1111 got HTTP 400 in 0.000832 seconds. Counted after that check and the checksum miss, the fast log had two lines, and the card prompt was absent. VmRSS on that process was 44,124 kB, five samples, two seconds apart, on a guest whose MemTotal was 984,552 kB. After a power cycle, systemd had the process active, /v1/models returned 200, and the new process read 38,340 kB.

The binary is the Linux amd64 build from the v1.7.0 release, 96,163,224 bytes, sha256 1b2fc01a2de29ca113bbb6258b5cae7b1eb3c724071c09d687a449132f2f42f8. agentgateway --version reported git 26f5aebe99a36a037fe54f8080a608aca5547e15, Rust 1.99.0, x86_64-unknown-linux-musl, and crypto_fips false. The guest was Ubuntu 24.04.5, kernel 6.8.0-147-generic, qemu -m 1024 -smp 1, SwapTotal 0. The upstreams were four Python listeners on the guest, and the completions came from them.

	On this guest
Listen	`127.0.0.1:4000` for `/v1/chat/completions`
Switch	`fast` to port 8101, `strong` to port 8102
Guard	builtin `creditCard`. The Luhn-valid number returned 400. The next digit returned 200.
Failover	Model `chat`. First response HTTP 500 from the primary mock. Next response `from backup`.
RAM	38,340 kB `VmRSS` after reboot. 44,124 kB after the calls.
Fee	The provider key is a file on the guest. OpenRouter bills a percentage when you buy credits.

How do you point one URL at two models?

The client keeps http://127.0.0.1:4000/v1/chat/completions. The JSON field model is the switch. fast and strong are names in the gateway file. Each one has its own baseUrl and its own key file. The mocks answered with the name of the port that received the body, so the route is visible in the completion.

Two curl calls to 127.0.0.1:4000. Model fast returns from fast in 0.006957 seconds. Model strong returns from strong in 0.002290 seconds. Both responses carry a Python BaseHTTP server header. — Same path. The model field changed the port. The Server header is the mock's, so the gateway passed that response through.

/v1/models listed three ids: strong, fast, and chat. primary and backup were missing from that list because their visibility is internal. The two public models came back owned_by openai, which is the provider field in the file, and chat came back owned_by virtual.

AgentGateway 1.7.0 models page. fast and strong show provider OpenAI and outgoing models mock-fast and mock-strong. primary and backup show one policy. chat is a virtual model with failover, two priorities and two targets. — The names the client can request, and the virtual model the client asks for as chat.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  database:
    url: sqlite:///home/ubuntu/agw/data.db
gateways:
  default:
    port: 4000
    bindAddress: 127.0.0.1
ui:
  gateways: default
llm:
  policies:
    guardrails:
      request:
      - regex:
          action: reject
          rules:
          - builtin: creditCard
        rejection:
          status: 400
          headers:
            set:
              content-type: application/json
          body: |
            {"error":{"message":"Request rejected: card number","type":"invalid_request_error","code":"credit_card"}}
  models:
  - name: fast
    provider: openAI
    params:
      model: mock-fast
      apiKey:
        file: /home/ubuntu/agw/fast.key
      baseUrl: http://127.0.0.1:8101/v1
  - name: strong
    provider: openAI
    params:
      model: mock-strong
      apiKey:
        file: /home/ubuntu/agw/strong.key
      baseUrl: http://127.0.0.1:8102/v1
  - name: primary
    visibility: internal
    provider: openAI
    params:
      model: mock-primary
      apiKey:
        file: /home/ubuntu/agw/fast.key
      baseUrl: http://127.0.0.1:8103/v1
    health:
      eviction:
        consecutiveFailures: 1
        duration: 60s
  - name: backup
    visibility: internal
    provider: openAI
    params:
      model: mock-backup
      apiKey:
        file: /home/ubuntu/agw/fast.key
      baseUrl: http://127.0.0.1:8104/v1
    health:
      eviction:
        consecutiveFailures: 1
        duration: 60s
  virtualModels:
  - name: chat
    routing:
      failover:
        targets:
        - model: primary
          priority: 0
        - model: backup
          priority: 1

The running file is longer than that copy. On first start the process wrote a config.session.key into it and logged that it would rather have SESSION_KEY set. stat showed mode 644 on config.yaml after that write. The key files were mode 600, and their contents stayed out of the process list. The schema describes baseUrl as the upstream base: https://api.openai.com/v1 sends completions to /v1/chat/completions. These four URLs were the mocks. A real account is a different base and a different file in apiKey.file.

Three later calls to fast, after the first pair, took 0.001079, 0.001062, and 0.001012 seconds. The same body sent straight to port 8101 took 0.000550, 0.000469, and 0.000440 seconds. The middle pair differs by 0.000593 seconds. That gap is the gateway on a loopback socket, with the mock in the same guest. It is the overhead of this process, and it says nothing about the time to a provider's network.

Does the card guard reject a number?

The shared request guard is the builtin creditCard rule, action reject. The regex guard docs say the built-in matches a card number that passes the Luhn checksum, with spaces or hyphens allowed, and that a number which fails the checksum is left alone.

4111 1111 1111 1111 passes. The gateway returned the rejection body from the file, HTTP 400, in 0.000832 seconds, and the status line was HTTP/1.1. 4111 1111 1111 1112 fails the checksum. That prompt came back HTTP/1.0 200 from the fast mock, from fast, in 0.001305 seconds. After those two prompts, plus the earlier say hi on fast, the fast log had two lines. The rejected prompt is the missing one.

A Luhn-valid card on model fast returns HTTP 400 and the credit card error. One digit later returns HTTP 200 from the fast mock. Model chat then returns HTTP 500 from the primary mock, and the following request returns from backup. — The 400 is the guard. The 200 under it is the checksum miss, and it reached the mock. The 500 and the 200 after that are the failover.

AgentGateway guardrails page. Request guards lists one built-in detector that rejects with status 400. Response guards says none are configured. — Request guards shows one built-in detector. Response guards is empty. The curl above is the detector against a card number.

The guard reads the body the client sent. A card number in the model's reply would need a response rule, and the response list in this file is empty. A provider moderation API is a separate guard in the same docs. This run stayed with the builtin.

What does failover do when the first model is down?

chat is a virtual model. Priority 0 is primary, which was a mock that returned HTTP 500 and the body upstream down. Priority 1 is backup. Both models set health.eviction.consecutiveFailures to 1 and duration to 60s, which is the shape in the virtual model docs.

The first request for chat returned that 500 to the client, Server: BaseHTTP/0.6 Python/3.12.3, in 0.001527 seconds. The primary log gained one line. The next request, a different body, returned from backup in 0.001557 seconds. The caller who hit the dead mock received the dead mock's error. The following caller got the backup. The docs describe the same order: the unhealthy target leaves the active set, and later requests use the next priority.

Four steps. The client posts to 127.0.0.1 port 4000. A credit card guard returns 400 for a Luhn-valid number. Model fast reaches port 8101 and model strong reaches port 8102. Model chat gets HTTP 500 once, then an answer from backup. — One URL. The guard sits in front of the mocks. Failover changed the port on the request after the 500.

What do you skip from a hosted router?

OpenRouter's pricing page says inference is billed at the provider's list price on every plan, and that OpenRouter's fee is charged when you buy credits: 5.5% on Standard and 8% on Business, on the purchase, with the per-request price left at the provider's list. The same page lists BYOK as $25,000 of list-price inference a month before a 5% fee, on Standard and on Business. 5.5% of a $100 Standard purchase is $5.50.

The license on the GitHub repo is Apache-2.0. The key the upstream sees is the file named in apiKey.file. You pay the provider for the tokens they generate. The percentage on the credit purchase stays on the hosted router's invoice, and the prompt passes through their proxy on the way. On this guest the moving parts are the process, the key files, and the guard in the file above.

RackNerd's specials page lists a 1 GB KVM VPS, 1 vCPU, 20 GB SSD, at $21.99 a year. Divided by 12, that is $1.83 a month. This guest was qemu -m 1024 -smp 1. The kernel reported MemTotal 984,552 kB, which is what a 1 GB machine has left after the kernel. The $5 in the title is a ceiling for a box that size. The special above is the price I could check, $1.83 a month.

Does the process fit, and what else is listening?

On the process that served the calls, VmRSS stayed at 44,124 kB and RssAnon at 10,956 kB across five readings. VmSize was 140,208 kB. After the reboot, the systemd process, with one /v1/models behind it, read VmRSS 38,340 kB and RssAnon 9,320 kB. With the gateway and the four mocks up, MemAvailable was 666,544 kB.

Two short bars on a track that ends at 984,552 kB. After reboot, VmRSS is 38,340 kB. After the calls, VmRSS is 44,124 kB. — The track is the guest. The teal is the gateway. The two readings are two process starts, the loaded one and the one systemd launched on the way back up.

The binary on disk is 96,163,224 bytes, larger than the resident set. SQLite, data.db, was 4,096 bytes after the calls. The ready line in the log was 127.530032 ms.

ss after the reboot:

systemctl reports agentgateway active, curl to /v1/models returns 200, and ss shows 127.0.0.1:4000, 127.0.0.1:15000, star port 15021, star port 15020, and ::1 port 15000. — The unit came up with the guest. 4000 and the admin port are on loopback. 15020 and 15021 are on every interface.

[Unit]
Description=agentgateway
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu/agw
ExecStart=/home/ubuntu/agw/agentgateway -f /home/ubuntu/agw/config.yaml
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

NRestarts was 0. I shut the guest down while the unit was active, booted the same disk with the same qemu -m 1024 -smp 1, and the first SSH session found it active.

A curl of http://127.0.0.1:15020/metrics returned Prometheus text, including agentgateway_config_synchronized 1. Port 15021 is the listener the log calls readiness. GET / on it returned 404 with an empty body. The admin UI is on 127.0.0.1:15000 and on [::1]:15000, and the same UI is mounted on the gateway, so with this file it is also at http://127.0.0.1:4000/ui.

AgentGateway overview. A banner says the UI is exposed without authentication. The LLM row shows 4 models, 1 virtual model, 0 shared providers, and port 4000. — The banner is the product's own warning. The listen address for that UI, in this file, is 127.0.0.1.

The overview opened on the banner. bindAddress is 127.0.0.1, and the only host port forwarded in the lab was SSH, so that page was reachable from the guest and from the SSH tunnel. A client off the machine wants a proxy on 443 and this port left on loopback, which is the shape in the Caddy, Traefik, and Nginx Proxy Manager comparison and the loopback publish in the Compose checklist. At the end of the run, 15020 was still on *.

Questions people ask about running this on the VPS

Do I pay OpenRouter's percentage if the key is mine?

On their pricing page, a credit purchase on Standard carries 5.5%, and Business carries 8%. BYOK on those plans includes $25,000 of list-price inference a month, then 5%. The gateway on this guest adds nothing to the price. The bill that remains is the provider's, at whatever baseUrl you put in the file.

Will every 16-digit number trip the card guard?

The Luhn-valid number returned 400. The number one digit away returned 200 and a line in the mock log. The builtin is a checksum, plus the spaces and hyphens the docs describe. The response guard list is empty, so a card number printed by the model would reach the client as the mock sent it.

Does failover hide the failure from the client that hit it?

The first chat request returned the primary mock's HTTP 500, body and Server header included. The next request returned from backup. Eviction changed the route for the following call. The call that got the 500 still got the 500.

Is the whole process bound to loopback?

4000 and 15000 were. 15020 and 15021 listened on *. A curl without an Authorization header still got /metrics. On a VPS that has a public address, that is a second port to close or to bind, separate from the bindAddress on the gateway.