All posts

A provider-side firewall does what ufw cannot

ufw 0.36.2 denied tcp/8080 and counted 0 packets. nginx:alpine returned 200. A bridge drop took guest softirq from 78.5% to 0 on a SYN flood.

Written by NodeRings Inc.

NodeRings Inc. writes these notes. The company runs the marketplace and the agent that stays on the provider’s side of the hypervisor.

ufw 0.36.2 listed 8080/tcp DENY IN Anywhere. I zeroed the counters and curled http://192.0.2.20:8080/ from a second guest. Status 200, in 0.001076 seconds, and the body was the nginx welcome page. The DROP in ufw-user-input read 0 packets.

ss showed docker-proxy on 0.0.0.0:8080 and on [::]:8080. The nat rule was -A DOCKER ! -i docker0 -p tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80. After that one request the filter DOCKER ACCEPT for 172.17.0.2 tcp/80 had 1 packet, 60 bytes, and DOCKER-CT had 6 packets, 390 bytes, for the established exchange. ufw's routed policy was deny as well. DOCKER-FORWARD sits ahead of ufw-before-forward, so the routed deny was on the wrong side of the accept.

ss shows docker-proxy listening on 0.0.0.0:8080 and on [::]:8080. ufw denies 8080. curl returns 200. The ufw DROP counter is 0, while DOCKER ACCEPT and DOCKER-CT have moved.
One curl from 192.0.2.30. The 22 ACCEPT line is the management SSH on the other NIC. The 8080 DROP stayed at 0.

The guests are qemu, Ubuntu 24.04.5 LTS, kernel 6.8.0-147-generic, one vCPU each. The target was -m 2048 and free -m printed 1,967 MiB. The attacker was -m 1024. Docker 29.1.3, iptables 1.8.10 on the nf_tables backend, image nginx:alpine at sha256:785835ea65db4bba6f021acc0f3be91b0fef76a53fe41f8588c6698d2aed0606.

They share an isolated bridge, br-fwlab, with the target at 192.0.2.20 and the attacker at 192.0.2.30. The panel stayed out of the lab. The outer drop later in this post is nftables on that bridge, which is the seat a hypervisor rule occupies.

Docker Compose on a VPS: the production checklist already caught this split on Postgres 5432, with nmap printing open while ufw denied the port. The bridge drop there turned the scan to filtered. That post left DOCKER-USER unmeasured. This one is nginx on 8080, with the counters above, and the flood below.

The port Docker rewrote

Published traffic is DNATed in PREROUTING, before the routing decision:

-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80

The packet then goes to FORWARD, and the destination port is already 80. ufw's user rules hang off INPUT. Docker's packet filtering page describes that split, and issue 690 is the long report of it. The iptables page says extra filtering on the guest belongs in DOCKER-USER, which is the first jump on FORWARD:

-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-FORWARD
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward

A drop on the published port, as written, misses. I inserted it, zeroed the counters, and curled again.

iptables -I DOCKER-USER -p tcp --dport 8080 -j DROP

Curl returned 200 in 0.001020 seconds. That rule's counter stayed at 0 packets. DOCKER ACCEPT had moved to 1 packet, 60 bytes. DNAT had already rewritten the port, so a match on 8080 was looking at a packet whose destination port was 80.

The match that sees the original port is conntrack:

iptables -I DOCKER-USER -p tcp -m conntrack --ctorigdstport 8080 -j DROP

Curl exited 28 after 3.002442 seconds, with status 000. DOCKER-USER read 3 packets, 180 bytes. DOCKER ACCEPT stayed at 0. Same URL, same source. That is the chain the iptables page documents for this job.

A DOCKER-USER drop on tcp dport 8080 stays at 0 packets and curl returns 200. A drop matching ctorigdstport 8080 shows 3 packets and curl times out at 3.002442 seconds.
First, the rule on the port after DNAT. Curl returned 200 and the counter stayed at 0. Second, the rule on the original port. The counter read 3 packets and curl timed out.

DOCKER-USER lives in the guest's filter table. Binding the publish to 127.0.0.1 is the other guest-side fix, and the Compose post measured it on 5432. The bridge rule in the next section sits outside the guest.

Four seats, and the one in front of the NIC

Four rows: provider edge, hypervisor bridge, guest firewall, application. The bridge row is the one that drops the frame before the guest NIC and stays up when the guest rules are flushed.
The lab measured the middle two rows. The edge row is upstream of every packet I sent.
SeatWhere the packet isWhat this lab saw
Provider edgeThe uplink, in front of the hostOutside these packets
Hypervisor bridgeForward hook, before the guest tapGuest RX at 0 while the bridge counted the drops
Guest firewallAfter the NIC: ufw on INPUT, DOCKER-USER on FORWARDufw's 8080 DROP at 0; ctorigdstport stopped the curl
ApplicationThe socket inside the containerHTTP 200 when the packet got that far

On Proxmox the hypervisor row is pve-firewall, running on each cluster node, with the VM rules in /etc/pve/firewall/<VMID>.fw. The VM zone is traffic from and to the VM, incoming and outgoing. The admin guide says forward rules work with the nftables proxmox-firewall, and that stock pve-firewall ignores them. The nftables option defaults to 0, and the same page calls that backend a tech preview. The guide also says the firewall filters IPv4 and IPv6 with one ruleset. The curls and the floods in this lab were IPv4.

NodeRings pushes a ruleset to that Proxmox VM firewall through the operator. The firewall doc puts the editor at /organization/firewall: a Firewall tab for ordered rules, an Address group tab, and a Virtual machine tab that binds a ruleset to guests. First match wins. The doc's example group is office-egress. A ruleset in that shape allows 22 from the office group, then 80 and 443 from anywhere, then UDP 51820, then a deny for the rest. This lab's rule was the bridge drop.

The same doc warns that a tight ruleset blocks SSH on 22 and ICMP, and that the way back is to detach or widen the ruleset from NodeRings, or to open a ticket. The troubleshooting page is where that recovery is written down. VirtFusion and SolusVM 2 stay on the provider's own networking. The NodeRings ruleset in that doc is the Proxmox path.

Removing an extra IPv4, on the additional IPv4 page, updates Proxmox ipfilter and takes the address out of the firewall address groups. The limit there is 10 extra addresses, each from the same prefix as the primary. The guest OS still has to be configured by hand on Proxmox.

A volumetric flood that fills the uplink is in front of the bridge rule. I sent SYNs from a guest on the same bridge. What fail2ban and CrowdSec each blocked in 30 days is the opposite seat: both of those act after the guest has the packet. With the bridge drop in place, the guest NIC counter for that flood stayed at 0. Those tools read what the guest received.

Ten seconds of SYNs

hping3 3.0.0-alpha-2, from 192.0.2.30, raw on enp0s3:

timeout --signal=INT 12 hping3 -S -p 2222 -I enp0s3 192.0.2.20 --flood

Port 2222 was closed. On the target I read /proc/stat and /proc/net/dev once a second, eleven times, and took the ten gaps. Softirq averaged 78.5% of the one vCPU. The quietest second was 61.7%, the busiest 89.4%. enp0s3 received 1,903,209 packets in those ten seconds and transmitted 1. hping's invocation, which started a fraction earlier and ran past the sample, printed 2,464,829 transmitted.

The same flood with one rule on the bridge:

nft add table bridge hyp
nft add chain bridge hyp forward '{ type filter hook forward priority -10; policy accept; }'
nft add rule bridge hyp forward ip daddr 192.0.2.20 tcp dport 2222 counter drop

Softirq was 0.0% on all ten seconds. Idle averaged 98.5%. enp0s3 received 0 packets. The bridge counter for that flood read 2,201,207 packets and 88,048,280 bytes, 40 bytes each, the header size hping prints. hping itself printed 2,201,206 transmitted.

nftables on the bridge drops tcp/2222 to 192.0.2.20 and counts 5 packets, 200 bytes. tcpdump on the guest NIC reports 0 packets captured.
Five SYNs, a separate check from the ten-second flood. The bridge counted 5 packets, 200 bytes. tcpdump on the guest NIC captured 0.

Port 8080 is the published nginx, so the guest has a socket and a proxy in the path. I ran hping for 16 seconds and kept ten one-second samples inside it. Softirq averaged 92.8%, from 81.5% to 99.1%. The NIC received 1,358,623 packets, 73,365,618 bytes, and transmitted 489,131. hping printed 2,408,689 transmitted for the whole invocation.

The bridge drop of tcp/8080, on its own run: softirq 0.0% every second, idle 98.9%, and 1 packet on the guest NIC across the ten seconds. The bridge counter read 2,574,361 packets and 102,974,440 bytes. hping printed 2,574,285 transmitted. Those two totals disagree by 76 packets, at the edge of the window. The guest NIC added 1 packet during the ten seconds, and softirq stayed at 0.

Bar chart of guest softirq. Closed port 2222 averages 78.5 percent and the published port 8080 averages 92.8 percent. Both sit at 0 percent when the bridge drops the port.
Mean of ten one-second samples. Packet counts are what the guest NIC received during those seconds.
FloodBridge ruleSoftirq, mean of 10 secondsPackets on enp0s3Packets transmitted by the guest
TCP 2222, closedOff78.5%1,903,2091
TCP 2222, closedDrop0.0%00
TCP 8080, publishedOff92.8%1,358,623489,131
TCP 8080, publishedDrop0.0%193

The last row transmitted 93 packets, and the receive counter moved by 1. The closed port spent the vCPU and transmitted 1 packet. The published port spent more of the core and transmitted 489,131. The drop left the target quiet while hping kept sending.

SSH on the filtered NIC

I put the same kind of drop on tcp/22 toward 192.0.2.20. From the attacker, nc hit its 2 second timeout and returned an empty buffer, exit 1. SSH through the other NIC, qemu's user-net forward, printed hostfwd-ok in that window, and ss on the guest still showed sshd listening on port 22. That second NIC stays off the filtered bridge. It is the lab's stand-in for a console, or for the provider panel, which talks to the hypervisor.

I deleted the bridge table. The next nc from the attacker received:

SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.20

On a NodeRings Proxmox VM the matching edit is detaching or widening the ruleset from the firewall page, which is the recovery the docs describe. ufw disable on the guest covers rules that live in the guest. A hypervisor drop stays where the bridge put it.

Where ufw still applies

ufw is the policy for packets that have reached the guest, and for traffic between containers on the guest bridges. East-west on docker0 is that path. A publish bound to 127.0.0.1 falls through to the INPUT drop, which is what the Compose post measured. A public publish needs DOCKER-USER with --ctorigdstport, or a drop in front of the NIC.

IPv6 on this lab stopped at the listening socket. ufw showed 8080/tcp (v6) DENY IN Anywhere (v6), and I sent the curls and the floods over IPv4. Proxmox's guide says one ruleset covers both families. If you write a NodeRings rule, write the v6 sources you mean to allow. I have a v4 measurement and a v6 line in ufw status.

An extra IPv4 is a guest address and an ipfilter entry. The additional-IP doc says removal updates the Proxmox filter and the address groups. Adding the address inside the guest is still your step on Proxmox.

When the guest's own firewall is the broken piece, the edit that still works is the one on the machine that owns the tap. In this lab that was a single nftables drop on br-fwlab. On a Proxmox VM bought through NodeRings, that seat is the ruleset at /organization/firewall.