Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions hole-punch-interop/router/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@

This directory contains a Debian-based router implemented on top of nftables.

It expects to be run with two network interfaces:
It expects to be run with two network interfaces: one "external" interface facing the
`internet` network and one "internal" interface facing the LAN.

- `eth0`: The "external" interface.
- `eth1`: The "internal" interface.

The order of these is important.
The router cannot possibly know which one is which and thus assumes that `eth0` is the external one and `eth1` the internal one.
The firewall is set up to take incoming traffic on `eth1` and forward + masquerade it to `eth0`.
The order of the interfaces is **not** important.
Docker does not guarantee that interface index order (`eth0`, `eth1`) matches the order
the networks are listed, so the router autodetects which interface is which at startup:
the external interface is the one that routes toward the `relay` (which lives on the
`internet` network), and the other inet-bearing interface is treated as internal.
The firewall is set up to take incoming traffic on the internal interface and forward +
masquerade it to the external one.

It also expects an env variable `DELAY_MS` to be set and will apply this delay as part of the routing process[^1].

Expand Down
49 changes: 33 additions & 16 deletions hole-punch-interop/router/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,43 @@

set -ex

if [ -z "$DELAY_MS" ]; then
echo "Error: DELAY_MS is not set!"
exit 1
fi

ADDR_EXTERNAL=$(ip -json addr show eth0 | jq '.[0].addr_info[0].local' -r)
SUBNET_INTERNAL=$(ip -json addr show eth1 | jq '.[0].addr_info[0].local + "/" + (.[0].addr_info[0].prefixlen | tostring)' -r)
[ -n "$DELAY_MS" ] || { echo "Error: DELAY_MS is not set!"; exit 1; }

# Docker does not guarantee eth0==internet / eth1==lan; the interface-to-network
# mapping can be assigned in either order. Derive the external interface as the one
# that routes toward the relay (which sits on the internet network) rather than
# assuming eth0. The relay name may not resolve yet at startup, so retry briefly.
relay_ip=""
i=0
while [ "$i" -lt 50 ]; do
relay_ip=$(getent hosts relay | head -n1 | cut -d' ' -f1)
[ -n "$relay_ip" ] && break
i=$((i + 1))
sleep 0.2
done
[ -n "$relay_ip" ] || { echo "Error: could not resolve relay"; exit 1; }

IFACE_EXTERNAL=$(ip -json route get "$relay_ip" | jq -r '.[0].dev')
IFACE_INTERNAL=$(ip -json addr show | jq -r --arg ext "$IFACE_EXTERNAL" \
'.[] | select(.ifname != "lo" and .ifname != $ext and any(.addr_info[]?; .family=="inet")) | .ifname' \
| head -n1)

ADDR_EXTERNAL=$(ip -json addr show "$IFACE_EXTERNAL" \
| jq -r '.[0].addr_info[] | select(.family=="inet") | .local' | head -n1)
SUBNET_INTERNAL=$(ip -json addr show "$IFACE_INTERNAL" \
| jq -r '.[0].addr_info[] | select(.family=="inet") | .local + "/" + (.prefixlen|tostring)' | head -n1)

# Set up NAT
nft add table ip nat
nft add chain ip nat postrouting { type nat hook postrouting priority 100 \; }
nft add rule ip nat postrouting ip saddr $SUBNET_INTERNAL oifname "eth0" snat $ADDR_EXTERNAL
nft add rule ip nat postrouting ip saddr $SUBNET_INTERNAL oifname "$IFACE_EXTERNAL" snat $ADDR_EXTERNAL

# tc can only apply delays on egress traffic. By setting a delay for both eth0 and eth1, we achieve the active delay passed in as a parameter.
half_of_delay=$(expr "$DELAY_MS" / 2 )
# tc can only apply delays on egress traffic. By setting a delay on both interfaces,
# we achieve the active delay passed in as a parameter.
half_of_delay=$(expr "$DELAY_MS" / 2)
param="${half_of_delay}ms"
tc qdisc add dev "$IFACE_EXTERNAL" root netem delay $param
tc qdisc add dev "$IFACE_INTERNAL" root netem delay $param

tc qdisc add dev eth0 root netem delay $param
tc qdisc add dev eth1 root netem delay $param

echo "1" > /tmp/setup_done # This will be checked by our docker HEALTHCHECK

tail -f /dev/null # Keep it running forever.
echo "1" > /tmp/setup_done # checked by the Docker HEALTHCHECK
tail -f /dev/null # keep running
Loading