Skip to content

Commit dcc1a92

Browse files
huang195huang195
authored
Merge pull request #484 from huang195/feat/proxy-init-enforce-drop
feat(proxy-init): add enforce-drop mode for proxy-sidecar egress enforcement
2 parents c856452 + f3101d7 commit dcc1a92

3 files changed

Lines changed: 324 additions & 24 deletions

File tree

‎authbridge/proxy-init/README.md‎

Lines changed: 98 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
11
# proxy-init
22

3-
The `proxy-init` container sets up iptables rules so that traffic in
4-
and out of an AuthBridge-injected pod is transparently redirected to
5-
the AuthBridge sidecar's listeners. It runs once at pod startup as a
6-
Kubernetes init container, then exits.
3+
The `proxy-init` container programs iptables rules for an
4+
AuthBridge-injected pod. It runs once at pod startup as a Kubernetes
5+
init container, then exits. It has two modes, selected by the `MODE`
6+
env var:
77

8-
**`proxy-init` is only used in `envoy-sidecar` mode.** In
9-
`proxy-sidecar` mode (the AuthBridge cluster default after
10-
kagenti-operator#361) traffic interception is done via `HTTP_PROXY`
11-
env vars on the workload container — no iptables, no init container.
8+
| `MODE` | Used by | What it does |
9+
|---|---|---|
10+
| `redirect` (default) | `envoy-sidecar` | Transparently **REDIRECT**s pod traffic to the Envoy listeners. |
11+
| `enforce-drop` | `proxy-sidecar` | Fail-closed egress guard — **DROP**s any egress that bypasses the forward proxy. |
1212

13-
## What it does
13+
## `redirect` mode (envoy-sidecar)
1414

1515
`init-iptables.sh` writes iptables rules that:
1616

@@ -28,21 +28,77 @@ env vars on the workload container — no iptables, no init container.
2828
`INBOUND_PORTS_EXCLUDE` env vars (commonly used to exclude
2929
Keycloak's port 8080 to avoid token-exchange loops).
3030

31+
## `enforce-drop` mode (proxy-sidecar)
32+
33+
In `proxy-sidecar` mode the workload is configured with `HTTP_PROXY`
34+
pointing at AuthBridge's forward proxy. On its own that is purely
35+
cooperative — an app that ignores `HTTP_PROXY` (or sets `NO_PROXY`)
36+
egresses directly and bypasses AuthBridge. `enforce-drop` closes that
37+
gap **without** transparently redirecting (you cannot REDIRECT raw
38+
traffic into a CONNECT forward proxy): it installs a fail-closed guard
39+
that DROPs any direct egress, forcing all external traffic through the
40+
proxy regardless of whether the app honors `HTTP_PROXY`.
41+
42+
`init-iptables.sh` builds a dedicated `AB_EGRESS` chain hooked from
43+
**`mangle` OUTPUT at position 1**, with this order:
44+
45+
1. `RETURN` ztunnel's own sockets (fwmark `0x539`) — keeps the mesh path working; a no-op when ambient is absent.
46+
2. `RETURN` the proxy's own re-originated egress (`--uid-owner $PROXY_UID`, default 1337).
47+
3. `RETURN` loopback (the app → proxy hop) and in-cluster CIDRs (`CLUSTER_CIDRS`, mesh/DNS).
48+
4. `DROP` everything else — direct external egress, including UDP (QUIC/HTTP-3).
49+
50+
An IPv6 mirror drops external v6 egress (allowing loopback, link-local,
51+
the proxy UID, and `CLUSTER_CIDRS6`).
52+
53+
> **`CLUSTER_CIDRS` is Kind-shaped by default.** The `10.0.0.0/8` default
54+
> covers Kind (pods `10.244.0.0/16` + services `10.96.0.0/16`). Other
55+
> distros differ — **OpenShift** uses services `172.30.0.0/16` and pods
56+
> `10.128.0.0/14`, and `172.30.0.0/16` is **outside** `10/8`, so the
57+
> default would drop in-cluster service traffic. On OCP/EKS/etc. you
58+
> **must** override `CLUSTER_CIDRS` with the cluster's real pod+service
59+
> ranges. The script logs the resolved value at startup, and the
60+
> operator wiring (follow-up PR) sets it from the cluster's CIDRs.
61+
62+
> **`enforce-drop` intentionally ignores `OUTBOUND_PORTS_EXCLUDE`** (a
63+
> `redirect`-mode knob). Any destination previously bypassed that way —
64+
> e.g. a direct LLM endpoint at `host.docker.internal:11434` — is now
65+
> dropped unless it goes through the forward proxy or falls within
66+
> `CLUSTER_CIDRS`. That is the point: `enforce-drop` closes direct-egress
67+
> holes. Operators relying on a bypass must route it through the proxy
68+
> (or, for in-cluster targets, include it in `CLUSTER_CIDRS`).
69+
70+
**Why `mangle` OUTPUT, not `filter`:** when Istio ambient is active it
71+
installs an in-pod `nat OUTPUT` REDIRECT (`ISTIO_OUTPUT` → ztunnel
72+
`:15001`). The netfilter OUTPUT hook order is `raw → mangle → nat →
73+
filter`, so a DROP in `mangle` evaluates the original destination and
74+
fires **before** ambient's nat redirect can rewrite it; a DROP in
75+
`filter` would run after nat and be defeated. `-I 1` also places the
76+
chain ahead of Istio's appended (`-A`) mangle chain. This makes the
77+
guard robust with no ambient, in-pod ambient, or node-level ambient.
78+
See [`test-enforce-drop.sh`](./test-enforce-drop.sh), which proves the
79+
preemption via packet counters.
80+
81+
## iptables backend
82+
3183
The script auto-detects `iptables-legacy` vs `iptables-nft` and uses
32-
whichever the host kernel exposes. Override with `IPTABLES_CMD` if
33-
needed.
84+
whichever the host kernel exposes. Override with `IPTABLES_CMD` (and
85+
`IP6TABLES_CMD`) if needed.
3486

3587
## Environment variables
3688

37-
| Variable | Default | Purpose |
38-
|---|---|---|
39-
| `PROXY_PORT` | `15123` | AuthBridge outbound listener port |
40-
| `INBOUND_PROXY_PORT` | `15124` | AuthBridge inbound listener port |
41-
| `PROXY_UID` | `1337` | UID of the AuthBridge sidecar process; excluded from outbound redirect |
42-
| `OUTBOUND_PORTS_EXCLUDE` | (empty) | Comma-separated outbound port list to skip (e.g. `8080`) |
43-
| `INBOUND_PORTS_EXCLUDE` | (empty) | Comma-separated inbound port list to skip |
44-
| `POD_IP` | (required) | Set via Downward API; used as the DNAT target for ambient-mesh inbound |
45-
| `IPTABLES_CMD` | auto-detected | Override iptables binary (`iptables-legacy` / `iptables-nft`) |
89+
| Variable | Default | Mode | Purpose |
90+
|---|---|---|---|
91+
| `MODE` | `redirect` | both | `redirect` (envoy-sidecar) or `enforce-drop` (proxy-sidecar) |
92+
| `PROXY_UID` | `1337` | both | UID of the AuthBridge sidecar process; exempted from redirect / drop |
93+
| `PROXY_PORT` | `15123` | redirect | AuthBridge outbound listener port |
94+
| `INBOUND_PROXY_PORT` | `15124` | redirect | AuthBridge inbound listener port |
95+
| `OUTBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated outbound port list to skip (e.g. `8080`) |
96+
| `INBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated inbound port list to skip |
97+
| `POD_IP` | (required in `redirect`) | redirect | Set via Downward API; DNAT target for ambient-mesh inbound. Not used by `enforce-drop`. |
98+
| `CLUSTER_CIDRS` | `10.0.0.0/8` | enforce-drop | Comma-separated in-cluster CIDRs allowed direct (pods/services/DNS) |
99+
| `CLUSTER_CIDRS6` | (empty) | enforce-drop | IPv6 in-cluster CIDRs (dual-stack); empty drops all external v6 egress |
100+
| `IPTABLES_CMD` | auto-detected | both | Override iptables binary (`iptables-legacy` / `iptables-nft`) |
101+
| `IP6TABLES_CMD` | derived from `IPTABLES_CMD` | enforce-drop | Override ip6tables binary |
46102

47103
## Required Kubernetes capabilities
48104

@@ -62,11 +118,31 @@ The image is published from CI as
62118
`ghcr.io/kagenti/kagenti-extensions/proxy-init:<tag>` (build defined
63119
in [`.github/workflows/build.yaml`](../../.github/workflows/build.yaml)).
64120

121+
## Testing
122+
123+
[`test-enforce-drop.sh`](./test-enforce-drop.sh) validates `enforce-drop`
124+
mode in a private network namespace (`unshare --net`): it asserts the
125+
`AB_EGRESS` rule structure and proves the `mangle` DROP preempts a
126+
simulated Istio ambient `nat OUTPUT` REDIRECT via packet counters.
127+
Requires root + iptables-nft on Linux (runs on CI; not macOS):
128+
129+
```sh
130+
sudo ./test-enforce-drop.sh
131+
```
132+
65133
## Where it gets injected
66134

67135
The kagenti-operator's mutating webhook injects the proxy-init
68-
container automatically when the resolved AuthBridge mode is
69-
`envoy-sidecar`. See
136+
container automatically:
137+
138+
- `redirect` mode (`MODE` unset) when the resolved AuthBridge mode is
139+
`envoy-sidecar`.
140+
- `enforce-drop` mode (`MODE=enforce-drop`) when `proxy-sidecar`
141+
egress enforcement is enabled (opt-in). _The operator wiring that
142+
sets this lands in the follow-up kagenti-operator PR; this PR only
143+
adds the mode to the image._
144+
145+
See
70146
[`authbridge/demos/weather-agent/demo-ui-advanced.md`](../demos/weather-agent/demo-ui-advanced.md)
71147
for an end-to-end demo and
72148
[`authbridge/demos/token-exchange-routes/README.md`](../demos/token-exchange-routes/README.md)

‎authbridge/proxy-init/init-iptables.sh‎

Lines changed: 132 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,22 @@
126126

127127
set -e
128128

129+
# --- Mode selection ---
130+
# MODE selects the interception strategy:
131+
# redirect (default) — envoy-sidecar: transparently REDIRECT pod traffic
132+
# to the Envoy listeners (the behavior documented above).
133+
# enforce-drop — proxy-sidecar: a fail-closed egress guard. The app is
134+
# configured with HTTP_PROXY pointing at AuthBridge's forward
135+
# proxy; this mode DROPs any egress that bypasses the proxy,
136+
# forcing all external traffic through AuthBridge regardless of
137+
# whether the app honors HTTP_PROXY. It installs no REDIRECT and
138+
# no PREROUTING/inbound rules. See setup_enforce_drop() below.
139+
MODE="${MODE:-redirect}"
140+
case "${MODE}" in
141+
redirect|enforce-drop) ;;
142+
*) echo "ERROR: unknown MODE='${MODE}' (expected: redirect | enforce-drop)" >&2; exit 1 ;;
143+
esac
144+
129145
# --- Auto-detect iptables backend ---
130146
# Prefer iptables-legacy for maximum compatibility with Kubernetes networking.
131147
# The nft backend sets rules in a different netfilter table that may not be
@@ -153,6 +169,17 @@ SSH_PORT="${SSH_PORT:-22}"
153169
OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}"
154170
INBOUND_PORTS_EXCLUDE="${INBOUND_PORTS_EXCLUDE:-}"
155171

172+
# enforce-drop mode: in-cluster destinations the agent may reach directly
173+
# (pods / services / DNS) — everything else egressing the pod is dropped.
174+
# Defaults to the RFC1918 10/8 block which covers typical Kind pod (10.244/16)
175+
# and service (10.96/16) CIDRs; override with the cluster's actual ranges.
176+
CLUSTER_CIDRS="${CLUSTER_CIDRS:-10.0.0.0/8}"
177+
CLUSTER_CIDRS6="${CLUSTER_CIDRS6:-}" # IPv6 in-cluster CIDRs (dual-stack); empty = none
178+
179+
# IPv6 counterpart of the detected iptables backend (iptables-legacy ->
180+
# ip6tables-legacy, iptables -> ip6tables). Override with IP6TABLES_CMD.
181+
IP6T="${IP6TABLES_CMD:-$(echo "${IPT}" | sed 's/iptables/ip6tables/')}"
182+
156183
# Istio ztunnel defaults
157184
ZTUNNEL_HBONE_PORT="${ZTUNNEL_HBONE_PORT:-15008}"
158185
ZTUNNEL_MARK="${ZTUNNEL_MARK:-0x539/0xfff}" # 0x539 = 1337 decimal, ztunnel's socket fwmark
@@ -162,12 +189,115 @@ ISTIO_HEALTH_PROBE_SRC="${ISTIO_HEALTH_PROBE_SRC:-169.254.7.127}"
162189
# It must be passed via the Kubernetes Downward API (status.podIP) or set manually.
163190
# We use DNAT to the pod IP instead of REDIRECT to avoid needing route_localnet=1,
164191
# which would require a privileged init container (to write to read-only /proc/sys).
165-
if [ -z "${POD_IP}" ]; then
166-
echo "ERROR: POD_IP environment variable is not set." >&2
192+
# POD_IP is only needed by redirect mode (DNAT target for the ambient inbound
193+
# rule). enforce-drop does no DNAT, so it does not require it.
194+
if [ "${MODE}" = "redirect" ] && [ -z "${POD_IP}" ]; then
195+
echo "ERROR: POD_IP environment variable is not set (required for redirect mode)." >&2
167196
echo "Set it via the Kubernetes Downward API (status.podIP) or manually." >&2
168197
exit 1
169198
fi
170199

200+
# =============================================================================
201+
# enforce-drop mode (proxy-sidecar fail-closed egress guard)
202+
# =============================================================================
203+
#
204+
# proxy-sidecar configures the app with HTTP_PROXY=127.0.0.1:<forward-proxy>.
205+
# Unlike redirect mode we do NOT transparently REDIRECT — you cannot redirect
206+
# raw traffic into a CONNECT forward proxy. Instead we DROP any egress that
207+
# leaves the pod without going through the proxy, forcing all external traffic
208+
# through AuthBridge regardless of whether the app honors HTTP_PROXY.
209+
#
210+
# Placement — a dedicated chain hooked from *mangle* OUTPUT at position 1:
211+
# * Istio ambient, when active, installs an in-pod `nat OUTPUT` REDIRECT
212+
# (ISTIO_OUTPUT -> ztunnel :15001). The netfilter OUTPUT hook order is
213+
# raw -> mangle -> nat -> filter, so a DROP in mangle evaluates the
214+
# ORIGINAL destination and fires BEFORE ambient's nat redirect can rewrite
215+
# it. A DROP in `filter` would run after nat and be defeated (dst already
216+
# rewritten to 127.0.0.1). -I 1 also places us ahead of Istio's appended
217+
# (-A) mangle ISTIO_OUTPUT chain.
218+
# * Works identically with no ambient, in-pod ambient, or node-level ambient
219+
# (in the node-level case our pod-netns rule runs before the packet ever
220+
# reaches the host netns).
221+
#
222+
# Rule order in the chain: RETURN ztunnel's own sockets (fwmark 0x539, a no-op
223+
# when ambient is absent) -> RETURN the proxy's own egress (PROXY_UID) ->
224+
# RETURN loopback (app -> proxy) -> RETURN in-cluster CIDRs (mesh/DNS) ->
225+
# DROP everything else (direct external egress, incl. UDP/QUIC).
226+
setup_enforce_drop() {
227+
CHAIN="AB_EGRESS"
228+
229+
echo "enforce-drop: installing fail-closed egress guard (mangle OUTPUT, chain ${CHAIN})"
230+
echo "enforce-drop: exempt proxy UID=${PROXY_UID}; allowed in-cluster CIDRs=${CLUSTER_CIDRS}"
231+
232+
# --- IPv4 ---
233+
${IPT} -t mangle -N "${CHAIN}" 2>/dev/null || true
234+
${IPT} -t mangle -F "${CHAIN}"
235+
# Replies to inbound connections (and related flows) are locally generated and
236+
# also traverse OUTPUT — a reply is never a "bypass". Let established/related
237+
# traffic through FIRST, so e.g. kubelet health-probe responses to an
238+
# off-cluster node IP (in Kind the node is 172.18.0.0/16, outside CLUSTER_CIDRS)
239+
# are not caught by the terminal DROP. Only NEW app-initiated flows are gated.
240+
${IPT} -t mangle -A "${CHAIN}" -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
241+
# ztunnel's own sockets (ambient) carry fwmark 0x539 — let them through so the
242+
# mesh/HBONE path keeps working. No-op when ambient is not installed.
243+
${IPT} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN
244+
# the AuthBridge proxy's own re-originated egress (must run as PROXY_UID).
245+
${IPT} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN
246+
# app -> proxy over loopback (HTTP_PROXY target), and any loopback traffic.
247+
${IPT} -t mangle -A "${CHAIN}" -o lo -j RETURN
248+
${IPT} -t mangle -A "${CHAIN}" -d 127.0.0.0/8 -j RETURN
249+
# in-cluster traffic (pods / services / DNS) — carried by the mesh, not the proxy.
250+
for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do
251+
[ -n "${cidr}" ] && ${IPT} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN
252+
done
253+
# everything else == direct external egress that bypassed the proxy. Drop it.
254+
# No -p filter, so UDP (QUIC/HTTP-3) is dropped as well as TCP.
255+
${IPT} -t mangle -A "${CHAIN}" -j DROP
256+
# Hook at position 1 so we run before any appended Istio mangle chain and
257+
# before nat OUTPUT.
258+
if ! ${IPT} -t mangle -C OUTPUT -j "${CHAIN}" 2>/dev/null; then
259+
${IPT} -t mangle -I OUTPUT 1 -j "${CHAIN}"
260+
fi
261+
echo "enforce-drop: IPv4 egress guard configured"
262+
263+
# --- IPv6 ---
264+
# Cluster is IPv4-only by default; until v6 cluster CIDRs are wired
265+
# (CLUSTER_CIDRS6), drop external v6 egress while allowing: established/related
266+
# replies, loopback, link-local unicast (fe80::/10) and link-local multicast
267+
# (ff02::/16, which carries NDP neighbor/router solicitations and MLD), the
268+
# proxy UID, and ztunnel's mark.
269+
if command -v "${IP6T%% *}" >/dev/null 2>&1 && ${IP6T} -t mangle -L >/dev/null 2>&1; then
270+
${IP6T} -t mangle -N "${CHAIN}" 2>/dev/null || true
271+
${IP6T} -t mangle -F "${CHAIN}"
272+
${IP6T} -t mangle -A "${CHAIN}" -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
273+
${IP6T} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN
274+
${IP6T} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN
275+
${IP6T} -t mangle -A "${CHAIN}" -o lo -j RETURN
276+
${IP6T} -t mangle -A "${CHAIN}" -d ::1/128 -j RETURN
277+
${IP6T} -t mangle -A "${CHAIN}" -d fe80::/10 -j RETURN
278+
${IP6T} -t mangle -A "${CHAIN}" -d ff02::/16 -j RETURN
279+
for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do
280+
[ -n "${cidr}" ] && ${IP6T} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN
281+
done
282+
${IP6T} -t mangle -A "${CHAIN}" -j DROP
283+
if ! ${IP6T} -t mangle -C OUTPUT -j "${CHAIN}" 2>/dev/null; then
284+
${IP6T} -t mangle -I OUTPUT 1 -j "${CHAIN}"
285+
fi
286+
echo "enforce-drop: IPv6 egress guard configured"
287+
else
288+
echo "enforce-drop: ip6tables unavailable — skipping IPv6 egress guard"
289+
fi
290+
291+
echo "enforce-drop: fail-closed egress guard active"
292+
}
293+
294+
# Dispatch enforce-drop here and exit; redirect mode falls through to the
295+
# transparent-interception logic below.
296+
if [ "${MODE}" = "enforce-drop" ]; then
297+
setup_enforce_drop
298+
exit 0
299+
fi
300+
171301
# =============================================================================
172302
# OUTBOUND traffic interception (nat OUTPUT)
173303
# =============================================================================

0 commit comments

Comments
 (0)