Merge pull request #498 from huang195/fix/in-cluster-egress-capture

Fix: Capture in-cluster TCP egress in enforce-redirect (keep DNS direct)

Author: huang195

CLAUDE.md

@@ -44,7 +44,7 @@ kagenti-extensions/
 │   │   ├── main.go
 │   │   ├── Dockerfile        #     proxy-sidecar lite combined image
 │   │   └── entrypoint.sh
-│   ├── proxy-init/           #   iptables init container (envoy-sidecar mode only)
+│   ├── proxy-init/           #   iptables init container (envoy-sidecar + proxy-sidecar enforce-redirect modes)
 │   │   ├── init-iptables.sh
 │   │   ├── Dockerfile.init
 │   │   ├── Makefile
@@ -76,7 +76,7 @@ kagenti-extensions/
 
 **Common:**
 - `authlib/` — shared auth library (JWT validation, token exchange, caching, routing, all listener implementations, all plugins).
-- `proxy-init/init-iptables.sh` — traffic interception setup (Istio ambient mesh compatible). Used by envoy-sidecar mode only.
+- `proxy-init/init-iptables.sh` — traffic interception setup (Istio ambient mesh compatible). Used by envoy-sidecar mode (`redirect`) and by proxy-sidecar mode's `enforce-redirect` egress guard.
 - `proxy-init/Dockerfile.init` — proxy-init container image.
 
 **Ports (envoy-sidecar):** 15123 (outbound), 15124 (inbound), 9090 (ext-proc), 9901 (admin)
@@ -149,13 +149,20 @@ Three mode-specific binaries, one Dockerfile per binary:
 
 ### PR Title Convention
 
-PRs must follow **conventional commits** format:
+PR titles must follow the format:
 
 ```
-<type>: <Subject starting with uppercase>
+<Prefix>: <Subject starting with uppercase>
 ```
 
-Types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, `revert`
+The CI title check (`deepakputhraya/action-pr-title`) is **case-sensitive** and
+requires a **capitalized** prefix from this set: `Build`, `Chore`, `CI`, `Docs`,
+`Feat`, `Fix`, `Perf`, `Refactor`, `Revert`, `Style`, `Test`, `Feature`,
+`Bug fix`, `Proposal`, `Breaking change`, `Other`, `Other/Misc`.
+
+Note: commit *message* examples elsewhere in this doc use lowercase
+(`feat:` / `fix:`) — that is fine for commits, but the PR *title* check rejects
+lowercase prefixes. Use `Fix:` / `Feat:` / `Docs:` in PR titles.
 
 ## Container Images
 
@@ -167,7 +174,7 @@ All images are pushed to `ghcr.io/kagenti/kagenti-extensions/` from
 | **`authbridge`** | **`authbridge/cmd/authbridge-proxy/Dockerfile`** | **proxy-sidecar combined image (default mode): authbridge-proxy (full plugin set incl. parsers) + spiffe-helper. No Envoy.** |
 | `authbridge-envoy` | `authbridge/cmd/authbridge-envoy/Dockerfile` | envoy-sidecar combined image: Envoy + authbridge-envoy (ext_proc, full plugin set) + spiffe-helper |
 | `authbridge-lite` | `authbridge/cmd/authbridge-lite/Dockerfile` | proxy-sidecar lite combined image: authbridge-lite (auth gates only, parsers dropped) + spiffe-helper. Same listener layout as `authbridge`; not yet referenced by the operator's default config |
-| `proxy-init` | `authbridge/proxy-init/Dockerfile.init` | Alpine + iptables init container (envoy-sidecar mode only) |
+| `proxy-init` | `authbridge/proxy-init/Dockerfile.init` | Alpine + iptables init container (envoy-sidecar + proxy-sidecar enforce-redirect modes) |
 
 In all three combined images, `spiffe-helper` is started conditionally
 based on the `SPIRE_ENABLED` env var (set by the operator when SPIRE

authbridge/CLAUDE.md

@@ -62,7 +62,7 @@ authbridge/
 │   ├── Dockerfile                    #   proxy-sidecar lite combined image
 │   └── entrypoint.sh
 │
-├── proxy-init/                       # iptables init container (envoy-sidecar mode only)
+├── proxy-init/                       # iptables init container (envoy-sidecar + proxy-sidecar enforce-redirect modes)
 │   ├── init-iptables.sh              #   iptables setup script
 │   ├── Dockerfile.init               #   proxy-init container image
 │   ├── Makefile                      #   docker-build-init + load-image targets
@@ -350,7 +350,7 @@ without needing a CR.
 ### Build images locally
 
 ```bash
-# Build the proxy-init iptables init container (envoy-sidecar mode only)
+# Build the proxy-init iptables init container (envoy-sidecar + proxy-sidecar enforce-redirect modes)
 cd authbridge/proxy-init
 make docker-build-init
 make load-image                     # Uses KIND_CLUSTER_NAME env var (default: kagenti)

authbridge/proxy-init/README.md

@@ -8,7 +8,7 @@ env var:
 | `MODE` | Used by | What it does |
 |---|---|---|
 | `redirect` (default) | `envoy-sidecar` | Transparently **REDIRECT**s pod traffic to the Envoy listeners. |
-| `enforce-redirect` | `proxy-sidecar`, `lite` | Fail-closed egress guard that **captures**: REDIRECTs external TCP that bypasses the forward proxy to AuthBridge's transparent listener; DROPs non-TCP external egress. |
+| `enforce-redirect` | `proxy-sidecar`, `lite` | Fail-closed egress guard that **captures**: REDIRECTs bypassing TCP — external **and** in-cluster (except in-cluster DNS) — to AuthBridge's transparent listener; DROPs non-TCP external egress; leaves in-cluster non-TCP (DNS/UDP) direct. |
 
 ## `redirect` mode (envoy-sidecar)
 
@@ -48,39 +48,50 @@ nat-table target but the nat table forbids `DROP` (`iptables` errors with
 
 - **`nat` OUTPUT / `AB_REDIRECT`** (position 1): `RETURN` ztunnel mark
   `0x539`, the proxy UID (`--uid-owner $PROXY_UID`, avoids the loop),
-  loopback, and `CLUSTER_CIDRS`; then `REDIRECT` external **TCP** to
-  `TRANSPARENT_PORT`.
+  loopback, and in-cluster **DNS-over-TCP** (`-p tcp --dport 53` to
+  `CLUSTER_CIDRS`, so cluster name resolution stays direct); then
+  `REDIRECT` all remaining **TCP** — external **and** in-cluster — to
+  `TRANSPARENT_PORT`, so agent→in-cluster calls (e.g. agent→tool) are
+  captured by the egress pipeline too.
 - **`mangle` OUTPUT / `AB_NOTCP`** (position 1): the same exemptions
   (plus `ESTABLISHED,RELATED` first, so UDP conntrack replies like DNS
-  pass), then `-p tcp -j RETURN` (TCP is handled by the nat REDIRECT) and
-  a terminal `DROP` for external **non-TCP** (UDP/QUIC), so HTTP/3 cannot
-  bypass — well-behaved clients fall back to TCP and get captured.
+  pass), **including the full `CLUSTER_CIDRS` `RETURN`** so in-cluster
+  non-TCP (DNS-over-UDP and any other in-cluster UDP) stays direct; then
+  `-p tcp -j RETURN` (TCP is handled by the nat REDIRECT) and a terminal
+  `DROP` for external **non-TCP** (UDP/QUIC), so HTTP/3 cannot bypass —
+  well-behaved clients fall back to TCP and get captured.
 
 Because the OUTPUT hook order is `raw → mangle → nat → filter`, the
 mangle chain drops non-TCP on its original destination while TCP falls
 through to the nat REDIRECT. Both chains are inserted at position 1,
-ahead of Istio's appended (`-A`) chains, so they preempt ambient's nat
-redirect for external destinations — exactly as `redirect` mode does for
-the Envoy path. IPv6 mirrors apply the same rules. See
+ahead of Istio's appended (`-A`) chains. The proxy's own re-originated
+egress (`--uid-owner $PROXY_UID`, `RETURN`ed) falls through to
+`ISTIO_OUTPUT` → ztunnel for transport mTLS under Istio ambient, and goes
+out plain without a mesh — so capturing in-cluster TCP **composes with**
+the mesh (AuthBridge does L7, ztunnel does transport) rather than
+bypassing it. IPv6 mirrors apply the same rules. See
 [`test-enforce-redirect.sh`](./test-enforce-redirect.sh), which proves
 the capture, the preemption, and the non-TCP drop via packet counters.
 
-> **`CLUSTER_CIDRS` is Kind-shaped by default.** The `10.0.0.0/8` default
-> covers Kind (pods `10.244.0.0/16` + services `10.96.0.0/16`). Other
-> distros differ — **OpenShift** uses services `172.30.0.0/16` and pods
-> `10.128.0.0/14`, and `172.30.0.0/16` is **outside** `10/8`, so the
-> default would drop in-cluster service traffic. On OCP/EKS/etc. you
-> **must** override `CLUSTER_CIDRS` with the cluster's real pod+service
-> ranges. The script logs the resolved value at startup, and the
-> operator sets it from the cluster's CIDRs.
+> **`CLUSTER_CIDRS` is Kind-shaped by default.** It now governs only what
+> stays **direct**: in-cluster DNS-over-TCP (`tcp/53`) and in-cluster
+> non-TCP (so cluster DNS keeps working). The `10.0.0.0/8` default covers
+> Kind (pods `10.244.0.0/16` + services `10.96.0.0/16`). Other distros
+> differ — **OpenShift** uses services `172.30.0.0/16` and pods
+> `10.128.0.0/14`, and `172.30.0.0/16` is **outside** `10/8`, so with the
+> default, cluster DNS (CoreDNS on `172.30.x`) is no longer left direct and
+> **resolution breaks**. On OCP/EKS/etc. you **must** override
+> `CLUSTER_CIDRS` with the cluster's real pod+service ranges. The script
+> logs the resolved value at startup, and the operator sets it from the
+> cluster's CIDRs.
 
 > **`enforce-redirect` intentionally ignores `OUTBOUND_PORTS_EXCLUDE`** (a
 > `redirect`-mode knob). Any destination previously bypassed that way —
 > e.g. a direct LLM endpoint at `host.docker.internal:11434` — is now
-> captured (external TCP) or dropped (external non-TCP) unless it falls
-> within `CLUSTER_CIDRS`. That is the point: `enforce-redirect` closes
-> direct-egress holes. Operators relying on a bypass for an in-cluster
-> target must include it in `CLUSTER_CIDRS`.
+> captured (external TCP) or dropped (external non-TCP). In-cluster TCP is
+> captured as well (only in-cluster DNS stays direct), so `CLUSTER_CIDRS`
+> is **not** a TCP bypass — it only keeps cluster DNS and in-cluster UDP
+> direct. That is the point: `enforce-redirect` closes direct-egress holes.
 
 ## iptables backend
 
@@ -100,7 +111,7 @@ whichever the host kernel exposes. Override with `IPTABLES_CMD` (and
 | `OUTBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated outbound port list to skip (e.g. `8080`) |
 | `INBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated inbound port list to skip |
 | `POD_IP` | (required in `redirect`) | redirect | Set via Downward API; DNAT target for ambient-mesh inbound. Not used by `enforce-redirect`. |
-| `CLUSTER_CIDRS` | `10.0.0.0/8` | enforce-redirect | Comma-separated in-cluster CIDRs allowed direct (pods/services/DNS) |
+| `CLUSTER_CIDRS` | `10.0.0.0/8` | enforce-redirect | Comma-separated in-cluster CIDRs kept direct **for DNS only**: DNS-over-TCP (`tcp/53`) + all in-cluster non-TCP (UDP). Other in-cluster TCP is captured. |
 | `CLUSTER_CIDRS6` | (empty) | enforce-redirect | IPv6 in-cluster CIDRs (dual-stack); empty drops all external v6 egress |
 | `IPTABLES_CMD` | auto-detected | all | Override iptables binary (`iptables-legacy` / `iptables-nft`) |
 | `IP6TABLES_CMD` | derived from `IPTABLES_CMD` | enforce-redirect | Override ip6tables binary |

authbridge/proxy-init/init-iptables.sh

@@ -178,11 +178,13 @@ SSH_PORT="${SSH_PORT:-22}"
 OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}"
 INBOUND_PORTS_EXCLUDE="${INBOUND_PORTS_EXCLUDE:-}"
 
-# enforce-redirect mode: in-cluster destinations the agent may reach directly
-# (pods / services / DNS) — external TCP is REDIRECTed to the transparent
-# listener and external non-TCP is dropped. Defaults to the RFC1918 10/8 block
-# which covers typical Kind pod (10.244/16) and service (10.96/16) CIDRs;
-# override with the cluster's actual ranges.
+# enforce-redirect mode: in-cluster CIDRs used to keep cluster DNS resolution
+# working — in-cluster DNS-over-TCP (TCP/53) and in-cluster non-TCP (UDP, incl.
+# DNS) are left direct, while all OTHER in-cluster TCP is now captured by the
+# egress pipeline so agent->in-cluster calls (e.g. agent->tool) are seen.
+# Defaults to the RFC1918 10/8 block (typical Kind pod 10.244/16 + service
+# 10.96/16); override with the cluster's actual ranges (e.g. OpenShift
+# 172.30.0.0/16) so cluster DNS is not captured.
 CLUSTER_CIDRS="${CLUSTER_CIDRS:-10.0.0.0/8}"
 CLUSTER_CIDRS6="${CLUSTER_CIDRS6:-}"   # IPv6 in-cluster CIDRs (dual-stack); empty = none
 
@@ -226,9 +228,11 @@ fi
 # Rule order: RETURN ztunnel's own sockets (fwmark 0x539, no-op without ambient)
 # -> RETURN the proxy's own re-originated egress (PROXY_UID, avoids the loop) ->
 # RETURN loopback (app -> forward proxy via HTTP_PROXY, and any loopback) ->
-# RETURN in-cluster CIDRs (mesh/DNS, left direct) -> REDIRECT external TCP to
-# TRANSPARENT_PORT -> DROP all other external egress (UDP/QUIC, so HTTP/3 can't
-# bypass; well-behaved clients fall back to TCP and get captured).
+# RETURN in-cluster DNS-over-TCP (TCP/53 to CLUSTER_CIDRS, left direct) ->
+# REDIRECT all remaining TCP -- external AND in-cluster -- to TRANSPARENT_PORT
+# (so agent->in-cluster calls are captured too) -> DROP all other external
+# egress (UDP/QUIC, so HTTP/3 can't bypass; clients fall back to TCP). In-cluster
+# non-TCP (UDP, incl. DNS) is left direct by the mangle chain below.
 #
 # The nat REDIRECT chain has no conntrack ESTABLISHED rule: nat only evaluates
 # the first packet of a flow, so replies and established connections are not
@@ -248,7 +252,7 @@ setup_enforce_redirect() {
 
   echo "enforce-redirect: installing fail-closed egress capture"
   echo "enforce-redirect: external TCP -> 127.0.0.1:${TRANSPARENT_PORT} (nat REDIRECT); external non-TCP -> DROP (mangle)"
-  echo "enforce-redirect: exempt proxy UID=${PROXY_UID}; direct in-cluster CIDRs=${CLUSTER_CIDRS}"
+  echo "enforce-redirect: exempt proxy UID=${PROXY_UID}; in-cluster TCP captured (DNS/53 + non-TCP left direct; CIDRs=${CLUSTER_CIDRS})"
 
   # --- IPv4: nat REDIRECT for TCP ---
   ${IPT} -t nat -N "${REDIR_CHAIN}" 2>/dev/null || true
@@ -261,11 +265,15 @@ setup_enforce_redirect() {
   # app -> forward proxy over loopback (HTTP_PROXY target), and any loopback.
   ${IPT} -t nat -A "${REDIR_CHAIN}" -o lo -j RETURN
   ${IPT} -t nat -A "${REDIR_CHAIN}" -d 127.0.0.0/8 -j RETURN
-  # in-cluster traffic (pods / services / DNS) — left direct, carried by the mesh.
+  # in-cluster DNS-over-TCP (TCP/53) — left direct so cluster name resolution is
+  # not captured. All OTHER in-cluster TCP falls through to the REDIRECT below,
+  # so the egress pipeline sees agent->in-cluster calls (e.g. agent->tool). The
+  # proxy's re-originated egress (PROXY_UID, RETURNed above) is exempt, and in an
+  # Istio ambient mesh it falls through to ISTIO_OUTPUT -> ztunnel for mTLS.
   for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do
-    [ -n "${cidr}" ] && ${IPT} -t nat -A "${REDIR_CHAIN}" -d "${cidr}" -j RETURN
+    [ -n "${cidr}" ] && ${IPT} -t nat -A "${REDIR_CHAIN}" -p tcp --dport 53 -d "${cidr}" -j RETURN
   done
-  # external TCP that bypassed the forward proxy — capture it transparently.
+  # all remaining TCP (external + in-cluster, minus in-cluster DNS) — capture it transparently.
   ${IPT} -t nat -A "${REDIR_CHAIN}" -p tcp -j REDIRECT --to-port "${TRANSPARENT_PORT}"
   if ! ${IPT} -t nat -C OUTPUT -j "${REDIR_CHAIN}" 2>/dev/null; then
     ${IPT} -t nat -I OUTPUT 1 -j "${REDIR_CHAIN}"
@@ -280,6 +288,9 @@ setup_enforce_redirect() {
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -o lo -j RETURN
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -d 127.0.0.0/8 -j RETURN
+  # in-cluster non-TCP (UDP, incl. DNS) — left direct. In-cluster TCP is now
+  # captured by the nat chain above; only in-cluster UDP relies on this RETURN
+  # (notably DNS-over-UDP, which the terminal DROP below would otherwise kill).
   for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do
     [ -n "${cidr}" ] && ${IPT} -t mangle -A "${NOTCP_CHAIN}" -d "${cidr}" -j RETURN
   done
@@ -305,8 +316,10 @@ setup_enforce_redirect() {
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -d ::1/128 -j RETURN
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -d fe80::/10 -j RETURN
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -d ff02::/16 -j RETURN
+    # in-cluster DNS-over-TCP (TCP/53) only — mirror of IPv4; all other in-cluster
+    # v6 TCP is captured below.
     for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do
-      [ -n "${cidr}" ] && ${IP6T} -t nat -A "${REDIR_CHAIN}" -d "${cidr}" -j RETURN
+      [ -n "${cidr}" ] && ${IP6T} -t nat -A "${REDIR_CHAIN}" -p tcp --dport 53 -d "${cidr}" -j RETURN
     done
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -p tcp -j REDIRECT --to-port "${TRANSPARENT_PORT}"
     if ! ${IP6T} -t nat -C OUTPUT -j "${REDIR_CHAIN}" 2>/dev/null; then

authbridge/proxy-init/test-enforce-redirect.sh

@@ -8,7 +8,8 @@
 #      position 1 with the expected RETURN exemptions and a `-p tcp` REDIRECT to
 #      TRANSPARENT_PORT (no DROP — the nat table forbids it); and the AB_NOTCP
 #      chain is hooked from mangle OUTPUT with `-p tcp RETURN` then a terminal
-#      DROP for external non-TCP egress.
+#      DROP for external non-TCP egress. In-cluster DNS-over-TCP (TCP/53) is left
+#      direct; all OTHER in-cluster TCP is captured (no blanket in-cluster RETURN).
 #   2. CAPTURE (not drop) + AMBIENT ROBUSTNESS — external TCP egress is
 #      REDIRECTed to TRANSPARENT_PORT, preempting a simulated Istio ambient
 #      "nat OUTPUT REDIRECT" appended after our chain. Proven via packet
@@ -65,7 +66,13 @@ assert "nat ztunnel mark RETURN"            'AB_REDIRECT .*mark.*0x539.*-j RETUR
 assert "nat proxy UID RETURN"               'AB_REDIRECT .*--uid-owner 1337 -j RETURN' "${natdump}"
 assert "nat loopback iface RETURN"          'AB_REDIRECT -o lo -j RETURN' "${natdump}"
 assert "nat loopback cidr RETURN"           'AB_REDIRECT -d 127.0.0.0/8 -j RETURN' "${natdump}"
-assert "nat cluster cidr RETURN"            'AB_REDIRECT -d 10.0.0.0/8 -j RETURN' "${natdump}"
+# in-cluster DNS-over-TCP (TCP/53) is left direct so cluster name resolution is
+# not captured; all OTHER in-cluster TCP now falls through to the REDIRECT.
+assert "nat in-cluster DNS-over-TCP RETURN" 'AB_REDIRECT.*10\.0\.0\.0/8.*dport 53.*RETURN' "${natdump}"
+# the blanket in-cluster RETURN must be gone — in-cluster TCP is now captured.
+if echo "${natdump}" | grep -qE '^-A AB_REDIRECT -d 10\.0\.0\.0/8 -j RETURN$'; then
+  echo "FAIL: nat AB_REDIRECT still blanket-RETURNs in-cluster (in-cluster TCP not captured)"; fail=1
+else echo "PASS: no blanket in-cluster RETURN — in-cluster TCP captured, only DNS/53 left direct"; fi
 assert "nat tcp REDIRECT to transparent"    "AB_REDIRECT -p tcp -j REDIRECT --to-ports ${TPORT}" "${natdump}"
 if echo "${natdump}" | grep -qE 'AB_REDIRECT -j DROP'; then
   echo "FAIL: nat AB_REDIRECT must not contain DROP (nat table forbids it)"; fail=1