Merge pull request #499 from huang195/fix/proxy-init-dns-resolvconf

Fix: Exempt DNS by resolv.conf nameservers, retire CLUSTER_CIDRS

Author: huang195

authbridge/proxy-init/README.md

@@ -48,18 +48,19 @@ nat-table target but the nat table forbids `DROP` (`iptables` errors with
 
 - **`nat` OUTPUT / `AB_REDIRECT`** (position 1): `RETURN` ztunnel mark
   `0x539`, the proxy UID (`--uid-owner $PROXY_UID`, avoids the loop),
-  loopback, and in-cluster **DNS-over-TCP** (`-p tcp --dport 53` to
-  `CLUSTER_CIDRS`, so cluster name resolution stays direct); then
-  `REDIRECT` all remaining **TCP** — external **and** in-cluster — to
+  loopback, and **DNS-over-TCP** (`-p tcp --dport 53`) to each
+  `/etc/resolv.conf` nameserver (so cluster name resolution stays direct);
+  then `REDIRECT` all remaining **TCP** — external **and** in-cluster — to
   `TRANSPARENT_PORT`, so agent→in-cluster calls (e.g. agent→tool) are
   captured by the egress pipeline too.
-- **`mangle` OUTPUT / `AB_NOTCP`** (position 1): the same exemptions
-  (plus `ESTABLISHED,RELATED` first, so UDP conntrack replies like DNS
-  pass), **including the full `CLUSTER_CIDRS` `RETURN`** so in-cluster
-  non-TCP (DNS-over-UDP and any other in-cluster UDP) stays direct; then
-  `-p tcp -j RETURN` (TCP is handled by the nat REDIRECT) and a terminal
-  `DROP` for external **non-TCP** (UDP/QUIC), so HTTP/3 cannot bypass —
-  well-behaved clients fall back to TCP and get captured.
+- **`mangle` OUTPUT / `AB_NOTCP`** (position 1): the same UID/mark/loopback
+  exemptions (plus `ESTABLISHED,RELATED` first, so UDP conntrack replies
+  like DNS pass), then **DNS-over-UDP** (`-p udp --dport 53`) to each
+  resolv.conf nameserver so cluster DNS keeps working; then `-p tcp -j
+  RETURN` (TCP is handled by the nat REDIRECT) and a terminal `DROP` for
+  all other **non-TCP** (UDP/QUIC), so HTTP/3 cannot bypass and non-DNS
+  in-cluster UDP is dropped too — well-behaved clients fall back to TCP and
+  get captured.
 
 Because the OUTPUT hook order is `raw → mangle → nat → filter`, the
 mangle chain drops non-TCP on its original destination while TCP falls
@@ -73,25 +74,26 @@ bypassing it. IPv6 mirrors apply the same rules. See
 [`test-enforce-redirect.sh`](./test-enforce-redirect.sh), which proves
 the capture, the preemption, and the non-TCP drop via packet counters.
 
-> **`CLUSTER_CIDRS` is Kind-shaped by default.** It now governs only what
-> stays **direct**: in-cluster DNS-over-TCP (`tcp/53`) and in-cluster
-> non-TCP (so cluster DNS keeps working). The `10.0.0.0/8` default covers
-> Kind (pods `10.244.0.0/16` + services `10.96.0.0/16`). Other distros
-> differ — **OpenShift** uses services `172.30.0.0/16` and pods
-> `10.128.0.0/14`, and `172.30.0.0/16` is **outside** `10/8`, so with the
-> default, cluster DNS (CoreDNS on `172.30.x`) is no longer left direct and
-> **resolution breaks**. On OCP/EKS/etc. you **must** override
-> `CLUSTER_CIDRS` with the cluster's real pod+service ranges. The script
-> logs the resolved value at startup, and the operator sets it from the
-> cluster's CIDRs.
+> **DNS stays direct by following the pod's actual resolvers — no CIDR
+> guessing.** The only thing left direct is DNS (`tcp/53` + `udp/53`) to the
+> `nameserver` IPs in `/etc/resolv.conf`, which kubelet writes per the pod's
+> `dnsPolicy`. This is cluster-agnostic by construction: it works whether the
+> resolver is a Kind/OpenShift/EKS service ClusterIP (any service CIDR — incl.
+> OpenShift's `172.30.0.0/16`, which is **outside** `10/8`) or a NodeLocal
+> DNSCache at a link-local `169.254.x` address. The script logs the resolved
+> nameservers at startup; override the file path with `RESOLV_CONF` (mainly
+> for tests). There is **no** in-cluster CIDR knob — `enforce-redirect` no
+> longer needs one. (A prior `CLUSTER_CIDRS` env was removed; its `10.0.0.0/8`
+> default silently dropped DNS on OpenShift, where the resolver sits outside
+> `10/8`.)
 
 > **`enforce-redirect` intentionally ignores `OUTBOUND_PORTS_EXCLUDE`** (a
 > `redirect`-mode knob). Any destination previously bypassed that way —
 > e.g. a direct LLM endpoint at `host.docker.internal:11434` — is now
 > captured (external TCP) or dropped (external non-TCP). In-cluster TCP is
-> captured as well (only in-cluster DNS stays direct), so `CLUSTER_CIDRS`
-> is **not** a TCP bypass — it only keeps cluster DNS and in-cluster UDP
-> direct. That is the point: `enforce-redirect` closes direct-egress holes.
+> captured as well (only DNS to the resolvers stays direct). That is the
+> point: `enforce-redirect` closes direct-egress holes, and the DNS
+> exemption is scoped to `port 53` to the resolver IPs — not a TCP bypass.
 
 ## iptables backend
 
@@ -111,8 +113,7 @@ whichever the host kernel exposes. Override with `IPTABLES_CMD` (and
 | `OUTBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated outbound port list to skip (e.g. `8080`) |
 | `INBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated inbound port list to skip |
 | `POD_IP` | (required in `redirect`) | redirect | Set via Downward API; DNAT target for ambient-mesh inbound. Not used by `enforce-redirect`. |
-| `CLUSTER_CIDRS` | `10.0.0.0/8` | enforce-redirect | Comma-separated in-cluster CIDRs kept direct **for DNS only**: DNS-over-TCP (`tcp/53`) + all in-cluster non-TCP (UDP). Other in-cluster TCP is captured. |
-| `CLUSTER_CIDRS6` | (empty) | enforce-redirect | IPv6 in-cluster CIDRs (dual-stack); empty drops all external v6 egress |
+| `RESOLV_CONF` | `/etc/resolv.conf` | enforce-redirect | Path read at init for `nameserver` IPs; DNS (`tcp/53` + `udp/53`) to those IPs is left direct (IPv4→`iptables`, IPv6→`ip6tables`). Override mainly for tests. |
 | `IPTABLES_CMD` | auto-detected | all | Override iptables binary (`iptables-legacy` / `iptables-nft`) |
 | `IP6TABLES_CMD` | derived from `IPTABLES_CMD` | enforce-redirect | Override ip6tables binary |
 

authbridge/proxy-init/init-iptables.sh

@@ -178,15 +178,28 @@ SSH_PORT="${SSH_PORT:-22}"
 OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}"
 INBOUND_PORTS_EXCLUDE="${INBOUND_PORTS_EXCLUDE:-}"
 
-# enforce-redirect mode: in-cluster CIDRs used to keep cluster DNS resolution
-# working — in-cluster DNS-over-TCP (TCP/53) and in-cluster non-TCP (UDP, incl.
-# DNS) are left direct, while all OTHER in-cluster TCP is now captured by the
-# egress pipeline so agent->in-cluster calls (e.g. agent->tool) are seen.
-# Defaults to the RFC1918 10/8 block (typical Kind pod 10.244/16 + service
-# 10.96/16); override with the cluster's actual ranges (e.g. OpenShift
-# 172.30.0.0/16) so cluster DNS is not captured.
-CLUSTER_CIDRS="${CLUSTER_CIDRS:-10.0.0.0/8}"
-CLUSTER_CIDRS6="${CLUSTER_CIDRS6:-}"   # IPv6 in-cluster CIDRs (dual-stack); empty = none
+# enforce-redirect mode: under the egress guard, cluster DNS must stay direct
+# (the forward proxy is HTTP-only and cannot carry DNS), while every other
+# in-cluster TCP flow is captured by the pipeline. Rather than guess in-cluster
+# CIDRs — the resolver may sit OUTSIDE them (OpenShift service net 172.30/172.31,
+# outside 10/8) or at a link-local address (NodeLocal DNSCache, 169.254.x) — we
+# read the pod's actual resolvers from /etc/resolv.conf and leave only DNS
+# (UDP/53 + TCP/53) to THOSE IPs direct. kubelet writes resolv.conf per the pod's
+# dnsPolicy, so it is the authoritative, cluster-agnostic source for "where is my
+# resolver". Override the path with RESOLV_CONF (e.g. for tests).
+RESOLV_CONF="${RESOLV_CONF:-/etc/resolv.conf}"
+
+# Emit the `nameserver` IPs from resolv.conf, one per line (IPv4 and IPv6 mixed;
+# callers split by address family). Empty output if the file is missing/unreadable.
+get_nameservers() {
+  [ -r "${RESOLV_CONF}" ] || return 0
+  while read -r _key _val _rest; do
+    [ "${_key}" = "nameserver" ] && [ -n "${_val}" ] && echo "${_val}"
+  done < "${RESOLV_CONF}"
+  # `read` returns non-zero at EOF; force success so `NS=$(get_nameservers)`
+  # under `set -e` does not abort the script (missing/empty resolv.conf is fine).
+  return 0
+}
 
 # IPv6 counterpart of the detected iptables backend (iptables-legacy ->
 # ip6tables-legacy, iptables -> ip6tables). Override with IP6TABLES_CMD.
@@ -228,11 +241,12 @@ fi
 # Rule order: RETURN ztunnel's own sockets (fwmark 0x539, no-op without ambient)
 # -> RETURN the proxy's own re-originated egress (PROXY_UID, avoids the loop) ->
 # RETURN loopback (app -> forward proxy via HTTP_PROXY, and any loopback) ->
-# RETURN in-cluster DNS-over-TCP (TCP/53 to CLUSTER_CIDRS, left direct) ->
+# RETURN DNS-over-TCP (TCP/53) to the resolv.conf nameservers (left direct) ->
 # REDIRECT all remaining TCP -- external AND in-cluster -- to TRANSPARENT_PORT
-# (so agent->in-cluster calls are captured too) -> DROP all other external
-# egress (UDP/QUIC, so HTTP/3 can't bypass; clients fall back to TCP). In-cluster
-# non-TCP (UDP, incl. DNS) is left direct by the mangle chain below.
+# (so agent->in-cluster calls are captured too) -> DROP all other egress
+# (UDP/QUIC, so HTTP/3 can't bypass; clients fall back to TCP). DNS-over-UDP
+# (UDP/53) to the resolvers is left direct by the mangle chain below; all other
+# non-TCP -- including non-DNS in-cluster UDP -- is dropped.
 #
 # The nat REDIRECT chain has no conntrack ESTABLISHED rule: nat only evaluates
 # the first packet of a flow, so replies and established connections are not
@@ -250,9 +264,22 @@ setup_enforce_redirect() {
   REDIR_CHAIN="AB_REDIRECT"
   NOTCP_CHAIN="AB_NOTCP"
 
+  # Fail closed and LOUD on zero resolvers: without a nameserver to exempt, the
+  # rules below would drop UDP/53 and capture TCP/53, leaving a running-but-DNS-
+  # dead pod that is far harder to triage than a failed init container. In a
+  # Kubernetes pod kubelet always populates resolv.conf, so an empty result means
+  # a real misconfiguration — surface it as Init:Error rather than silent breakage.
+  NAMESERVERS=$(get_nameservers)
+  if [ -z "${NAMESERVERS}" ]; then
+    echo "enforce-redirect: ERROR: no nameservers found in ${RESOLV_CONF}" >&2
+    echo "enforce-redirect: refusing to start — DNS egress would be dropped (UDP/53) / captured (TCP/53), silently breaking name resolution." >&2
+    echo "enforce-redirect: set RESOLV_CONF to a file with valid 'nameserver' entries if running outside Kubernetes." >&2
+    exit 1
+  fi
+
   echo "enforce-redirect: installing fail-closed egress capture"
   echo "enforce-redirect: external TCP -> 127.0.0.1:${TRANSPARENT_PORT} (nat REDIRECT); external non-TCP -> DROP (mangle)"
-  echo "enforce-redirect: exempt proxy UID=${PROXY_UID}; in-cluster TCP captured (DNS/53 + non-TCP left direct; CIDRs=${CLUSTER_CIDRS})"
+  echo "enforce-redirect: exempt proxy UID=${PROXY_UID}; in-cluster TCP captured; DNS/53 to resolvers left direct (resolvers=$(echo "${NAMESERVERS}" | tr '\n' ' '))"
 
   # --- IPv4: nat REDIRECT for TCP ---
   ${IPT} -t nat -N "${REDIR_CHAIN}" 2>/dev/null || true
@@ -265,13 +292,15 @@ setup_enforce_redirect() {
   # app -> forward proxy over loopback (HTTP_PROXY target), and any loopback.
   ${IPT} -t nat -A "${REDIR_CHAIN}" -o lo -j RETURN
   ${IPT} -t nat -A "${REDIR_CHAIN}" -d 127.0.0.0/8 -j RETURN
-  # in-cluster DNS-over-TCP (TCP/53) — left direct so cluster name resolution is
-  # not captured. All OTHER in-cluster TCP falls through to the REDIRECT below,
-  # so the egress pipeline sees agent->in-cluster calls (e.g. agent->tool). The
-  # proxy's re-originated egress (PROXY_UID, RETURNed above) is exempt, and in an
-  # Istio ambient mesh it falls through to ISTIO_OUTPUT -> ztunnel for mTLS.
-  for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do
-    [ -n "${cidr}" ] && ${IPT} -t nat -A "${REDIR_CHAIN}" -p tcp --dport 53 -d "${cidr}" -j RETURN
+  # DNS-over-TCP (TCP/53) to the pod's resolvers — left direct so cluster name
+  # resolution is not captured (the forward proxy can't carry DNS). All OTHER
+  # in-cluster TCP falls through to the REDIRECT below, so the egress pipeline
+  # sees agent->in-cluster calls (e.g. agent->tool). The proxy's re-originated
+  # egress (PROXY_UID, RETURNed above) is exempt, and in an Istio ambient mesh it
+  # falls through to ISTIO_OUTPUT -> ztunnel for mTLS.
+  for ns in ${NAMESERVERS}; do
+    case "${ns}" in *:*) continue ;; esac   # IPv6 resolver — handled in the v6 block
+    ${IPT} -t nat -A "${REDIR_CHAIN}" -p tcp --dport 53 -d "${ns}" -j RETURN
   done
   # all remaining TCP (external + in-cluster, minus in-cluster DNS) — capture it transparently.
   ${IPT} -t nat -A "${REDIR_CHAIN}" -p tcp -j REDIRECT --to-port "${TRANSPARENT_PORT}"
@@ -288,11 +317,13 @@ setup_enforce_redirect() {
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -o lo -j RETURN
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -d 127.0.0.0/8 -j RETURN
-  # in-cluster non-TCP (UDP, incl. DNS) — left direct. In-cluster TCP is now
-  # captured by the nat chain above; only in-cluster UDP relies on this RETURN
-  # (notably DNS-over-UDP, which the terminal DROP below would otherwise kill).
-  for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do
-    [ -n "${cidr}" ] && ${IPT} -t mangle -A "${NOTCP_CHAIN}" -d "${cidr}" -j RETURN
+  # DNS-over-UDP (UDP/53) to the pod's resolvers — left direct (the terminal DROP
+  # below would otherwise kill cluster name resolution). Scoped to the resolvers
+  # and port 53: all other non-TCP, including non-DNS in-cluster UDP, is dropped
+  # so nothing can bypass the pipeline over UDP.
+  for ns in ${NAMESERVERS}; do
+    case "${ns}" in *:*) continue ;; esac   # IPv6 resolver — handled in the v6 block
+    ${IPT} -t mangle -A "${NOTCP_CHAIN}" -p udp --dport 53 -d "${ns}" -j RETURN
   done
   # TCP is handled by the nat REDIRECT above — let it pass mangle untouched.
   ${IPT} -t mangle -A "${NOTCP_CHAIN}" -p tcp -j RETURN
@@ -304,9 +335,9 @@ setup_enforce_redirect() {
   echo "enforce-redirect: IPv4 egress capture configured"
 
   # --- IPv6 ---
-  # Mirror of IPv4. Until v6 cluster CIDRs are wired (CLUSTER_CIDRS6), allow
-  # loopback + link-local (fe80::/10 unicast, ff02::/16 NDP/MLD multicast) and
-  # the proxy UID / ztunnel mark; REDIRECT external v6 TCP; DROP other v6 egress.
+  # Mirror of IPv4: allow loopback + link-local (fe80::/10 unicast, ff02::/16
+  # NDP/MLD multicast) and the proxy UID / ztunnel mark; leave DNS to any IPv6
+  # resolv.conf nameservers direct; REDIRECT external v6 TCP; DROP other v6 egress.
   if command -v "${IP6T%% *}" >/dev/null 2>&1 && ${IP6T} -t nat -L >/dev/null 2>&1; then
     ${IP6T} -t nat -N "${REDIR_CHAIN}" 2>/dev/null || true
     ${IP6T} -t nat -F "${REDIR_CHAIN}"
@@ -316,10 +347,11 @@ setup_enforce_redirect() {
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -d ::1/128 -j RETURN
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -d fe80::/10 -j RETURN
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -d ff02::/16 -j RETURN
-    # in-cluster DNS-over-TCP (TCP/53) only — mirror of IPv4; all other in-cluster
-    # v6 TCP is captured below.
-    for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do
-      [ -n "${cidr}" ] && ${IP6T} -t nat -A "${REDIR_CHAIN}" -p tcp --dport 53 -d "${cidr}" -j RETURN
+    # DNS-over-TCP (TCP/53) to IPv6 resolvers only — mirror of IPv4; all other
+    # in-cluster v6 TCP is captured below.
+    for ns in ${NAMESERVERS}; do
+      case "${ns}" in *:*) ;; *) continue ;; esac   # IPv4 resolver — handled in the v4 block
+      ${IP6T} -t nat -A "${REDIR_CHAIN}" -p tcp --dport 53 -d "${ns}" -j RETURN
     done
     ${IP6T} -t nat -A "${REDIR_CHAIN}" -p tcp -j REDIRECT --to-port "${TRANSPARENT_PORT}"
     if ! ${IP6T} -t nat -C OUTPUT -j "${REDIR_CHAIN}" 2>/dev/null; then
@@ -335,8 +367,10 @@ setup_enforce_redirect() {
     ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -d ::1/128 -j RETURN
     ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -d fe80::/10 -j RETURN
     ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -d ff02::/16 -j RETURN
-    for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do
-      [ -n "${cidr}" ] && ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -d "${cidr}" -j RETURN
+    # DNS-over-UDP (UDP/53) to IPv6 resolvers only — mirror of IPv4.
+    for ns in ${NAMESERVERS}; do
+      case "${ns}" in *:*) ;; *) continue ;; esac   # IPv4 resolver — handled in the v4 block
+      ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -p udp --dport 53 -d "${ns}" -j RETURN
     done
     ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -p tcp -j RETURN
     ${IP6T} -t mangle -A "${NOTCP_CHAIN}" -j DROP