refactor(egress): split mitmproxy config into yaml (static) vs env (dynamic)

Move fleet-wide, rarely-changing mitmproxy options into a baked-in
config.yaml under the standard mitm confdir layout, so launch.go only
emits per-deployment dynamic overrides via --set. This eliminates two
classes of bug along the way:

- stream_large_bodies was set in two places (launch.go --set 1m and
  custom.py ctx.options 10m), with the addon silently winning — making
  the launch.go line dead code. Now declared once in config.yaml (10m).
- ignore_hosts was env-driven with `;`-separated values, but each value
  was passed as a separate --set, and mitmproxy --set on a list option
  REPLACES the list — so configuring multiple bypass patterns silently
  only kept the last one. config.yaml uses a native YAML list with no
  override semantics.

Static options now in /var/lib/mitmproxy/.mitmproxy/config.yaml:
  mode, listen_host, connection_strategy (eager),
  stream_large_bodies (10m), http2, ignore_hosts (empty default),
  ssl_verify_upstream_trusted_confdir (default).

Dynamic overrides remain env-driven and applied as --set in launch.go
(precedence: --set > config.yaml > mitm defaults):
  OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT  (toggle)
  OPENSANDBOX_EGRESS_MITMPROXY_PORT
  OPENSANDBOX_EGRESS_MITMPROXY_SCRIPT
  OPENSANDBOX_EGRESS_MITMPROXY_SSL_INSECURE
  OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_TRUST_DIR

Removed env vars (no internal use, replaced by config.yaml):
  OPENSANDBOX_EGRESS_MITMPROXY_CONFDIR  — confdir is the mitm user's
    home (/var/lib/mitmproxy), which is also where config.yaml lives;
    splitting them via env created an unused escape hatch that would
    have broken config.yaml discovery.
  OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS  — replaced by ignore_hosts
    in config.yaml (native list, no covert-overwrite bug).

The mitmproxy.Config struct loses its ConfDir field accordingly.
SyncRootCA still accepts an optional confDirEnv argument so the existing
candidate-path search behavior is preserved if a future caller needs to
plumb it back in.

Author: Pangjiping

components/egress/Dockerfile

@@ -104,13 +104,22 @@ RUN apt-get update \
     && rm -rf /var/lib/apt/lists/*
 
 # Python mitmproxy (transparent mode): mitmdump runs as user mitmproxy; iptables skips this uid.
+# /var/lib/mitmproxy is mitm's home, used as the confdir (CA + config.yaml live under .mitmproxy/).
 RUN useradd -r -u 10042 -d /var/lib/mitmproxy -s /usr/sbin/nologin mitmproxy \
-    && mkdir -p /var/lib/mitmproxy \
-    && chown mitmproxy:mitmproxy /var/lib/mitmproxy \
+    && mkdir -p /var/lib/mitmproxy/.mitmproxy \
+    && chown -R mitmproxy:mitmproxy /var/lib/mitmproxy \
     && pip3 install --no-cache-dir --break-system-packages 'mitmproxy>=10,<11' \
     && (command -v mitmdump && mitmdump --version) \
     && mkdir -p /var/egress/mitmscripts
 
+# Static mitmproxy options (mode, listen_host, connection_strategy, stream_large_bodies,
+# http2, ignore_hosts, ssl_verify_upstream_trusted_confdir). mitmdump auto-loads
+# config.yaml from its confdir. Dynamic per-deployment options stay env-driven and
+# are applied as --set by launch.go (which overrides values declared here).
+COPY components/egress/mitmproxy/config.yaml /var/lib/mitmproxy/.mitmproxy/config.yaml
+RUN chown mitmproxy:mitmproxy /var/lib/mitmproxy/.mitmproxy/config.yaml \
+    && chmod 0644 /var/lib/mitmproxy/.mitmproxy/config.yaml
+
 # All egress runtime artifacts live under one directory to keep paths grouped.
 COPY --from=builder /out/egress /opt/opensandbox-egress/egress
 COPY --from=builder /out/opensandbox-supervisor /opt/opensandbox-egress/supervisor

components/egress/docs/mitmproxy-transparent.md

@@ -32,28 +32,48 @@ export OPENSANDBOX_EGRESS_MITMPROXY_PORT=18081
 
 # Optional: load an additional user-defined mitm addon (loaded after the system addon)
 export OPENSANDBOX_EGRESS_MITMPROXY_SCRIPT=/path/to/your/addon.py
-
-# Optional: bypass decryption for selected domains (semicolon-separated regex list)
-export OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS='.*\.log\.aliyuncs\.com;.*\.example\.internal'
 ```
 
+To bypass decryption for selected domains, edit the baked-in
+`components/egress/mitmproxy/config.yaml` and rebuild the image — see
+"Static Configuration (config.yaml)" below.
+
 ## Configuration Reference
 
+### Environment Variables (Per-Deployment Overrides)
+
 | Variable | Required | Purpose | Default |
 |------|----------|------|--------|
 | `OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT` | Yes | Enable transparent mitmproxy (`1/true/on`, etc.) | Disabled |
 | `OPENSANDBOX_EGRESS_MITMPROXY_PORT` | No | mitmdump listen port; `iptables` redirects `80/443` here | `18081` |
 | `OPENSANDBOX_EGRESS_MITMPROXY_SCRIPT` | No | Additional user mitm addon script path (`-s`); loaded after the system addon | Empty |
-| `OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS` | No | Host/IP regex list for TLS pass-through (`;` separated) | Empty |
-| `OPENSANDBOX_EGRESS_MITMPROXY_CONFDIR` | No | mitm config and CA directory (passed as `--set confdir=`, also used as `HOME`) | Default directory under `/var/lib/mitmproxy` |
-| `OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_TRUST_DIR` | No | Trust directory for upstream TLS verification (OpenSSL style) | `/etc/ssl/certs` |
+| `OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_TRUST_DIR` | No | Trust directory for upstream TLS verification (OpenSSL style); overrides the config.yaml default | `/etc/ssl/certs` |
+| `OPENSANDBOX_EGRESS_MITMPROXY_SSL_INSECURE` | No | Skip upstream TLS verification (`1/true/on`); use when clients connect by IP and SNI is unavailable | Disabled |
 
 Notes:
 
-- `OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS` means **no decryption**, not “completely bypass mitm process”.
 - In transparent mode, mitmproxy generally recommends matching by IP/range; verify SNI/resolve behavior if using domain regex only.
 - Before mitm, `iptables`, and CA export are ready, `GET /healthz` returns `503 (mitm not ready)` to prevent premature readiness.
 
+### Static Configuration (config.yaml)
+
+Fleet-wide, rarely-changing mitm options live in
+`components/egress/mitmproxy/config.yaml`, baked into the image at
+`/var/lib/mitmproxy/.mitmproxy/config.yaml` and auto-loaded by mitmdump.
+This is the single source of truth for:
+
+- `mode` (`transparent`)
+- `listen_host` (`127.0.0.1`)
+- `connection_strategy` (`eager`)
+- `stream_large_bodies` (`10m`)
+- `http2` (`true`)
+- `ignore_hosts` (regex list for TLS pass-through; empty by default — append entries here rather than via env, because `--set` on a list option REPLACES the entire list)
+- `ssl_verify_upstream_trusted_confdir` (default `/etc/ssl/certs`; overridable per-deployment via env)
+
+Precedence: command-line `--set` (from env overrides) > `config.yaml` > mitmproxy built-in defaults.
+
+To change a static option for the whole fleet: edit `config.yaml`, rebuild the egress image, redeploy. To bypass decryption for a specific host **temporarily** in one deployment, the option is to edit and remount `config.yaml` rather than pass an env override.
+
 ## Common Configuration Templates
 
 ### 1) Enable Transparent MITM Only
@@ -81,11 +101,18 @@ The user addon is loaded after the system addon (`-s system.py -s user.py`), so
 
 ### 4) Bypass Decryption for Specific Domains (e.g. log upload)
 
-```bash
-export OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT=true
-export OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS='.*\.log\.aliyuncs\.com'
+Edit `components/egress/mitmproxy/config.yaml` and append to `ignore_hosts`,
+then rebuild the egress image:
+
+```yaml
+ignore_hosts:
+  - '.*\.log\.aliyuncs\.com'
 ```
 
+`ignore_hosts` means **no decryption**, not "completely bypass mitm process":
+mitm still proxies the TCP connection, it just forwards bytes without
+breaking TLS, and addons do not see request/response content.
+
 ### 5) Use a Fixed CA (consistent fingerprint across replicas)
 
 If CA files already exist in `confdir`, mitmproxy reuses them instead of regenerating on each startup. Typical paths:

components/egress/mitmproxy/config.yaml

@@ -0,0 +1,71 @@
+# Baked-in static mitmproxy options for the OpenSandbox egress sidecar.
+#
+# Loaded automatically by mitmdump from <confdir>/config.yaml. The mitmproxy
+# user's home is /var/lib/mitmproxy, so the effective path inside the image is
+# /var/lib/mitmproxy/.mitmproxy/config.yaml (mitm's default confdir layout).
+#
+# This file is the single source of truth for fleet-wide, rarely-changing
+# mitmproxy options. Per-deployment overrides (listen port, debug flags,
+# user addon path, upstream TLS trust dir) remain env-driven and are
+# applied as `--set key=value` by launch.go, which takes precedence over
+# values declared here.
+#
+# Precedence: command line --set  >  this file  >  mitm built-in defaults
+#
+# DO NOT put per-sandbox or per-pipeline-run values here. DO NOT put
+# secrets here. This file ships in the image and is identical for every
+# replica.
+
+mode:
+  - transparent
+
+# Loopback only: transparent mode receives traffic via iptables REDIRECT
+# from inside the same network namespace; never expose mitm on 0.0.0.0.
+listen_host: 127.0.0.1
+
+# Eager: open the upstream connection alongside the client connection so
+# mitm's IO loop continuously observes upstream FIN/RST. Lazy defers the
+# upstream open until the full request is buffered and checks the
+# upstream pool per-request; on h1 keepalive sessions that exposes a
+# stale-connection race where the second request on a reused TCP picks
+# a peer-closed conn, surfacing as a silent transport error on the client
+# (e.g. `git fetch` exiting 128 with empty stderr after POST
+# /git-upload-pack right after GET /info/refs). Eager trades a wasted
+# TCP/TLS handshake for the small fraction of requests denied by the
+# egress addon, which is acceptable because a denied flow already
+# short-circuits with `flow.response = ...` before any HTTP write
+# reaches upstream.
+connection_strategy: eager
+
+# Threshold above which mitm streams response bodies chunk-by-chunk
+# instead of buffering them in memory. Set conservatively to keep RSS
+# bounded; chunked / SSE responses are forced to stream regardless via
+# the system addon's responseheaders hook.
+stream_large_bodies: 10m
+
+# Path-style trust store for upstream TLS verification (OpenSSL c_rehash
+# layout). Debian / RHEL ship the system CA bundle directory here.
+ssl_verify_upstream_trusted_confdir: /etc/ssl/certs
+
+# HTTP/2 negotiation enabled (default). Keep ALPN h2/http1.1 negotiation
+# on so upstreams that only advertise h2 are reachable.
+http2: true
+
+# Hosts (regex) for which mitm performs TLS pass-through: TCP forwarded
+# without decryption, addons do not see request/response content.
+#
+# Use this for high-throughput hosts where mitm interception adds no
+# value and only risks performance/race issues — e.g. code hosting that
+# does not need L7 policy, large file uploads, hosts known to behave
+# poorly through a TLS-terminating proxy.
+#
+# Each entry is a Python regex matched against the host. Use anchors and
+# escape dots. Example:
+#   ignore_hosts:
+#     - 'gitlab\.example\.com'
+#     - '.*\.internal-registry\.example\.com'
+#
+# Empty by default. Operators extending the image should append entries
+# here rather than passing --set on the command line, because --set on
+# a list option REPLACES the entire list.
+ignore_hosts: []

components/egress/mitmproxy_transparent.go

@@ -106,7 +106,6 @@ func startMitmproxyTransparentIfEnabled() (*mitmTransparent, error) {
 	cfg := mitmproxy.Config{
 		ListenPort: mpPort,
 		UserName:   mitmproxy.RunAsUser,
-		ConfDir:    strings.TrimSpace(os.Getenv(constants.EnvMitmproxyConfDir)),
 		ScriptPath: strings.TrimSpace(os.Getenv(constants.EnvMitmproxyScript)),
 	}
 	// Buffer absorbs OnExit events from a retry storm so OnExit goroutines
@@ -131,8 +130,7 @@ func startMitmproxyTransparentIfEnabled() (*mitmTransparent, error) {
 	}
 	log.Infof("mitmproxy: transparent intercept active (OUTPUT tcp 80,443 -> %d; trust mitm CA in clients)", mpPort)
 
-	confDir := strings.TrimSpace(os.Getenv(constants.EnvMitmproxyConfDir))
-	if err := mitmproxy.SyncRootCA(confDir, mpHome); err != nil {
+	if err := mitmproxy.SyncRootCA("", mpHome); err != nil {
 		return nil, fmt.Errorf("mitm CA export: %w", err)
 	}
 	return &mitmTransparent{

components/egress/pkg/constants/configuration.go

@@ -36,12 +36,13 @@ const (
 	EnvNameserverExempt        = "OPENSANDBOX_EGRESS_NAMESERVER_EXEMPT"
 
 	// MITM: mitmdump transparent; Linux + CAP_NET_ADMIN, runs as a dedicated user.
+	// Static mitm options (mode, listen_host, connection_strategy, stream_large_bodies,
+	// http2, ignore_hosts, ssl_verify_upstream_trusted_confdir default) live in
+	// /var/lib/mitmproxy/.mitmproxy/config.yaml; only per-deployment overrides are env-driven.
 	EnvMitmproxyTransparent      = "OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT"
 	EnvMitmproxyPort             = "OPENSANDBOX_EGRESS_MITMPROXY_PORT"
-	EnvMitmproxyConfDir          = "OPENSANDBOX_EGRESS_MITMPROXY_CONFDIR"
 	EnvMitmproxyScript           = "OPENSANDBOX_EGRESS_MITMPROXY_SCRIPT"
 	EnvMitmproxyUpstreamTrustDir = "OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_TRUST_DIR"
-	EnvMitmproxyIgnoreHosts      = "OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS"
 	EnvMitmproxySslInsecure      = "OPENSANDBOX_EGRESS_MITMPROXY_SSL_INSECURE"
 
 	// Comma-separated upstream resolvers: literal IP only (optional :port) — no hostnames (see dnsproxy REDIRECT note).

components/egress/pkg/mitmproxy/launch.go

@@ -32,17 +32,23 @@ import (
 const RunAsUser = "mitmproxy"
 
 // Loopback: transparent mode receives via REDIRECT; do not listen on 0.0.0.0 in the netns.
+// Kept as a Go constant only for the startup log line; the actual listen_host is set in
+// /var/lib/mitmproxy/.mitmproxy/config.yaml (shipped via the egress Dockerfile).
 const listenHostLoopback = "127.0.0.1"
 
 // systemScriptPath: bundled system addon shipped via the egress Dockerfile
 // (COPY components/egress/mitmscripts /var/egress/mitmscripts). Always loaded.
 const systemScriptPath = "/var/egress/mitmscripts/system.py"
 
-// Config: mitmdump --mode transparent; UserName must match iptables ! --uid-owner, ConfDir is mitm state/CA.
+// Config: mitmdump --mode transparent. Static options (mode, listen_host,
+// connection_strategy, stream_large_bodies, http2, ignore_hosts,
+// ssl_verify_upstream_trusted_confdir) live in
+// /var/lib/mitmproxy/.mitmproxy/config.yaml and are auto-loaded by mitmdump.
+// This struct carries only per-launch dynamic values that override those
+// defaults via `--set`.
 type Config struct {
 	ListenPort int
 	UserName   string
-	ConfDir    string
 	// ScriptPath is an optional user-supplied addon, loaded after the system addon.
 	ScriptPath string
 	// OnExit is called (if non-nil) when mitmdump exits. Called from a background goroutine.
@@ -92,24 +98,21 @@ func Launch(cfg Config) (*Running, error) {
 		return nil, fmt.Errorf("mitmproxy: lookup user %q: %w", uname, err)
 	}
 
+	// Only per-launch dynamic values are passed on the command line. Static
+	// options (mode, listen_host, connection_strategy, stream_large_bodies,
+	// http2, ignore_hosts, ssl_verify_upstream_trusted_confdir) come from
+	// /var/lib/mitmproxy/.mitmproxy/config.yaml shipped in the egress image.
+	// `--set` overrides config.yaml, so the env-driven overrides below take
+	// precedence at runtime without rebuilding the image.
 	args := []string{
-		"--mode", "transparent",
-		"--listen-host", listenHostLoopback,
 		"--listen-port", strconv.Itoa(cfg.ListenPort),
 	}
 
-	trustDir := strings.TrimSpace(os.Getenv(constants.EnvMitmproxyUpstreamTrustDir))
-	if trustDir == "" {
-		trustDir = "/etc/ssl/certs"
+	// Upstream cert trust path override. Default in config.yaml is /etc/ssl/certs;
+	// override per-deployment when the upstream uses a private CA bundle.
+	if trustDir := strings.TrimSpace(os.Getenv(constants.EnvMitmproxyUpstreamTrustDir)); trustDir != "" {
+		args = append(args, "--set", "ssl_verify_upstream_trusted_confdir="+trustDir)
 	}
-	args = append(args, "--set", "ssl_verify_upstream_trusted_confdir="+trustDir)
-
-	// Stream large bodies instead of buffering them in memory (OOM prevention).
-	args = append(args, "--set", "stream_large_bodies=1m")
-
-	// Lazy connection strategy: defer upstream connection until the request is fully received,
-	// which avoids unnecessary connections for blocked/filtered requests.
-	args = append(args, "--set", "connection_strategy=lazy")
 
 	// Transparent mode redirects TCP to IP addresses. Clients connecting to IPs
 	// do not send SNI, so upstream TLS cert hostname verification fails with
@@ -119,34 +122,21 @@ func Launch(cfg Config) (*Running, error) {
 		args = append(args, "--set", "ssl_insecure=true")
 	}
 
-	homeEnv := home
-	if strings.TrimSpace(cfg.ConfDir) != "" {
-		cd := strings.TrimSpace(cfg.ConfDir)
-		args = append(args, "--set", "confdir="+cd)
-		homeEnv = cd
-	}
 	// Load the system addon first so user addons can observe / override its hooks.
 	args = append(args, "-s", systemScriptPath)
 	if user := strings.TrimSpace(cfg.ScriptPath); user != "" {
 		args = append(args, "-s", user)
 	}
 
-	// Upstream passthrough: each pattern becomes --set ignore_hosts= (regex; IP ranges are practical in transparent mode).
-	for _, p := range strings.Split(os.Getenv(constants.EnvMitmproxyIgnoreHosts), ";") {
-		p = strings.TrimSpace(p)
-		if p == "" {
-			continue
-		}
-		args = append(args, "--set", "ignore_hosts="+p)
-	}
-
 	cmd := exec.Command("mitmdump", args...)
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
 	cmd.SysProcAttr = &syscall.SysProcAttr{
 		Credential: &syscall.Credential{Uid: uid, Gid: gid},
 	}
-	cmd.Env = append(os.Environ(), "HOME="+homeEnv)
+	// HOME determines mitm's confdir (~/.mitmproxy) which holds both the CA
+	// and the baked-in config.yaml.
+	cmd.Env = append(os.Environ(), "HOME="+home)
 
 	if err := cmd.Start(); err != nil {
 		return nil, fmt.Errorf("mitmproxy: start mitmdump: %w", err)