opensandbox-group · hittyt · Jun 5, 2026 · Jun 5, 2026
@@ -55,18 +55,46 @@ Optional advanced features:
 - Nameserver bypass: `OPENSANDBOX_EGRESS_NAMESERVER_EXEMPT`
 - Denied hostname webhook: `OPENSANDBOX_EGRESS_DENY_WEBHOOK`, `OPENSANDBOX_EGRESS_SANDBOX_ID`
 - DoH/DoT controls: `OPENSANDBOX_EGRESS_BLOCK_DOH_443`, `OPENSANDBOX_EGRESS_DOH_BLOCKLIST`
+- Custom DNS upstream: `OPENSANDBOX_EGRESS_DNS_UPSTREAM` (comma-separated IPs, optional `:port`), `OPENSANDBOX_EGRESS_DNS_UPSTREAM_TIMEOUT` (default `5` seconds)
+- DNS upstream health probe: `OPENSANDBOX_EGRESS_DNS_UPSTREAM_PROBE` (enable), `OPENSANDBOX_EGRESS_DNS_UPSTREAM_PROBE_INTERVAL_SEC`
+
+### Always-Rules Files
+
+Static rule files under `/var/egress/rules/` are loaded at startup and take priority over dynamic API rules:
+
+| File | Purpose |
+|------|---------|
+| `/var/egress/rules/deny.always` | Domains always denied, overrides user and allow rules |
+| `/var/egress/rules/allow.always` | Domains always allowed, overrides user rules |
+| `/var/egress/rules/log_skip.always` | Domain patterns whose DNS blocks are not logged (noise reduction) |
+
+Format: one domain per line (supports wildcards like `*.example.com`). Lines starting with `#` are comments. Missing files are silently ignored.
+
+Rule precedence: `deny.always` > `allow.always` > user policy (API/env).
+
+Always-rules are hot-reloaded: the sidecar polls the files once per minute and applies changes without restart.
 
 ### Runtime HTTP API
 
-- `GET /policy`: get current policy
-- `POST /policy`: replace policy (`{}`, `null`, empty body => reset to deny-all)
-- `PATCH /policy`: merge/append rules (body is JSON array of egress rules)
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/policy` | Get current policy and enforcement mode |
+| `POST` | `/policy` | Replace policy (`{}`, `null`, empty body => reset to deny-all) |
+| `PUT` | `/policy` | Alias for `POST` |
+| `PATCH` | `/policy` | Merge/append rules (body is JSON array of egress rules) |
+| `DELETE` | `/policy` | Remove specific targets (body is JSON string array, e.g. `["*.example.com"]`) |
+| `GET` | `/healthz` | Health check; returns `200 ok` or `503 mitmproxy not ready` (when transparent MITM is enabled but not yet initialized) |
 
 Quick example:
 
 ```bash
+# Replace policy
 curl -XPOST http://127.0.0.1:18080/policy \
   -d '{"defaultAction":"deny","egress":[{"action":"allow","target":"*.example.com"}]}'
+
+# Remove specific targets
+curl -XDELETE http://127.0.0.1:18080/policy \
+  -d '["*.example.com"]'
 ```
 
 ### Experimental: Transparent MITM (mitmproxy)
@@ -128,7 +156,7 @@ curl -I https://github.com
 
 ## Development
 
-- **Language**: Go 1.24+
+- **Language**: Go 1.25+
 - **Key Packages**:
     - `pkg/dnsproxy`: DNS server and policy matching logic.
     - `pkg/iptables`: `iptables` rule management.
@@ -152,6 +180,21 @@ An end-to-end benchmark compares **dns** (pass-through, no nft write) and **dns+
 
 More details in [docs/benchmark.md](docs/benchmark.md).
 
+## Process Supervisor
+
+The egress container runs under [`opensandbox-supervisor`](../../components/internal/supervisor/README.md), a lightweight process wrapper that restarts the egress worker on crash with exponential backoff, a crashloop circuit breaker, and structured JSONL event logging.
+
+```
+ENTRYPOINT: supervisor --pre-start=cleanup.sh --name=egress --grace-period=20s -- /opt/opensandbox-egress/egress
+```
+
+Egress-specific configuration:
+
+- **`--grace-period=20s`**: Egress needs extra time to drain DNS connections and tear down iptables/nft rules on shutdown (default is 10 s).
+- **Pre-start hook** (`cleanup.sh`): Reaps orphaned `mitmdump` processes from a previous crash so the new egress can bind the MITM listen port. Intentionally does NOT tear down iptables/nft rules — keeping enforcement active during the backoff window protects the workload.
+
+For full supervisor documentation (all flags, backoff behavior, crashloop breaker, event log schema, library API), see the [supervisor README](../../components/internal/supervisor/README.md).
+
 ## Troubleshooting
 
 - **"iptables setup failed"**: ensure sidecar has `--cap-add=NET_ADMIN`.

@@ -47,6 +47,7 @@ export OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS='.*\.log\.aliyuncs\.com;.*\.exa
 | `OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS` | No | Host/IP regex list for TLS pass-through (`;` separated) | Empty |
 | `OPENSANDBOX_EGRESS_MITMPROXY_CONFDIR` | No | mitm config and CA directory (passed as `--set confdir=`, also used as `HOME`) | Default directory under `/var/lib/mitmproxy` |
 | `OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_TRUST_DIR` | No | Trust directory for upstream TLS verification (OpenSSL style) | `/etc/ssl/certs` |
+| `OPENSANDBOX_EGRESS_MITMPROXY_SSL_INSECURE` | No | Skip upstream TLS certificate verification (`1/true/on`). Needed when clients connect by IP (no SNI → hostname mismatch). | Disabled |
 
 Notes:
 
@@ -112,3 +113,59 @@ Limits:
 - Currently IPv4 `iptables` only; IPv6 is not automatically handled.
 - Non-Linux environments (for example local macOS runtime) are not supported for transparent mode.
 - Full HTTPS decryption introduces CPU/memory and certificate trust overhead; benchmark before production rollout.
+
+## Process Supervisor (Crash Recovery)
+
+The egress sidecar includes a built-in supervisor that monitors the `mitmdump` child process and automatically restarts it on unexpected exits.
+
+### Restart behavior
+
+When `mitmdump` exits unexpectedly, the supervisor restarts it with **exponential backoff**: 1 s, 2 s, 4 s, ..., capped at **30 s**. Retries continue indefinitely until the process starts successfully or the egress sidecar itself shuts down.
+
+A successful restart requires two conditions:
+
+1. `mitmdump` process starts without error.
+2. The listen port (`127.0.0.1:<port>`) accepts TCP connections within 15 seconds.
+
+If the listener does not come up in time, the half-started process is gracefully terminated (SIGTERM → wait → SIGKILL) before the next attempt, so the port is released cleanly.
+
+### Generation tagging
+
+Each `mitmdump` launch is assigned a monotonically increasing **generation number**. When a process exits, the exit event carries the generation it was launched with. The supervisor compares this against the currently-live generation:
+
+- **Match**: the live process just died — trigger restart.
+- **Mismatch**: a stale process from a previous failed attempt was reaped — ignore.
+
+This prevents restart storms where multiple rapid failures queue up cascading restart attempts.
+
+### Health gate integration
+
+When transparent mitmproxy is enabled:
+
+- `/healthz` returns **503** until the full mitm stack is ready (process started, listener up, iptables installed, CA exported).
+- On crash, the health gate is set back to not-ready (503) immediately.
+- After a successful restart and listener readiness, the health gate is restored.
+
+Kubernetes readiness probes that hit `/healthz` will stop routing traffic to the sandbox during the restart window.
+
+### Graceful shutdown
+
+When the egress sidecar receives `SIGTERM` or `SIGINT`:
+
+1. The supervisor watcher goroutine exits (context cancelled).
+2. `iptables` transparent redirect rules are removed.
+3. `mitmdump` receives `SIGTERM`; if it does not exit within 5 seconds, `SIGKILL` is sent.
+
+Any `OnExit` callbacks still blocked on the restart channel are unblocked via a dedicated shutdown channel, preventing goroutine leaks.
+
+### Observability
+
+All supervisor activity is logged with the `[mitmproxy]` prefix:
+
+| Log pattern | Meaning |
+|-------------|---------|
+| `mitmdump exited (gen=N): <error>; restarting...` | Live process crashed; restart initiated |
+| `ignoring stale exit event (gen=N, current=M)` | Old generation reaped; no action needed |
+| `restart attempt N failed; retrying in Xs` | Launch or listener wait failed; backing off |
+| `mitmdump restarted (pid P, gen N, attempt M)` | Successful restart |
+| `dropping exit event during shutdown` | Exit event discarded because egress is shutting down |
@@ -0,0 +1,162 @@
+# opensandbox-supervisor
+
+A lightweight process supervisor that wraps a single worker with restart backoff, lifecycle hooks, a crashloop circuit breaker, and a structured event log. Designed to run as a container `ENTRYPOINT` or as a child of another process; it does not assume PID 1 and performs no zombie reaping.
+
+## Usage
+
+```
+opensandbox-supervisor [flags] -- <worker-cmd> [worker-args...]
+```
+
+Everything after `--` is the worker command. The supervisor starts the worker, monitors it, and restarts it on unexpected exits.
+
+### Example (egress sidecar)
+
+```dockerfile
+ENTRYPOINT ["/opt/opensandbox-egress/supervisor", \
+            "--pre-start=/opt/opensandbox-egress/cleanup.sh", \
+            "--name=egress", \
+            "--grace-period=20s", \
+            "--", \
+            "/opt/opensandbox-egress/egress"]
+```
+
+## Flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--pre-start` | _(none)_ | Executable to run before each worker launch (repeatable). No shell expansion; wrap in a script if needed. |
+| `--post-exit` | _(none)_ | Executable to run after each worker exit (repeatable). Receives `WORKER_*` env vars. Failures are logged, not fatal. |
+| `--event-log` | stderr | Path to JSONL event log file. Supports rotation via lumberjack. |
+| `--backoff-min` | `1s` | Minimum restart backoff. |
+| `--backoff-max` | `30s` | Maximum restart backoff (exponential growth capped here). |
+| `--backoff-jitter` | `0.1` | Jitter fraction (±10%). Set to `0` to disable. |
+| `--stable-after` | `60s` | Worker uptime after which backoff resets to minimum. |
+| `--burst-window` | `5m` | Sliding window for crashloop detection. |
+| `--burst-max` | `10` | Maximum launches allowed within `burst-window` before the breaker trips. |
+| `--on-burst-exit` | `true` | `true`: supervisor exits non-zero when burst budget trips (lets kubelet react). `false`: keep retrying indefinitely. |
+| `--grace-period` | `10s` | Time between SIGTERM and SIGKILL when shutting the worker down. |
+| `--pre-start-timeout` | `30s` | Timeout for each pre-start hook execution. |
+| `--post-exit-timeout` | `30s` | Timeout for each post-exit hook execution. |
+| `--name` | _(basename of worker cmd)_ | Worker name shown in logs and events. |
+| `--log-level` | `info` | Supervisor diagnostic log level (`debug`\|`info`\|`warn`\|`error`). |
+
+## Restart Behavior
+
+### Exponential Backoff
+
+When the worker exits unexpectedly, the supervisor sleeps before restarting:
+
+```
+1s → 2s → 4s → 8s → 16s → 30s → 30s → ...
+```
+
+Each delay is perturbed by ±`backoff-jitter` (default ±10%) to avoid thundering herds. After the worker has been alive at least `stable-after` (default 60 s), the backoff resets to `backoff-min`.
+
+### Crashloop Circuit Breaker
+
+A sliding-window counter tracks launches. If more than `burst-max` (default 10) launches occur within `burst-window` (default 5 min), the supervisor either:
+
+- **Exits non-zero** (`--on-burst-exit=true`, default) — surfacing the crashloop via Kubernetes pod status instead of silently retrying.
+- **Continues retrying** (`--on-burst-exit=false`) — for environments without an outer restart supervisor.
+
+## Lifecycle Hooks
+
+### Pre-start hooks
+
+Run **before each worker launch**. A non-zero exit aborts that launch attempt and counts toward the crashloop budget. Use for cleanup tasks like reaping orphaned child processes from a previous crash.
+
+### Post-exit hooks
+
+Run **after the worker has been reaped**. Failures are logged but do not block the restart loop. Post-exit hooks run to completion even during shutdown (bounded by `--post-exit-timeout`) so cleanup paths are not aborted.
+
+Post-exit hooks receive these environment variables:
+
+| Variable | Description |
+|----------|-------------|
+| `WORKER_EXIT_CODE` | Worker's exit code (`-1` if not available) |
+| `WORKER_SIGNAL` | Signal name if worker was signaled (e.g. `terminated`, `killed`) |
+| `WORKER_DURATION_MS` | Wall-clock worker runtime in milliseconds |
+| `WORKER_PID` | Worker's PID |
+| `WORKER_ATTEMPT` | Launch attempt number (1-based) |
+
+## Graceful Shutdown
+
+On context cancellation (typically from `SIGTERM` or `SIGINT`):
+
+1. Supervisor sends `SIGTERM` to the worker.
+2. Waits up to `--grace-period` for the worker to exit on its own.
+3. Sends `SIGKILL` if the worker does not exit in time.
+
+### Signal Handling
+
+- The supervisor does **not** install `signal.Notify` itself; the caller (e.g. `cmd/supervisor/main.go`) translates OS signals into context cancellation.
+- `SIGINT` and `SIGTERM` both result in `SIGTERM` to the worker.
+- Other signals (`SIGHUP`, `SIGUSR1`, etc.) are **not forwarded**. Add forwarding in the caller if the worker needs them.
+
+### Process Group Isolation
+
+The worker is started with `Setpgid=true` on Unix so signals delivered to the supervisor's process group do not reach the worker by side channel. The supervisor signals the worker explicitly via its PID.
+
+## Structured Event Log
+
+One JSONL record per lifecycle event, written to stderr by default or to the file specified by `--event-log` (with automatic rotation).
+
+### Event Kinds
+
+| Event | When | Key Fields |
+|-------|------|------------|
+| `start` | Worker process launched | `pid`, `gen`, `attempt` |
+| `exit` | Worker exited | `pid`, `gen`, `attempt`, `exit_code`, `signal`, `duration_ms`, `reason` |
+| `prestart` | Pre-start hook ran | `hook`, `exit_code`, `duration_ms` |
+| `postexit` | Post-exit hook ran | `hook`, `exit_code`, `duration_ms` |
+| `backoff` | Sleeping before next restart | `sleep_ms`, `next_attempt` |
+| `stable` | Worker uptime exceeded `stable-after`; backoff reset | `pid`, `gen`, `duration_ms`, `reset_backoff` |
+| `burst_exit` | Crashloop budget exceeded | `attempts`, `window` |
+| `shutdown` | Supervisor shutting down | `reason` |
+
+### Example Events
+
+```jsonl
+{"ts":"2026-01-15T10:30:00Z","name":"egress","event":"start","pid":42,"gen":1,"attempt":1}
+{"ts":"2026-01-15T10:30:00.15Z","name":"egress","event":"exit","pid":42,"gen":1,"attempt":1,"exit_code":1,"duration_ms":150,"reason":"crashed"}
+{"ts":"2026-01-15T10:30:00.15Z","name":"egress","event":"backoff","sleep_ms":1000,"next_attempt":2}
+{"ts":"2026-01-15T10:30:01.15Z","name":"egress","event":"prestart","hook":"cleanup.sh","exit_code":0,"duration_ms":50}
+{"ts":"2026-01-15T10:30:01.2Z","name":"egress","event":"start","pid":43,"gen":2,"attempt":2}
+```
+
+### Exit Reasons
+
+| Reason | Meaning |
+|--------|---------|
+| `exited` | Worker exited with code 0 |
+| `crashed` | Worker exited with non-zero code |
+| `signaled` | Worker killed by signal |
+| `shutdown` | Supervisor-initiated stop (context cancelled) |
+| `launch_failed` | Worker binary could not be started |
+| `no_processstate` | Unexpected: no process state available |
+
+## Library Usage
+
+The `internal/supervisor` package can be used programmatically:
+
+```go
+import "github.com/alibaba/opensandbox/internal/supervisor"
+
+spec := supervisor.Spec{
+    Name:        "my-worker",
+    Cmd:         "/usr/local/bin/worker",
+    Args:        []string{"--config", "/etc/worker.toml"},
+    PreStart:    []supervisor.Hook{{Argv: []string{"/usr/local/bin/cleanup.sh"}}},
+    BackoffMin:  time.Second,
+    BackoffMax:  30 * time.Second,
+    GracePeriod: 15 * time.Second,
+}
+
+ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+defer cancel()
+
+err := supervisor.Run(ctx, spec)
+```
+
+`Run` blocks until context cancellation or `ErrBurstExceeded`. Zero-valued fields receive sensible defaults (see Flags table above for values).