diff --git a/components/egress/README.md b/components/egress/README.md index 3e837b139..a190c7611 100644 --- a/components/egress/README.md +++ b/components/egress/README.md @@ -55,18 +55,46 @@ Optional advanced features: - Nameserver bypass: `OPENSANDBOX_EGRESS_NAMESERVER_EXEMPT` - Denied hostname webhook: `OPENSANDBOX_EGRESS_DENY_WEBHOOK`, `OPENSANDBOX_EGRESS_SANDBOX_ID` - DoH/DoT controls: `OPENSANDBOX_EGRESS_BLOCK_DOH_443`, `OPENSANDBOX_EGRESS_DOH_BLOCKLIST` +- Custom DNS upstream: `OPENSANDBOX_EGRESS_DNS_UPSTREAM` (comma-separated IPs, optional `:port`), `OPENSANDBOX_EGRESS_DNS_UPSTREAM_TIMEOUT` (default `5` seconds) +- DNS upstream health probe: `OPENSANDBOX_EGRESS_DNS_UPSTREAM_PROBE` (enable), `OPENSANDBOX_EGRESS_DNS_UPSTREAM_PROBE_INTERVAL_SEC` + +### Always-Rules Files + +Static rule files under `/var/egress/rules/` are loaded at startup and take priority over dynamic API rules: + +| File | Purpose | +|------|---------| +| `/var/egress/rules/deny.always` | Domains always denied, overrides user and allow rules | +| `/var/egress/rules/allow.always` | Domains always allowed, overrides user rules | +| `/var/egress/rules/log_skip.always` | Domain patterns whose DNS blocks are not logged (noise reduction) | + +Format: one domain per line (supports wildcards like `*.example.com`). Lines starting with `#` are comments. Missing files are silently ignored. + +Rule precedence: `deny.always` > `allow.always` > user policy (API/env). + +Always-rules are hot-reloaded: the sidecar polls the files once per minute and applies changes without restart. ### Runtime HTTP API -- `GET /policy`: get current policy -- `POST /policy`: replace policy (`{}`, `null`, empty body => reset to deny-all) -- `PATCH /policy`: merge/append rules (body is JSON array of egress rules) +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/policy` | Get current policy and enforcement mode | +| `POST` | `/policy` | Replace policy (`{}`, `null`, empty body => reset to deny-all) | +| `PUT` | `/policy` | Alias for `POST` | +| `PATCH` | `/policy` | Merge/append rules (body is JSON array of egress rules) | +| `DELETE` | `/policy` | Remove specific targets (body is JSON string array, e.g. `["*.example.com"]`) | +| `GET` | `/healthz` | Health check; returns `200 ok` or `503 mitmproxy not ready` (when transparent MITM is enabled but not yet initialized) | Quick example: ```bash +# Replace policy curl -XPOST http://127.0.0.1:18080/policy \ -d '{"defaultAction":"deny","egress":[{"action":"allow","target":"*.example.com"}]}' + +# Remove specific targets +curl -XDELETE http://127.0.0.1:18080/policy \ + -d '["*.example.com"]' ``` ### Experimental: Transparent MITM (mitmproxy) @@ -128,7 +156,7 @@ curl -I https://github.com ## Development -- **Language**: Go 1.24+ +- **Language**: Go 1.25+ - **Key Packages**: - `pkg/dnsproxy`: DNS server and policy matching logic. - `pkg/iptables`: `iptables` rule management. @@ -152,6 +180,21 @@ An end-to-end benchmark compares **dns** (pass-through, no nft write) and **dns+ More details in [docs/benchmark.md](docs/benchmark.md). +## Process Supervisor + +The egress container runs under [`opensandbox-supervisor`](../../components/internal/supervisor/README.md), a lightweight process wrapper that restarts the egress worker on crash with exponential backoff, a crashloop circuit breaker, and structured JSONL event logging. + +``` +ENTRYPOINT: supervisor --pre-start=cleanup.sh --name=egress --grace-period=20s -- /opt/opensandbox-egress/egress +``` + +Egress-specific configuration: + +- **`--grace-period=20s`**: Egress needs extra time to drain DNS connections and tear down iptables/nft rules on shutdown (default is 10 s). +- **Pre-start hook** (`cleanup.sh`): Reaps orphaned `mitmdump` processes from a previous crash so the new egress can bind the MITM listen port. Intentionally does NOT tear down iptables/nft rules — keeping enforcement active during the backoff window protects the workload. + +For full supervisor documentation (all flags, backoff behavior, crashloop breaker, event log schema, library API), see the [supervisor README](../../components/internal/supervisor/README.md). + ## Troubleshooting - **"iptables setup failed"**: ensure sidecar has `--cap-add=NET_ADMIN`. diff --git a/components/egress/docs/mitmproxy-transparent.md b/components/egress/docs/mitmproxy-transparent.md index 3df101527..d9b34600d 100644 --- a/components/egress/docs/mitmproxy-transparent.md +++ b/components/egress/docs/mitmproxy-transparent.md @@ -47,6 +47,7 @@ export OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS='.*\.log\.aliyuncs\.com;.*\.exa | `OPENSANDBOX_EGRESS_MITMPROXY_IGNORE_HOSTS` | No | Host/IP regex list for TLS pass-through (`;` separated) | Empty | | `OPENSANDBOX_EGRESS_MITMPROXY_CONFDIR` | No | mitm config and CA directory (passed as `--set confdir=`, also used as `HOME`) | Default directory under `/var/lib/mitmproxy` | | `OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_TRUST_DIR` | No | Trust directory for upstream TLS verification (OpenSSL style) | `/etc/ssl/certs` | +| `OPENSANDBOX_EGRESS_MITMPROXY_SSL_INSECURE` | No | Skip upstream TLS certificate verification (`1/true/on`). Needed when clients connect by IP (no SNI → hostname mismatch). | Disabled | Notes: @@ -112,3 +113,59 @@ Limits: - Currently IPv4 `iptables` only; IPv6 is not automatically handled. - Non-Linux environments (for example local macOS runtime) are not supported for transparent mode. - Full HTTPS decryption introduces CPU/memory and certificate trust overhead; benchmark before production rollout. + +## Process Supervisor (Crash Recovery) + +The egress sidecar includes a built-in supervisor that monitors the `mitmdump` child process and automatically restarts it on unexpected exits. + +### Restart behavior + +When `mitmdump` exits unexpectedly, the supervisor restarts it with **exponential backoff**: 1 s, 2 s, 4 s, ..., capped at **30 s**. Retries continue indefinitely until the process starts successfully or the egress sidecar itself shuts down. + +A successful restart requires two conditions: + +1. `mitmdump` process starts without error. +2. The listen port (`127.0.0.1:`) accepts TCP connections within 15 seconds. + +If the listener does not come up in time, the half-started process is gracefully terminated (SIGTERM → wait → SIGKILL) before the next attempt, so the port is released cleanly. + +### Generation tagging + +Each `mitmdump` launch is assigned a monotonically increasing **generation number**. When a process exits, the exit event carries the generation it was launched with. The supervisor compares this against the currently-live generation: + +- **Match**: the live process just died — trigger restart. +- **Mismatch**: a stale process from a previous failed attempt was reaped — ignore. + +This prevents restart storms where multiple rapid failures queue up cascading restart attempts. + +### Health gate integration + +When transparent mitmproxy is enabled: + +- `/healthz` returns **503** until the full mitm stack is ready (process started, listener up, iptables installed, CA exported). +- On crash, the health gate is set back to not-ready (503) immediately. +- After a successful restart and listener readiness, the health gate is restored. + +Kubernetes readiness probes that hit `/healthz` will stop routing traffic to the sandbox during the restart window. + +### Graceful shutdown + +When the egress sidecar receives `SIGTERM` or `SIGINT`: + +1. The supervisor watcher goroutine exits (context cancelled). +2. `iptables` transparent redirect rules are removed. +3. `mitmdump` receives `SIGTERM`; if it does not exit within 5 seconds, `SIGKILL` is sent. + +Any `OnExit` callbacks still blocked on the restart channel are unblocked via a dedicated shutdown channel, preventing goroutine leaks. + +### Observability + +All supervisor activity is logged with the `[mitmproxy]` prefix: + +| Log pattern | Meaning | +|-------------|---------| +| `mitmdump exited (gen=N): ; restarting...` | Live process crashed; restart initiated | +| `ignoring stale exit event (gen=N, current=M)` | Old generation reaped; no action needed | +| `restart attempt N failed; retrying in Xs` | Launch or listener wait failed; backing off | +| `mitmdump restarted (pid P, gen N, attempt M)` | Successful restart | +| `dropping exit event during shutdown` | Exit event discarded because egress is shutting down | diff --git a/components/internal/supervisor/README.md b/components/internal/supervisor/README.md new file mode 100644 index 000000000..694e37c53 --- /dev/null +++ b/components/internal/supervisor/README.md @@ -0,0 +1,162 @@ +# opensandbox-supervisor + +A lightweight process supervisor that wraps a single worker with restart backoff, lifecycle hooks, a crashloop circuit breaker, and a structured event log. Designed to run as a container `ENTRYPOINT` or as a child of another process; it does not assume PID 1 and performs no zombie reaping. + +## Usage + +``` +opensandbox-supervisor [flags] -- [worker-args...] +``` + +Everything after `--` is the worker command. The supervisor starts the worker, monitors it, and restarts it on unexpected exits. + +### Example (egress sidecar) + +```dockerfile +ENTRYPOINT ["/opt/opensandbox-egress/supervisor", \ + "--pre-start=/opt/opensandbox-egress/cleanup.sh", \ + "--name=egress", \ + "--grace-period=20s", \ + "--", \ + "/opt/opensandbox-egress/egress"] +``` + +## Flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--pre-start` | _(none)_ | Executable to run before each worker launch (repeatable). No shell expansion; wrap in a script if needed. | +| `--post-exit` | _(none)_ | Executable to run after each worker exit (repeatable). Receives `WORKER_*` env vars. Failures are logged, not fatal. | +| `--event-log` | stderr | Path to JSONL event log file. Supports rotation via lumberjack. | +| `--backoff-min` | `1s` | Minimum restart backoff. | +| `--backoff-max` | `30s` | Maximum restart backoff (exponential growth capped here). | +| `--backoff-jitter` | `0.1` | Jitter fraction (±10%). Set to `0` to disable. | +| `--stable-after` | `60s` | Worker uptime after which backoff resets to minimum. | +| `--burst-window` | `5m` | Sliding window for crashloop detection. | +| `--burst-max` | `10` | Maximum launches allowed within `burst-window` before the breaker trips. | +| `--on-burst-exit` | `true` | `true`: supervisor exits non-zero when burst budget trips (lets kubelet react). `false`: keep retrying indefinitely. | +| `--grace-period` | `10s` | Time between SIGTERM and SIGKILL when shutting the worker down. | +| `--pre-start-timeout` | `30s` | Timeout for each pre-start hook execution. | +| `--post-exit-timeout` | `30s` | Timeout for each post-exit hook execution. | +| `--name` | _(basename of worker cmd)_ | Worker name shown in logs and events. | +| `--log-level` | `info` | Supervisor diagnostic log level (`debug`\|`info`\|`warn`\|`error`). | + +## Restart Behavior + +### Exponential Backoff + +When the worker exits unexpectedly, the supervisor sleeps before restarting: + +``` +1s → 2s → 4s → 8s → 16s → 30s → 30s → ... +``` + +Each delay is perturbed by ±`backoff-jitter` (default ±10%) to avoid thundering herds. After the worker has been alive at least `stable-after` (default 60 s), the backoff resets to `backoff-min`. + +### Crashloop Circuit Breaker + +A sliding-window counter tracks launches. If more than `burst-max` (default 10) launches occur within `burst-window` (default 5 min), the supervisor either: + +- **Exits non-zero** (`--on-burst-exit=true`, default) — surfacing the crashloop via Kubernetes pod status instead of silently retrying. +- **Continues retrying** (`--on-burst-exit=false`) — for environments without an outer restart supervisor. + +## Lifecycle Hooks + +### Pre-start hooks + +Run **before each worker launch**. A non-zero exit aborts that launch attempt and counts toward the crashloop budget. Use for cleanup tasks like reaping orphaned child processes from a previous crash. + +### Post-exit hooks + +Run **after the worker has been reaped**. Failures are logged but do not block the restart loop. Post-exit hooks run to completion even during shutdown (bounded by `--post-exit-timeout`) so cleanup paths are not aborted. + +Post-exit hooks receive these environment variables: + +| Variable | Description | +|----------|-------------| +| `WORKER_EXIT_CODE` | Worker's exit code (`-1` if not available) | +| `WORKER_SIGNAL` | Signal name if worker was signaled (e.g. `terminated`, `killed`) | +| `WORKER_DURATION_MS` | Wall-clock worker runtime in milliseconds | +| `WORKER_PID` | Worker's PID | +| `WORKER_ATTEMPT` | Launch attempt number (1-based) | + +## Graceful Shutdown + +On context cancellation (typically from `SIGTERM` or `SIGINT`): + +1. Supervisor sends `SIGTERM` to the worker. +2. Waits up to `--grace-period` for the worker to exit on its own. +3. Sends `SIGKILL` if the worker does not exit in time. + +### Signal Handling + +- The supervisor does **not** install `signal.Notify` itself; the caller (e.g. `cmd/supervisor/main.go`) translates OS signals into context cancellation. +- `SIGINT` and `SIGTERM` both result in `SIGTERM` to the worker. +- Other signals (`SIGHUP`, `SIGUSR1`, etc.) are **not forwarded**. Add forwarding in the caller if the worker needs them. + +### Process Group Isolation + +The worker is started with `Setpgid=true` on Unix so signals delivered to the supervisor's process group do not reach the worker by side channel. The supervisor signals the worker explicitly via its PID. + +## Structured Event Log + +One JSONL record per lifecycle event, written to stderr by default or to the file specified by `--event-log` (with automatic rotation). + +### Event Kinds + +| Event | When | Key Fields | +|-------|------|------------| +| `start` | Worker process launched | `pid`, `gen`, `attempt` | +| `exit` | Worker exited | `pid`, `gen`, `attempt`, `exit_code`, `signal`, `duration_ms`, `reason` | +| `prestart` | Pre-start hook ran | `hook`, `exit_code`, `duration_ms` | +| `postexit` | Post-exit hook ran | `hook`, `exit_code`, `duration_ms` | +| `backoff` | Sleeping before next restart | `sleep_ms`, `next_attempt` | +| `stable` | Worker uptime exceeded `stable-after`; backoff reset | `pid`, `gen`, `duration_ms`, `reset_backoff` | +| `burst_exit` | Crashloop budget exceeded | `attempts`, `window` | +| `shutdown` | Supervisor shutting down | `reason` | + +### Example Events + +```jsonl +{"ts":"2026-01-15T10:30:00Z","name":"egress","event":"start","pid":42,"gen":1,"attempt":1} +{"ts":"2026-01-15T10:30:00.15Z","name":"egress","event":"exit","pid":42,"gen":1,"attempt":1,"exit_code":1,"duration_ms":150,"reason":"crashed"} +{"ts":"2026-01-15T10:30:00.15Z","name":"egress","event":"backoff","sleep_ms":1000,"next_attempt":2} +{"ts":"2026-01-15T10:30:01.15Z","name":"egress","event":"prestart","hook":"cleanup.sh","exit_code":0,"duration_ms":50} +{"ts":"2026-01-15T10:30:01.2Z","name":"egress","event":"start","pid":43,"gen":2,"attempt":2} +``` + +### Exit Reasons + +| Reason | Meaning | +|--------|---------| +| `exited` | Worker exited with code 0 | +| `crashed` | Worker exited with non-zero code | +| `signaled` | Worker killed by signal | +| `shutdown` | Supervisor-initiated stop (context cancelled) | +| `launch_failed` | Worker binary could not be started | +| `no_processstate` | Unexpected: no process state available | + +## Library Usage + +The `internal/supervisor` package can be used programmatically: + +```go +import "github.com/alibaba/opensandbox/internal/supervisor" + +spec := supervisor.Spec{ + Name: "my-worker", + Cmd: "/usr/local/bin/worker", + Args: []string{"--config", "/etc/worker.toml"}, + PreStart: []supervisor.Hook{{Argv: []string{"/usr/local/bin/cleanup.sh"}}}, + BackoffMin: time.Second, + BackoffMax: 30 * time.Second, + GracePeriod: 15 * time.Second, +} + +ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) +defer cancel() + +err := supervisor.Run(ctx, spec) +``` + +`Run` blocks until context cancellation or `ErrBurstExceeded`. Zero-valued fields receive sensible defaults (see Flags table above for values).