fix(egress): purge stale mitmproxy CA on startup#1371
Conversation
mitmproxy's CA lives in the egress container's ephemeral confdir (/var/lib/mitmproxy/.mitmproxy/), so every egress container restart regenerates a fresh CA. The previous generation's public cert may still be on the pod-scoped shared volume; without a purge, an agent container starting or restarting alongside egress passes its bootstrap readiness check on the stale file and installs a CA that no longer matches what mitmproxy will sign with. Erase /opt/opensandbox/mitmproxy-ca-cert.pem as early as possible in egress main() (immediately after logger initialization, before any telemetry / policy / iptables / mitm setup) so the agent's wait-loop blocks until this generation's SyncRootCA has completed. Scope: - Closes the race for pod-container co-restart (both containers restarting together with an emptyDir that survived). - Does NOT address the egress-alone container restart case where the agent's bootstrap has long finished and never re-reads the file; that still requires an agent-side reload path. Refs opensandbox-group#1370
github-actions
Bot
added
component/egress
size/M
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b49e52342
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 73526d7485
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
73526d7 to
2b49e52
Compare
2 participants
Summary
Fixes part of #1370.
mitmproxy's CA lives in the egress container's ephemeral confdir (
/var/lib/mitmproxy/.mitmproxy/). Every time the egress container restarts, a fresh CA is generated with a new fingerprint. The previous generation's public cert, however, may still be on the pod-scoped shared volume at/opt/opensandbox/mitmproxy-ca-cert.pem(anemptyDirunder Kubernetes / a docker named volume, both survive container restarts).Without any purge, an agent container starting or restarting alongside egress passes its bootstrap readiness check
[ -f "$MITM_CA" ] && [ -s "$MITM_CA" ]on the stale file before egress has had a chance to overwrite it, and installs a CA that no longer matches what mitmproxy will sign with. Subsequent HTTPS from agent then fails withunable to get local issuer certificate.What this change does
Erase
/opt/opensandbox/mitmproxy-ca-cert.pemas early as possible inegressmain()— immediately after logger initialization, before any telemetry / policy / dnsproxy / iptables / mitmproxy setup. This forces the agent's bootstrap wait-loop to block until this generation'sSyncRootCAwrites the current CA back to the shared volume.Log level is
warn: on a healthy first pod startup the file does not exist and nothing is logged; aremoved stale exported CAwarning is real signal that egress just restarted, which is useful when diagnosing HTTPS-from-agent problems.Scope
Fixes: pod-container co-restart (both containers restarting together within the same pod, with
emptyDirsurviving). This is scenario C in #1370.Does NOT fix: egress-alone container restart (scenario B in #1370). In that case, the agent's bootstrap has long finished and never re-reads the file, so no amount of egress-side purging helps. That case still requires an agent-side reload path (watcher in
execd), which is intentionally out of scope for this PR.Not affected: whole-pod recreation (scenario A) already goes through a fresh shared volume and works correctly.
In-place egress process restart (uncommon)
If the egress process is somehow restarted while its container stays up (uncommon under normal K8s / Docker lifecycles, since egress is PID 1), mitmproxy will reuse the CA already in confdir. The purge deletes the shared-volume copy,
SyncRootCAwrites back the same bytes moments later, and any agent bootstrap running concurrently only blocks for the duration of that gap. Functionally a no-op; thewarnline correctly reports "previous egress generation left it behind".Tests
Added unit tests for the pure helper
purgeStaleExportedCAFromcovering:merged-ca-certificates.pem) are left untouched.Local verification:
All pass.
Files changed
components/egress/main.go— callmitmproxy.PurgeStaleExportedCA()afterwithLogger, before telemetry init.components/egress/pkg/mitmproxy/cacert_export.go— new exportedPurgeStaleExportedCA()and internal testablepurgeStaleExportedCAFrom.components/egress/pkg/mitmproxy/cacert_export_test.go— three focused unit tests.