Update runner doctor catalog for ARC/DinD DNS, gVisor MCP 403s, and sbx home exposure#6409
Conversation
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (2 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
Updates Runner Doctor diagnostics for ARC/DinD, gVisor, and sbx runtime failures.
Changes:
- Refines B5/B6 diagnostic guidance.
- Adds D8/D9 runtime failure modes and quick lookups.
- Extends mirror-alignment tests.
Show a summary per file
| File | Description |
|---|---|
scripts/ci/self-hosted-runner-doctor-workflow.test.ts |
Checks new catalog entries across mirrors. |
.github/workflows/shared/self-hosted-failure-modes.md |
Updates the shared failure catalog. |
.github/workflows/self-hosted-runner-doctor.md |
Adds workflow diagnostic guidance. |
.github/agents/self-hosted-runner-doctor.md |
Updates the portable doctor mirror. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 4
- Review effort level: Medium
| | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 | | ||
| | B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 | | ||
| | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la <firewall-logs-dir>` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 | | ||
| | B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Added `detectDnsResolutionFailure()` in `container-startup-diagnostics.ts`. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. `awmg-cli-proxy`) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure `dockerd --dns <cluster-dns-ip>`. | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix. For ARC/DinD: `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | |
| | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 | | ||
| | B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 | | ||
| | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la <firewall-logs-dir>` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 | | ||
| | B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Added `detectDnsResolutionFailure()` in `container-startup-diagnostics.ts`. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. `awmg-cli-proxy`) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure `dockerd --dns <cluster-dns-ip>`. | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix. For ARC/DinD: `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | |
| | D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 | | ||
| | D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` was added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11); it requires the `sbx` CLI to be installed on the runner. AWF's `isSbxAvailable()` preflight fails if `sbx` is absent and the run aborts before the agent starts. | Install the `sbx` CLI on the runner before invoking AWF; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard `--allow-domains` whitelisting applies. | `command -v sbx && sbx version` — absence of `sbx` confirms; `docker info` — confirm Docker daemon is accessible from the runner | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 | | ||
| | D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 | | ||
| | D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`; `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | |
| | D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 | | ||
| | D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11) requires the `sbx` CLI on the runner; `isSbxAvailable()` preflight fails if `sbx` is absent | Install the `sbx` CLI on the runner; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard domain whitelisting applies. | `command -v sbx && sbx version` — absence confirms; `docker info` | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 | | ||
| | D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 | | ||
| | D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`; `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | |
|
@copilot address review feedback |
Addressed in a10e221. The runner doctor catalog now splits the ARC/DinD DNS-isolation case from the startup-ordering deadlock, and the gVisor guidance now reflects raw |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
✅ Smoke Claude passed |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
✅ Contribution Check completed successfully! PR #6409 follows the applicable contribution guidelines; no review comment needed. |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
Smoke Test: Copilot PAT Auth
Overall: FAIL — workflow template variables were not substituted before agent invocation. Auth mode: PAT (COPILOT_GITHUB_TOKEN) Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🔬 Smoke Test Results
Overall: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Copilot BYOK (Direct) Mode✅ GitHub MCP connectivity - Environment ready Overall: PASS — Running in direct BYOK mode via api-proxy sidecar. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Claude Engine Validation
Overall Result: PASS ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Smoke test: Copilot network isolation egress EGRESS_RESULT allow=pass deny=pass
Overall: PASS — Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"
- "example.com"See Network Configuration for more information.
|
github-actions
Bot
added
the
smoke-copilot-network-isolation
Smoke Test Results
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
|
Protect Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"
- "registry.npmjs.org"See Network Configuration for more information.
|
Smoke Test: Services Connectivity
Overall: FAIL
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: API Proxy OpenTelemetry Tracing
All 5 scenarios pass. OTEL tracing integration is working correctly. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Smoke test: Copilot BYOK (Direct) Mode — Azure OpenAI (Foundry)
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Chroot Version Comparison Results
ALL_TESTS_PASSED: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🧪 Smoke Test: Docker Sbx — PASS
Overall: PASS cc
|
This updates the Self-Hosted Runner Doctor knowledge base with four lessons from the 2026-07-10 to 2026-07-19 scan: two existing failure-mode refinements and two new runtime-specific modes. It keeps the shared catalog, workflow playbook, and portable agent aligned so the same diagnoses and quick-lookups are available in every entry point.
Catalog updates: B5/B6
detectDnsResolutionFailure()follow-up, actionable remediation, and thenslookup awmg-cli-proxyprobe.#6328warning semantics.New runtime failure modes: D8/D9
403 ERR_ACCESS_DENIEDwhen the iptables bypass is unavailable.$HOMEinto the microVM and exposing credential-bearing paths before the home-whitelist/scrub fix.Quick-lookup expansion
403 ERR_ACCESS_DENIEDon MCP tool calls under--container-runtime gvisorMirror consistency
shared/self-hosted-failure-modes.mdself-hosted-runner-doctor.md.github/agents/self-hosted-runner-doctor.mdExample of the new lookup coverage: