Skip to content

🩺 Runner Doctor UpdateRunner Doctor: 4 new/updated failure-mode entries (2026-07-10 to 2026-07-19 scan) #6405

Description

@github-actions

Summary


Proposed knowledge-base changes

For .github/workflows/shared/self-hosted-failure-modes.md

Update B5 — add citations #6326, #6328 and improved diagnosis note

Add citation #6326, #6328. Extend the "Fix / flag" cell:

PR #6328 (merged 2026-07-17): Added detectDnsResolutionFailure() in container-startup-diagnostics.ts. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. awmg-cli-proxy) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for EAI_AGAIN/ENOTFOUND, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure dockerd --dns <cluster-dns-ip>.

Extend the "Probe" cell:

For ARC/DinD: docker run --rm alpine nslookup awmg-cli-proxyNXDOMAIN/SERVFAIL confirms the Kubernetes DNS is not reachable from DinD containers.

Updated citations: #5543, #5542, #6326, #6328


Update B6 — add citation #6328 and noise-reduction note

Add citation #6328. Extend the "Fix / flag" cell:

PR #6328 (merged 2026-07-17): Benign permission errors (Operation not permitted, EPERM, EACCES) from the artifact repair container are now logged at debug with an "expected on restricted runners" note. Only genuinely unexpected failures still emit [WARN]. Post-#6328: a [WARN] Rootless artifact permission repair failed message indicates a genuine failure, not a restricted-runner non-issue.

Updated citations: #5545, #5542, #6328


New row D8 — gVisor: MCP gateway 403 (iptables DNAT absent)

Insert after D7:

| D8 | MCP tool calls (safeoutputs, github) return 403 ERR_ACCESS_DENIED under --container-runtime gvisor; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct /dev/tcp connections fail with No route to host | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT RETURN bypass rule (installed by awf-iptables-init) that normally lets the agent reach the MCP gateway (172.30.0.1:8080) directly never fires under gVisor. MCP requests follow HTTP_PROXY into Squid and receive 403 ERR_ACCESS_DENIED because the MCP gateway IP is not in the domain allowlist. | Fixed in AWF (PR #6401): runtimeUsesIptables() returns false for gvisor/sbx; awf-iptables-init is skipped for these runtimes (AWF_SKIP_IPTABLES_INIT=1); the MCP gateway (172.30.0.1) and host.docker.internal are added to NO_PROXY so proxy-aware MCP clients connect directly. Caveat: proxy-unaware tools using raw sockets (e.g. /dev/tcp) still get No route to host under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for 403 on 172.30.0.1; check NO_PROXY env inside the agent container — should include 172.30.0.1 on patched AWF | #6401, #6326 |


New row D9 — sbx: entire $HOME mounted into microVM (credential exposure)

Insert after D8:

| D9 | On --container-runtime sbx, credential files (~/.aws/credentials, ~/.ssh/id_rsa, ~/.docker/config.json, ~/.kube/config, ~/.azure/, ~/.gnupg/, ~/.netrc, ~/.config/gh/hosts.yml, ~/.config/gcloud/, ~/.cargo/credentials.toml, ~/.claude/.credentials.json, ~/.copilot/config.json, ~/.gemini/oauth_creds.json) are visible to the agent inside the sbx microVM | AWF before PR #6336 mounted the entire host $HOME (read-write) into the sbx microVM. Unlike compose mode (empty home volume + /dev/null overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | Fixed in AWF (PR #6336): sbx-manager.ts mounts only whitelisted tool/cache + agent-state subdirs (HOME_TOOL_SUBDIRS + .copilot/.gemini). For whitelisted dirs that contain nested credential files (e.g. ~/.config/gh/hosts.yml, ~/.cargo/credentials.toml, ~/.copilot/config.json), scrubHomeCredentials() moves them aside to .awf-sbx-cred-backup-<pid> before sbx create and restoreHomeCredentials() restores them after teardown. Workaround (older AWF): do not use --container-runtime sbx with real credentials in $HOME. | ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes #6336 | #6336 |


Error-string quick lookup additions

Add two entries to the quick-lookup table:

| 403 ERR_ACCESS_DENIED for MCP tool calls (safeoutputs, github) to (172.30.0.1/redacted) under --container-runtime gvisor; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in NO_PROXY; fixed in github/gh-aw-firewall#6401) | | Credential files (/.aws, /.ssh, /.docker/config.json, /.kube, /.config/gh, /.cargo/credentials.toml, etc.) visible inside sbx microVM under --container-runtime sbx| D9 (entire$HOMEmounted into sbx microVM virtiofs; fixed in github/gh-aw-firewall#6336 with home-whitelist +scrubHomeCredentials()`) |


Proposed doctor changes

For .github/workflows/self-hosted-runner-doctor.md

Apply the same B5, B6 updates and D8, D9 new rows, plus the two new error-string lookup entries.


Proposed portable agent changes

For .github/agents/self-hosted-runner-doctor.md

Apply the same B5 update (citations + DNS-isolation note + probe), B6 update (citations + noise-reduction note), D8 new row, D9 new row, and the two new error-string lookup entries — keeping this file in sync with the two catalog files above.


Source issues and PRs

Source What it teaches
#6326 ARC/DinD gaps: Gap 3 = DinD DNS isolation; Gap 4 = chmod noise on restricted runners
#6328 Fixes Gap 3 (detectDnsResolutionFailure()) and Gap 4 (benign chmod downgrade to debug)
#6336 sbx microVM exposed entire $HOME; fixed with home-whitelist + scrubHomeCredentials()
#6401 gVisor iptables bypass absent; MCP gateway 403; fixed with NO_PROXY + skip awf-iptables-init

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Runner Doctor Updater · 167.4 AIC · ⊞ 16K ·

  • expires on Aug 18, 2026, 7:25 PM UTC

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions