From 9c419e93533c244fe1179235b7bc4f1ee1cdf5af Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 19 Jul 2026 20:04:50 +0000 Subject: [PATCH 1/3] Initial plan From c330725d09d0bb1254e425eeb581ab01f2162bd3 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 19 Jul 2026 20:10:51 +0000 Subject: [PATCH 2/3] docs: update runner doctor failure mode catalog --- .github/agents/self-hosted-runner-doctor.md | 16 +++++++++++++--- .github/workflows/self-hosted-runner-doctor.md | 8 +++++++- .../shared/self-hosted-failure-modes.md | 8 ++++++-- .../self-hosted-runner-doctor-workflow.test.ts | 18 ++++++++++++++++++ 4 files changed, 44 insertions(+), 6 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index f9ff89efc..c071b0529 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -74,8 +74,10 @@ Prefer the narrowest match. Examples: - `unknown shorthand flag: 'd' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`) - `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount) - `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16 (sysroot filter was over-broad and dropped the workspace mount) -- `EAI_AGAIN ` in network-isolation + topology-attach → B5 +- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach → B5 - `EACCES` in upload-artifact after sudo:false → B6 +- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8 (gVisor userspace netstack bypasses the usual iptables DNAT path; patched AWF adds `172.30.0.1` to `NO_PROXY`) +- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9 (older AWF mounted the entire host `$HOME` into sbx; fixed in github/gh-aw-firewall#6336) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -107,6 +109,10 @@ B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compo B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`. +D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`/`sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. + +D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. + C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved. C8 / github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens). @@ -184,8 +190,8 @@ Establish these facts before matching a failure mode: | B2 | All outbound traffic fails behind a mandatory corporate proxy | AWF must chain Squid through the upstream proxy | Set `https_proxy` / `http_proxy` on the host or use `--upstream-proxy`. **Note:** AWF ≤ v0.27.32 (before PR github/gh-aw-firewall#6267) silently blocked proxy environment variables (`NO_PROXY`, `HTTP_PROXY`, `HTTPS_PROXY`, etc.) when passed via `--env` if `enableApiProxy` is active, because `PROXY_ENV_VARS` were in the credential exclusion set. **Fixed in PR github/gh-aw-firewall#6267**: proxy vars are now checked against the `PROXY_ENV_VARS` allowlist and passed through `additionalEnv` even when credential isolation is active. | `env | grep -i proxy`; inspect Squid config for `cache_peer` | #1975, github/gh-aw-firewall#6267 | | B3 | Squid exits with `FATAL: http_port: IPv6 is not available` | Docker IPv6 is disabled but Squid tries to bind an IPv6 listener | Enable Docker/kernel IPv6 (required with current AWF builds), or use a custom AWF build that removes the `[::]` listener | `docker info | grep -i ipv6`; inspect `/proc/sys/net/ipv6/conf/all/disable_ipv6` | #2139 | | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 | -| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 | -| B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la ` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 | +| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Added `detectDnsResolutionFailure()` in `container-startup-diagnostics.ts`. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. `awmg-cli-proxy`) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure `dockerd --dns `. | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix. For ARC/DinD: `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | +| B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn`. **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Benign permission errors (`Operation not permitted`, `EPERM`, `EACCES`) from the artifact repair container are now logged at `debug` with an "expected on restricted runners" note. Only genuinely unexpected failures still emit `[WARN]`. Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure, not a restricted-runner non-issue. | `ls -la ` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, github/gh-aw-firewall#5542, github/gh-aw-firewall#6328 | | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf--chroot-home/` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 | | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 | | B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 | @@ -216,6 +222,8 @@ Establish these facts before matching a failure mode: | D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 | | D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11) requires the `sbx` CLI on the runner; `isSbxAvailable()` preflight fails if `sbx` is absent | Install the `sbx` CLI on the runner; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard domain whitelisting applies. | `command -v sbx && sbx version` — absence confirms; `docker info` | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 | | D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 | +| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`; `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | +| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.copilot/config.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`, `~/.copilot/config.json`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 | ## Error-string quick lookup @@ -249,6 +257,8 @@ Establish these facts before matching a failure mode: | `HTTP 404` / `404 Not Found` downloading `runsc` from `storage.googleapis.com` during gVisor install | D5 | | `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` with `--container-runtime sbx` | D6 | | `SIGSEGV` / `SIGABRT` signal crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail | D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) | +| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in `NO_PROXY`; fixed in github/gh-aw-firewall#6401) | +| Credential files (`~/.aws`, `~/.ssh`, `~/.docker/config.json`, `~/.kube`, `~/.config/gh`, `~/.cargo/credentials.toml`, etc.) visible inside sbx microVM under `--container-runtime sbx` | D9 (entire `$HOME` mounted into sbx microVM virtiofs; fixed in github/gh-aw-firewall#6336 with home-whitelist + `scrubHomeCredentials()`) | ## Known unresolved items diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 8e75c3799..58989301e 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -102,8 +102,10 @@ Prefer the narrowest match. Examples: - `unknown shorthand flag: 'd' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`) - `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount) - `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16 (sysroot filter was over-broad and dropped the workspace mount) -- `EAI_AGAIN ` in network-isolation + topology-attach → B5 +- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach → B5 - `EACCES` in upload-artifact after sudo:false → B6 +- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8 (gVisor userspace netstack bypasses the usual iptables DNAT path; patched AWF adds `172.30.0.1` to `NO_PROXY`) +- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9 (older AWF mounted the entire host `$HOME` into sbx; fixed in github/gh-aw-firewall#6336) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -135,6 +137,10 @@ B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compo B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were too opaque and could mislead triage when the agent already exited non-zero. **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: repair-container stderr is now included in the `[WARN]` message, and chroot-home cleanup noise is reduced by downgrading that log to `debug`. +D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`/`sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. + +D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. + C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved. C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens). diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 048d06f9c..57a20aff0 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -43,8 +43,8 @@ Establish these facts before matching a failure mode: | B2 | All outbound traffic fails behind a mandatory corporate proxy | AWF must chain Squid through the upstream proxy | Set `https_proxy` / `http_proxy` on the host or use `--upstream-proxy`. **Note:** AWF ≤ v0.27.32 (before PR github/gh-aw-firewall#6267) silently blocked proxy environment variables (`NO_PROXY`, `HTTP_PROXY`, `HTTPS_PROXY`, etc.) when passed via `--env` if `enableApiProxy` is active, because `PROXY_ENV_VARS` were in the credential exclusion set. **Fixed in PR github/gh-aw-firewall#6267**: proxy vars are now checked against the `PROXY_ENV_VARS` allowlist and passed through `additionalEnv` even when credential isolation is active. | `env | grep -i proxy`; inspect Squid config for `cache_peer` | #1975, github/gh-aw-firewall#6267 | || B3 | Squid exits with `FATAL: http_port: IPv6 is not available` | Docker IPv6 is disabled but Squid tries to bind an IPv6 listener | Enable Docker/kernel IPv6 (required with current AWF builds), or use a custom AWF build that removes the `[::]` listener | `docker info | grep -i ipv6`; inspect `/proc/sys/net/ipv6/conf/all/disable_ipv6` | #2139 | | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 | -| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 | -| B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la ` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 | +| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Added `detectDnsResolutionFailure()` in `container-startup-diagnostics.ts`. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. `awmg-cli-proxy`) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure `dockerd --dns `. | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix. For ARC/DinD: `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | +| B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn`. **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Benign permission errors (`Operation not permitted`, `EPERM`, `EACCES`) from the artifact repair container are now logged at `debug` with an "expected on restricted runners" note. Only genuinely unexpected failures still emit `[WARN]`. Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure, not a restricted-runner non-issue. | `ls -la ` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, github/gh-aw-firewall#5542, github/gh-aw-firewall#6328 | | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf--chroot-home/` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 | | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 | | B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 | @@ -75,6 +75,8 @@ Establish these facts before matching a failure mode: | D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 | | D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` was added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11); it requires the `sbx` CLI to be installed on the runner. AWF's `isSbxAvailable()` preflight fails if `sbx` is absent and the run aborts before the agent starts. | Install the `sbx` CLI on the runner before invoking AWF; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard `--allow-domains` whitelisting applies. | `command -v sbx && sbx version` — absence of `sbx` confirms; `docker info` — confirm Docker daemon is accessible from the runner | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 | | D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 | +| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`; `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | +| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.copilot/config.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`, `~/.copilot/config.json`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 | ## Error-string quick lookup @@ -108,6 +110,8 @@ Establish these facts before matching a failure mode: | `HTTP 404` / `404 Not Found` downloading `runsc` from `storage.googleapis.com` during gVisor install | D5 | | `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` with `--container-runtime sbx` | D6 | | `SIGSEGV` / `SIGABRT` signal crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail | D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) | +| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in `NO_PROXY`; fixed in github/gh-aw-firewall#6401) | +| Credential files (`~/.aws`, `~/.ssh`, `~/.docker/config.json`, `~/.kube`, `~/.config/gh`, `~/.cargo/credentials.toml`, etc.) visible inside sbx microVM under `--container-runtime sbx` | D9 (entire `$HOME` mounted into sbx microVM virtiofs; fixed in github/gh-aw-firewall#6336 with home-whitelist + `scrubHomeCredentials()`) | ## Known unresolved items diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index f3933e9fb..8217b5b33 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -64,11 +64,29 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('**Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**'); expect(content).toContain('The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it.'); expect(content).toContain('| `[WARN] Rootless artifact permission repair failed ... (exit 1)` with little/no stderr detail, plus cleanup warnings around chroot-home removal and `Command completed with exit code: 1` | B11 |'); + expect(content).toContain('`detectDnsResolutionFailure()`'); + expect(content).toContain('docker run --rm alpine nslookup awmg-cli-proxy'); + expect(content).toContain('github/gh-aw-firewall#6326, github/gh-aw-firewall#6328'); + expect(content).toContain('expected on restricted runners'); + expect(content).toContain('Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure'); + expect(content).toContain('| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` |'); + expect(content).toContain('`runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`'); + expect(content).toContain('AWF_SKIP_IPTABLES_INIT=1'); + expect(content).toContain('| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.copilot/config.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM |'); + expect(content).toContain('`scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-`'); + expect(content).toContain('| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; agent finishes but safe-output validation fails | D8'); + expect(content).toContain('| Credential files (`~/.aws`, `~/.ssh`, `~/.docker/config.json`, `~/.kube`, `~/.config/gh`, `~/.cargo/credentials.toml`, etc.) visible inside sbx microVM under `--container-runtime sbx` | D9'); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(source).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); + expect(source).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8'); + expect(source).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); + expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); + expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.'); expect(portableAgent).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(portableAgent).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); + expect(portableAgent).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8'); + expect(portableAgent).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); }); }); From a10e221133c1c95469aa4bf3cd329f98552bafcf Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 19 Jul 2026 20:22:22 +0000 Subject: [PATCH 3/3] docs: address runner doctor review feedback --- .github/agents/self-hosted-runner-doctor.md | 24 +++++++++++-------- .../workflows/self-hosted-runner-doctor.md | 8 ++++--- .../shared/self-hosted-failure-modes.md | 14 ++++++----- ...self-hosted-runner-doctor-workflow.test.ts | 24 +++++++++++-------- 4 files changed, 41 insertions(+), 29 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index c071b0529..3fe1b654d 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -74,9 +74,9 @@ Prefer the narrowest match. Examples: - `unknown shorthand flag: 'd' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`) - `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount) - `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16 (sysroot filter was over-broad and dropped the workspace mount) -- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach → B5 +- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5 - `EACCES` in upload-artifact after sudo:false → B6 -- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8 (gVisor userspace netstack bypasses the usual iptables DNAT path; patched AWF adds `172.30.0.1` to `NO_PROXY`) +- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; safe-output validation fails even though the agent completed → D8 (gVisor userspace netstack bypasses the usual iptables DNAT path; patched AWF adds `172.30.0.1` to `NO_PROXY`) - credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9 (older AWF mounted the entire host `$HOME` into sbx; fixed in github/gh-aw-firewall#6336) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) @@ -107,9 +107,11 @@ B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible i B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compound `tag@digest` ref timeout is **fixed** in AWF version including github/gh-aw-firewall#6025. `resolvePermFixerImageRef()` now returns tag-only refs, eliminating registry I/O during `--pull never` repair. -B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`. +B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were too opaque and could mislead triage when the agent already exited non-zero. **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: repair-container stderr is now included in the `[WARN]` message, and chroot-home cleanup noise is reduced by downgrading that log to `debug`. -D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`/`sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. +B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix. `detectDnsResolutionFailure()` now augments the startup error with the unresolved host and recommends using the proxy IP or `dockerd --dns `. + +D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. @@ -190,13 +192,14 @@ Establish these facts before matching a failure mode: | B2 | All outbound traffic fails behind a mandatory corporate proxy | AWF must chain Squid through the upstream proxy | Set `https_proxy` / `http_proxy` on the host or use `--upstream-proxy`. **Note:** AWF ≤ v0.27.32 (before PR github/gh-aw-firewall#6267) silently blocked proxy environment variables (`NO_PROXY`, `HTTP_PROXY`, `HTTPS_PROXY`, etc.) when passed via `--env` if `enableApiProxy` is active, because `PROXY_ENV_VARS` were in the credential exclusion set. **Fixed in PR github/gh-aw-firewall#6267**: proxy vars are now checked against the `PROXY_ENV_VARS` allowlist and passed through `additionalEnv` even when credential isolation is active. | `env | grep -i proxy`; inspect Squid config for `cache_peer` | #1975, github/gh-aw-firewall#6267 | | B3 | Squid exits with `FATAL: http_port: IPv6 is not available` | Docker IPv6 is disabled but Squid tries to bind an IPv6 listener | Enable Docker/kernel IPv6 (required with current AWF builds), or use a custom AWF build that removes the `[::]` listener | `docker info | grep -i ipv6`; inspect `/proc/sys/net/ipv6/conf/all/disable_ipv6` | #2139 | | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 | -| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Added `detectDnsResolutionFailure()` in `container-startup-diagnostics.ts`. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. `awmg-cli-proxy`) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure `dockerd --dns `. | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix. For ARC/DinD: `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | +| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs, and the topology peer becomes reachable once attached to `awf-net` | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → transient `EAI_AGAIN`/`ENOTFOUND` → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat transient `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). If ARC/DinD `nslookup awmg-cli-proxy` still fails even after the ordering fix, match B12 instead. | Confirm `topologyAttach` is non-empty; inspect whether the topology peer is attached to `awf-net` before the cli-proxy health check runs. On ARC/DinD, run `docker run --rm alpine nslookup awmg-cli-proxy` first — if it returns `NXDOMAIN`/`SERVFAIL`, prefer B12 over B5. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn`. **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Benign permission errors (`Operation not permitted`, `EPERM`, `EACCES`) from the artifact repair container are now logged at `debug` with an "expected on restricted runners" note. Only genuinely unexpected failures still emit `[WARN]`. Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure, not a restricted-runner non-issue. | `ls -la ` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, github/gh-aw-firewall#5542, github/gh-aw-firewall#6328 | | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf--chroot-home/` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 | | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 | | B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 | | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025 | | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: `; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | +| B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns ` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | ## Category C — GHES / GHEC / `ghe.com` @@ -215,14 +218,14 @@ Establish these facts before matching a failure mode: | ID | Signal | Root cause | Status | Probe | Citations | |---|---|---|---|---|---| -| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | Alternative runtimes can differ from AWF's default runtime assumptions | **gVisor support landed** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor` (maps to Docker runtime `runsc`) and AWF enables static-DNS host mappings for that runtime. Raw `runsc` remains an unknown passthrough runtime (no gVisor-specific capability profile lookup). **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; run AWF with `--container-runtime gvisor` (not raw `runsc`) on a build including github/gh-aw-firewall#6093 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093 | +| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | Alternative runtimes can differ from AWF's default runtime assumptions | **gVisor support landed** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor` (maps to Docker runtime `runsc`), or raw `--container-runtime runsc` which now aliases to the same gVisor capability profile (static-DNS host mappings plus non-iptables networking). **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; run AWF with `--container-runtime gvisor` or raw `runsc` on a build including github/gh-aw-firewall#6401 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093, github/gh-aw-firewall#6401 | | D2 | cli-proxy fails on IPv6-enabled runners | The tunnel was bound only to `127.0.0.1` | Fixed by dual-stack binding | Check `ss -tlnp` inside `awf-cli-proxy` | #4626 | | D3 | `--enable-dind` still exists after DinD removal | Legacy flag cleanup is incomplete | Known unresolved cleanup item | `awf --help | grep enable-dind` | #1727 | | D4 | Enterprise LLM gateway needs an injected auth header | API proxy lacks a user extension point for that hop | Known unresolved proposal | No general probe; capture the required header flow in the report | #4849 | | D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 | | D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11) requires the `sbx` CLI on the runner; `isSbxAvailable()` preflight fails if `sbx` is absent | Install the `sbx` CLI on the runner; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard domain whitelisting applies. | `command -v sbx && sbx version` — absence confirms; `docker info` | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 | | D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 | -| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`; `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | +| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor` or raw `runsc`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor` and its raw `runsc` alias (plus `sbx`); `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` when runtime is `gvisor` or raw `runsc` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | | D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.copilot/config.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`, `~/.copilot/config.json`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 | ## Error-string quick lookup @@ -247,7 +250,8 @@ Establish these facts before matching a failure mode: | `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 | | `ENOENT ... /host/usr/local/bin/copilot` | A8 | | `mkdirat ... : read-only file system` during chroot agent startup | A12 | -| `getaddrinfo EAI_AGAIN ` with `awf-cli-proxy could not connect to the external DIFC proxy` | B5 | +| `getaddrinfo EAI_AGAIN ` with `awf-cli-proxy could not connect to the external DIFC proxy`, but DinD `nslookup` succeeds once the peer is attached | B5 | +| `getaddrinfo EAI_AGAIN awmg-cli-proxy` / `ENOTFOUND awmg-cli-proxy` on ARC/DinD, and `docker run --rm alpine nslookup awmg-cli-proxy` fails | B12 | | `EACCES` in `upload-artifact` after `sudo: false` (`--network-isolation`) AWF run | B6 | | `EACCES` / `unlink` on path containing `/tmp/awf-...-chroot-home/` during AWF cleanup (not in an `upload-artifact` step) | B7 | | `chroot: failed to run command '/bin/sh'` on glibc daemon (not musl — confirmed by `ldd --version`) | A13 | @@ -257,14 +261,14 @@ Establish these facts before matching a failure mode: | `HTTP 404` / `404 Not Found` downloading `runsc` from `storage.googleapis.com` during gVisor install | D5 | | `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` with `--container-runtime sbx` | D6 | | `SIGSEGV` / `SIGABRT` signal crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail | D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) | -| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in `NO_PROXY`; fixed in github/gh-aw-firewall#6401) | +| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in `NO_PROXY`; fixed in github/gh-aw-firewall#6401) | | Credential files (`~/.aws`, `~/.ssh`, `~/.docker/config.json`, `~/.kube`, `~/.config/gh`, `~/.cargo/credentials.toml`, etc.) visible inside sbx microVM under `--container-runtime sbx` | D9 (entire `$HOME` mounted into sbx microVM virtiofs; fixed in github/gh-aw-firewall#6336 with home-whitelist + `scrubHomeCredentials()`) | ## Known unresolved items Flag these explicitly instead of implying there is a complete fix: -- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093) +- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093; raw `runsc` aliases to the same profile in github/gh-aw-firewall#6401) - D3 / #1727 — lingering `--enable-dind` cleanup - D4 / #4849 — enterprise header injection extension point - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 58989301e..cd6d0e117 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -102,9 +102,9 @@ Prefer the narrowest match. Examples: - `unknown shorthand flag: 'd' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`) - `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount) - `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16 (sysroot filter was over-broad and dropped the workspace mount) -- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach → B5 +- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5 - `EACCES` in upload-artifact after sudo:false → B6 -- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8 (gVisor userspace netstack bypasses the usual iptables DNAT path; patched AWF adds `172.30.0.1` to `NO_PROXY`) +- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; safe-output validation fails even though the agent completed → D8 (gVisor userspace netstack bypasses the usual iptables DNAT path; patched AWF adds `172.30.0.1` to `NO_PROXY`) - credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9 (older AWF mounted the entire host `$HOME` into sbx; fixed in github/gh-aw-firewall#6336) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) @@ -137,7 +137,9 @@ B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compo B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were too opaque and could mislead triage when the agent already exited non-zero. **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: repair-container stderr is now included in the `[WARN]` message, and chroot-home cleanup noise is reduced by downgrading that log to `debug`. -D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`/`sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. +B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix. `detectDnsResolutionFailure()` now augments the startup error with the unresolved host and recommends using the proxy IP or `dockerd --dns `. + +D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 57a20aff0..870715980 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -43,13 +43,14 @@ Establish these facts before matching a failure mode: | B2 | All outbound traffic fails behind a mandatory corporate proxy | AWF must chain Squid through the upstream proxy | Set `https_proxy` / `http_proxy` on the host or use `--upstream-proxy`. **Note:** AWF ≤ v0.27.32 (before PR github/gh-aw-firewall#6267) silently blocked proxy environment variables (`NO_PROXY`, `HTTP_PROXY`, `HTTPS_PROXY`, etc.) when passed via `--env` if `enableApiProxy` is active, because `PROXY_ENV_VARS` were in the credential exclusion set. **Fixed in PR github/gh-aw-firewall#6267**: proxy vars are now checked against the `PROXY_ENV_VARS` allowlist and passed through `additionalEnv` even when credential isolation is active. | `env | grep -i proxy`; inspect Squid config for `cache_peer` | #1975, github/gh-aw-firewall#6267 | || B3 | Squid exits with `FATAL: http_port: IPv6 is not available` | Docker IPv6 is disabled but Squid tries to bind an IPv6 listener | Enable Docker/kernel IPv6 (required with current AWF builds), or use a custom AWF build that removes the `[::]` listener | `docker info | grep -i ipv6`; inspect `/proc/sys/net/ipv6/conf/all/disable_ipv6` | #2139 | | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 | -| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Added `detectDnsResolutionFailure()` in `container-startup-diagnostics.ts`. On ARC/DinD, DinD Docker containers cannot resolve Kubernetes Service names (e.g. `awmg-cli-proxy`) because the DinD network does not forward DNS to the cluster resolver. The function scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with an actionable explanation and concrete fixes: address the DIFC proxy by IP, or configure `dockerd --dns `. | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix. For ARC/DinD: `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | +| B5 | `getaddrinfo EAI_AGAIN ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs, and the topology peer becomes reachable once attached to `awf-net` | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → transient `EAI_AGAIN`/`ENOTFOUND` → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat transient `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B). If ARC/DinD `nslookup awmg-cli-proxy` still fails even after the ordering fix, match B12 instead. | Confirm `topologyAttach` is non-empty; inspect whether the topology peer is attached to `awf-net` before the cli-proxy health check runs. On ARC/DinD, run `docker run --rm alpine nslookup awmg-cli-proxy` first — if it returns `NXDOMAIN`/`SERVFAIL`, prefer B12 over B5. | #5543, github/gh-aw-firewall#5542, github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn`. **PR github/gh-aw-firewall#6328 (merged 2026-07-17):** Benign permission errors (`Operation not permitted`, `EPERM`, `EACCES`) from the artifact repair container are now logged at `debug` with an "expected on restricted runners" note. Only genuinely unexpected failures still emit `[WARN]`. Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure, not a restricted-runner non-issue. | `ls -la ` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, github/gh-aw-firewall#5542, github/gh-aw-firewall#6328 | | B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf--chroot-home/` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 | | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 | | B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 | | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025 | | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: `; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | +| B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND ` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns ` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | ## Category C — GHES / GHEC / `ghe.com` @@ -68,14 +69,14 @@ Establish these facts before matching a failure mode: | ID | Signal | Root cause | Status | Probe | Citations | |---|---|---|---|---|---| -| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | Alternative runtimes can differ from AWF's default runtime assumptions | **gVisor support landed** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor` (maps to Docker runtime `runsc`) and AWF enables static-DNS host mappings for that runtime. Raw `runsc` remains an unknown passthrough runtime (no gVisor-specific capability profile lookup). **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; run AWF with `--container-runtime gvisor` (not raw `runsc`) on a build including github/gh-aw-firewall#6093 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093 | +| D1 | AWF does not start or isolation is ineffective under gVisor / Kata | Alternative runtimes can differ from AWF's default runtime assumptions | **gVisor support landed** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor` (maps to Docker runtime `runsc`), or raw `--container-runtime runsc` which now aliases to the same gVisor capability profile (static-DNS host mappings plus non-iptables networking). **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; run AWF with `--container-runtime gvisor` or raw `runsc` on a build including github/gh-aw-firewall#6401 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093, github/gh-aw-firewall#6401 | | D2 | cli-proxy fails on IPv6-enabled runners | The tunnel was bound only to `127.0.0.1` | Fixed by dual-stack binding | Check `ss -tlnp` inside `awf-cli-proxy` | #4626 | | D3 | `--enable-dind` still exists after DinD removal | Legacy flag cleanup is incomplete | Known unresolved cleanup item | `awf --help | grep enable-dind` | #1727 | | D4 | Enterprise LLM gateway needs an injected auth header | API proxy lacks a user extension point for that hop | Known unresolved proposal | No general probe; capture the required header flow in the report | #4849 | | D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 | | D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` was added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11); it requires the `sbx` CLI to be installed on the runner. AWF's `isSbxAvailable()` preflight fails if `sbx` is absent and the run aborts before the agent starts. | Install the `sbx` CLI on the runner before invoking AWF; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard `--allow-domains` whitelisting applies. | `command -v sbx && sbx version` — absence of `sbx` confirms; `docker info` — confirm Docker daemon is accessible from the runner | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 | | D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 | -| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`; `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | +| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor` or raw `runsc`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor` and its raw `runsc` alias (plus `sbx`); `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` when runtime is `gvisor` or raw `runsc` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 | | D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.copilot/config.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`, `~/.copilot/config.json`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 | ## Error-string quick lookup @@ -100,7 +101,8 @@ Establish these facts before matching a failure mode: | `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 | | `ENOENT ... /host/usr/local/bin/copilot` | A8 | | `mkdirat ... : read-only file system` during chroot agent startup | A12 | -| `getaddrinfo EAI_AGAIN ` with `awf-cli-proxy could not connect to the external DIFC proxy` | B5 | +| `getaddrinfo EAI_AGAIN ` with `awf-cli-proxy could not connect to the external DIFC proxy`, but DinD `nslookup` succeeds once the peer is attached | B5 | +| `getaddrinfo EAI_AGAIN awmg-cli-proxy` / `ENOTFOUND awmg-cli-proxy` on ARC/DinD, and `docker run --rm alpine nslookup awmg-cli-proxy` fails | B12 | | `EACCES` in `upload-artifact` after `sudo: false` (`--network-isolation`) AWF run | B6 | | `EACCES` / `unlink` on path containing `/tmp/awf-...-chroot-home/` during AWF cleanup (not in an `upload-artifact` step) | B7 | | `chroot: failed to run command '/bin/sh'` on glibc daemon (not musl — confirmed by `ldd --version`) | A13 | @@ -110,14 +112,14 @@ Establish these facts before matching a failure mode: | `HTTP 404` / `404 Not Found` downloading `runsc` from `storage.googleapis.com` during gVisor install | D5 | | `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` with `--container-runtime sbx` | D6 | | `SIGSEGV` / `SIGABRT` signal crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail | D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) | -| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in `NO_PROXY`; fixed in github/gh-aw-firewall#6401) | +| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; agent finishes but safe-output validation fails | D8 (gVisor userspace netstack: iptables DNAT bypass absent, MCP gateway not in `NO_PROXY`; fixed in github/gh-aw-firewall#6401) | | Credential files (`~/.aws`, `~/.ssh`, `~/.docker/config.json`, `~/.kube`, `~/.config/gh`, `~/.cargo/credentials.toml`, etc.) visible inside sbx microVM under `--container-runtime sbx` | D9 (entire `$HOME` mounted into sbx microVM virtiofs; fixed in github/gh-aw-firewall#6336 with home-whitelist + `scrubHomeCredentials()`) | ## Known unresolved items Flag these explicitly instead of implying there is a complete fix: -- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093) +- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093; raw `runsc` aliases to the same profile in github/gh-aw-firewall#6401) - D3 / #1727 — lingering `--enable-dind` cleanup - D4 / #4849 — enterprise header injection extension point - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index 8217b5b33..91f69fe9c 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -52,11 +52,11 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).not.toMatch(/^- A15 \/ /m); // D1 gVisor support phrasing expect(content).toContain('**gVisor support landed** in PR github/gh-aw-firewall#6093'); - expect(content).toContain('use `--container-runtime gvisor` (maps to Docker runtime `runsc`)'); - expect(content).toContain('Raw `runsc` remains an unknown passthrough runtime'); + expect(content).toContain('use `--container-runtime gvisor` (maps to Docker runtime `runsc`), or raw `--container-runtime runsc` which now aliases to the same gVisor capability profile'); + expect(content).not.toContain('Raw `runsc` remains an unknown passthrough runtime'); expect(content).toContain('**Kata Containers** remain an unresolved research area.'); - expect(content).toContain('github/gh-aw-firewall#3264, github/gh-aw-firewall#6093'); - expect(content).toContain('- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093)'); + expect(content).toContain('github/gh-aw-firewall#3264, github/gh-aw-firewall#6093, github/gh-aw-firewall#6401'); + expect(content).toContain('- D1 / #3264 — Kata Containers compatibility research (gVisor resolved in github/gh-aw-firewall#6093; raw `runsc` aliases to the same profile in github/gh-aw-firewall#6401)'); expect(content).not.toContain('- D1 / #3264 — gVisor and Kata compatibility research'); // B11 new failure mode expect(content).toContain('| B11 |'); @@ -64,29 +64,33 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('**Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**'); expect(content).toContain('The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it.'); expect(content).toContain('| `[WARN] Rootless artifact permission repair failed ... (exit 1)` with little/no stderr detail, plus cleanup warnings around chroot-home removal and `Command completed with exit code: 1` | B11 |'); + expect(content).toContain('| B12 | `getaddrinfo EAI_AGAIN ` or `ENOTFOUND `'); expect(content).toContain('`detectDnsResolutionFailure()`'); expect(content).toContain('docker run --rm alpine nslookup awmg-cli-proxy'); expect(content).toContain('github/gh-aw-firewall#6326, github/gh-aw-firewall#6328'); expect(content).toContain('expected on restricted runners'); expect(content).toContain('Post-#6328: a `[WARN] Rootless artifact permission repair failed` message indicates a genuine failure'); - expect(content).toContain('| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` |'); - expect(content).toContain('`runtimeUsesIptables()` returns `false` for `gvisor`/`sbx`'); + expect(content).toContain('| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor` or raw `runsc`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` |'); + expect(content).toContain('`runtimeUsesIptables()` returns `false` for `gvisor` and its raw `runsc` alias (plus `sbx`)'); expect(content).toContain('AWF_SKIP_IPTABLES_INIT=1'); expect(content).toContain('| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.copilot/config.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM |'); expect(content).toContain('`scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-`'); - expect(content).toContain('| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; agent finishes but safe-output validation fails | D8'); + expect(content).toContain('| `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; agent finishes but safe-output validation fails | D8'); expect(content).toContain('| Credential files (`~/.aws`, `~/.ssh`, `~/.docker/config.json`, `~/.kube`, `~/.config/gh`, `~/.cargo/credentials.toml`, etc.) visible inside sbx microVM under `--container-runtime sbx` | D9'); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(source).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); - expect(source).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8'); + expect(source).toContain('- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5'); + expect(source).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; safe-output validation fails even though the agent completed → D8'); expect(source).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); - expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); + expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.'); + expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.'); expect(portableAgent).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(portableAgent).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); - expect(portableAgent).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor`; safe-output validation fails even though the agent completed → D8'); + expect(portableAgent).toContain('- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5'); + expect(portableAgent).toContain('- `403 ERR_ACCESS_DENIED` for MCP tool calls (`safeoutputs`, `github`) to `172.30.0.1/redacted` under `--container-runtime gvisor` or raw `runsc`; safe-output validation fails even though the agent completed → D8'); expect(portableAgent).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); }); });