Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions containers/agent/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,11 @@ wait_for_iptables() {
#
# In network-isolation (topology) mode there is no iptables-init container —
# egress is enforced by Docker network topology — so skip the handshake.
if [ "${AWF_NETWORK_ISOLATION:-}" = "1" ]; then
echo "[entrypoint] Network-isolation mode: skipping iptables init container wait"
# Likewise for runtimes whose network stack can't be governed by host-netns
# iptables (e.g. gVisor's isolated netstack): AWF_SKIP_IPTABLES_INIT is set and
# egress relies on the HTTP_PROXY/HTTPS_PROXY env vars instead.
if [ "${AWF_NETWORK_ISOLATION:-}" = "1" ] || [ "${AWF_SKIP_IPTABLES_INIT:-}" = "1" ]; then
echo "[entrypoint] iptables-init skipped (proxy-based egress): skipping init container wait"
else
echo "[entrypoint] Waiting for iptables initialization from init container..."
INIT_TIMEOUT=300 # 300 * 0.1s = 30 seconds
Expand Down
69 changes: 55 additions & 14 deletions docs/gvisor-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@ gVisor is registered with `executionModel: 'compose'`:

```ts
const RUNTIME_REGISTRY = {
gvisor: { executionModel: 'compose', dockerRuntime: 'runsc', needsStaticDns: true },
sbx: { executionModel: 'microvm', dockerRuntime: undefined, needsStaticDns: false },
gvisor: { executionModel: 'compose', dockerRuntime: 'runsc', needsStaticDns: true, usesIptables: false },
sbx: { executionModel: 'microvm', dockerRuntime: undefined, needsStaticDns: false, usesIptables: false },
};
```

Expand Down Expand Up @@ -204,14 +204,51 @@ every static hostname into `/host/etc/hosts`. A new netstack-based runtime must
account for both files.
:::

### iptables DNAT must work inside the sandbox
### gVisor skips iptables entirely (proxy-based egress)

gVisor's userspace netstack is **isolated from the host network namespace**, so
the host-netns iptables DNAT/RETURN rules that `awf-iptables-init` installs
(port 80/443 → Squid, plus the NAT `RETURN` bypass for the MCP gateway) never
govern the sandbox's traffic. AWF therefore **does not run the iptables-init
container for gVisor at all** — the runtime is registered with
`usesIptables: false` in `container-runtime.ts`, which:

- skips `assembleIptablesInitService()` (`optional-services.ts`), and
- sets `AWF_SKIP_IPTABLES_INIT=1` so `entrypoint.sh` doesn't wait for the (absent)
init-container ready-file handshake.

**Internet egress** for proxy-aware clients is routed through Squid via the
`HTTP_PROXY`/`HTTPS_PROXY` env vars. There is one deliberate exception: because
the iptables NAT bypass for the MCP gateway is gone, `proxy-environment.ts` adds
the network gateway IP (e.g. `172.30.0.1`) and `host.docker.internal` to
`NO_PROXY` for non-iptables runtimes, so proxy-aware MCP clients (rmcp) connect
to the gateway **directly** instead of being routed through Squid and rejected
with `403 ERR_ACCESS_DENIED`.

Note that `NO_PROXY`/`HTTP_PROXY` are **client-side routing hints, not a security
boundary** — the agent controls its own environment and could reach the gateway
with or without them. The enforced egress boundary is external to the agent and
runtime-independent:

- **Strict/default mode** (`networkIsolation: true`): the agent sits on an
internal Docker network with no route off-host except through the dual-homed
Squid; the network topology is the boundary.
- **Legacy mode** (`networkIsolation: false`): host-level iptables in the
`DOCKER-USER` (FORWARD) chain default-denies container→internet traffic
(`host-iptables-rules.ts`) — this is what produces "No route to host" and is
unaffected by which runtime the agent uses. The container-level iptables that
gVisor skips was only ever a defense-in-depth DNAT fallback, never the primary
boundary.

:::caution Proxy-unaware tools under gVisor
With no iptables DNAT fallback, tools that ignore `HTTP_PROXY`/`HTTPS_PROXY`
(e.g. a raw `/dev/tcp` connection) have no route to external hosts and will fail
with "No route to host". Egress under gVisor requires proxy-aware clients.
:::

AWF's defense-in-depth relies on iptables DNAT (port 80/443 → Squid:3128) applied
inside the agent's network namespace. gVisor must support those rules for that
fallback to hold. `.github/workflows/test-gvisor-compat.yml` is a **manual,
non-gating diagnostic probe** that exercises iptables DNAT and proxy reachability
inside a `runsc` sandbox; it is useful evidence, but not an enforced guarantee in
CI.
`.github/workflows/test-gvisor-compat.yml` remains a **manual, non-gating
diagnostic probe** for iptables/proxy reachability inside a `runsc` sandbox; it
documents the historical behavior but is not part of the enforced egress path.

### Runtime-specific compatibility shims

Expand Down Expand Up @@ -265,6 +302,7 @@ myruntime: {
executionModel: 'compose',
dockerRuntime: 'my-oci-runtime', // the name registered in daemon.json
needsStaticDns: false, // true if its netstack can't reach 127.0.0.11
usesIptables: true, // false if host-netns iptables can't govern its traffic
},
```

Expand All @@ -282,11 +320,14 @@ Set `needsStaticDns: true` only if the runtime cannot reach Docker's embedded DN
`/etc/hosts` via `extra_hosts` *and* the chrooted `/host/etc/hosts`) and ensure
topology peers are patched into both.

### 3. Confirm the network fallback works
### 3. Confirm the network egress model

AWF's iptables DNAT-to-Squid path must function inside the runtime's network
namespace. Add a compat check modeled on `test-gvisor-compat.yml` before relying
on it.
If the runtime shares the host network namespace (`usesIptables: true`), AWF's
iptables DNAT-to-Squid path must function inside it — add a compat check modeled
on `test-gvisor-compat.yml`. If the runtime has an isolated netstack
(`usesIptables: false`, like gVisor), AWF skips iptables entirely and relies on
the `HTTP_PROXY`/`HTTPS_PROXY` env vars plus a `NO_PROXY` entry for the MCP
gateway; ensure the agent's tools are all proxy-aware.

### 4. Ensure the runtime is installed

Expand All @@ -302,7 +343,7 @@ viable and may be faster.
| --- | --- | --- |
| Host-kernel / syscall isolation | ✅ owns it (Sentry) | selects the runtime |
| In-sandbox network stack | ✅ netstack | works around its DNS limits |
| Domain egress ACL | — | ✅ Squid + iptables DNAT |
| Domain egress ACL | — | ✅ Squid (via `HTTP_PROXY`/`HTTPS_PROXY`; no iptables under gVisor) |
| Credential injection | — | ✅ api-proxy (`COPILOT_*`) |
| Chroot + capability drop | — | ✅ entrypoint / capsh |
| Lifecycle | OCI runtime under compose | ✅ `docker compose` + `docker wait` |
Expand Down
39 changes: 39 additions & 0 deletions src/compose-generator.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,45 @@ describe('generateDockerCompose', () => {
});
});

describe('gVisor runtime (non-iptables compose agent)', () => {
it('omits iptables-init but keeps the compose agent when networkIsolation is false', () => {
const config = {
...mockConfig,
containerRuntime: 'gvisor',
networkIsolation: false,
};
const result = generateDockerCompose(config, mockNetworkConfig);

expect(result.services.agent).toBeDefined();
expect(result.services['iptables-init']).toBeUndefined();
});

it('sets AWF_SKIP_IPTABLES_INIT (not AWF_NETWORK_ISOLATION) in the agent environment', () => {
const config = {
...mockConfig,
containerRuntime: 'gvisor',
networkIsolation: false,
};
const result = generateDockerCompose(config, mockNetworkConfig);

expect(result.services.agent.environment?.AWF_SKIP_IPTABLES_INIT).toBe('1');
expect(result.services.agent.environment?.AWF_NETWORK_ISOLATION).toBeUndefined();
});

it('treats the raw runsc runtime name the same as gvisor', () => {
const config = {
...mockConfig,
containerRuntime: 'runsc',
networkIsolation: false,
};
const result = generateDockerCompose(config, mockNetworkConfig);

expect(result.services.agent).toBeDefined();
expect(result.services['iptables-init']).toBeUndefined();
expect(result.services.agent.environment?.AWF_SKIP_IPTABLES_INIT).toBe('1');
});
});

describe('microVM runtime (sbx)', () => {
it('omits compose agent and agent-only helper services', () => {
const config = {
Expand Down
55 changes: 50 additions & 5 deletions src/container-runtime.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { resolveDockerRuntime, runtimeNeedsStaticDns, runtimeUsesComposeAgent } from './container-runtime';
import { resolveDockerRuntime, runtimeNeedsStaticDns, runtimeUsesComposeAgent, runtimeUsesIptables, isGvisorRuntime } from './container-runtime';
import { sanitizeEnvForSbx } from './sbx-manager';

describe('container-runtime', () => {
Expand All @@ -13,9 +13,12 @@ describe('container-runtime', () => {

it('passes through unknown runtime names unchanged', () => {
expect(resolveDockerRuntime('kata')).toBe('kata');
expect(resolveDockerRuntime('runsc')).toBe('runsc');
expect(resolveDockerRuntime('custom-runtime')).toBe('custom-runtime');
});

it('resolves the runsc alias to gVisor (docker runtime runsc)', () => {
expect(resolveDockerRuntime('runsc')).toBe('runsc');
});
});

describe('runtimeNeedsStaticDns', () => {
Expand All @@ -29,7 +32,10 @@ describe('container-runtime', () => {

it('returns false for unknown runtimes', () => {
expect(runtimeNeedsStaticDns('kata')).toBe(false);
expect(runtimeNeedsStaticDns('runsc')).toBe(false);
});

it('returns true for the runsc alias (same as gvisor)', () => {
expect(runtimeNeedsStaticDns('runsc')).toBe(true);
});

it('returns false for undefined/empty', () => {
Expand All @@ -38,8 +44,30 @@ describe('container-runtime', () => {
});
});

describe('runtimeUsesComposeAgent', () => {
it('returns true when no runtime is configured', () => {
describe('runtimeUsesIptables', () => {
it('returns false for gvisor (isolated netstack)', () => {
expect(runtimeUsesIptables('gvisor')).toBe(false);
});

it('returns false for sbx (microVM manages own egress)', () => {
expect(runtimeUsesIptables('sbx')).toBe(false);
});

it('returns true for unknown runtimes (share host netns)', () => {
expect(runtimeUsesIptables('kata')).toBe(true);
});

it('returns false for the runsc alias (same as gvisor)', () => {
expect(runtimeUsesIptables('runsc')).toBe(false);
});

it('returns true for undefined/empty (default runc)', () => {
expect(runtimeUsesIptables(undefined)).toBe(true);
expect(runtimeUsesIptables('')).toBe(true);
});
});

describe('runtimeUsesComposeAgent', () => { it('returns true when no runtime is configured', () => {
expect(runtimeUsesComposeAgent(undefined)).toBe(true);
});

Expand All @@ -56,6 +84,23 @@ describe('container-runtime', () => {
expect(runtimeUsesComposeAgent('runsc')).toBe(true);
});
});

describe('isGvisorRuntime', () => {
it('returns true for the gvisor friendly name', () => {
expect(isGvisorRuntime('gvisor')).toBe(true);
});

it('returns true for the raw runsc runtime name', () => {
expect(isGvisorRuntime('runsc')).toBe(true);
});

it('returns false for other/undefined runtimes', () => {
expect(isGvisorRuntime('sbx')).toBe(false);
expect(isGvisorRuntime('kata')).toBe(false);
expect(isGvisorRuntime(undefined)).toBe(false);
expect(isGvisorRuntime('')).toBe(false);
});
});
});

describe('sanitizeEnvForSbx', () => {
Expand Down
80 changes: 77 additions & 3 deletions src/container-runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,21 @@ interface RuntimeCapabilities {
* @see https://github.com/google/gvisor/issues/7469
*/
readonly needsStaticDns: boolean;

/**
* When `true`, AWF sets up egress control via the `awf-iptables-init` container,
* which applies host-netns iptables DNAT/RETURN rules inside the agent's network
* namespace (redirect port 80/443 → Squid, and bypass Squid for the MCP gateway).
*
* When `false`, those rules cannot govern the agent's traffic — e.g. gVisor's
* userspace netstack is isolated from the host network namespace, so the
* iptables-init container is skipped entirely and egress relies solely on the
* `HTTP_PROXY`/`HTTPS_PROXY` env vars plus `NO_PROXY` (the MCP gateway is reached
* directly instead of via an iptables bypass).
*
* @see https://github.com/google/gvisor/issues/7469
*/
readonly usesIptables: boolean;
}

/**
Expand All @@ -86,15 +101,46 @@ const RUNTIME_REGISTRY: Readonly<Record<string, RuntimeCapabilities>> = {
executionModel: 'compose',
dockerRuntime: 'runsc',
needsStaticDns: true,
// gVisor's isolated netstack can't be governed by host-netns iptables rules,
// so skip the iptables-init container and route egress via proxy env vars.
usesIptables: false,
},
// Future: Docker sbx microVM backend
sbx: {
executionModel: 'microvm',
dockerRuntime: undefined,
needsStaticDns: false, // sbx manages its own DNS
usesIptables: false, // microVM manages its own network egress
},
};

/**
* Aliases for runtime names that should resolve to the same capabilities as a
* canonical registry entry. This lets users pass either the friendly name
* (`gvisor`) or the raw Docker OCI runtime name (`runsc`) and get identical
* behavior — without an alias, `runsc` would fall through to the unknown-runtime
* defaults (iptables-capable), reintroducing the gVisor egress bug.
*/
const RUNTIME_ALIASES: Readonly<Record<string, string>> = {
runsc: 'gvisor',
};

/**
* Canonicalizes a user-facing runtime name to its registry key, following the
* alias table. Unknown names are returned unchanged.
*/
function canonicalRuntime(runtime: string): string {
return RUNTIME_ALIASES[runtime] ?? runtime;
}

/**
* Looks up the capabilities for a runtime name, following aliases. Returns
* `undefined` for unknown names (raw Docker runtime identifiers / default runc).
*/
function lookupCapabilities(runtime: string): RuntimeCapabilities | undefined {
return RUNTIME_REGISTRY[canonicalRuntime(runtime)];
}

// ─── Public API ──────────────────────────────────────────────────────────────

/**
Expand All @@ -105,7 +151,7 @@ const RUNTIME_REGISTRY: Readonly<Record<string, RuntimeCapabilities>> = {
* runtime field.
*/
export function resolveDockerRuntime(runtime: string): string | undefined {
const entry = RUNTIME_REGISTRY[runtime];
const entry = lookupCapabilities(runtime);
if (entry) return entry.dockerRuntime;
// Unknown name — pass through as a raw Docker runtime identifier
return runtime;
Expand All @@ -121,7 +167,24 @@ export function resolveDockerRuntime(runtime: string): string | undefined {
*/
export function runtimeNeedsStaticDns(runtime: string | undefined): boolean {
if (!runtime) return false;
return RUNTIME_REGISTRY[runtime]?.needsStaticDns ?? false;
return lookupCapabilities(runtime)?.needsStaticDns ?? false;
}

/**
* Returns `true` when AWF should set up egress control for the runtime via the
* `awf-iptables-init` container (host-netns iptables DNAT/RETURN rules).
*
* Returns `false` for runtimes whose network stack cannot be governed by those
* rules (e.g. gVisor's isolated netstack, or microVM backends that manage their
* own egress). For these, AWF skips the iptables-init container and relies on
* proxy env vars + `NO_PROXY`.
*
* Defaults to `true` for unknown/undefined runtimes (raw Docker runtime names
* and the default runc), which share the host network namespace.
*/
export function runtimeUsesIptables(runtime: string | undefined): boolean {
if (!runtime) return true;
return lookupCapabilities(runtime)?.usesIptables ?? true;
}

/**
Expand All @@ -136,7 +199,18 @@ export function runtimeNeedsStaticDns(runtime: string | undefined): boolean {
*/
export function runtimeUsesComposeAgent(runtime: string | undefined): boolean {
if (!runtime) return true;
const entry = RUNTIME_REGISTRY[runtime];
const entry = lookupCapabilities(runtime);
if (!entry) return true; // unknown runtime → assume compose
return entry.executionModel === 'compose';
}

/**
* Returns `true` when the configured runtime is gVisor, accepting either the
* friendly name (`gvisor`) or the raw Docker OCI runtime name (`runsc`). Use
* this instead of a direct `=== 'gvisor'` comparison so both spellings are
* handled consistently.
*/
export function isGvisorRuntime(runtime: string | undefined): boolean {
if (!runtime) return false;
return canonicalRuntime(runtime) === 'gvisor';
}
Loading
Loading