feat(v1): execution-time network isolation for Docker (Linux+macOS) with allow/block egress lists

docs/v1/evaluation.md

@@ -55,3 +55,64 @@ disabled_tools = ["shell_tool"]
 ```
 
 The names of these tools are set by the respective harness. Consult the relevant documentation for the given harness for the relevant name(s). Some harnesses do not offer support to disable tools.
+
+## Runtime network policies
+
+Prime and Modal expose provider-native `network_access` switches. Docker instead uses
+URL-level `allow` and `block` lists, with a trusted setup phase before enforcement; the
+same policy vocabulary can extend to other runtimes when their sandbox APIs support it.
+
+### Docker URL policies
+
+Docker harnesses can keep trusted setup online, then restrict the agent to declared
+HTTP(S) destinations:
+
+```toml
+[env.agent.harness.runtime]
+type = "docker"
+allow = ["https://*.wikipedia.org"]
+block = ["https://upload.wikimedia.org"]
+```
+
+Docker is deny-by-default: an empty `allow` list permits only the interception URL and
+every MCP URL, which are added automatically before user entries. A bare `"*"` with no
+block entries opts out of filtering and keeps Docker's host-network behavior; adding a
+block entry enables filtering and narrows the wildcard. User block rules win over user
+allow rules; framework interception and MCP routes always remain reachable. Under every
+filtered policy, non-global destinations—including host-loopback, private, and link-local
+addresses—are reserved for framework routes, so user `allow` rules cannot expose host/LAN
+services or cloud metadata endpoints.
+
+Filtered Docker runtimes are single-rollout. Reusing one would require reopening trusted
+setup networking to processes left by the previous agent, so each rollout gets a fresh box.
+
+Rules may be bare host patterns (`*.example.com`) or URL origins
+(`https://example.com:8443`). A scheme or port in a rule narrows the match; URL paths
+are ignored. `*.example.com` includes `example.com` itself. HTTPS proxy tunnels use
+port 443 by default; an `allow` entry with an explicit HTTPS origin authorizes another
+port. Both the CONNECT authority and the TLS ClientHello SNI must satisfy the policy.
+
+The enforcement shape follows
+[Docker Sandboxes network isolation](https://docs.docker.com/ai/sandboxes/security/isolation/):
+HTTP(S) leaves through a policy proxy and direct non-HTTP egress is removed. As in
+[Docker's policy evaluation](https://docs.docker.com/ai/sandboxes/governance/concepts/),
+user deny rules win over user allows.
+
+Per-task `TaskData.network_allow` and `TaskData.network_block` entries are merged into
+the Docker runtime lists. The task's default `network_allow=["*"]` is neutral and leaves
+the evaluator policy intact. A concrete list replaces an evaluator wildcard; two
+concrete lists are combined, all block entries are retained, and block entries win. Any
+non-wildcard task URL policy requires Docker's framework-aware policy support.
+
+The restriction begins after task and harness setup and remains active through agent
+execution, finalization, and scoring. Debug actions apply it after task setup as well.
+The policy proxy runs in the Verifiers process, not a sidecar. Linux puts its listening
+socket on container loopback and removes every non-loopback route. macOS keeps one route
+to the host and limits it to the proxy port. One-shot helpers place the listener and
+apply the cut, then exit, so the harness remains the only running container. This
+prevents direct proxy bypass, peer-container access, arbitrary host access, and non-HTTP
+egress.
+
+Colocated MCP servers remain available on container loopback. The in-process proxy dials
+host-local interception and MCP endpoints directly; shared and external MCP URLs pass
+through the same policy without requiring a sidecar or public tunnel.

docs/v1/harbor.md

@@ -69,10 +69,28 @@ resource_multiplier = 2.0
 
 The `timeout_multiplier` multiplies both the agent and verifier timeout, while the `resource_multiplier` multiplies the task's CPU, memory and disk space. You might want to use these multipliers when the tasks set too tight limits and/or the agent is slow.
 
+## Network policies
+
+Harbor's effective agent network policy is applied to Docker harness runtimes. An
+`[agent].network_mode` override takes precedence over the `[environment]` baseline;
+legacy `[environment].allow_internet` is normalized by Harbor's schema.
+
+| Harbor mode | Docker execution policy |
+|---|---|
+| `public` | Sets the task allowlist to `["*"]`, leaving the evaluator policy intact. |
+| `no-network` | Sets the task allowlist to `[]` (framework routes only). |
+| `allowlist` | Sets the task allowlist to `allowed_hosts`. |
+
+Trusted task and harness setup remains online. The policy starts immediately before the
+agent and stays active through finalization and scoring. Interception and MCP URLs are
+added automatically; evaluator-provided `allow` entries add exceptions and `block`
+entries can narrow them. Restricted Harbor tasks require the Docker runtime because the
+other runtimes do not provide framework-aware URL filtering.
+
 ## Shortcomings
 
 verifiers does not have parity with Harbor yet, so some features are missing and currently being worked on. The most notable missing features right now are:
 
-- `no-network` support for sandbox runtimes ([Harbor Docs](https://www.harborframework.com/docs/tasks/network-policy))
+- Switching to a different verifier-phase network policy ([Harbor Docs](https://www.harborframework.com/docs/tasks/network-policy))
 - Shared & separate verifiers ([Harbor Docs](https://www.harborframework.com/docs/tasks#verifier-environment-shared-vs-separate))
 - Multi-step tasks ([Harbor Docs](https://www.harborframework.com/docs/tasks/multi-step))

environments/gsm8k_v1/gsm8k_v1/taskset.py

@@ -26,6 +26,12 @@ class GSM8KData(vf.TaskData):
 
 
 class GSM8KTask(vf.Task[GSM8KData]):
+    async def setup(self, runtime: vf.Runtime) -> None:
+        # Pre-provision the verifier's uv env while the runtime still has internet —
+        # with a restricted Docker allowlist it is cut before the agent runs, and
+        # scoring happens after that.
+        await runtime.prepare_uv_script(VERIFY)
+
     @vf.reward(weight=1.0)
     async def correct(self, trace: vf.Trace, runtime: vf.Runtime) -> float:
         prediction = trace.last_reply

skills/evaluate-environments/references/REFERENCE.md

@@ -322,7 +322,8 @@ Run on the host in a fresh `/tmp/<name>` workspace per rollout. **No extra field
 
 ### `DockerConfig` — `type: "docker"`
 
-Local Docker container sharing the host network (`--network host`).
+Local Docker container. A bare wildcard with no block entries shares the host network;
+every narrower policy uses an isolated bridge during agent execution.
 
 | Field | Type | Default | Notes |
 | --- | --- | --- | --- |
@@ -332,6 +333,8 @@ Local Docker container sharing the host network (`--network host`).
 | `memory` | `float \| None` | `None` | Hard memory limit in GB (`docker --memory`). None = unlimited. |
 | `gpu` | `str \| None` | `None` | GPU spec, e.g. `"A100"` or `"2"` (`docker --gpus` uses the count; needs the nvidia toolkit). |
 | `disk` | `float \| None` | `None` | Advisory disk request in GB. Docker has no portable per-container size limit, so accepted but **not enforced**. |
+| `allow` | `list[str]` | `[]` | URL origins or host patterns allowed during execution, e.g. `"https://*.wikipedia.org"`. Empty permits only automatically added interception/MCP routes; bare `"*"` with no block entries opts out of filtering. Wildcards are supported; `*.example.com` also matches the apex. URL paths are ignored. An explicit HTTPS origin authorizes a nonstandard CONNECT port; CONNECT authority and TLS SNI must both match policy. Under filtered policies, non-global addresses (including host loopback, private, and link-local) are reserved for framework routes. |
+| `block` | `list[str]` | `[]` | URL origins or host patterns denied during execution. Block wins over user `allow`; interception and MCP routes always remain reachable. |
 
 ### `PrimeConfig` — `type: "prime"`
 
@@ -377,6 +380,10 @@ Before each rollout or validation check, `resolve_runtime_config` combines the s
   class's default. Any non-default runtime-config workdir wins.
 - Non-`None` `TaskData.resources` values similarly fill supported runtime fields only while those
   fields remain at their defaults. Any non-default runtime-config resource value wins.
+- Non-wildcard task URL policy fields require framework-aware Docker policy support.
+  `TaskData.network_allow=["*"]` is neutral; a concrete task list replaces an evaluator
+  wildcard, otherwise concrete task/runtime lists combine. `TaskData.network_block`
+  combines with runtime `block`, and every block rule wins over allow rules.
 - A resource field unsupported by the chosen runtime is ignored; evaluation warns once per
   runtime/field combination. Docker and Modal accept `disk` so portable task data validates, but neither enforces a disk limit.
 
@@ -423,6 +430,8 @@ Per-row wall-clock timeout requests, in seconds, one for each rollout stage. For
 | `system_prompt` | `str \| None` | `None` | Optional system prompt. Harnesses with `APPENDS_SYSTEM_PROMPT` emit a real system message; otherwise a string prompt is prefixed with a warning. A separate system prompt cannot be folded into `Messages` or `None`. |
 | `image` | `str \| None` | `None` | Required container/sandbox image for this row. It replaces the base runtime image; subprocess is refused when set. |
 | `workdir` | `str \| None` | `None` | Working directory for harness execution and task hooks. Applied when the runtime supports it and its config remains at the default. |
+| `network_allow` | `list[str]` | `["*"]` | Docker destinations needed by the task. The wildcard is neutral and leaves evaluator policy intact; empty requests framework-only access. |
+| `network_block` | `list[str]` | `[]` | Destinations merged into Docker's `block` list. A non-empty list requires Docker filtering. |
 | `timeout` | `TaskTimeout` | `TaskTimeout()` | Per-stage timeout requests described above. |
 | `resources` | `TaskResources` | `TaskResources()` | Portable runtime resource requests described above. |
 

verifiers/v1/agent.py

@@ -26,6 +26,7 @@
 from verifiers.v1.retries import RetryConfig, backoff, trace_should_retry
 from verifiers.v1.rollout import RolloutRun
 from verifiers.v1.runtimes import (
+    DockerConfig,
     Runtime,
     RuntimeConfig,
     SubprocessConfig,
@@ -93,11 +94,37 @@ def _resolve_harness(cls, data):
         return data
 
 
-def _check_borrowed_placement(task: Task, runtime: Runtime) -> None:
+def _check_borrowed_placement(
+    task: Task, runtime: Runtime, base_config: RuntimeConfig
+) -> None:
     """A borrowed box is never re-provisioned, so a task's placement fields can't
-    be honored. A task `image` on a subprocess box raises (a wiring bug — it goes
-    to the caller, not the trace); a container box whose image differs only warns,
-    since placing a run into an existing world is the point of borrowing."""
+    be honored. Reject requirements that cannot be applied to the running box; an
+    image mismatch on a container only warns, since sharing its world is the point."""
+    task_policy = "*" not in task.data.network_allow or bool(task.data.network_block)
+    base_docker = base_config if isinstance(base_config, DockerConfig) else None
+    if task_policy or (base_docker is not None and base_docker.network_isolated):
+        config = runtime.config
+        if not isinstance(config, DockerConfig):
+            raise ValueError(
+                f"task {task.data.idx!r} requires a Docker URL network policy, but "
+                f"borrowed runtime {runtime.name!r} is not Docker-backed; use "
+                "agent.provision(task)"
+            )
+        policy_base = (
+            base_docker if base_docker is not None else DockerConfig(allow=["*"])
+        )
+        expected = resolve_runtime_config(policy_base, task)
+        assert isinstance(expected, DockerConfig)
+        # Do not inherit extra destinations from a box provisioned for another task.
+        if set(config.allow) != set(expected.allow) or set(config.block) != set(
+            expected.block
+        ):
+            raise ValueError(
+                f"task {task.data.idx!r} requires allow={expected.allow!r} and "
+                f"block={expected.block!r}, but borrowed runtime {runtime.name!r} "
+                f"has allow={config.allow!r} and block={config.block!r}; use "
+                "agent.provision(task)"
+            )
     if task.data.image is None:
         return
     if isinstance(runtime.config, SubprocessConfig):
@@ -299,7 +326,7 @@ def _rollout_params(
     ) -> dict:
         """Resolve one run's runtime config, pairing checks, timeouts, interception."""
         if runtime is not None:
-            _check_borrowed_placement(task, runtime)
+            _check_borrowed_placement(task, runtime, self.runtime_config)
             runtime_config = runtime.config
             run_is_local = runtime.is_local
         else:

verifiers/v1/cli/debug.py

@@ -230,6 +230,7 @@ async def debug_task(task: Task, config: DebugConfig) -> tuple[Trace, bool]:
             invoke(task.setup, {"trace": trace, "runtime": runtime}),
             setup_timeout,
         )
+        await runtime.prepare_execution([])
         trace.timing.setup.end = time.time()
 
         trace.timing.generation.start = time.time()

verifiers/v1/interception/__init__.py

@@ -34,13 +34,10 @@ def requires_tunnel(
     server_configs: Iterable[BaseConfig] = (),
     shared: "Iterable[SharedToolServer]" = (),
 ) -> bool:
-    """Whether the interception must be exposed via a tunnel — some consumer is off the
-    host network: the harness itself, a live `shared` server in a remote runtime, or a
-    tool/user server config placing one there (each reaches the `/state` channel from
-    its own runtime). Skipped as non-consumers: a `colocated` server (shares the
-    harness's runtime, covered by `harness_is_local`), a config-`url` server (external —
-    it connects out), and an `external` shared server (outside the state machinery
-    entirely). False means every consumer reaches the server at localhost."""
+    """Whether interception needs a public tunnel because some consumer cannot use a
+    host-local URL: the harness itself, a live `shared` server in a remote runtime, or a
+    tool/user server placed there. Colocated servers, configured URLs, and external shared
+    servers add no consumer. False means local URL translation is sufficient."""
     if not harness_is_local:
         return True
     if any(not s.external and not s.local for s in shared):
@@ -57,10 +54,8 @@ def make_interception(
     config: InterceptionConfig, *, requires_tunnel: bool
 ) -> Interception:
     """The interception for a config, picked by type (the host-side counterpart to
-    `make_runtime`). With `requires_tunnel` (some consumer is off the host network — see
-    the `requires_tunnel` util) each server is exposed via its configured tunnel; without
-    it they get none and are reached at localhost. The caller computes it (see
-    `Env._requires_tunnel`)."""
+    `make_runtime`). With `requires_tunnel`, each server is exposed through its configured
+    tunnel; otherwise it remains on host loopback. The caller computes this requirement."""
     if isinstance(config, InterceptionServerConfig):
         return InterceptionServer(config, requires_tunnel)
     if isinstance(config, StaticInterceptionPoolConfig):

verifiers/v1/interception/server.py

@@ -121,11 +121,9 @@ class InterceptionServerConfig(BaseInterceptionConfig):
 
 class InterceptionServer(Interception):
     """A server that proxies model calls for one or more rollouts — and is itself the
-    single-server `Interception` (the pools compose several of these). With
-    `requires_tunnel` (some consumer is off the host network) it mints its configured
-    tunnel; on `start` it then binds where the tunnel says (`bind_host`/`bind_port`) and
-    sets `base_url` — the one URL every consumer reaches it at — to the tunnel's public
-    URL. Without, every consumer is on the host network: it binds loopback, tunnel-free."""
+    single-server `Interception` (the pools compose several of these). When a consumer
+    needs a public URL, it mints the configured tunnel and binds where that tunnel says;
+    otherwise it stays on host loopback."""
 
     def __init__(
         self,
@@ -203,9 +201,8 @@ async def start(self) -> None:
         self.runner = web.AppRunner(app)
         await self.runner.setup()
         self.stack.push_async_callback(self.runner.cleanup)
-        # No tunnel → every consumer shares the host network: bind loopback on any ephemeral
-        # port. Otherwise the tunnel says where to bind for it to reach the port, and
-        # `expose` publishes it.
+        # Without a tunnel, local URL translation reaches an ephemeral loopback port.
+        # Otherwise the tunnel determines the bind address and publishes it.
         if self.tunnel is None:
             self.host, bind_port = "127.0.0.1", 0
         else:

verifiers/v1/interception/tunnel/base.py

@@ -1,12 +1,9 @@
 """The Tunnel contract: expose a host interception port to a remote consumer.
 
-The interception server runs on the host. A *local* consumer (host network) reaches it at
-localhost — no tunnel, whatever the config. A *remote* one needs the port published outward:
-that's a `Tunnel`. It says where the server must bind for the tunnel to reach it
-(`bind_host`/`bind_port`) and `expose`s the bound port as a public URL. So a tunnel knows
-nothing about locality — the caller (`InterceptionServer.start`) uses it only in the remote
-case. The host-side counterpart to a `Runtime`: `Runtime.expose` publishes a port *inside* a
-sandbox; `Tunnel.expose` a *host* port.
+The interception server runs on the host. Local runtimes use a host-local URL directly or
+through their runtime translation; remote runtimes need the port published outward. A
+`Tunnel` supplies the bind address and exposes that host port as a public URL. It is the
+host-side counterpart to `Runtime.expose`, which publishes a port inside a sandbox.
 """
 
 import contextlib