feat(sandbox-envs): per-environment egress policy UI (#1336)

Author: srtab

CHANGELOG.md

@@ -9,7 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Added per-environment network egress policy for sandbox environments. A `SandboxEnvironment` can now carry an `egress_policy` (default/intercept + per-host rules) and encrypted `egress_secrets` (named header credentials); when set, the agent provisions them onto the daiv-sandbox MITM egress proxy at session start so credentials are injected by the sidecar and never enter the sandbox container. The egress proxy is mandatory for network-enabled sessions: if the sandbox deployment has no egress proxy configured, a network-enabled run is rejected rather than falling back to unrestricted network. Note: a network-enabled environment **without** an egress policy receives the sidecar's default deny-all (no connectivity) — configure an egress policy on those environments. Set programmatically via the ORM for now (admin/API/form UI to follow).
+- Sandbox environments now automatically allow and authenticate their repository's git platform (GitLab/GitHub) for git operations over HTTPS in the sandbox. DAIV injects an `Authorization: Basic` credential at the egress proxy, built from a short-lived token — the same project-scoped ephemeral clone token for GitLab, and a `contents:write` installation token for GitHub — so `git fetch`/clone/push of the repository works without configuring a per-environment host rule. Because DAIV pushes from inside the sandbox, this applies even to **Network Off** environments when a push token exists: the environment is opened solely for its git platform (everything else stays blocked) so publishing always works, while token-less eval/benchmark runs stay fully network-isolated. The rule is applied at runtime (never stored on the environment) and takes precedence over a user-defined rule for the same host.
+- Added a per-environment **network egress policy** for sandbox environments, configurable from the environment form when network is enabled. Restrict outbound traffic to an allow-list of hosts (glob patterns), attach per-host HTTP credentials (header name + value, encrypted at rest and never rendered back), scope each host to **all methods** or **read-only** (`GET, HEAD, OPTIONS`), and set the `default` (deny/allow) reachability for unlisted hosts. The proxy always TLS-inspects every reachable host, so per-host credentials and method rules are always enforced. Credentials are injected by the daiv-sandbox egress proxy and never enter the sandbox container. The egress proxy is mandatory for network-enabled sessions: if the sandbox deployment has no egress proxy configured, a network-enabled run is rejected rather than falling back to unrestricted network.
 - Added `release_orphan_queued_threads` management command to recover `QUEUED` activities left behind on threads with no active sibling (rare TOCTOU loss).
 - Added per-domain auth headers for the `web_fetch` tool to the configuration UI under **Web Fetch → Per-domain auth headers**. Each row pairs a domain (exact match) with an HTTP header name and a header value (encrypted at rest). Replaces the previous env-only `AUTOMATION_WEB_FETCH_AUTH_HEADERS` setting; the new env override is `DAIV_WEB_FETCH_AUTH_HEADERS` (same JSON shape). Operators using the old name must rename it.
 - Added a "Start a run" page at `/dashboard/runs/new/` for launching new agent runs from the UI, and a "Retry" button on terminal non-webhook activities that pre-fills the form with the original prompt, repository, ref, and max-mode flag.
@@ -41,6 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Hardened the agent's workspace file tools (`read_file`/`write_file`/`edit_file`/`ls`/`glob`/`grep`) against sandbox transport faults: a transient failure — most notably the `409 "Session is busy"` the sandbox returns when batched tool calls contend on the one-at-a-time session — now degrades to a retryable tool-result error instead of aborting the whole run, matching the `bash` tool. The `bash` tool's own classifier now also treats `409` as transient (retry once) rather than permanent.
 - Hardened MCP tool loading so a single broken or slow MCP server can no longer freeze chats and runs. Each server's tools are now loaded independently with a per-server timeout (`MCP_TOOL_LOAD_TIMEOUT`, default 30s); a server that times out or errors is skipped instead of blocking the whole agent.
+- GitLab clones now self-heal from a stale cached clone token. The short-lived project-scoped clone token is cached for a day, but the underlying project access token can die sooner (revoked, or the project/instance reset); previously the dead token was served to — and failed — every clone of that project until the cache window closed. A clone rejected for authentication while using the ephemeral token now drops it and retries once with a freshly minted one (the PAT fallback is not retried, since re-minting cannot change that credential).
 - Enforced "at most one active (`READY`/`RUNNING`) API/MCP Activity per `thread_id`" at the DB layer via a partial unique constraint. Concurrent submissions on the same thread cleanly fall back to `QUEUED` instead of both running.
 - Made the FIFO dispatcher race-safe: an atomic compare-and-swap (`UPDATE filter(status=QUEUED) → READY`) prevents two terminal events on the same thread from double-promoting the same queued sibling. Dispatch failures now set `finished_at` and iterate via a loop instead of recursive signal re-entry. The loop bails after `MAX_CONSECUTIVE_DISPATCH_FAILURES` (3) consecutive failures so a transient broker outage does not mass-fail an entire QUEUED backlog — remaining rows stay QUEUED for `release_orphan_queued_threads`.
 - Guarded the post-enqueue `task_result_id` save in both `asubmit_batch_runs` and `dispatch_next_in_thread`: a DB blip after a successful broker enqueue now marks the row FAILED (and releases queued siblings) instead of stranding it in READY with no task linkage.

daiv/accounts/templates/accounts/_sidebar.html

@@ -1,7 +1,7 @@
 {% load i18n icon_tags nav_tags %}
 
 <aside data-testid="app-sidebar"
-       class="hidden w-60 shrink-0 overflow-y-auto border-r border-white/[0.06] bg-[#030712] sm:flex sm:flex-col"
+       class="shrink-0 overflow-y-auto bg-[#030712] {{ sidebar_root_class|default:'hidden w-60 border-r border-white/[0.06] sm:flex sm:flex-col' }}"
        x-cloak>
   <div class="flex items-center gap-2.5 px-5 pt-5 pb-5">
     <span class="inline-block h-6 w-6 rounded-md bg-gradient-to-br from-violet-500 to-indigo-500"></span>

daiv/accounts/templates/base_app.html

@@ -17,7 +17,7 @@
          x-transition:leave-start="translate-x-0"
          x-transition:leave-end="-translate-x-full"
          class="relative flex h-full w-[85%] max-w-[260px] flex-col bg-[#030712] shadow-2xl">
-      {% include "accounts/_sidebar.html" %}
+      {% include "accounts/_sidebar.html" with sidebar_root_class="flex w-full flex-col" %}
     </div>
   </div>
 

daiv/automation/agent/git_manager.py

@@ -10,6 +10,7 @@
 from git import GitCommandError
 
 from automation.agent.constants import REPO_PATH
+from core.utils import is_git_auth_error_text
 
 if TYPE_CHECKING:
     from git import Repo
@@ -366,31 +367,14 @@ class GitPushPermissionError(RuntimeError):
 class GitPushNetworkError(RuntimeError):
     """Raised when pushing fails because the remote host is unreachable.
 
-    Typically a sandbox-authoritative run whose sandbox environment is configured with
-    ``network_enabled=False``: git runs (and therefore pushes) from inside the sandbox, so the
-    sandbox must have network access for the push to reach ``origin``.
+    git runs (and therefore pushes) from inside the sandbox, so the sandbox must be able to reach
+    ``origin``'s git platform. DAIV opens that host automatically whenever a platform token can be
+    minted — even on a network-off env — so this typically means the egress proxy is unavailable, or
+    the run is one that legitimately has no platform token (e.g. an eval/benchmark run) and so stays
+    fully network-isolated.
     """
 
 
-def _is_push_auth_error_text(output: str) -> bool:
-    """
-    Check if git push output indicates an authentication or permission failure.
-    """
-    text = output.lower()
-    return any(
-        marker in text
-        for marker in (
-            "returned error: 403",
-            "authentication failed",
-            "permission denied",
-            "access denied",
-            "http basic: access denied",
-            "could not read username",
-            "not authorized",
-        )
-    )
-
-
 def _is_push_network_error_text(output: str) -> bool:
     """Check if git push output indicates the remote host was unreachable (network failure).
 
@@ -420,7 +404,7 @@ def _raise_for_push_failure(push_args: list[str], result: _GitResult) -> None:
     Auth/permission → ``GitPushPermissionError``; an unreachable host → ``GitPushNetworkError``;
     anything else → the raw ``GitCommandError``. Auth is checked first so it always wins.
     """
-    if _is_push_auth_error_text(result.output):
+    if is_git_auth_error_text(result.output):
         logger.warning("git push auth failure: %s", result.output)
         raise GitPushPermissionError(
             "Failed to push changes to the remote repository due to authentication or permission issues. "
@@ -432,7 +416,7 @@ def _raise_for_push_failure(push_args: list[str], result: _GitResult) -> None:
         logger.warning("git push network failure: %s", result.output)
         raise GitPushNetworkError(
             "Failed to push changes: the remote host is unreachable. Sandbox-authoritative auto-commit "
-            "pushes from inside the sandbox, so the sandbox environment must run with network access "
-            "(network_enabled=True)."
+            "pushes from inside the sandbox, so the sandbox environment must run as an egress-enabled "
+            "sandbox."
         )
     raise GitCommandError(["git", *push_args], result.exit_code, result.output)

daiv/automation/agent/middlewares/sandbox.py

@@ -20,7 +20,7 @@
 from automation.agent.conf import settings as agent_settings
 from automation.agent.constants import BUILTIN_SKILLS_PATH
 from automation.agent.middlewares.file_system import SandboxFileBackend  # noqa: TC001
-from codebase.context import RuntimeCtx, SandboxRuntime  # noqa: TC001
+from codebase.context import RuntimeCtx  # noqa: TC001
 from core.conf import settings
 from core.sandbox.client import DAIVSandboxClient, is_transient_sandbox_error
 from core.sandbox.command_parser import CommandParseError, parse_command
@@ -293,11 +293,19 @@ def _make_global_skills_archive() -> bytes | None:
 
 class SandboxEgressUnavailableError(RuntimeError):
     """Raised when a session's resolved egress policy cannot be provisioned because the sandbox has
-    no egress proxy configured (HTTP 404 'Egress proxy not configured' — no shared egress CA).
+    no egress proxy configured (no shared egress CA). daiv-sandbox rejects such a session up front
+    with HTTP 400 and a detail naming the egress proxy (see its ``POST /session/`` handler).
     Fail-closed: an environment that requires a restricted egress policy must not run without that
     policy in force."""
 
 
+# Substring that distinguishes the "egress proxy not configured" 400 from any other create-time 400.
+# Couples to daiv-sandbox's FastAPI ``detail`` ("egress requires the egress proxy, which is not
+# configured on this deployment" — its ``POST /session/`` handler). The sandbox exposes no
+# machine-readable error code, so this is the single greppable point of that prose coupling.
+_EGRESS_PROXY_UNAVAILABLE_MARKER = "egress proxy"
+
+
 class BashFailure(Enum):
     """Why a bash invocation produced no result, mapped to the guidance the agent gets.
 
@@ -502,36 +510,6 @@ async def _session_exists(client: DAIVSandboxClient, session_id: str) -> bool:
             )
             return False
 
-    @staticmethod
-    async def _provision_egress(client: DAIVSandboxClient, session_id: str, sb: SandboxRuntime | None) -> None:
-        """Provision the sidecar egress policy for a network-enabled session that requires it.
-
-        No-op when there is no resolved sandbox runtime, the env defines no egress policy, or network
-        is off. Only a **404** (egress proxy not configured on the sandbox — no shared egress CA,
-        unambiguous and permanent) is converted to the actionable ``SandboxEgressUnavailableError``.
-        On a fresh create this 404 is effectively unreachable: a network-enabled ``start_session`` is
-        rejected up front (400) when the sandbox has no egress proxy, so the run aborts before reaching
-        here. It survives for warm reuse — if the shared CA is rotated out between the session's start
-        and a later turn's re-provision, ``configure_egress`` 404s and we still want the actionable
-        diagnosis. Every other status propagates unchanged, including a 409 — on the egress endpoint 409
-        is ambiguous ("Session is busy" transient lock contention, or "Session has no egress proxy") and
-        the project already classifies 409 as transient (``TRANSIENT_SANDBOX_STATUS``), so it must not be
-        mislabeled as a permanent "configure the proxy" diagnosis. A propagated error still fails closed:
-        the caller's setup path aborts the run.
-        """
-        if sb is None or sb.egress is None or not sb.network_enabled:
-            return
-        try:
-            await client.configure_egress(session_id, sb.egress)
-        except httpx.HTTPStatusError as exc:
-            if exc.response.status_code == 404:
-                raise SandboxEgressUnavailableError(
-                    "The resolved sandbox environment requires the egress proxy, but the sandbox returned "
-                    "404 for egress provisioning. Configure the shared egress CA (EGRESS_CA_CERT_FILE + "
-                    "EGRESS_CA_KEY_FILE) on the sandbox deployment to enable the egress proxy."
-                ) from exc
-            raise
-
     async def abefore_agent(self, state: StateT, runtime: Runtime[RuntimeCtx]) -> dict[str, str] | None:
         """
         Bind a sandbox session — reusing the prior turn's warm session when possible.
@@ -555,34 +533,44 @@ async def abefore_agent(self, state: StateT, runtime: Runtime[RuntimeCtx]) -> di
         prior_session_id = state.get("session_id")
         if prior_session_id and await self._session_exists(client, prior_session_id):
             self._bind_session(prior_session_id)
-            # Re-provision egress on every warm reuse so the fail-closed guarantee holds on resumed
-            # turns too. A failure here propagates and aborts the run (fail-closed). Unlike the
-            # fresh-create path below we deliberately do NOT force-close the container: it's a healthy
-            # resumable session, the failure may be transient (e.g. a 409 "Session is busy"), and the
-            # next turn re-runs this check (or the reaper reclaims it). Force-closing here would throw
-            # away a good warm container on a transient blip.
-            await self._provision_egress(client, prior_session_id, runtime.context.sandbox)
             logger.info("Reusing warm sandbox session %s", prior_session_id)
             return {"session_id": prior_session_id}
 
         sb = runtime.context.sandbox
-        session_id = await client.start_session(
-            StartSessionRequest(
-                base_image=sb.base_image,
-                network_enabled=sb.network_enabled,
-                memory_bytes=sb.memory_bytes,
-                cpus=sb.cpus,
-                environment=sb.env_vars or None,
+        try:
+            session_id = await client.start_session(
+                StartSessionRequest(
+                    base_image=sb.base_image,
+                    egress=sb.egress,
+                    memory_bytes=sb.memory_bytes,
+                    cpus=sb.cpus,
+                    environment=sb.env_vars or None,
+                )
             )
-        )
+        except httpx.HTTPStatusError as exc:
+            # A network-enabled env on a sandbox with no egress proxy (no shared CA) is rejected up
+            # front with HTTP 400 (see _EGRESS_PROXY_UNAVAILABLE_MARKER). Match that specific signal so
+            # an unrelated 400 (e.g. an invalid base image) re-raises as-is instead of being mislabelled.
+            detail = (exc.response.text or "").lower()
+            if exc.response.status_code == 400 and _EGRESS_PROXY_UNAVAILABLE_MARKER in detail:
+                logger.error(
+                    "Sandbox rejected egress-required session (400): the egress proxy/CA is not configured "
+                    "on the sandbox deployment. Aborting run (fail-closed)."
+                )
+                raise SandboxEgressUnavailableError(
+                    "The resolved sandbox environment requires the egress proxy, but the sandbox rejected "
+                    "the session (400). Configure the shared egress CA (EGRESS_CA_CERT_FILE + "
+                    "EGRESS_CA_KEY_FILE) on the sandbox deployment to enable the egress proxy."
+                ) from exc
+            raise
         try:
             working_dir = Path(runtime.context.gitrepo.working_dir)
             repo_archive, skills_archive = await asyncio.gather(
                 asyncio.to_thread(_make_repo_archive, str(working_dir)), asyncio.to_thread(_make_global_skills_archive)
             )
             await client.seed_session(session_id, repo_archive=repo_archive, skills_archive=skills_archive)
-            await self._provision_egress(client, session_id, sb)
         except Exception:
+            # Build/seed failure on an already-created session (the egress-unavailable case fails earlier).
             logger.exception("Failed to build or seed sandbox session %s", session_id)
             try:
                 await client.close_session(session_id, force=True)