fix: classify Modal sandbox / tests-dir loss as transient_infra

The harbor backend's error taxonomy records any unrecognized case failure as
TASK_FAILURE: an informative, scoreable sample at the failure value with status
SUCCESS. That is right for a candidate that crashes on its own, but Modal
sandbox lifecycle failures and a missing held-out tests fixture are
infrastructure the candidate did not cause.

Seen on a swe-atlas-qna run (2026-07-24): all 396 cases returned status=success,
error_category=task_failure, score=0.0. Terminal exceptions were AddTestsDirError
x144, Modal sandbox NotFound x125, RuntimeError x67, BadRequestError x60.
Classified as task failures, the aggregate reported error_rate 0.0 and objective
feasible at 0.0, so a fully broken environment looked like a real, shippable
zero and the optimizer ran with no gradient (baseline and every candidate 0.0).

Add the harness/Modal environment-loss signals (sandbox lifecycle,
AddTestsDirError / "failed to add tests directory", container fetch failures,
StreamTerminatedError) to the TRANSIENT_INFRA patterns. Those cases are now
excluded from the aggregate and counted toward invalidity, so a collapsed
environment surfaces as an invalid run instead of a silent zero. A candidate's
own bug (ValueError, KeyError, no answer) is still a task failure, unchanged.
Covered by two new tests; existing behavior asserted unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: shehabyasser-scale

vero/src/vero/evaluation/error_taxonomy.py

@@ -158,6 +158,22 @@ def policy(category: ErrorCategory) -> CategoryPolicy:
         ),
         ErrorCategory.TRANSIENT_INFRA,
     ),
+    (
+        # Harness / Modal environment loss: the task sandbox died, was never
+        # created, or its container or held-out tests fixture could not be
+        # provisioned. This is infrastructure the candidate did not cause, so
+        # it is excluded from the aggregate and counted toward invalidity,
+        # never scored as an informative task failure. Without this, a run
+        # whose sandboxes all collapsed reports a fully "successful" 0.0
+        # instead of an invalid aggregate.
+        re.compile(
+            r"sandbox|streamterminated|"
+            r"failed.?to.?add.?tests.?directory|addtestsdirerror|"
+            r"loading.?container|fetchspec",
+            re.IGNORECASE,
+        ),
+        ErrorCategory.TRANSIENT_INFRA,
+    ),
 ]
 
 

vero/tests/test_v05_error_taxonomy.py

@@ -58,3 +58,40 @@ def test_policy_encodes_the_intended_treatment():
     task = policy(ErrorCategory.TASK_FAILURE)
     assert task.is_informative_sample
     assert not task.counts_toward_invalidity and not task.terminating
+
+
+def test_classify_signal_recognizes_harness_environment_loss():
+    # Modal sandbox lifecycle failures and a missing held-out tests fixture are
+    # infrastructure the candidate did not cause, not an informative zero.
+    assert classify_signal("AddTestsDirError") is ErrorCategory.TRANSIENT_INFRA
+    assert (
+        classify_signal("Failed to add tests directory to environment.")
+        is ErrorCategory.TRANSIENT_INFRA
+    )
+    assert (
+        classify_signal("The Sandbox is unavailable. It may have already shut down.")
+        is ErrorCategory.TRANSIENT_INFRA
+    )
+    assert (
+        classify_signal("Modal Sandbox with container ID ta-01 not found.")
+        is ErrorCategory.TRANSIENT_INFRA
+    )
+    assert (
+        classify_signal("FetchSpec failed: loading container: file does not exist")
+        is ErrorCategory.TRANSIENT_INFRA
+    )
+
+
+def test_classify_case_treats_environment_loss_as_infra_not_task_failure():
+    # Regression for the swe-atlas-qna flat-zero: the majority of cases died on
+    # Modal sandbox / tests-dir provisioning yet were masked as scoreable task
+    # failures at 0.0. They must be excluded infrastructure, not informative.
+    assert classify_case(["AddTestsDirError"]) is ErrorCategory.TRANSIENT_INFRA
+    assert (
+        classify_case(
+            ["NotFoundError", "Modal Sandbox with container ID ta-01 not found."]
+        )
+        is ErrorCategory.TRANSIENT_INFRA
+    )
+    # A candidate's own bug is still an informative task failure, unchanged.
+    assert classify_case(["ValueError"]) is ErrorCategory.TASK_FAILURE