Re-pin all five held-out baselines from fresh 3-round measurements

The pins had gone stale: the seed agents moved 4-10 commits after the original
pinning (one commit touched all five) and nothing recorded the dependency.
Re-measured with scripts/rescore_candidate.py --seed, which reuses the original
path exactly -- plain harbor run over the explicit test task list, 3 rounds
pooled, no gateway, harbor's default timeout multiplier.

    benchmark        was      now      delta
    tau3             0.6111   0.7321   +0.121   real (sd 0.0099)
    gaia             0.5736   0.6205   +0.047   inside noise (sd 0.0524)
    swe-atlas-qna    0.0966   0.0676   -0.029   inside noise (sd 0.0257)
    officeqa         0.3603   0.3412   -0.019   inside noise (sd 0.0330)
    browsecomp-plus  0.4495   0.4619   +0.012   inside noise (sd 0.0283)

Only tau3 moved materially, so the earlier claim that all five were stale was too
broad -- but that was not knowable without measuring, which is the point.

Two benchmarks turn out to be too noisy for single-run comparisons, and this is
the more useful finding. gaia's three rounds spanned 0.554-0.682. tau3's optimizer
scored one *unchanged* harness at 0.800 and 0.547 on development, because its
user-simulator and NL-assertion grader are both LLMs whose variance rides on every
eval. Deltas under ~0.1 on either benchmark should be read as unresolved.

Their splits are not the cause: domain mix matches to the percentage point across
all three partitions (airline 13%, banking 26%, retail 30%, telecom 31%), as does
telecom persona difficulty. Recorded so the next person does not re-derive it.

Verified by compile on all five.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: varunursekar

harness-engineering-bench/CONFIGURATION.md

@@ -160,7 +160,7 @@ benchmark can be checked against the others at a glance.
 | | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
 |---|---|---|---|---|---|
 | target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
-| held-out baseline (K=3) ◆ | 0.574 ±0.010 | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 |
+| held-out baseline (K=3) ◆ | 0.621 ±0.052 | 0.341 ±0.033 | 0.068 ±0.026 (agg 0.632) | 0.732 ±0.010 | 0.462 ±0.028 |
 | split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
 | dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
 | val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 |
@@ -267,6 +267,10 @@ deepseek benchmarks logged zero exceptions over 945 trials, swe-atlas lost
 5/150 to gpt-oss 128k context overflow, and gaia lost 4/198 (infra) after the
 agent's reason/search-only-turn crash was fixed.
 
+**Re-pin these whenever a seed agent changes.** The first set went stale because the seed agents moved 4-10 commits afterwards -- one commit touched all five -- and nothing recorded the dependency. Re-measured 2026-07-28 with `scripts/rescore_candidate.py --seed`, which reuses the original path exactly. Only tau3 moved materially (+0.121 against an sd of 0.0099, so a genuine seed improvement); the other four shifted inside their own noise.
+
+**gaia and tau3 are too noisy for single-run comparisons.** gaia's own three rounds spanned 0.554-0.682 (sd 0.052), and tau3's optimizer scored one *unchanged* harness at 0.800 and 0.547 on development -- its user-simulator and NL-assertion grader are both LLMs, so their variance rides on every eval. Treat a gaia or tau3 delta under ~0.1 as unresolved. Their splits are not the problem: domain mix matches to the percentage point across all three partitions (airline 13%, banking 26%, retail 30%, telecom 31%), as does telecom persona difficulty.
+
 ◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and
 that model is text-only. Verified against the same litellm endpoint the gateway
 proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent

harness-engineering-bench/browsecomp-plus/baseline/build.yaml

@@ -31,7 +31,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
-    baseline_reward: 0.4495  # held-out K=3 mean (see CONFIGURATION.md)
+    baseline_reward: 0.4619  # re-pinned 0.424 / 0.470 / 0.492 (sd 0.0283); was 0.4495. See runs/BASELINES.md
     failure_value: 0.0
     max_attempts: 1
     # The held-out eval is noisy: score the selected candidate 3x per case and

harness-engineering-bench/gaia/baseline/build.yaml

@@ -31,7 +31,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
-    baseline_reward: 0.5736  # held-out K=3 mean (see CONFIGURATION.md)
+    baseline_reward: 0.6205  # re-pinned 0.554 / 0.625 / 0.682 (sd 0.0524); was 0.5736. See runs/BASELINES.md
     failure_value: 0.0
     max_attempts: 1
     # The held-out eval is noisy: score the selected candidate 3x per case and

harness-engineering-bench/officeqa/baseline/build.yaml

@@ -36,7 +36,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
-    baseline_reward: 0.3603  # held-out K=3 mean (see CONFIGURATION.md)
+    baseline_reward: 0.3412  # re-pinned 0.354 / 0.296 / 0.374 (sd 0.0330); was 0.3603. See runs/BASELINES.md
     failure_value: 0.0
     max_attempts: 1
     # The held-out eval is noisy: score the selected candidate 3x per case and

harness-engineering-bench/scripts/rescore_candidate.py

@@ -255,6 +255,15 @@ def main() -> int:
     log(f"{len(tasks)} {args.partition} case(s), {args.rounds} round(s), "
         f"{args.attempts} attempt(s)/case, concurrency {args.concurrency}")
 
+    # Some tasks reference the base URL under litellm's alias rather than
+    # OPENAI_BASE_URL -- swe-atlas-qna's rubric judge declares
+    # `EVAL_BASE_URL = "${OPENAI_API_BASE}"` in [verifier.env], and harbor aborts
+    # the whole job with "Missing Environment Variables" before running a single
+    # trial if it is unset. Mirror it so a benchmark's own judge can start.
+    if os.environ.get("OPENAI_BASE_URL") and not os.environ.get("OPENAI_API_BASE"):
+        os.environ["OPENAI_API_BASE"] = os.environ["OPENAI_BASE_URL"]
+        log("mirrored OPENAI_BASE_URL -> OPENAI_API_BASE for task verifier env")
+
     if "OPENAI_API_KEY" not in os.environ and not args.dry_run:
         sys.exit("OPENAI_API_KEY is not set. Source the run's secrets.env first: "
                  "the target agent talks to the upstream directly here, exactly "

harness-engineering-bench/swe-atlas-qna/baseline/build.yaml

@@ -31,7 +31,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
-    baseline_reward: 0.0966  # held-out K=3 mean (see CONFIGURATION.md)
+    baseline_reward: 0.0676  # re-pinned 0.061 / 0.102 / 0.040 (sd 0.0257); was 0.0966. See runs/BASELINES.md
     failure_value: 0.0
     max_attempts: 1
     # The held-out eval is noisy: score the selected candidate 3x per case and

harness-engineering-bench/tau3/baseline/build.yaml

@@ -30,7 +30,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
-    baseline_reward: 0.6111  # held-out K=3 mean (see CONFIGURATION.md)
+    baseline_reward: 0.7321  # re-pinned 0.738 / 0.740 / 0.718 (sd 0.0099); was 0.6111. See runs/BASELINES.md
     failure_value: 0.0
     max_attempts: 1
     # The held-out eval is noisy: score the selected candidate 3x per case and