Pin held-out baselines (K=3) into baseline_reward; disable score_baseline

Full held-out (test) pass of the seed harness, 3 independent rounds, under the
new targets (deepseek-v4-flash; gpt-oss-120b on swe-atlas):

  swe-atlas  0.097 ±0.011  (agg_score 0.632)   [gpt-oss-120b]
  tau3       0.611 ±0.021                       [deepseek-v4-flash]
  officeqa   0.360 ±0.042                       [deepseek-v4-flash]
  browsecomp 0.449 ±0.007                       [deepseek-v4-flash]

Each pinned into its target's baseline_reward with score_baseline: false, so
finalization uses the number instead of re-scoring the seed every run. The
three deepseek benchmarks logged zero exceptions over 945 trials (max_retries
absorbed the shared-quota 429s); swe-atlas lost 5/150 to gpt-oss 128k overflow.
CONFIGURATION.md documents the numbers, method, and the swe-atlas binary-vs-
agg_score note. gaia deferred.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: varunursekar

harness-engineering-bench/CONFIGURATION.md

@@ -54,6 +54,7 @@ benchmark can be checked against the others at a glance.
 | | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
 |---|---|---|---|---|---|
 | target model | deepseek-v4-flash | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
+| held-out baseline (K=3) ◆ | — | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 |
 | split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
 | dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
 | val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 |
@@ -82,6 +83,18 @@ benchmark can be checked against the others at a glance.
   emitting a timeout/connection error. Coverage gaps (no trial produced) and
   gateway budget/auth exhaustion remain excluded/terminating for both.
 
+◆ Held-out baseline of the seed harness on the **test** partition, mean over
+K=3 independent rounds; ± is the stdev across the three round means. Pinned
+into each target's `baseline_reward` with `score_baseline: false`, so runs use
+this number instead of re-scoring the seed every finalization. gaia is not yet
+measured (deferred). swe-atlas's `reward` is a binary pass/fail over a rubric
+and sits near the floor (0.097); the continuous `agg_score` (0.632, sd 0.011)
+is the far more informative signal — a candidate `reward_key` switch, pending
+the verifier emitting `agg_score` as a selectable key. Measured with
+deepseek-v4-flash (gpt-oss-120b on swe-atlas); the three deepseek benchmarks
+logged zero exceptions over 945 trials, swe-atlas lost 5/150 to gpt-oss 128k
+context overflow.
+
 † Sized from stock-agent probes (codex on the target model, 3 development
 tasks each, full declared timeouts): tau3 trials took 202-211s (900s budget
 ≈ 4x headroom) and swe-atlas trials 379-602s (1800s ≈ 3x headroom over the

harness-engineering-bench/browsecomp-plus/baseline/build.yaml

@@ -31,6 +31,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
+    baseline_reward: 0.4495  # held-out K=3 mean (see CONFIGURATION.md)
     failure_value: 0.0
     max_attempts: 1
 
@@ -41,7 +42,7 @@ objective:
   direction: maximize
 reward_mode: submit
 baseline_floor: false
-score_baseline: true
+score_baseline: false
 rescore_top_k: 3
 rescore_attempts: 1
 

harness-engineering-bench/officeqa/baseline/build.yaml

@@ -36,6 +36,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
+    baseline_reward: 0.3603  # held-out K=3 mean (see CONFIGURATION.md)
     failure_value: 0.0
     max_attempts: 1
 
@@ -46,7 +47,7 @@ objective:
   direction: maximize
 reward_mode: submit  # agent picks; falls back to auto_best, then current version
 baseline_floor: false  # gates on validation while reward is on test; opt-in only
-score_baseline: true
+score_baseline: false
 rescore_top_k: 3
 rescore_attempts: 1
 

harness-engineering-bench/swe-atlas-qna/baseline/build.yaml

@@ -31,6 +31,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
+    baseline_reward: 0.0966  # held-out K=3 mean (see CONFIGURATION.md)
     failure_value: 0.0
     max_attempts: 1
 
@@ -41,7 +42,7 @@ objective:
   direction: maximize
 reward_mode: submit  # agent picks; falls back to auto_best, then current version
 baseline_floor: false  # gates on validation while reward is on test; opt-in only
-score_baseline: true
+score_baseline: false
 rescore_top_k: 3
 rescore_attempts: 1
 

harness-engineering-bench/tau3/baseline/build.yaml

@@ -30,6 +30,7 @@ selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
+    baseline_reward: 0.6111  # held-out K=3 mean (see CONFIGURATION.md)
     failure_value: 0.0
     max_attempts: 1
 
@@ -40,7 +41,7 @@ objective:
   direction: maximize
 reward_mode: submit  # agent picks; falls back to auto_best, then current version
 baseline_floor: false  # gates on validation while reward is on test; opt-in only
-score_baseline: true
+score_baseline: false
 rescore_top_k: 3
 rescore_attempts: 1