Skip to content

Commit ad2c081

Browse files
committed
Docs: note infra retry/exclusion is trusted-only in benchmark config
1 parent 54b21d6 commit ad2c081

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

harness-engineering-bench/CONFIGURATION.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,12 @@ benchmark can be checked against the others at a glance.
6666
- **Verifier timeout** is 2× `timeout_seconds`: finalization runs the
6767
candidate and the baseline test evaluations.
6868
- **Case budgets** are 4× the partition size, i.e. four full passes.
69+
- **`infrastructure_max_attempts: 3`** applies only to trusted finalization
70+
re-scores. For competitive (agent) evaluations, whole-sub-run infrastructure
71+
retry is disabled and a within-trial transient-infra failure is scored at the
72+
failure value rather than excluded — a candidate cannot inflate its mean by
73+
emitting a timeout/connection error. Coverage gaps (no trial produced) and
74+
gateway budget/auth exhaustion remain excluded/terminating for both.
6975

7076
† Sized from stock-agent probes (codex on the target model, 3 development
7177
tasks each, full declared timeouts): tau3 trials took 202-211s (900s budget

0 commit comments

Comments
 (0)