File tree Expand file tree Collapse file tree
harness-engineering-bench Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -66,6 +66,12 @@ benchmark can be checked against the others at a glance.
6666- ** Verifier timeout** is 2× ` timeout_seconds ` : finalization runs the
6767 candidate and the baseline test evaluations.
6868- ** Case budgets** are 4× the partition size, i.e. four full passes.
69+ - ** ` infrastructure_max_attempts: 3 ` ** applies only to trusted finalization
70+ re-scores. For competitive (agent) evaluations, whole-sub-run infrastructure
71+ retry is disabled and a within-trial transient-infra failure is scored at the
72+ failure value rather than excluded — a candidate cannot inflate its mean by
73+ emitting a timeout/connection error. Coverage gaps (no trial produced) and
74+ gateway budget/auth exhaustion remain excluded/terminating for both.
6975
7076† Sized from stock-agent probes (codex on the target model, 3 development
7177tasks each, full declared timeouts): tau3 trials took 202-211s (900s budget
You can’t perform that action at this time.
0 commit comments