3/3 Add harness-engineering-bench (GAIA + benchmarks) by varunursekar · Pull Request #46 · scaleapi/vero

varunursekar · 2026-07-23T17:57:21Z

Third of three stacked PRs splitting #41. Base: pr2-add-vero.

Adds harness-engineering-bench/ with each benchmark as a top-level sibling (gaia, ale-bench, officeqa, swe-atlas-qna, tau3) + scripts. Pure adds (71 files).

Union of 1/3 + 2/3 + 3/3 == v0.5 plus a legacy/ archive (verified).

🤖 Generated with Claude Code

Greptile Summary

This PR adds the harness-engineering-bench/ tree with six benchmarks (GAIA, OfficeQA, SWE-Atlas-QnA, tau3, BrowseComp-Plus, SWE-bench-Pro) as top-level siblings, and ships the supporting vero-core changes needed to run them reliably. The vero changes address several observed production failures: silent W&B sink loss when WANDB_BASE_URL lacks a scheme, missing-model 404s being misclassified as informative task failures, sandbox-collapse signals going unclassified, and optimizer-trial teardown needing build-declared outer-run flags.

SWE-bench-Pro baseline is the most fully hardened new agent: Responses API with retry-and-backoff, strict:True schemas, parallel_tool_calls:False, a correct turn-budget else-branch, and a binary-output guard — all patterns that were retrofit-flagged on other agents in earlier PRs.
optimizer_harbor_args threads build-declared flags through to the outer harbor run (not the nested eval run), with a new preflight model-availability check that discriminates route-level from model-level 404s before a full trial burns.
UPSTREAM_MODEL_UNAVAILABLE error category and sandbox-loss patterns are added to the error taxonomy to prevent the harness from misattributing infrastructure failures as informative candidate zeros.

Confidence Score: 4/5

Safe to merge after addressing the optimizer_harbor_args validator gap; all other changes are well-tested and directionally correct.

The new validate_optimizer_harbor_args validator guards short-form flags (-a, -e, -m, -p) but not their long-form equivalents. Because optimizer_harbor_args is appended after vero's own flags and Harbor uses last-value-wins, a build declaring ["--agent", "other-agent"] silently overrides the outer run's agent with no validation error. The rest of the PR — the SWE-bench-Pro agent, the model-preflight check, the error taxonomy additions, the W&B scheme fix, and the deployment artifact recording — are all well-tested and tackle real observed failures.

Files Needing Attention: vero/src/vero/harbor/build/config.py — the validate_optimizer_harbor_args method needs long-form flag variants added to its controlled set.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/build/config.py	Adds `optimizer_harbor_args` field with a validator that guards short-form flags only; long-form equivalents (--agent, --environment, --model) are not blocked and can silently override the outer-run agent
vero/src/vero/harbor/cli.py	Adds model-preflight check (_preflight_models) that distinguishes route-level vs model-level 404s, and wires optimizer_harbor_args into the outer harbor run command; both changes are well-tested
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py	New Responses-API-based SWE-bench-Pro agent; uses strict:true schemas, parallel_tool_calls:false, retry-with-backoff, and a correct turn-budget else-branch — avoids the RuntimeError/missing-metadata patterns flagged in other agents
vero/src/vero/evaluation/scoring/error_taxonomy.py	Adds UPSTREAM_MODEL_UNAVAILABLE category and sandbox/container-loss patterns to TRANSIENT_INFRA; priority ordering in classify_case correctly places the new deterministic category before infra
vero/src/vero/runtime/wandb.py	Adds normalize_wandb_base_url() helper that auto-prefixes https:// to a scheme-less WANDB_BASE_URL before wandb.init(); called in both _open_wandb_run and WandbEventSink.init
vero/src/vero/harbor/deployment.py	Records W&B init failures to session artifacts so a silently-disabled sink is distinguishable from a healthy run that logged nothing
harness-engineering-bench/scripts/rescore_candidate.py	New operational script for re-scoring a candidate from a recovered session; aggregation matches runs/recompute.py; --seed path re-pins baselines using the same measurement path
vero/tests/test_v05_benchmark_configs.py	New invariant tests for all six benchmark configs; avoids pinning literal model names by asserting structural relationships, and skips (not fails) when task sources are unvendored
vero/tests/test_v05_cli.py	Adds thorough tests for _preflight_models (route vs model 404 discrimination, fallback spelling, inconclusive cases) and for optimizer_harbor_args command-line forwarding

Sequence Diagram

sequenceDiagram
    participant CLI as vero harbor run
    participant PF as _preflight_models
    participant UP as Upstream API
    participant HB as harbor run (outer)
    participant EV as harbor run (nested eval)

    CLI->>PF: probe each allowed_models entry
    PF->>UP: POST /responses (1 token)
    alt route-level 404
        UP-->>PF: 404 (no model mention in body)
        PF->>UP: POST /chat/completions (1 token)
    end
    alt model-level 404
        UP-->>PF: 404 + DeploymentNotFound
        PF-->>CLI: ClickException — abort
    else 200 / inconclusive
        UP-->>PF: 200 or timeout
        PF-->>CLI: proceed
    end

    CLI->>HB: harbor run [optimizer_harbor_args] [extra_args]
    note over HB: outer trial runs optimizer agent
    HB->>EV: harbor run [extra_harbor_args]
    note over EV: nested eval scores one candidate
    EV-->>HB: reward
    HB-->>CLI: finalized result

_{Reviews (66): Last reviewed commit: "Merge branch 'pr2-add-vero' into pr3-har..." | Re-trigger Greptile}

socket-security · 2026-07-23T17:57:54Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/tqdm@4.68.4
	pypi/tqdm@4.69.0
	pypi/tqdm@4.69.1
	pypi/click@8.4.2
	pypi/uvicorn@0.51.0
	pypi/tiktoken@0.13.0
	pypi/requests@2.34.2
	pypi/fastapi@0.139.2
	pypi/fastapi@0.140.0

View full report

shehabyasser-scale · 2026-07-24T11:36:12Z

Approve with nits. The split is the part I care most about for experiment validity, and it checks out.

GAIA split is genuinely leak-safe (verified):

Deterministic stratified partition keyed by sha256("vero-gaia-v1:{task}") (partition_gaia.py:116), no RNG, with a --check mode that fails CI if the committed manifest drifts.
Verified disjoint: development 33 / validation 66 / test 66, zero pairwise overlap, union 165.
Wiring is correct: development at disclosure: full, validation at aggregate (min_aggregate_cases: 5, expose_case_resources: false), selection on validation, and reward/target on the test partition which is never agent-visible. A proper held-out test set.
k-anonymity floors are safe in every build.yaml (tau3 / officeqa / swe-atlas-qna / gaia all set 5). The ale-bench endswith("ACCEPTED") judge is fine too: upstream JudgeResult has no NOT_ACCEPTED member, so no failing verdict ends in that string.

One real defect (baseline quality, not integrity): the tau3 baseline drops conversation history after every plain-text turn. tau3_agent/agent.py:370 sets previous_response_id = None in the non-tool branch, while the tool branch threads it correctly (:347). After any non-terminal customer reply the model loses the domain policy (only present in the turn-0 input), prior tool results, and reasoning state, which depresses tau3 from the first follow-up onward. The Responses API supports threading after a message output, so None isn't required. Suggest previous_response_id = response.id on the text branch as well. Confirmed tau3-specific; the GAIA baseline has no such reset.

One thing to note for readers: every build.yaml ships infrastructure_max_attempts: 3 + aggregate_attempts: best, so the competitive-selection integrity of these benchmarks depends on the aggregate/retry fix over on #45, not on any change here.

The probe only spoke the Responses API and treated any 404 as proof the deployment was absent. A route-level 404 (the upstream does not implement that path) and a model-level 404 (the deployment does not exist) share a status code and mean opposite things, so the preflight would hard-block a run against a Chat-Completions-only upstream that would have succeeded. That is now a live risk in this repo: officeqa, tau3, swe-atlas-qna and browsecomp-plus were ported to Chat Completions, leaving gaia as the only agent still on Responses. Try both routes, and only call a model missing when the 404 body names the model (`DeploymentNotFound`, `model_not_found`, or "model ... does not exist") rather than the route. If every route 404s on the route itself, the result is inconclusive and the run proceeds. Same fix for the model name: the provider prefix routes on a proxy and is meaningless to a single-provider endpoint, so probe the configured spelling first and fall back to the bare one before reporting anything missing. Previously the prefix was stripped unconditionally, which is wrong for `fireworks_ai/...` names. Verified against the live Azure endpoint: `gpt-4o` 200 on both routes; `openai/gpt-5.3-codex` 404s prefixed and passes preflight via the bare fallback; `fireworks_ai/gpt-oss-120b` is still correctly blocked; and a model 404 body is distinguished from a bare route 404. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The harbor backend's error taxonomy records any unrecognized case failure as TASK_FAILURE: an informative, scoreable sample at the failure value with status SUCCESS. That is right for a candidate that crashes on its own, but Modal sandbox lifecycle failures and a missing held-out tests fixture are infrastructure the candidate did not cause. Seen on a swe-atlas-qna run (2026-07-24): all 396 cases returned status=success, error_category=task_failure, score=0.0. Terminal exceptions were AddTestsDirError x144, Modal sandbox NotFound x125, RuntimeError x67, BadRequestError x60. Classified as task failures, the aggregate reported error_rate 0.0 and objective feasible at 0.0, so a fully broken environment looked like a real, shippable zero and the optimizer ran with no gradient (baseline and every candidate 0.0). Add the harness/Modal environment-loss signals (sandbox lifecycle, AddTestsDirError / "failed to add tests directory", container fetch failures, StreamTerminatedError) to the TRANSIENT_INFRA patterns. Those cases are now excluded from the aggregate and counted toward invalidity, so a collapsed environment surfaces as an invalid run instead of a silent zero. A candidate's own bug (ValueError, KeyError, no answer) is still a task failure, unchanged. Covered by two new tests; existing behavior asserted unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

#53 stopped a dead sandbox being scored as an informative zero. The same masking survives one layer up, for a cause that is even easier to fix. In the 2026-07-25 swe-atlas-qna run, 145 of 469 cases ended on: NotFoundError: Error code: 404 - {'error': {'type': 'invalid_request_error', 'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. ...'}} The configured eval model is not provisioned on the Azure resource. The agent's every call 404s, it writes no answer, and `classify_case` — finding nothing in the signal it recognises — returns TASK_FAILURE, whose policy is `is_informative_sample=True`. So each case was recorded as a genuine score of 0.0 and the harness reported that the candidate failed the task. Add UPSTREAM_MODEL_UNAVAILABLE: never an informative sample, never retried, terminating (every remaining case fails identically, so there is nothing left to measure), and counted toward invalidity so the aggregate still comes out invalid if the terminating path is ever bypassed. It is ranked above TRANSIENT_INFRA, so a case that saw both a dead sandbox and a missing deployment reports the deterministic, operator-fixable cause. The pattern deliberately matches the provider error codes and the exact Azure sentence rather than a bare "does not exist": 102 cases in that same run failed with `FetchSpec failed: loading container: file does not exist`, which is genuine infrastructure and must stay TRANSIENT_INFRA. There is a test pinning both sides. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The scaffold could not resolve a task: `task_source` was the literal `scale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST`, the three partition files were empty, and the split script refused to run behind a `TOTAL_TASKS = 0` stub guard. SWE-bench-Pro is not a package dataset. It ships as `swebenchpro` in Harbor's default registry (731 instances over 11 projects), whose tasks resolve to git-backed ids under laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1, so the pin is a bare `name@version` rather than a content digest. That distinction drives the rest of the change: - `_read_tasks` reads the stratification key from `tests/config.json` (`repo`), not `[metadata].repository`: these task.toml files carry no `[task].name`, and Harbor derives the canonical name from the task directory, which is what `-i` matches. - `_fetch_registry_refs` goes through `RegistryClientFactory` instead of `PackageDatasetClient`, recording `<git_commit>:<path>` as the per-task ref, which identifies content the way a digest would. - `expose_case_resources` is false on the development partition. VeRO materializes case resources through `PackageDatasetClient`, which only accepts `<org>/<name>` refs, so a registry dataset cannot be exposed that way. - 731 splits 20/40/40 as 146.2/292.4/292.4; the leftover goes by largest remainder and the .4/.4 tie breaks toward test, matching how officeqa (246 -> 49/98/99) and swe-atlas-qna (124 -> 25/49/50) resolved it. Hence 146/292/293. - `task_agent_timeout_seconds` drops 3600 -> 3000 to mirror the task package's own `[agent] timeout_sec`, making the runner's --agent-timeout-multiplier exactly 1800/3000 = 0.6. - `requires-python` floors at 3.12 because harbor 0.20.0 requires it; ">=3.11" left the package's own `uv run pytest` unresolvable. The <3.14 cap stays (litellm/pyo3 fails to build on 3.14). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ward Each of these silently biases the measurement rather than failing loudly, so the optimizer would have hill-climbed against a broken baseline. 1. REPO_DIR was `/app/repo`. The swebenchpro task images set `WORKDIR /app` and reset the checkout in place there, and the task instruction says so verbatim ("I've uploaded a code repository in the directory /app"). Every shell command ran in a directory that does not exist. 2. `reasoning: {effort: high}` was sent unconditionally. Non-reasoning models reject it with a hard 400 on the very first turn, so a gpt-4o run could not complete a single trial. Gated on capability, matching the sibling agents. 3. `_write_file` joined a model-supplied path onto `logs_dir`. Models routinely answer with the absolute path they just saw in shell output, and `Path(logs_dir) / "staged" / "/app/x.py"` discards the left side entirely, so the write landed on the HOST root and killed the trial with `OSError [Errno 30] Read-only file system: '/app'`. Seen on 2 of the first 12 held-out trials. Paths are now normalized repo-relative and traversal is refused. 4. Exhausting the turn budget raised RuntimeError. Reward comes from the task's hidden suite run against whatever the agent left in the repository, so raising discarded real edits, errored the trial, made harbor re-run the whole thing (9 of the first 49 held-out trials) and scored a hard 0 for a fix that may well have been correct. It now records the exhaustion in metadata and lets the verifier grade, which is exactly what the "model stopped calling tools" branch already did. 5. Harbor decodes exec output as strict UTF-8. These are real repositories, so a model that cats or greps a binary raised UnicodeDecodeError out of `exec` and killed the trial: 5 of the first 49 held-out trials. A non-decodable stream is now a tool error the model can react to, with a hint to re-run through `strings`/`file`. Verified end to end: a single-case smoke on instance_ansible__ansible-0ea40e with a gpt-4o agent completed with reward 1.0, all 16 required tests PASSED, 0 errored trials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The config-invariant suite enumerates benchmarks explicitly, so swe-bench-pro was silently unchecked. Now that its task_source resolves, it can join the other five. All three parametrized cases pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

gaia and swe-bench-pro send `reasoning: {effort: ...}` unconditionally. A non-reasoning model rejects it with HTTP 400, so every call fails and the case is scored as an honest-looking task failure. Measured on a live gaia run: 715 of 2376 calls returned 400 (514 evaluation, 201 finalization). Gate it on the model being reasoning-capable. No behavior change for reasoning models; gpt-4o stops 400ing. Scoped to the two agents that have no gate at all. officeqa, tau3 and swe-atlas-qna gained `if "fireworks" not in self._api_model` in b7a5b80, so their hunks are dropped here rather than rewritten. That gate is provider-shaped rather than capability-shaped and still sends reasoning_effort to any non-Fireworks non-reasoning model such as Azure gpt-4o, which is raised as review on the PR rather than changed unilaterally. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Restores the hunks dropped when this PR was restacked. b7a5b80 added `if "fireworks" not in self._api_model` to officeqa, tau3 and swe-atlas-qna, which is provider-shaped rather than capability-shaped: it correctly spares Fireworks-served open models and still sends `reasoning_effort` to any non-Fireworks model that cannot accept it. That is not theoretical. Measured today against the configured Azure endpoint, running the swe-atlas-qna seed agent over its 50-case held-out split: gpt-5.3-codex, provider gate 50/50 BadRequestError 400 "The requested operation is unsupported" (that model has no chat/completions surface on this resource at all) gpt-4o, provider gate 50/50 BadRequestError 400 "Unrecognized request argument supplied: reasoning_effort" gpt-4o, capability gate 0 BadRequestError, agent inference succeeded So the provider gate makes these three benchmarks unrunnable against any non-Fireworks upstream, which is every endpoint we currently have. The capability form also excludes `fireworks_ai/*`, so it is a strict superset of the current behaviour and changes nothing for the default configuration. officeqa gates two arguments on that one condition. Only `reasoning_effort` moves: `parallel_tool_calls` is a separate axis that Fireworks rejects and gpt-4o supports, so it keeps the provider check. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The four Chat Completions agents send `max_tokens` unconditionally. Every gpt-5 model rejects it: "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead." So even with the reasoning_effort gate corrected, these agents could not target any modern OpenAI reasoning model, only Fireworks-served models and gpt-4o. Select the parameter name from the same capability test, and fold the two copies of that test into one `_is_reasoning_model` helper per agent so they cannot drift apart. browsecomp-plus's reasoning_effort check moves onto the helper as well; it had the same provider-shaped gate as the other three. Verified live against the configured Azure endpoint, sending exactly the shapes the patched agents now build: gpt-4o ['max_tokens'] 200 gpt-5.4-mini-2026-03-17 ['max_completion_tokens','reasoning_effort'] 200 and the predicate classifies the suite's own targets correctly: `fireworks_ai/deepseek-v4-flash` and `fireworks_ai/gpt-oss-120b` are both non-reasoning, so the default configuration keeps the legacy shape and is unaffected. gaia and swe-bench-pro are untouched here: they use the Responses API, whose `max_output_tokens` has no such split. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The tau3 optimizer trial kept dying ~22 min in, after codex had finished, with `grpclib StreamTerminatedError: Connection lost` raised from harbor's finalization verifier exec (harbor/verifier/verifier.py:199 -> environments/dind_compose.py -> modal.py `_sdk_exec`). Harbor logs its own remedy at trial start: "DinD mode uses host networking ... Use --ek modal_vm_runtime=true to use the VM runtime instead". There was no way to declare that. `extra_harbor_args` reaches the NESTED `harbor run` that scores a candidate (build/config.py -> HarborBackendConfig -> backend.py `_harbor_command`), not the OUTER `harbor run` that hosts the optimizer trial, which is where the stream drops. Add `optimizer_harbor_args`, forwarded by `vero harbor run` to the outer command ahead of any trailing CLI args (harbor's `--ek` is last-value-wins, so the command line still overrides), and set tau3's to `--ek modal_vm_runtime=true`. This supersedes the previous version of this PR, which raised tau3's `max_retries` 1 -> 3. That knob is the nested per-case retry and was proven not to be the failing layer: the 2026-07-25 run still errored with `retries=0` and "Not retrying trial because the maximum number of retries has been reached", because the outer trial never receives `--max-retries`. Reverted to 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Four live runs, each ~40 min on Modal, all ending the same way: default DinD 08:13 StreamTerminatedError (verifier exec) modal_vm_runtime 09:18 StreamTerminatedError (codex agent exec) modal_sandbox_v2 10:02 StreamTerminatedError --max-retries 1 10:22 StreamTerminatedError on both attempts So tau3's optimizer trial gets no `optimizer_harbor_args` value. None of the four is a fix, and a committed config value that implies otherwise is worse than an empty one. What the experiments did establish: - The mechanism works. With `--max-retries 1` the trial logged "failed with exception StreamTerminatedError. Retrying in 1.00 seconds" where every earlier run logged "Not retrying ... maximum number of retries has been reached". The outer layer is now configurable, which it was not before. - Retry is not a cure here. The second attempt died the same way, so this converts one guaranteed loss into two. - It is not a networking mode. The failure moved between phases under vm_runtime but kept an identical bottom of stack, and sandbox_v2 changed nothing. - It looks deterministic, not flaky: 39m55s, 39m45s and 38m14s to failure across three independent runs is too tight for a random stream drop. The common factor is that harbor runs each phase as one exec and streams its stdout for that phase's whole duration. No harbor timeout matches ~40 min, so the deadline, if there is one, is not in harbor's config surface. This PR is therefore just the `optimizer_harbor_args` plumbing plus its tests. The tau3 failure belongs upstream with harbor. Refs #55. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two tests in test_v05_harbor_http.py build the build-config as a SimpleNamespace carrying only the attributes run_command reads, so adding a field to HarborBuildConfig breaks them with an AttributeError that surfaces only as a non-zero exit code. Same convention as when agent_env was added: the stub grows with the config. The mirror of that also applied to this branch's own test, whose _Config predates agent_env and so broke once both fields were in the same code path. Suite now matches the base branch exactly: 11 pre-existing failures, none new. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every harbor benchmark run has reported nothing to the self-hosted W&B: `shehab-yasser` listed zero projects, and each run's exported `artifacts/wandb/state.json` showed a run_id minted with `evaluation_ids: []`, `next_step: 0`, `request_log_files: {}` — even for a session that completed 11 evaluations. That fingerprint is exactly what `SidecarWandbSink.__init__` leaves when `wandb.init()` raises: state is written (wandb.py:225) before the run is opened (wandb.py:230), and `deployment.py` catches anything from the constructor and continues without W&B. Reproduced locally against scaleai.wandb.io, $0, deterministic: WANDB_BASE_URL=scaleai.wandb.io pydantic_core.ValidationError: 1 validation error for Settings base_url Input should be a valid URL, relative URL without a base [type=url_parsing, input_value='scaleai.wandb.io'] W&B parses `base_url` as a URL, so the natural way to write a self-hosted host silently costs the run all of its reporting. With `https://` prepended, the same script logs metrics, uploads an artifact and auto-creates the project. Egress was never the problem. Two changes: - `normalize_wandb_base_url()` prepends `https://` to a scheme-less `WANDB_BASE_URL` (and warns), called before both `wandb.init()` sites. Already-qualified values, including plain `http://localhost`, pass through. - The swallowed init failure now also lands in `session/artifacts/wandb/init-error.json`. The existing `logger.warning` goes to the sidecar container's stderr, which no run artifact captures, so a disabled sink was indistinguishable from a healthy run that logged nothing. Observability still never takes the eval path down. Refs #52. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix: only send reasoning.effort to models that support it

feat: let a build configure its outer optimizer trial's harbor flags

fix: repair a scheme-less WANDB_BASE_URL and stop losing W&B sink failures

…fold feat: make the SWE-bench-Pro baseline runnable for the vero optimizer

… candidate Re-pinning a baseline and re-scoring a candidate are the same measurement with a different starting tree, so --seed reuses the whole verified path rather than introducing a second scorer whose equivalence we would have to argue: plain harbor run over the explicit test task list, N rounds pooled, no gateway, and no --agent-timeout-multiplier so harbor's default 1.0 applies exactly as it did when the baselines were first pinned. Aggregation still matches runs/recompute.py. The seed is copied out of the benchmark's agent_repo before running so a measurement cannot mutate the checkout, and __pycache__/.venv/.git are excluded. Needed because the pinned baselines are of uncertain currency: gaia's seed scored 0.6414 through a run's finalization against a pinned 0.5736. Whether that is a stale pin or gaia's own variance is what re-measuring settles -- officeqa's first fresh round already lands at 0.3535 against its pinned 0.3603, which suggests the pins are healthier than the gaia figure implied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The pins had gone stale: the seed agents moved 4-10 commits after the original pinning (one commit touched all five) and nothing recorded the dependency. Re-measured with scripts/rescore_candidate.py --seed, which reuses the original path exactly -- plain harbor run over the explicit test task list, 3 rounds pooled, no gateway, harbor's default timeout multiplier. benchmark was now delta tau3 0.6111 0.7321 +0.121 real (sd 0.0099) gaia 0.5736 0.6205 +0.047 inside noise (sd 0.0524) swe-atlas-qna 0.0966 0.0676 -0.029 inside noise (sd 0.0257) officeqa 0.3603 0.3412 -0.019 inside noise (sd 0.0330) browsecomp-plus 0.4495 0.4619 +0.012 inside noise (sd 0.0283) Only tau3 moved materially, so the earlier claim that all five were stale was too broad -- but that was not knowable without measuring, which is the point. Two benchmarks turn out to be too noisy for single-run comparisons, and this is the more useful finding. gaia's three rounds spanned 0.554-0.682. tau3's optimizer scored one *unchanged* harness at 0.800 and 0.547 on development, because its user-simulator and NL-assertion grader are both LLMs whose variance rides on every eval. Deltas under ~0.1 on either benchmark should be read as unresolved. Their splits are not the cause: domain mix matches to the percentage point across all three partitions (airline 13%, banking 26%, retail 30%, telecom 31%), as does telecom persona difficulty. Recorded so the next person does not re-derive it. Verified by compile on all five. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both branches had accumulated real vero/src work the other lacked -- 12 commits here (wandb URL repair, optimizer-trial harbor flags, model preflight, transient infra classification) against 5 on pr2 (opencode step limit, litellm base-url alias, outer Modal app naming, terminal budget status, backend in the eval plan). The stacked-PR assumption that vero/ lives only on pr2 had quietly stopped holding, and the cost was concrete: benchmark runs launch from this worktree, so three fixes committed on pr2 were simply absent at run time. It cost a wasted mini-swe-agent attempt and a gaia run launched without the step limit it needed. Merging rather than rebasing: PR #57 is already merged into this branch, so replaying commits over it would rewrite merged history for no benefit, and a merge resolves the overlap once instead of per-commit. Three conflicts, all where both sides edited the same lines: - The run command gained both sets of appended flags. The derived Modal app name now defers to an explicit app_name from *either* source, since a build can declare one in optimizer_harbor_args just as a caller can pass one on the command line, and appending ours after the build's would have silently won on harbor's last-value-per-key rule. - Three config stubs needed both `optimizer_harbor_args` and `name`; they were written before the other side's field existed. - Three assertions expecting `_opencode_gateway_args` to return nothing are now wrong by design: the step limit is unconditional, so a bare model name and a gateway-less task both still get it. Dropped those and kept every other test from both sides. 453 passed, 5 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The gateway ignored Anthropic's cache_read/cache_creation input fields, so producer input metered 15,000x to 47,000x low. The request log captures each response body, so the true figures are recoverable and no run needs repeating -- which matters for the five runs currently in flight and for the grid cost table. Reports metered against recovered per scope, and prints coverage: the log keeps only the head and tail of each body, so anything under 100% makes the recovered total a lower bound. Takes the high water mark per record rather than summing matches, mirroring the fixed gateway, since a streaming response carries input in message_start and a cumulative output in message_delta. Leaves non-Anthropic scopes alone -- they were always right. Validated against a live run: evaluation scope reported as already correct, producer 112 metered against 3,965,279 recovered at 100% coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The local-capacity footnote quoted a CPU-based estimate that runs well above what container memory actually allows, so anyone sizing a parallel sweep off that line over-schedules and gets OOM-killed -- which surfaces as a misleading rate-limit error rather than as memory pressure. Says memory binds before CPU and to measure the local VM, without quoting figures from one machine that would not transfer anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

# Conflicts: # vero/src/vero/harbor/cli.py # vero/tests/test_v05_cli.py

Two handoff gaps. First, each harness addresses the gateway differently and the knowledge lived only in code comments: claude-code wants the scope root with /v1 stripped, opencode wants it kept, litellm wants a different base per provider, kimi-cli needs its config file written. Getting it wrong yields a 401 or 403 that reads like a credential fault, and finding that out costs a run each time. CONFIGURATION.md now carries the table, flags codex as the one harness never yet run, and records the two kimi-specific traps -- the deliberately doubled model prefix, and the allow-list needing the wire form via an explicit --param. Second, `scaleai.wandb.io` appeared in eight tracked files ahead of this going public. No hardcoded default was involved -- the code reads WANDB_BASE_URL -- but one secrets example shipped the host as a filled-in value while every other template is blank, and the rest named it in comments and a test fixture. All now use a neutral example host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

varunursekar mentioned this pull request Jul 23, 2026

Redesign VeRO as a generic program optimization harness #41

Closed

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread harness-engineering-bench/ale-bench/environment/sidecar/dind-entrypoint.sh Outdated

Comment thread harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

Comment thread harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

varunursekar force-pushed the pr3-harness-bench branch 3 times, most recently from e0a1efb to ab4c52f Compare July 23, 2026 20:00

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

varunursekar force-pushed the pr3-harness-bench branch 2 times, most recently from 573c1ac to c86351e Compare July 23, 2026 22:58

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread harness-engineering-bench/ale-bench/environment/sidecar/harness/score.py Outdated

varunursekar force-pushed the pr3-harness-bench branch 3 times, most recently from 745ed8e to 6788d76 Compare July 24, 2026 03:49

greptile-apps Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread harness-engineering-bench/officeqa/baseline/build.yaml Outdated

varunursekar force-pushed the pr3-harness-bench branch 4 times, most recently from 214f1aa to a2fcd19 Compare July 24, 2026 06:41

shehabyasser-scale approved these changes Jul 24, 2026

View reviewed changes

varunursekar force-pushed the pr3-harness-bench branch 4 times, most recently from 768de08 to ad2c081 Compare July 24, 2026 22:28

greptile-apps Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py

greptile-apps Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread harness-engineering-bench/scripts/per_trial_tokens.py

greptile-apps Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py

varunursekar force-pushed the pr3-harness-bench branch from 741bfa6 to 95fed89 Compare July 25, 2026 19:52

greptile-apps Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread harness-engineering-bench/browsecomp-plus/task-template/tests/evaluate.py

varunursekar force-pushed the pr3-harness-bench branch from 51c152d to ced19f9 Compare July 26, 2026 04:26

shehabyasser-scale and others added 23 commits July 28, 2026 06:50

Merge pull request #54 from scaleapi/fix/judge-reasoning-effort

2aff9f4

fix: only send reasoning.effort to models that support it

Merge pull request #55 from scaleapi/fix/tau3-trial-resilience

c8b6871

feat: let a build configure its outer optimizer trial's harbor flags

Merge pull request #57 from scaleapi/fix/wandb-base-url-normalization

16fbc2a

fix: repair a scheme-less WANDB_BASE_URL and stop losing W&B sink failures

Merge pull request #50 from scaleapi/feat/swe-bench-pro-baseline-scaf…

d93c696

…fold feat: make the SWE-bench-Pro baseline runnable for the vero optimizer

Merge branch 'pr2-add-vero' into pr3-harness-bench

7fed76c

Merge branch 'pr2-add-vero' into pr3-harness-bench

d34bf7b

shehabyasser-scale mentioned this pull request Jul 28, 2026

ci: run pytest for vero and the six baseline agents #63

Open

varunursekar and others added 6 commits July 28, 2026 09:19

Merge branch 'pr2-add-vero' into pr3-harness-bench

68ab9b2

# Conflicts: # vero/src/vero/harbor/cli.py # vero/tests/test_v05_cli.py

Merge branch 'pr2-add-vero' into pr3-harness-bench

3d505c9

Merge branch 'pr2-add-vero' into pr3-harness-bench

7ea8772

Merge branch 'pr2-add-vero' into pr3-harness-bench

838374a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

3/3 Add harness-engineering-bench (GAIA + benchmarks) - #46

3/3 Add harness-engineering-bench (GAIA + benchmarks)#46
varunursekar wants to merge 92 commits into
pr2-add-verofrom
pr3-harness-bench

varunursekar commented Jul 23, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

socket-security Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shehabyasser-scale commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

varunursekar commented Jul 23, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

socket-security Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shehabyasser-scale commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

varunursekar commented Jul 23, 2026 •

edited by greptile-apps Bot

Loading

socket-security Bot commented Jul 23, 2026 •

edited

Loading