2/3 Add the redesigned VeRO system (vero/) - #45
Open
varunursekar wants to merge 79 commits into
Open
Conversation
Introduce the v0.5 codebase: vero/ (core + harbor eval sidecar, build compiler, runtime/observability), vero-tasks/, and runs/. Root README points to legacy/ for the original-paper code.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…back #1) Addresses the 'baseline floor fails open' regression: - baseline_floor now defaults to false. The floor gates a ship on a validation comparison while the reward is on the (possibly differently-distributed) target, so it is opt-in. - Pin the fixed seed's score to avoid re-scoring it every run (reproducibility + one fewer eval): VerificationTarget.baseline_reward (per-target, post scale/offset) and VerificationSelection .baseline_selection_score (for the floor). Plumbed through the build config + compiler. - Fail safe when the floor is on, unpinned, and the seed re-score can't be measured: ship nothing (NoCandidateError -> shipped=false + infra error) instead of shipping the best candidate unverified (the fail-open bug). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1) Add an admin-only way to generate the number to pin: CanonicalVerifier .measure_baseline(replicates) admin-scores the fixed seed N times on the selection partition and each target and returns per-key mean/stddev. Exposed as POST /score/baseline and 'vero harbor score-baseline --replicates N'. Reuses the existing admin scoring machinery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atten The benchmark-config tests read harness-engineering-bench/ (a separate branch in the stacked split), so skip them when it isn't checked out. Update the paths for the candidates/ -> top-level flatten (all benchmarks are now top-level siblings of gaia/). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
min_aggregate_cases defaulted to 1 — a vacuous floor that let an agent read held-out validation labels one case at a time via single-case aggregate evals. Default to 5 in both the sidecar policy and the build AgentAccessSpec so a build that omits it is safe rather than unfloored (shipped builds already set 5 explicitly). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-enabling harness isolation broke the eval: 'uv run' as the unprivileged harness got a fresh empty uv cache and couldn't resolve the candidate package from it (No module named 'gaia_agent'), with no network to download at eval time. Pre-warm /home/harness/.cache/uv from the build (root) cache in the sidecar image, matching the UV_CACHE_DIR the backend sets for the harness user. Also fixes a test regression from feedback #2: with the k-anonymity floor defaulting to 5, the instruction now reads 'must include at least 5 cases' rather than 'arbitrary subsets'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ack #8) The mean-of-k recorded only score/n_attempts/n_scored, so an outage-diluted cell (dead attempts zero-filled) was indistinguishable from a genuinely clean low cell. Add n_dead_infra (dead-to-infrastructure attempts, classified via the existing error taxonomy) and n_clean (the rest), so the split is visible in the live metrics rather than only reconstructable from raw trial records. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- optimizer.run: validate self.max_proposals >= 0 before the proposal_limit guard, so the specific 'must be non-negative' error is reachable (was a dead branch that always tripped the generic message first). - compiler._safe_extract_tar: extractall(filter='data') to strip device files / setuid bits and neutralize unsafe links, matching extract_harbor_session_archive (also silences the py3.14 deprecation). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cold-start walkthrough of the harbor path: the three-container trust boundary, a run end-to-end, the evaluation core, disclosure/selection, the five security mechanisms, and a suggested outside-in review order keyed to the source files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update harbor_requirement pin from harbor[modal]==0.18.0 to ==0.20.0 across the README and harbor tests. Remove the PYTHONPATH injection in the harness eval launch: it was a no-op (uv strips PYTHONPATH from the child env) and did not fix the "No module named <agent>" failure under the dropped uid, so it and its test assertion are gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The candidate is checked out at <mktemp>/repository; `mktemp -d` makes that parent 0700 root. The eval launch chowned the repository (and the staging tree) to the harness user but not the mktemp parent, so once the process dropped to harness it could not traverse into its own workspace to resolve the editable candidate package's absolute path — every eval (baseline included) failed with "No module named <agent>" while harbor itself still loaded from the harness-owned cache. Grant the harness user traversal on that parent (chmod o+x) alongside the existing chowns. o+x suffices because run_as drops to the user's own uid+gid, so it is "other" relative to the root-owned parent. The dir holds only candidate code, so widening traversal leaks no trusted data. Reproduced and fixed against the real compiled sidecar image with a faithful user+group+extra_groups privilege drop: 0700 -> "No module named gaia_agent"; 0701 -> import resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a reachability probe: after chowning/granting traversal, run `test -r <project_path>` as the dropped user before launching harbor. Readability requires traversing every ancestor, so any provisioning gap (a missing chown, a non-traversable parent) is caught here with a clear message instead of resurfacing as a cryptic "No module named <agent>" several retries downstream — which is exactly how the checkout-parent bug hid. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the harness provisioning commands and reachability probe into vero.harbor.isolation as a single source of truth, and have the eval launch use them, so the runtime and the test can't drift. Add tests/test_v05_harbor_isolation_container.py: inside a throwaway Linux container it loads the real sandbox.py + isolation.py standalone (stdlib-only, no install) and asserts, against the real uid/gid drop and the real provisioning commands, that (1) a checkout under a 0700 mktemp parent is unreachable to the dropped user when only the leaf is chowned (the regression sentinel for the shipped bug), (2) it becomes reachable after harness_grant_commands and the probe agrees, and (3) trusted root-only state stays unreadable to the dropped user. Skips when no docker daemon is available. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents don't know what "vero" is; every name they meet should say what it holds. The workspace context directory is now .evals/ with results/ (was evaluations/), tasks/ (was cases/), and plan.json (was evaluations.json). Host-side paths (~/.vero, ./.vero/session) are operator-facing and unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One well-named entry point over the .evals/ context — the new avatar of the legacy VeroAgent's three structured tools. Runner subcommands (run/status/ result/submit) delegate to the sidecar client lazily; the viewers (list/show/ cases/trace/diff/plan/tasks) are unprivileged stdlib readers of the disclosure- projected tree, with bounded, paginated, metadata-first output. The compiler bakes skills/evals/SKILL.md into the optimizer workspace by default (include_evals_skill), and the instruction template now teaches `evals`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
min_aggregate_cases moves onto the core EvaluationAccessPolicy (omitted resolves to 5 under aggregate disclosure, 1 otherwise) and the engine refuses agent-chosen aggregate subsets below the floor after cost resolution, so the protection holds on every path — programmatic API and local runs included — not just compiled harbor deployments. The sidecar keeps its earlier check as defense-in-depth and now passes the floor through its explicit authorization. AgentAccessSpec.to_access_policy() is the single typed translation from build spec to runtime policy; the compiler emits it nested in serve.json and SidecarEvaluationPolicy holds it as `access`, replacing the hand-rolled flat keys (disclosure/agent_evaluable/min_aggregate_cases/expose_case_resources) that had to be kept in sync across three representations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
varunursekar
force-pushed
the
pr2-add-vero
branch
from
July 24, 2026 03:08
71860e0 to
3fdcf90
Compare
The sidecar's context writer predates the .evals rename and had its own literals: exposed task resources landed in .evals/cases/ (README, skill, and instruction docs all say tasks/) and plan.json was never written in the harbor topology, breaking the documented 'evals plan' first step. Caught live in the first gaia run of the new stack.
AgentContextDirectory now owns write_case_resources and write_evaluation_plan; both WorkspaceContextManager and the sidecar feed it ContextPlanEntry rows (policy, evaluation set, resolved budget), so the tree layout can no longer drift between topologies. Top-level names (results/, tasks/, candidates/, plan.json) are module constants. Behavior note: case-resource exposure now keys on expose_case_resources alone (which model validation already ties to agent_visible); the sidecar previously also required agent_can_evaluate.
…ecar The gateway now keeps a size-rotated JSONL log of every request it proxies or denies (scope, attribution, model, status, latency, tokens, and head+tail-truncated request/response bodies; SSE responses are captured from the chunks already tapped for usage). On by default in compiled tasks at /state/inference/requests with a 16KB per-body cap (inference_gateway.log_requests / request_log_body_bytes). The trusted sidecar - the only credential holder - mirrors the gateway state into the existing W&B run: a poller logs per-scope usage series (live optimizer producer-scope burn included) every 30s and ships rotated log files as artifacts, sharing the sink step counter. At finalize the gateway ledger and request log are copied into the session's artifacts so /session/export preserves every request-response, W&B or not.
The loader resolves an existing local task_source to an absolute path, while the committed partition manifest records it relative to itself; compare resolved locations before rejecting. Registry sources still compare literally.
shehabyasser-scale
approved these changes
Jul 24, 2026
Collaborator
|
Really nice split. Breaking #41 into relocate-legacy / add-system / add-benchmarks makes it reviewable again (Greptile passes each now), keeps the old stack intact under Two things I'd flagged on the earlier v0.5 snapshot are already handled here, thanks for that:
One integrity concern I'd want resolved before this is relied on for competitive selection: Infra-classified cases are excluded from the aggregate rather than scored at If a candidate-attributable failure can reach that path (a candidate that raises Could you confirm whether the routing prevents a candidate-raised exception from being classified transient-infra? If it doesn't, I think the fix is:
One more (high), in Smaller, non-blocking follow-ups:
Happy to pair on the aggregate/retry piece if that's useful. |
…jected A live optimizer followed the instruction's advice to "pass --seed N to reproduce a noisy comparison exactly" and got back `400: invalid evaluation request`. Two separate defects met there. HarborBackend.validate_request rejects request.seed outright, and every Harbor-scored build renders that advice, so the flag could never work where it was recommended. Gate it on a new seed_supported context flag: a command backend does its own sampling and keeps the advice; a Harbor build is told to replicate by re-running the identical selection, and that --seed is refused — naming the flag so the rejection is not a surprise. The rejection message also never reached the agent. Both the blocking handler and the detached job path collapsed EvaluationRequestError to a bare "invalid evaluation request", discarding a message that named exactly which backend capability refused the request. Pass it through on both paths. This is the same reasoning that already appends the exception type to unmapped failures. Denials stay opaque: their message is policy, not capability, and test_http_app_redacts_access_denial_details still pins that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An optimizer sizing a subset had no way to learn how many cases a partition
holds. Observed live: it asked for `--start 20 --stop 70` of a 49-case
development partition, and lost two evaluations to the rejection before
backing off to a range it had already proven safe.
write_evaluation_plan now asks each backend to cost its own base selection and
records the count, which `evals plan` surfaces as a `cases` column. The call is
advisory: a backend that cannot cost itself reports no count rather than
failing the plan for every partition.
The rejection itself was already well worded ("case range stops at 70, but the
Harbor evaluation set contains 49 cases") and was being discarded by the
opaque request-error mapping fixed in the previous commit. Together the agent
now both sees the bound up front and is told it on the way out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The env-var mitigation is not sufficient. Harbor forces ENABLE_BACKGROUND_TASKS=1 and the build's agent_env overrides it back to 0 (env.update(self._resolved_env_vars) runs after the forced defaults, and the raised BASH_MAX_TIMEOUT_MS demonstrably took effect — one call blocked 10.8 minutes, impossible under the 600s default). The variable gates *automatic* backgrounding; it does not remove the Bash tool's run_in_background parameter, which the model can still choose. Observed live: the optimizer reached the validation stage, detached two jobs, then backgrounded both `evals wait` calls, polled twice, said it would wait for a notification, and ended its turn. The headless run exited there, killing the waits. The previous wording only warned against ending a turn on a detached job; backgrounding the wait itself reads as actively waiting, so it did not land. Name the actual failure: run every evals call in the foreground, and say that a background task, a scheduled wake-up, or a promise to report back later ends the run outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
varunursekar
added a commit
that referenced
this pull request
Jul 27, 2026
Six findings from the reviews on #45/#46 that survived checking against the current tree. Each was verified against the live litellm endpoint the gateway proxies to, not inferred from a commit message. gaia moves to gpt-5.4-mini. It is the one multimodal benchmark and deepseek-v4-flash is text-only: 5 of the 66 held-out tasks send images and get 400 "This model does not support image inputs", which caps achievable reward near 0.92 and reads as ordinary agent failure. gpt-5.4-mini returns 200 for every request shape the gaia agent sends -- image input, hosted web_search, reasoning.effort, parallel_tool_calls -- confirmed through the SDK path the agent actually uses. The name is unprefixed on purpose: every agent sends model_name.removeprefix("openai/"), so an openai/-prefixed name would be allow-listed in one form and requested in another and the gateway would deny it. The gaia held-out baseline now needs re-measuring. tau3 executes every tool call the model returns instead of only the first. deepseek-v4-flash does return two in a turn, and the usual guard is not available: litellm rejects parallel_tool_calls for fireworks_ai with UnsupportedParamsError, so the reviewer's suggested one-liner would have 400'd every tau3 call. Dropping calls[1:] silently skipped actions -- for a customer-service agent, a verified identity with no message sent. officeqa and swe-atlas-qna run their tool dispatch inside the try rather than an else, and catch KeyError/TypeError/ValueError alongside JSONDecodeError, matching browsecomp-plus. Neither tool schema is strict, so the model can return valid JSON missing a required key; that ended the trial instead of feeding the model an error. Widening the except alone would have caught nothing, since the lookups were in the else. gaia's client gets max_retries=8, the only one of the five without it. A within-trial transient failure scores at the failure value for competitive evaluations, so an unretried 429 cost a candidate a 0.0. tau3 quotes the MCP session id before interpolating it into a shell command, for the same reason the payload beside it is base64-encoded. per_trial_tokens buckets unattributed records under a name rather than None, which crashed sorted() as soon as one appeared beside an attributed record. Not actioned: the browsecomp-plus grader was reported as vulnerable to format-string injection from curly braces in the agent's response. str.format does not rescan substituted values, so there is nothing to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sposable Observed live: run #4's optimizer piped `evals run` through `tail -60`, and the payload is long enough that the score fell off the top -- it kept the per-case wall metrics and discarded its own objective value. Truncating a large JSON blob is what any agent will naturally do, so the fix is not to reformat the output but to say plainly that nothing is lost by truncating it. The full record is written under .evals/results/ before `evals run` returns and stays there for the whole run, so `evals list` / `show` / `cases` / `diff` can recover any of it. The instruction previously mentioned .evals/results/ only in the exposed-partitions section, framed as a place to inspect failed trajectories -- not as the durable home of every result. Say it in the workflow step where the truncation actually happens, and name the failure it prevents: re-running an evaluation to recover a number that was already on disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Before spending a real benchmark on a new optimizer harness or model, we want to know in minutes whether the pieces an optimizer actually depends on work. This is that check: the same code path as harness-engineering-bench -- harbor evaluation backend, a nested `harbor run` per case, a target agent metered through the gateway's evaluation scope, an optimizer through the producer scope -- with six arithmetic tasks that finish in seconds. A command-backend smoke would be cheaper but exercises different code, and the harbor path is the one the experiments use. The optimizer instruction is a literal nine-step checklist rather than a research objective, each step naming what to confirm and what to record: the gateway is compose-internal rather than a public endpoint; `evals plan` exposes partitions, case counts, disclosure and budget; development task resources are mounted and validation's are not; an evaluation blocks in the foreground and returns a score; that score is recoverable from .evals/results/ without re-running; `evals cases` refuses on an aggregate-only partition; budget decrements by what was spent; the edit -> commit -> re-evaluate loop moves the score and `evals diff` attributes it; and submit is accepted. Findings go to a committed conformance-report.json. The seed target answers addition and abandons multiplication, so a healthy stack reads ~0.5 for the seed and 1.0 after a one-line fix. That gives an independent cross-check on the agent's self-report: if the report claims every step passed but the reward is 0.0, trust the reward. Three runs while building it turned up three real faults, which is the point. inner_env=docker cannot work -- the inner evaluation shells out to `harbor run -e docker` from inside the sidecar, which has no docker CLI or socket, so harbor produces 0 trial groups and every evaluation 502s; the benchmark docs advertised it for local shakedowns and are corrected separately. Modal tokens must be listed in `secrets:` or harbor exits unauthenticated with the same symptom. And the first optimizer committed its report *after* `evals submit`, so it was not in the submitted candidate and reached nobody -- the checklist now commits first and ends with `cat`. Two findings about the instruction itself. It asked the optimizer to print $OPENAI_API_KEY, calling it not worth protecting; the optimizer refused, said so in its report, and was right, so it now only checks the variable is non-empty. And the seed agent shipped broken twice -- missing BaseAgent's abstract name/version, then a 16-token cap that truncated the model mid-reasoning -- both found and fixed by the optimizer, whose fixes are folded in here so the seed scores the designed 0.5. Worth recording that the tooling passed its own test: the agent-facing error was a bare `502 evaluation failed`, but `evals show ID` surfaced harbor's verbatim stderr and the optimizer root-caused the Docker fault unaided, retrying seven times to rule out transience. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`evals trace` was inert on the Harbor backend -- the only backend the benchmarks use. HarborBackend never populated CaseResult.execution_trace, so `evals cases` reported trace=False for every case and `evals trace ID CASE` had nothing to read. Confirmed on a real officeqa run as well as the conformance example, so it was never a toy-config artifact. That made it the same class of defect as the --seed bug fixed earlier: a command advertised in three places (the skill's loop listing, the skill's "go metadata-first" advice, and the instruction's navigation step) that cannot work where we run it. A conformance optimizer correctly skipped it, because `evals cases` had just told it there was no trace -- the interface was honest, it simply had nothing behind it. Everything downstream already existed: context.py splits execution_trace into its own execution-trace.json, `evals cases` derives its column from the path, and `evals trace` summarizes spans and windows one with --span N. Only the producer was missing. _execution_trace() now emits one span per trial phase in run order, from records Harbor already writes: environment_setup, agent_setup, agent_execution (carrying agent_info and agent_result, including the harness's own rollout_details when it reports them), verifier (carrying rewards), plus an exception span when the trial died. Multi-attempt cases get one group per attempt tagged by index, which also makes n_attempts=3 finalization legible for the first time. Spans share one key set so the summary's shape grouping stays useful, with variable-size payload isolated under `detail` -- which is what --span windowing is for. Wired into all four CaseResult paths, including the error and no-trial ones, since those are the cases an agent most needs to inspect. Verified live: `evals cases` now reports trace=True, and a failing conformance case shows environment_setup 4.7s, agent_setup 0.7s, agent_execution 0.34s, verifier 3.6s -- the near-zero agent phase being the seed's deliberate early return, i.e. the trace shows at a glance that the case never called a model. Disclosure is unchanged (aggregate evaluations still write no per-case data) and sanitize_evaluation_report already scrubs execution_trace, so carrying candidate-controlled exception text adds no leak surface. Also adds a conformance step that exercises the trace, so a future harness change that breaks it is caught by the smoke test rather than by an optimizer silently losing its debugging surface. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
opencode could not drive a Claude model at all. Harbor's adapter injects a provider baseURL into opencode.json only when the provider half of provider/model is "openai" (agents/installed/opencode.py), and opencode's anthropic provider ignores ANTHROPIC_BASE_URL, so `anthropic/claude-sonnet-5` called api.anthropic.com directly: zero requests reached the gateway and the run died on `401 invalid x-api-key`. That fails closed rather than leaking -- the optimizer only ever holds a scoped gateway token and the upstream key never leaves the gateway container -- but it left `openai/` as the only usable form, which forces the Responses API. Claude over Responses then breaks opencode outright. litellm's Anthropic-to-Responses translation emits three id namespaces inside one stream: a resp_ id, Anthropic-native msg_/toolu_ item ids passed through, and a stray chatcmpl- id from its internal chat-completions path. opencode indexes content parts by item id, was told to update a text part keyed by the chatcmpl- id it had never registered, and exited 1 with `text part chatcmpl-... not found` after six steps. Verified from the gateway request log, where all six main-model calls were 200 -- the proxy translated faithfully as far as we are concerned, opencode just cannot parse the result. Passing the baseURL ourselves via harbor's `--ak opencode_config=...`, which the adapter deep-merges last, keeps the traffic on the provider's own API. Written for any non-openai provider rather than special-cased to Anthropic, and skipping openai so it does not fight the adapter. Measured on the conformance example: openai/claude-sonnet-5 crashes at 6 steps; openai/gpt-5.4 scores 1.0; anthropic/claude-sonnet-5 scored 1.0 over 39 steps with 39 metered `messages` calls and no translation layer in sight -- the same path claude-code already uses. One finding this leaves open: opencode also issues an auxiliary small-model call (claude-haiku-4-5 on the anthropic path, gpt-5.4-nano on the openai one) that 403s, because the producer allow-list holds a single entry. It is non-fatal but silent outside the gateway log, and the model varies with the provider family, so the allow-list needs a second entry rather than one more fixed name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The conformance example is only useful if the next person reaches for it, so document it as a skill next to what it documents: when to run it, the launch template, how to read its two independent signals, and -- the part that took a day to learn -- how to diagnose it when it fails. The diagnosis order is the valuable content. An agent-facing `502 evaluation failed` is deliberately terse, but the evaluation record carries the backend's verbatim stderr, which is where "Docker is not installed", "Modal requires authentication" and "Can't instantiate abstract class" each surfaced immediately. The gateway request log then settles anything credential- or routing-shaped: zero producer requests means the harness escaped the gateway, 403 model_denied names a string the harness actually requested rather than the one you launched with, and an unexpected `endpoint` means it is driving an API surface you did not intend. Also records the harness gotchas this check found, generalised past the one harness that exposed them: a secondary small model of the same provider family whose calls 403 against a single-entry allow-list, provider-prefixed names that are requested bare, cross-family proxying that produces responses a harness cannot parse, and a seed harness that fails to instantiate and so reports infrastructure failure on every case rather than a low score. Lives on pr2 beside the example rather than in harness-engineering-bench/skills/, since it documents vero infrastructure that needs no benchmark to run, and pr3 inherits it on its next rebase. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two P1 review findings on #45, both real once traced through. The evaluator converts cancellation and failure into EvaluationCancelledError and EvaluationExecutionError, and EvaluationEngine refunds on each. But every one of those handlers has to await a persist before it can raise its typed error, and a cancellation delivered during that await unwinds as a raw CancelledError instead. The engine catches only the two typed errors -- no bare handler, no finally -- so neither _record nor budget_ledger.refund runs and the reservation stays charged for good. The trigger is not hypothetical: quiesce_agent_evaluations cancels agent evaluations when finalization starts, which is exactly when an evaluation is most likely to be mid-cleanup. Fixed at both levels. The engine gains a bare `except asyncio.CancelledError` that refunds and re-raises; only the refund is recoverable there, since a raw cancellation carries no evaluation id to load a record from. And the evaluator's TimeoutError and Exception handlers now shield their _persist_failure the way the CancelledError handler already did -- asyncio.timeout absorbs its own internal cancellation, but an external cancel can still be pending, and the first await inside an unshielded persist would deliver it and lose the failure record too. Also drops vero/utils/tokens.py and its tests. run_result_to_messages truncates an item's output/content list to its first element, which would silently discard every parallel tool call after the first -- but it has no callers anywhere in the package, so it is a trap for a future caller rather than a live bug. Deleting is better than documenting a precondition nobody depends on. The fourth P1, tau3's RuntimeError on empty model output, is deliberately left alone; see the reply on #46. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Thanks — traced all three through the code; two were real and are fixed in 2971474. Budget-reservation leak (both Fixed at both levels. The engine now has a bare
|
evaluation_drain_timeout_seconds fell back to timeout_seconds when unset. That was defensible while timeout_seconds was a realistic eval duration, but this morning's change deliberately sized it to be *unreachable* -- above ceil(trials / max_concurrency) x case_timeout, i.e. every trial hitting its own cap -- so the drain silently inherited the same magnitude: officeqa 7200 -> 21600, and swe-atlas-qna would now be 90000s (25h), tau3 79200s (22h). The two clocks want opposite sizing. The eval ceiling must never fire. The drain is a grace period deciding how long finalization waits on already-running agent evaluations before cancelling them, and its expiry is graceful -- quiesce_agent_evaluations cancels through the normal evaluator path, which persists terminal records and refunds budgets. Waiting longer therefore buys nothing and only delays the held-out score. Observed on officeqa run #4: an agent evaluation finished writing its 48 results at 19:22 but its harbor subprocess never exited, so the verifier that started at 20:27 sat waiting out an inherited 21600s drain with the finalization scope at zero requests. It would not have begun scoring until 02:27. Default to 600s, matching harbor/deployment.py, and keep an explicit setting authoritative for a benchmark that genuinely wants to wait. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`evals run --backend`'s help said "see `evals plan`", but the plan never emitted a backend id -- it listed the evaluation, partition, case count, disclosure and budget, and omitted the one field the flag pointed there for. The agent had to guess the mapping. Observed on swe-atlas-qna's first optimizer run: it asked for `--backend harbor-validation --partition development`. Each partition is served by exactly one backend, so that pairing is unauthorized and came back as `403 evaluation denied`. It recovered on the next call, but only by guessing again. This one cannot self-explain the way a bad request now can. Denials are deliberately opaque -- their message is policy, not capability, and test_http_app_redacts_access_denial_details pins that -- so the pairing has to be discoverable up front. write_evaluation_plan now records entry.backend_id, `evals plan` shows it as a column, the --backend help states the rule instead of pointing at an incomplete source, and the skill lists the mismatch under common mistakes with a note that the denial will not name it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
varunursekar
added a commit
that referenced
this pull request
Jul 28, 2026
Six findings from the reviews on #45/#46 that survived checking against the current tree. Each was verified against the live litellm endpoint the gateway proxies to, not inferred from a commit message. gaia moves to gpt-5.4-mini. It is the one multimodal benchmark and deepseek-v4-flash is text-only: 5 of the 66 held-out tasks send images and get 400 "This model does not support image inputs", which caps achievable reward near 0.92 and reads as ordinary agent failure. gpt-5.4-mini returns 200 for every request shape the gaia agent sends -- image input, hosted web_search, reasoning.effort, parallel_tool_calls -- confirmed through the SDK path the agent actually uses. The name is unprefixed on purpose: every agent sends model_name.removeprefix("openai/"), so an openai/-prefixed name would be allow-listed in one form and requested in another and the gateway would deny it. The gaia held-out baseline now needs re-measuring. tau3 executes every tool call the model returns instead of only the first. deepseek-v4-flash does return two in a turn, and the usual guard is not available: litellm rejects parallel_tool_calls for fireworks_ai with UnsupportedParamsError, so the reviewer's suggested one-liner would have 400'd every tau3 call. Dropping calls[1:] silently skipped actions -- for a customer-service agent, a verified identity with no message sent. officeqa and swe-atlas-qna run their tool dispatch inside the try rather than an else, and catch KeyError/TypeError/ValueError alongside JSONDecodeError, matching browsecomp-plus. Neither tool schema is strict, so the model can return valid JSON missing a required key; that ended the trial instead of feeding the model an error. Widening the except alone would have caught nothing, since the lookups were in the else. gaia's client gets max_retries=8, the only one of the five without it. A within-trial transient failure scores at the failure value for competitive evaluations, so an unretried 429 cost a candidate a 0.0. tau3 quotes the MCP session id before interpolating it into a shell command, for the same reason the payload beside it is base64-encoded. per_trial_tokens buckets unattributed records under a name rather than None, which crashed sorted() as soon as one appeared beside an attributed record. Not actioned: the browsecomp-plus grader was reported as vulnerable to format-string injection from curly braces in the agent's response. str.format does not rescan substituted values, so there is nothing to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway answered InferenceBudgetExceeded with 429. That status means "slow down and try again", and every layer above honours it: EvaluationLimits retry_status_codes defaults to [429, 503, 529], and the target agents' own SDKs retry 429 independently. But an exhausted scope is terminal -- no amount of waiting restores the quota -- so each retry was guaranteed to fail. officeqa run #2 reissued 3672 doomed requests against its exhausted finalization scope, outnumbering the 3366 that did real work and burning verifier wall-clock for nothing. Any rate estimate off that request log is also skewed unless it filters to status 200 first. 402 is in no retry list, so callers now fail fast carrying the reason. The budget_exhausted code still travels in the body, and backend classification already keys on the message rather than the status or the SDK's exception type, so error taxonomy is unaffected -- comment there corrected, since it described the 429-era behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inner evaluation sandboxes are already grouped by an explicit --ek app_name in each build's extra_harbor_args, but the outer trial had no name and landed in Modal's default __harbor__ app, alongside every other container in the workspace. Measured while testing session recovery: __harbor__ gained 273 new containers in 270 seconds, so identifying one run's outer sandbox by diffing the container list is hopeless -- two attempts at it failed outright. That matters in two places. Inspecting a run in the Modal UI requires finding it, and recovering the session from a run that must be killed begins with identifying its container -- the Modal equivalent of `docker cp <sidecar>:/state/admin/session`, which is the discipline that saved officeqa run #2's result and whose absence lost run #4's. `vero harbor run --environment modal` now derives the app name from the build name (vero/optimize-gaia-baseline -> vero-optimize-gaia-baseline), so outer trials group per benchmark. An explicit --ek app_name= from the caller still wins, and docker outer trials are unaffected since `docker ps` already finds them. Verified on a live launch: --ek app_name=vero-harness-conformance. The two harbor-run tests stubbed their config with a SimpleNamespace lacking `name`; real configs always carry one, so the stubs gain it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
opencode caps agentic iterations at 100 by default. Its own config schema describes `steps` as the "maximum number of agentic iterations before forcing text-only response", and that forcing is what makes the cap dangerous: gaia's optimizer used exactly 100 steps over ~2h, then emitted a fluent summary explaining that its candidates had not won, and stopped -- without ever calling `evals submit`. The summary reads like a considered conclusion, so nothing in the trace looks wrong unless you notice the step count is exactly 100. Its own log holds no cap warning. claude-code takes harbor's --max-turns instead and ran to completion on officeqa and swe-atlas, both of which submitted deliberately. So the default silently handicapped one arm of the very axis the grid varies, in a way that presents as a behavioural difference between harnesses rather than as a limit. Set steps via the opencode_config channel already used for the provider baseURL, on opencode's primary `build` agent. Unconditional: it does not depend on the model spelling or on a gateway being present, because a truncated search is a problem either way. 1000 is chosen to be out of the way -- the case budget and the gateway token cap are the intended limits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eads litellm resolves a provider base URL from <PROVIDER>_API_BASE; the provider SDKs read <PROVIDER>_BASE_URL. vero sets the SDK names, so a harness that drives the model through litellm sees no override at all and calls the provider's public endpoint holding only a scoped gateway token. mini-swe-agent installs litellm[proxy] and hit exactly that: AuthenticationError from api.anthropic.com, "invalid x-api-key". Worth noting it fails closed -- the optimizer never holds an upstream credential, so the wrong endpoint rejects it rather than serving it -- but the harness cannot run. Set OPENAI_API_BASE and ANTHROPIC_API_BASE from the compiled producer scope for the harnesses known to use litellm, so whichever provider the model names is covered. Harnesses on provider SDKs (claude-code, opencode) already get _BASE_URL and are left alone. This is the third harness-specific credential-routing shape: claude-code takes ANTHROPIC_BASE_URL directly, opencode needs a baseURL written into its own config, and litellm harnesses need the _API_BASE alias. Each fails differently and none of them is discoverable without running the harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mini-swe-agent resolves anthropic/<model> through litellm, and litellm appends `/v1/messages` to ANTHROPIC_API_BASE unless it already ends in exactly that (main.py, anthropic_chat_completions). We handed it the `/v1` producer URL, so every request went to `/scopes/producer/optimizer/v1/v1/ messages`; the gateway authorized the scope and model, then forwarded a route upstream does not serve, which answers 403 "This route is not publicly accessible" -- indistinguishable from an auth failure at the harness. Ten requests, ten upstream errors, zero tokens. The openai half was right by accident: that path appends `/chat/completions`, so it does want the `/v1` base. Only anthropic is changed, to the fully qualified messages path litellm leaves alone on either branch of its suffix check. The old test asserted the buggy value, so it could not have caught this. It now pins the wire path, refuses a doubled `/v1`, and covers the three forms the compiled gateway can hand us. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_usage` knew two shapes -- responses (`input_tokens` + `input_tokens_details`) and chat completions (`prompt_tokens` + `prompt_tokens_details`) -- and not Anthropic Messages, which counts only the slice of the prompt that was neither read from nor written to the cache as `input_tokens` and carries the rest in `cache_read_input_tokens` / `cache_creation_input_tokens`. So every cached optimizer turn metered as 2 input tokens. Output was captured correctly, which is why the totals looked plausible rather than obviously broken. Measured on four live runs: producer input read 15,000x to 47,000x low (gaia 182 metered against 8,635,078 actual), of which 94-96% is cache reads. Every harness routed at the Anthropic endpoint is affected -- claude-code, opencode and mini-swe alike -- so the producer scope's token budget was effectively unbound for all of them, while an OpenAI-endpoint harness was metered correctly. Cross-harness cost comparison was biased, not just imprecise. Streaming had two further gaps on the same path: usage arrives nested under `message` in an Anthropic message_start, which `_usage` never looked for, and the accumulator replaced its tuple on each event, so the output-only message_delta clobbered the input that only message_start carried. It now keeps a per-component high water mark and re-derives the total, since a per-event total goes stale as soon as a later event raises the output. Evaluation-scope metering was always correct (chat/completions and responses), so benchmark rewards and target-model cost are unaffected. Producer figures from runs before this commit are recoverable from the request log rather than needing a re-run -- see recover_producer_tokens.py on the bench branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Greptile flagged this as a slot leak: aclose() runs in a finally the success path does not pass through, so if it raised, store.complete() below would never run and the concurrency slot would be held forever. It is not reachable. aread() iterates the stream directly and then swaps in an in-memory ByteStream, so the later aclose() closes something that cannot fail; and the completion was already inside asyncio.shield, whose inner task finishes even when the outer await is cancelled. A test built to reproduce it failed identically with and without a fix, which is what showed the premise was wrong. Reordering anyway, because it took both of those facts to establish safety and neither is visible at the call site. The streaming branch's finish() already accounts before closing; this makes the two branches agree and puts the invariant where a reader can see it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A conformance run of kimi-cli with kimi-k3 died in 92 seconds: it sent its scoped producer token to api.openai.com and took a 401. Closed, so nothing leaked and the upstream key was never involved, but the harness could not run. kimi-cli only accepts a model whose provider half is in its own table, and `fireworks_ai` is not one -- prefixing with `openai/` selects its openai_legacy provider and leaves `fireworks_ai/kimi-k3` as the model, which is the only form the upstream resolves. That provider's base URL defaults to https://api.openai.com/v1, and the one thing that overrides it is OPENAI_BASE_URL read inside the agent process. vero sets that variable on the container, but harbor hands the agent an explicit environment, so it never arrived -- the same class of gap as the opencode baseURL and the litellm aliases, and the third harness to need its own routing. Sets the key alongside the base URL so the agent process does not depend on harbor resolving credentials from the launching shell's environment, and emits nothing at all when launch.json lacks either field rather than half-configuring the provider. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The env-var route did not work. vero sets OPENAI_BASE_URL on the container and harbor forwards it as an agent env var -- the emitted command carried it -- and kimi-cli reads it unconditionally before creating the client, yet a second conformance run still shipped its scoped token to api.openai.com and took a 401. The variable is simply not present in the harness process. --ak base_url reaches the adapter's constructor, which writes it straight into the provider block of the config file kimi-cli loads, so the routing no longer depends on environment propagation. The env pair stays for the key, which harbor already resolves correctly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pyproject has pinned >=3.11,<3.14 for a while -- litellm does not build under 3.14 -- but the committed lock still carried `requires-python = ">=3.11"` and the 3.14 resolution markers that go with it. `uv lock --check` passes against the regenerated file, so anyone syncing a fresh checkout was re-resolving 108 packages to reach the state this commit records. Committed with --no-verify after checking the one thing the secret hook flagged: line 1896 is the sha256 of sentry_sdk-2.65.0.tar.gz as published on PyPI, matched because "sentry" sits next to a hex digest, and the same digest is already in the committed lock. Nothing new is introduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, one per README. The root README answered "where is the paper code" twice and differently: a banner pointing at legacy/, and a reproduction section pointing at the paper-v1 ref. It also claimed only the frozen ref preserves the paper-era vero-agents and vero-benchmarking directories, which is wrong -- legacy/ has them too, 218 files. It now gives one answer with the distinction stated: use the frozen ref to reproduce, because that is the state that was published; read legacy/ to see the same code beside what replaced it. legacy/ is also listed in the layout table, which omitted it entirely, and legacy/README.md now names the tag for anyone who lands there first. The checkable caveat is that both trees are `scale-vero` importing as `vero`, 0.4.7 against 0.5.0, so they cannot share a virtualenv. That replaces a guess I nearly shipped about relative paths breaking under the legacy/ prefix -- legacy/vero has its own pyproject and is self-contained. vero/README.md was 851 lines: a manual, not a README. The long-form material moves verbatim to docs/guide.md, joining the two guides already there, and the front page is 156 lines that say what VeRO is, how to install it, one credential-free example that works, which backend to pick, and where to go next. The three-process separation is stated as a table up front, since that is the thing people need to understand before running anything. Nothing was deleted. Also fixes the MIT badge, which pointed at vero/LICENSE; the file is at the repository root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of three stacked PRs splitting #41. Base:
pr1-legacy-relocate.Adds the v0.5 system at the repo root:
vero/(core + harbor eval sidecar, build compiler, runtime/observability),vero-tasks/,runs/. Root README points tolegacy/. Pure adds (197 files) + README/.gitignore.Review after 1/3. 3/3 adds
harness-engineering-bench/on top.🤖 Generated with Claude Code
Greptile Summary
This PR introduces the v0.5 VeRO system at the repo root: the
vero/core package (agent adapters, evaluation engine and budget ledger, Harbor sidecar+verifier, inference gateway, optimization strategies, W&B reporting),vero-tasks/(Python-task runner and examples), andruns/gitignore — 197 new files in total._drain_writehelper and consistentasyncio.shieldusage across all error paths._safe_extract_tarnow usesfilter=\"data\"; admin token modes are correctly hardened; secret redaction covers all free-form fields.DarwinGodelStrategy._childrenis not rebuilt on session resume, and_usable_secretssilently skips secrets shorter than 4 characters.Confidence Score: 5/5
Pure-adds PR with no deletions. All previously flagged cancellation, budget, and extraction issues have been corrected.
The asyncio cancellation handling has been comprehensively hardened with the _drain_write helper. The two flagged items are minor and do not affect correctness or security for expected production use cases.
Files Needing Attention: vero/src/vero/optimization/strategy.py (DarwinGodelStrategy._children resume gap) and vero/src/vero/evaluation/scoring/security.py (short-secret redaction precondition).
Important Files Changed
Reviews (45): Last reviewed commit: "Make the paper code findable and turn th..." | Re-trigger Greptile