feat(v1): support Harbor's separate verifier environments#2067
feat(v1): support Harbor's separate verifier environments#2067xeophon wants to merge 16 commits into
Conversation
Honor `[verifier].environment_mode = "separate"` / `[verifier.environment]` in the v1 Harbor taskset: grade in a fresh container/sandbox on the same provider (docker/prime/modal), derived from the agent runtime's resolved config. Mirrors Harbor semantics: mode resolution and validation, artifact transfer (/logs/artifacts + top-level artifacts, main service only) at identical absolute paths, reward.json-first read order in separate mode, and the pre_artifacts.sh capture hook used by Pier-style datasets (e.g. DeepSWE v1.1). Shared-mode behavior is unchanged; no new config knobs.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review This PR adds a substantial new feature for separate verifier environments, introducing new runtime lifecycle methods, modified scoring flow, and artifact transfer logic between runtimes. The scope of new capability and runtime behavior changes warrants human review. You can customize Macroscope's approvability policy. Learn more. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 662b48db5b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- move verifier.start() inside the try so a failed/cancelled start still reaches the finally's stop() (no leaked sandbox/container) - _read_reward: catch TypeError too (non-object reward.json, null reward) so it falls back to reward.txt instead of aborting scoring - _transfer_artifacts: check both tar exit codes and raise SandboxError instead of grading against silently incomplete artifacts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61beb0b5b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- _grade wipes /tests before extracting the staged tests tarball, so files baked into a verifier image can't linger and contaminate grading - parse_verifier fails fast on separate-mode tasks declaring [[verifier.collect]] hooks (compose sidecars aren't supported) instead of silently grading without the collected evidence
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8c5eeb62b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A separate-mode task whose artifacts entries target a compose service can't be graded faithfully without the sidecar's files — fail fast like the collect-hook guard instead of warn-and-skip.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 789a40d02e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- HarborData.verifier: VerifierConfig | None (image/resources/workdir) — None grades shared; replaces the flat verifier_* fields (review) - TODO to port the separate verifier runtime into a judge env once multi-agent lands (review) - separate-mode reward.json without a 'reward' key but with an all-numeric mapping (Harbor's multi-metric form) is returned as keyed rewards instead of falling through to 0.0 - warn at load when [verifier]/[verifier.environment] declares a network policy (not enforced; rejecting would break DeepSWE's uniform allow_internet=false, which its graders don't depend on)
The probe always exits 0 when it runs, so a nonzero exit means the agent runtime is gone (e.g. dead container) — raise SandboxError instead of skipping the transfer and grading a clean verifier.
Harbor applies excludes per entry, relative to the artifact root (-C <src> .); the single combined transfer tar can't reproduce that scoping, so fail fast at load instead of materializing files the task explicitly excluded.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7caabea43
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Structured artifacts entries may declare exclude patterns; apply them the way Harbor does — per entry, relative to the artifact's root, and to directory artifacts only. Plain entries still share one combined tar; an entry with excludes gets its own tar (-C <source> .) so the patterns scope correctly. File artifacts with excludes travel with the combined tar (Harbor ignores excludes on files too).
A present [verifier.environment] replaces the task's [environment] wholesale (Harbor): resource fields it omits are no longer inherited from the agent task. Fields whose value matches the task's declared resources (task-set) reset to the runtime default; cli/toml overrides (mismatches) keep winning. Mode-only separate stays a fresh copy and keeps inheriting.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 48ed242ed9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…envs - keyed reward.json values must be int/float by exact type (JSON booleans are not rewards; fall back to reward.txt) - relative artifact sources resolve against the agent runtime's workdir before probing/archiving, so the combined tar packs and re-materializes the right absolute path - an explicit [verifier.environment] without docker_image is rejected even without tests/Dockerfile (Harbor would build or fail; the agent image is not a faithful fallback unless ignore_dockerfile opts in)
A failed hook doesn't invalidate /logs/artifacts (also the agent's own publish directory), so partial artifacts still transfer — say that instead of claiming the base state is graded.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e83057ab44
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d20fa0fc8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 845cd50. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 845cd5033b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: adff2dcc3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ["tar", "-czf", ARTIFACTS_TAR, "-C", "/", "--"] | ||
| + [(path.lstrip("/") or ".") for path in combined], |
There was a problem hiding this comment.
When a separate-mode task declares a plain directory artifact that contains /tmp/.vf-artifacts.tgz — for example artifacts = ["/tmp"] or a relative artifact resolving under /tmp — this command writes the archive inside the tree being archived. GNU tar exits nonzero in that shape (file changed as we read it), so _archive_tar() raises and the verifier never runs even though Harbor allows directory artifacts at their original paths; create the temporary archive outside any collected source or exclude the archive path.
Useful? React with 👍 / 👎.
| f"{task_dir.name}: [verifier.environment] requires a separate verifier " | ||
| '(environment_mode = "separate", or no explicit mode, which it implies)' | ||
| ) | ||
| fields: dict[str, Any] = {"verifier_env": verifier.get("env", {})} |
There was a problem hiding this comment.
When a separate Harbor task sets [verifier].user to a user different from the verifier image default, this parser preserves only verifier.env and never applies or rejects the requested user. Harbor’s task schema documents this field for verifier execution, but Docker/Prime will run bash test.sh as the image/default user here, so permission-sensitive graders can pass or fail differently from Harbor; thread the user through runtime execution or fail fast for separate mode when it is set.
Useful? React with 👍 / 👎.
2 participants
Overview
Adds Harbor v1 support for tasks that grade in a separate verifier environment. Shared tasks keep grading in the agent runtime; separate tasks run their verifier in a fresh runtime on the same provider, using the task's Harbor configuration without adding new run-time knobs.
Details
[verifier].environment_modeand[verifier.environment]./logs/artifactsand declared main-service artifacts at their original paths, including per-artifact exclude patterns.pre_artifacts.shcapture hook before artifact transfer.reward.jsonvalues or numeric metric maps, then falls back toreward.txt; shared-mode reward behavior remains unchanged.This enables Harbor datasets with isolated grading environments, including fresh-copy and dedicated verifier-image configurations, while preserving the explicit artifact boundary between agent work and grading.
Note
Overview
Adds Harbor v1 support for tasks that grade in a separate verifier environment. Shared tasks keep grading in the agent runtime; separate tasks run their verifier in a fresh runtime on the same provider, using the task's Harbor configuration without adding new run-time knobs.
Details
[verifier].environment_modeand[verifier.environment]./logs/artifactsand declared main-service artifacts at their original paths, including per-artifact exclude patterns.pre_artifacts.shcapture hook before artifact transfer.reward.jsonvalues or numeric metric maps, then falls back toreward.txt; shared-mode reward behavior remains unchanged.This enables Harbor datasets with isolated grading environments, including fresh-copy and dedicated verifier-image configurations, while preserving the explicit artifact boundary between agent work and grading.
Changes since #2067 opened
tasksets.harbor.HarborTask[32de5e2]tasksets.harbor.HarborTask._read_rewardmethod [32de5e2]tasksets.harbor.HarborTask._transfer_artifactsmethod [32de5e2]tasksets.harbor.parse_verifierfunction [32de5e2]tasksets.harborpydantic models and methods [32de5e2]HarborTask._grademethod to insert end-of-options marker and handle empty path strings [0d20fa0]parse_verifierfunction to use top-level configuration environment as fallback [adff2dc]Note
Medium Risk
Changes rollout scoring order and lifecycle for tasks that return a scoring runtime config, and adds multi-runtime Harbor flows that depend on confirmed teardown and artifact tar/restore correctness.
Overview
Adds Harbor “separate verifier” grading: tasks can score in a fresh docker/prime runtime instead of reusing the agent sandbox, with artifacts copied across after confirmed agent teardown.
Rollout / task plumbing:
Task.scoring_runtime_config()lets a task opt into a replacement scoring runtime. When set, rollout scores the harness on the live agent runtime first, then runs task scoring (which may swap runtimes) instead of parallelgather. Scoring timeouts useasyncio.timeoutinstead ofwait_foraroundgather.Runtimes: Docker and Prime implement
stop_confirmed()so Harbor can tear down the agent environment only after deletion is verified before starting the verifier.Harbor: Parses
[verifier].environment_mode,[verifier.environment], and[[artifacts]]intoVerifierConfig/ArtifactonHarborData. Separate mode runs optionalpre_artifacts.sh, tars/logs/artifactsand declared paths (with exclude patterns),stop_confirmed()on the agent, boots a derived verifier runtime, restores archives, stagestests/, and runstest.sh. Rewards prefer numericreward.jsonin separate mode; shared mode still usesreward.txt. Unsupported cases (subprocess runtimes, Dockerfile-only verifier builds, compose sidecars, collect hooks) fail clearly; declared verifier network policies only warn.Reviewed by Cursor Bugbot for commit adff2dc. Bugbot is set up for automated code reviews on this repo. Configure here.
Fixes RES-1093