From 72bb132fe1284ec21d236f817a16026aa729f10b Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Tue, 28 Jul 2026 08:09:39 +0300 Subject: [PATCH] feat: add a subsampled swe-bench-pro variant (33/66/66) for optimizer runs Additive only. The canonical baseline/build.yaml and its 146/292/293 split are byte-unchanged; this adds a second config beside them. Why: the full split is unaffordable for an optimizer loop. Each case builds a real repository and runs its test suite, so the full-split held-out baseline alone cost ~7.8h of wall clock for three rounds, and an optimizer evaluates candidates many times over. 33/66/66 matches gaia and browsecomp-plus, the suite's tightest split, which is the right reference for its most expensive benchmark. The sample is NESTED, which is the load-bearing detail. `--sample` computes the canonical split first, unchanged, then narrows each partition within itself, so partitions/sample/test.json is a strict subset of partitions/test.json. Two things follow. No instance can migrate from test into development: a flat re-split at the smaller size does move them, and the first version of this change produced a "held-out" set with zero overlap with the real one, silently drawn from what had been dev and validation. And the baseline needed no re-measuring. baseline_reward: 0.2923 over the 66 sampled cases, recomputed from the very same K=3 trials that produced the canonical 0.2940 +-0.0082. The recomputation reproduces 0.2940 +-0.0082 exactly on the full set first, which is what validates it. The point estimate barely moves; the spread widens to +-0.0332 because n falls 293 -> 66. CAVEAT recorded in the file: those trials targeted qwen-3.6-27b while the variant targets deepseek-v4-flash, so 0.2923 is provisional until a K=3 deepseek pass over the 66 replaces it (~1.8h at max_concurrency 24). Repository stratification and sha256 ordering are unchanged and apply within each partition, so all 11 upstream repos appear in every sampled partition. The sample shares the canonical manifest rather than duplicating it: it pins the same 731 refs and the build loader reads only names from it, so a second copy would just be a second thing to keep in sync. `--tasks-dir` is now optional, since the manifest already records every name and stratum. The variant also carries the settings the canonical config is missing relative to its siblings: deepseek-v4-flash (prefixed-only, because agents strip just `openai/` before the gateway matches exactly), max_concurrency 24, 4x case budgets (132/264), case_timeout equal to task_agent_timeout at 3000 so the multiplier is 1.0, score_baseline false with the baseline pinned, and n_attempts 3 / mean on the test target. CONFIGURATION.md gains the swe-bench-pro column it never had, describing the canonical config, plus a footnote covering the variant and two pre-existing inconsistencies in the canonical file that are deliberately left alone here: case_timeout_seconds 1800 implies a 0.6 multiplier though the pinned 0.294 was measured at 1.0 (all 879 sampled trial configs say so), and model: gpt-4o though that baseline was measured against qwen-3.6-27b. Verified: canonical partitions, manifest and build.yaml show no diff against pr3; canonical --check passes; --sample regenerates byte-identically; every sampled partition is a strict subset of its canonical counterpart; both configs compile side by side (full -> 146/292/293, gpt-4o, 1800; sample -> 33/66/66, deepseek, 3000, baseline 0.2923, budgets 132/264, n_attempts 3/mean on test only); 13 passed, 6 skipped in test_v05_benchmark_configs.py. --- harness-engineering-bench/CONFIGURATION.md | 86 +++++++--- .../swe-bench-pro/baseline/build.sample.yaml | 155 ++++++++++++++++++ .../partitions/sample/development.json | 35 ++++ .../swe-bench-pro/partitions/sample/test.json | 68 ++++++++ .../partitions/sample/validation.json | 68 ++++++++ .../scripts/partition_swe_bench_pro.py | 133 ++++++++++++++- 6 files changed, 518 insertions(+), 27 deletions(-) create mode 100644 harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml create mode 100644 harness-engineering-bench/swe-bench-pro/partitions/sample/development.json create mode 100644 harness-engineering-bench/swe-bench-pro/partitions/sample/test.json create mode 100644 harness-engineering-bench/swe-bench-pro/partitions/sample/validation.json diff --git a/harness-engineering-bench/CONFIGURATION.md b/harness-engineering-bench/CONFIGURATION.md index e28dc749..1cbd1549 100644 --- a/harness-engineering-bench/CONFIGURATION.md +++ b/harness-engineering-bench/CONFIGURATION.md @@ -157,25 +157,25 @@ benchmark can be checked against the others at a glance. ## Per-benchmark values -| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus | -|---|---|---|---|---|---| -| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash | -| held-out baseline (K=3) ◆ | 0.621 ±0.052 | 0.341 ±0.033 | 0.068 ±0.026 (agg 0.632) | 0.732 ±0.010 | 0.462 ±0.028 | -| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 | -| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 | -| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 | -| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B | -| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 | -| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 | -| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 | -| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 | -| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | -| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | -| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | -| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | -| harness_user | harness | harness | null ‡ | null ‡ | null ‡ | -| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | -| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | +| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus | swe-bench-pro | +|---|---|---|---|---|---| --- | +| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash | gpt-4o ◈ | +| held-out baseline (K=3) ◆ | 0.621 ±0.052 | 0.341 ±0.033 | 0.068 ±0.026 (agg 0.632) | 0.732 ±0.010 | 0.462 ±0.028 | 0.294 ±0.008 ◈ | +| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 | 146/292/293 ◈ | +| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 | 100 / 146 ◈ | +| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 | 100 / 292 ◈ | +| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B | 100 M | +| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 | 8 ◈ | +| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 | 28800 | +| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 | 1800 ◈ | +| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 | 3000 | +| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | n/a (registry dataset) | +| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | n/a (registry dataset) | +| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 28800 | +| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | n/a | +| harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness | +| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false | +| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | registry dataset; `expose_case_resources: false`; sampled variant ◈ | ## Conventions @@ -271,6 +271,52 @@ agent's reason/search-only-turn crash was fixed. **gaia and tau3 are too noisy for single-run comparisons.** gaia's own three rounds spanned 0.554-0.682 (sd 0.052), and tau3's optimizer scored one *unchanged* harness at 0.800 and 0.547 on development -- its user-simulator and NL-assertion grader are both LLMs, so their variance rides on every eval. Treat a gaia or tau3 delta under ~0.1 as unresolved. Their splits are not the problem: domain mix matches to the percentage point across all three partitions (airline 13%, banking 26%, retail 30%, telecom 31%), as does telecom persona difficulty. +◈ swe-bench-pro is the newest benchmark and the only one with **two** configs. +The column above describes the canonical `baseline/build.yaml` (full 731-instance +dataset, split 146/292/293). Alongside it sits `baseline/build.sample.yaml`, a +subsampled variant for optimizer runs; the canonical config is unchanged by it. + +*Why a variant.* The full split is unaffordable for an optimizer loop: each case +builds a real repository and runs its test suite, so the held-out baseline alone +cost ~7.8h of wall clock for three rounds, and an optimizer evaluates candidates +many times over. The variant is 33/66/66, matching gaia and browsecomp-plus, on +`fireworks_ai/deepseek-v4-flash`, `max_concurrency: 24`, 4x case budgets +(132/264), and `case_timeout_seconds: 3000`. + +*The sample is nested.* The full split is computed first and each partition is +then narrowed within itself (`partition_swe_bench_pro.py --sample`), so +`partitions/sample/test.json` is a strict subset of `partitions/test.json`. No +instance can migrate from test into development, and the baseline needed no +re-measuring: **0.292 ±0.033** over the 66 sampled cases is the same K=3 trials +that produced the full-partition 0.294 ±0.008, recomputed. The point estimate +barely moves; the spread widens roughly fourfold because n falls 293 → 66, which +matters when reading a candidate's margin over that floor. + +*Three known inconsistencies in the canonical config*, all pre-existing and left +alone here rather than silently changed: + +- `case_timeout_seconds: 1800` against `task_agent_timeout_seconds: 3000` implies + a 0.6 multiplier, but the pinned 0.294 was measured at **1.0**: all 879 sampled + trial configs record `agent_timeout_multiplier=1.0`. The config value is wrong, + not the measurement. Every sibling has these two equal. +- `model: gpt-4o`, but the 0.294 baseline was measured against + **qwen-3.6-27b**. So the recomputed 0.292 the variant pins is also a qwen + number while the variant targets deepseek; treat it as provisional until a K=3 + deepseek pass over the 66 replaces it (~1.8h at `max_concurrency: 24`). +- **The pinned 0.294 was measured on a different agent than the one in the tree**, + which is the same staleness the note above describes, in its strongest form. The + qwen-era agent was 630 lines on Chat Completions with an explicit local + `messages` list and edited files on 571 of 879 cases (write_file 1108, + apply_patch 899, submit 85). The committed seed is Responses-only, delegates its + conversation to `previous_response_id`, and edits nothing: over 66 sampled cases + it called write_file, apply_patch, and submit **zero** times and scored 0.0000. + So this benchmark is the one the re-pin above did not cover, and it needs + `scripts/rescore_candidate.py --seed` more than any of the five that did. + +Also unlike the siblings, swe-bench-pro's canonical config still has +`score_baseline: true`, so it re-measures the seed every run instead of pinning +it. The variant sets `false` and pins, as everything else does. + ◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and that model is text-only. Verified against the same litellm endpoint the gateway proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent @@ -320,7 +366,7 @@ per benchmark from its own case counts; see each `build.yaml` for the arithmetic the per-key bucket, so more keys buy proportionally more parallel runs: | counter | limit | scope | -|---|---|---| +|---|---|---| --- | | `x-ratelimit-api_key-limit-tokens` | 10 M TPM | **per key** | | `x-ratelimit-api_key-limit-requests` | 5 000 RPM | **per key** | diff --git a/harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml b/harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml new file mode 100644 index 00000000..4599e2c3 --- /dev/null +++ b/harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml @@ -0,0 +1,155 @@ +# Subsampled SWE-bench-Pro variant, for optimizer runs. +# +# This is a SECOND configuration alongside build.yaml, not a replacement. The +# canonical build.yaml keeps the full 146/292/293 split and its held-out +# baseline of 0.2940 +-0.0082; nothing here touches it. +# +# Why a variant exists: the full split is unaffordable for an optimizer loop. +# Each case builds a real repository and runs its (often slow) test suite, so +# the full-split held-out baseline alone cost ~7.8h of wall clock for three +# rounds, and an optimizer evaluates candidates many times over. 33/66/66 +# matches gaia and browsecomp-plus, the suite's tightest split. +# +# The sampled partitions are NESTED inside the canonical ones: the full split is +# computed first and each partition is then narrowed within itself, so +# partitions/sample/test.json is a strict subset of partitions/test.json. No +# instance can migrate from test into development, and the baseline below did +# not need re-measuring. +name: vero/optimize-swe-bench-pro-sample +description: >- + Improve a code-editing SWE-bench-Pro agent while preserving the Harbor agent + interface. The target must edit a checked-out repository so the task's hidden + test suite passes; the task-source verifier runs the suite for the reward. +agent_repo: target +# SWE-bench-Pro ships as the `swebenchpro` dataset in the DEFAULT Harbor registry +# (731 instances), not as an `/@sha256:` package like +# swe-atlas-qna or tau3. Its tasks resolve to git-backed task ids under +# laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1, +# so the version pin is the registry version rather than a content digest. +task_source: swebenchpro@1.0 +# Shared with the canonical split on purpose: the manifest pins all 731 refs and +# the build loader reads only task names from it, so a second copy would just be +# a second thing to keep in sync. +task_manifest: ../partitions/manifest.json +agent_import_path: swebench_pro_agent.agent:SweBenchProAgent +harbor_requirement: harbor[modal]==0.20.0 + +partition_files: + development: ../partitions/sample/development.json + validation: ../partitions/sample/validation.json + test: ../partitions/sample/test.json + +agent_access: + - partition: development + disclosure: full + # false, unlike the sibling benchmarks: VeRO materializes case resources via + # PackageDatasetClient, which only accepts `/` package refs. A + # registry dataset such as swebenchpro@1.0 cannot be exposed that way. + expose_case_resources: false + total_runs: 100 + total_cases: 132 # 4x the partition, the suite-wide budget convention + - partition: validation + disclosure: aggregate + expose_case_resources: false + min_aggregate_cases: 5 + total_runs: 100 + total_cases: 264 # 4x the partition, the suite-wide budget convention + +selection_partition: validation +targets: + - partition: test + reward_key: reward + # Held-out K=3 mean over the 66 sampled cases. Recomputed from the very same + # trials that produced the canonical full-partition 0.2940 +-0.0082, which is + # sound only because the sample is a strict subset; the recomputation + # reproduces 0.2940 +-0.0082 exactly on the full set first. + # + # CAVEAT: those trials targeted qwen-3.6-27b, whereas this variant targets + # deepseek-v4-flash. Treat 0.2923 as provisional until a K=3 pass on deepseek + # over these 66 cases replaces it (~1.8h at max_concurrency 24). + # + # The +-0.0332 is not a typo: it is 4x the full split's +-0.0082 because n + # falls 293 -> 66. Worth remembering when reading a candidate's margin. + baseline_reward: 0.2923 + failure_value: 0.0 + max_attempts: 1 + # The held-out eval is noisy: score the selected candidate 3x per case and + # average, so the final reward is comparable to the pinned baseline, which was + # itself pooled over 3 rounds. Without this the candidate carries ~sqrt(3) more + # standard error than the floor it is judged against. + # Per-target override - search/validation keep the global n_attempts (1). + n_attempts: 3 + aggregate_attempts: mean + +evaluation_set_name: swe-bench-pro-sample +objective: + selector: + metric: score + direction: maximize +reward_mode: submit # agent picks; falls back to auto_best, then current version +baseline_floor: false # gates on validation while reward is on test; opt-in only +score_baseline: false # pinned as baseline_reward above, like every sibling +rescore_top_k: 3 +rescore_attempts: 1 + +model: fireworks_ai/deepseek-v4-flash +environment_name: ${inner_env:-modal} +# inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default +extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"] +harbor_python_version: "3.12" +n_attempts: 1 +max_retries: 1 +infrastructure_max_attempts: 3 +infrastructure_retry_delay_seconds: 5 +aggregate_attempts: best +feedback_transcripts: true +feedback_max_bytes: 16000 +expose_attempt_detail: false +# Timeouts are bumped well above the GAIA baseline: SWE-bench-Pro tasks build a +# real repository and run its (often slow) test suite, so both the per-case and +# the verifier budgets need substantially more headroom than a short-answer task. +# Both mirror the task package's own `[agent] timeout_sec = 3000`, so the +# multiplier is 1.0, matching every sibling benchmark. (The canonical build.yaml +# still reads 1800 here, implying 0.6, but its published 0.2940 was in fact +# measured at 1.0: all 879 sampled trial configs record +# agent_timeout_multiplier=1.0. That file's value is wrong and worth a separate +# one-line fix; this variant is simply spelled correctly from the start.) +timeout_seconds: 28800 +case_timeout_seconds: 3000 +task_agent_timeout_seconds: 3000 +max_concurrency: 24 # matching the rest of the suite +error_rate_threshold: 0.1 +verifier_timeout_seconds: 28800 +secrets: + - MODAL_TOKEN_ID + - MODAL_TOKEN_SECRET + - WANDB_API_KEY + - WANDB_BASE_URL # self-hosted W&B (scaleai.wandb.io) + +# candidate harness runs as an unprivileged uid, unable to read held-out state. +harness_user: harness + +wandb: + project: vero-swe-bench-pro + group: swe-bench-pro-sample # distinct from the full-split group + log_traces: true + +inference_gateway: + upstream_api_key_env: OPENAI_API_KEY + upstream_base_url_env: OPENAI_BASE_URL + producer: + # NOTE: gpt-5.3-codex has no /responses deployment on the LiteLLM proxy + # (/chat/completions is 200, /responses is 404 DeploymentNotFound), and the + # codex agent is Responses-native, so launch with + # `--model gpt-5.6-sol` (or gpt-4.1) until that deployment exists. + allowed_models: ["${optimizer_model:-gpt-5.6-sol}"] + max_concurrency: 8 + evaluation: + # Prefixed form only: the agents strip just `openai/` before calling the + # gateway, which then matches this allow-list as an exact string. + allowed_models: [fireworks_ai/deepseek-v4-flash] + max_requests: 15000 + max_tokens: 100000000 + max_concurrency: 64 +instruct_multifidelity: true +instruct_exhaust_budget: true diff --git a/harness-engineering-bench/swe-bench-pro/partitions/sample/development.json b/harness-engineering-bench/swe-bench-pro/partitions/sample/development.json new file mode 100644 index 00000000..14439df1 --- /dev/null +++ b/harness-engineering-bench/swe-bench-pro/partitions/sample/development.json @@ -0,0 +1,35 @@ +[ + "instance_ansible__ansible-12734fa21c08a0ce8c84e533abdc560db2eb1955-v7eee2454f617569fd6889f2211f75bc02a35f9f8", + "instance_ansible__ansible-1a4644ff15355fd696ac5b9d074a566a80fe7ca3-v30a923fb5c164d6cd18280c02422f75e611e8fb2", + "instance_ansible__ansible-34db57a47f875d11c4068567b9ec7ace174ec4cf-v1055803c3a812189a1133297f7f5468579283f86", + "instance_ansible__ansible-b8025ac160146319d2b875be3366b60c852dd35d-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_ansible__ansible-fb144c44144f8bd3542e71f5db62b6d322c7bd85-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5", + "instance_element-hq__element-web-27139ca68eb075a4438c18fca184887002a4ffbc-vnan", + "instance_element-hq__element-web-56c7fc1948923b4b3f3507799e725ac16bcf8018-vnan", + "instance_flipt-io__flipt-8bd3604dc54b681f1f0f7dd52cbc70b3024184b6", + "instance_flipt-io__flipt-967855b429f749c28c112b8cb1b15bc79157f973", + "instance_flipt-io__flipt-b2cd6a6dd73ca91b519015fd5924fde8d17f3f06", + "instance_flipt-io__flipt-f1bc91a1b999656dbdb2495ccb57bf2105b84920", + "instance_future-architect__vuls-0ec945d0510cdebf92cdd8999f94610772689f14", + "instance_future-architect__vuls-7e91f5ef7e5712b1a3d7d5066ad6607e9debc21c", + "instance_future-architect__vuls-d576b6c6c15e56c47cc3e26f5878867677d4a9ea", + "instance_gravitational__teleport-0ecf31de0e98b272a6a2610abe1bbedd379a38a3-vce94f93ad1030e3136852817f2423c1b3ac37bc4", + "instance_gravitational__teleport-1a77b7945a022ab86858029d30ac7ad0d5239d00-vee9b09fb20c43af7e520f57e9239bbcf46b7113d", + "instance_gravitational__teleport-8302d467d160f869b77184e262adbe2fbc95d9ba-vce94f93ad1030e3136852817f2423c1b3ac37bc4", + "instance_internetarchive__openlibrary-630221ab686c64e75a2ce253c893c033e4814b2e-v93c53c13d5f9b383ebb411ee7750b49dcd1a34c6", + "instance_internetarchive__openlibrary-bdba0af0f6cbaca8b5fc3be2a3080f38156d9c92-ve8c8d62a2b60610a3c4631f5f23ed866bada9818", + "instance_internetarchive__openlibrary-d40ec88713dc95ea791b252f92d2f7b75e107440-v13642507b4fc1f8d234172bf8129942da2c2ca26", + "instance_internetarchive__openlibrary-d8162c226a9d576f094dc1830c4c1ffd0be2dd17-v76304ecdb3a5954fcf13feb710e8c40fcf24b73c", + "instance_navidrome__navidrome-29bc17acd71596ae92131aca728716baf5af9906", + "instance_navidrome__navidrome-8d56ec898e776e7e53e352cb9b25677975787ffc", + "instance_nodebb__nodebb-8ca65b0c78c67c1653487c02d1135e1b702185e1-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e", + "instance_nodebb__nodebb-97c8569a798075c50e93e585ac741ab55cb7c28b-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e", + "instance_protonmail__webclients-6e1873b06df6529a469599aa1d69d3b18f7d9d37", + "instance_protonmail__webclients-c5a2089ca2bfe9aa1d85a664b8ad87ef843a1c9c", + "instance_protonmail__webclients-fc9d535e9beb3ae30a52a7146398cadfd6e30606", + "instance_qutebrowser__qutebrowser-1a9e74bfaf9a9db2a510dc14572d33ded6040a57-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-2e961080a85d660148937ee8f0f6b3445a8f2c01-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d", + "instance_qutebrowser__qutebrowser-ebfe9b7aa0c4ba9d451f993e08955004aaec4345-v059c6fdc75567943479b23ebca7c07b5e9a7f34c", + "instance_qutebrowser__qutebrowser-ff1c025ad3210506fc76e1f604d8c8c27637d88e-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d", + "instance_tutao__tutanota-db90ac26ab78addf72a8efaff3c7acc0fbd6d000-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf" +] diff --git a/harness-engineering-bench/swe-bench-pro/partitions/sample/test.json b/harness-engineering-bench/swe-bench-pro/partitions/sample/test.json new file mode 100644 index 00000000..b7291c28 --- /dev/null +++ b/harness-engineering-bench/swe-bench-pro/partitions/sample/test.json @@ -0,0 +1,68 @@ +[ + "instance_ansible__ansible-4c5ce5a1a9e79a845aff4978cfeb72a0d4ecf7d6-v1055803c3a812189a1133297f7f5468579283f86", + "instance_ansible__ansible-709484969c8a4ffd74b839a673431a8c5caa6457-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5", + "instance_ansible__ansible-8127abbc298cabf04aaa89a478fc5e5e3432a6fc-v30a923fb5c164d6cd18280c02422f75e611e8fb2", + "instance_ansible__ansible-83fb24b923064d3576d473747ebbe62e4535c9e3-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5", + "instance_ansible__ansible-a6e671db25381ed111bbad0ab3e7d97366395d05-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_ansible__ansible-d2f80991180337e2be23d6883064a67dcbaeb662-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5", + "instance_ansible__ansible-d62496fe416623e88b90139dc7917080cb04ce70-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_ansible__ansible-de01db08d00c8d2438e1ba5989c313ba16a145b0-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5", + "instance_element-hq__element-web-71fe08ea0f159ccb707904d87f0a4aef205a167c-vnan", + "instance_element-hq__element-web-9bf77963ee5e036d54b2a3ca202fbf6378464a5e-vnan", + "instance_element-hq__element-web-ca8b1b04effb4fec0e1dd3de8e3198eeb364d50e-vnan", + "instance_element-hq__element-web-d06cf09bf0b3d4a0fbe6bd32e4115caea2083168-vnan", + "instance_element-hq__element-web-d405160080bbe804f7e9294067d004a7d4dad9d6-vnan", + "instance_flipt-io__flipt-2eac0df47b5ecc8bb05002d80383ceb08ab3620a", + "instance_flipt-io__flipt-507170da0f7f4da330f6732bffdf11c4df7fc192", + "instance_flipt-io__flipt-524f277313606f8cd29b299617d6565c01642e15", + "instance_flipt-io__flipt-56a620b8fc9ef7a0819b47709aa541cdfdbba00b", + "instance_flipt-io__flipt-b433bd05ce405837804693bebd5f4b88d87133c8", + "instance_flipt-io__flipt-c154dd1a3590954dfd3b901555fc6267f646a289", + "instance_flipt-io__flipt-d966559200183b713cdf3ea5007a7e0ba86a5afb", + "instance_flipt-io__flipt-dae029cba7cdb98dfb1a6b416c00d324241e6063", + "instance_future-architect__vuls-01441351c3407abfc21c48a38e28828e1b504e0c", + "instance_future-architect__vuls-54e73c2f5466ef5daec3fb30922b9ac654e4ed25", + "instance_future-architect__vuls-5af1a227339e46c7abf3f2815e4c636a0c01098e", + "instance_future-architect__vuls-61c39637f2f3809e1b5dad05f0c57c799dce1587", + "instance_future-architect__vuls-be7b9114cc9545e68fb0ee7bc63d7ec53d1a00ad", + "instance_future-architect__vuls-f0b3a8b1db98eb1bd32685f1c36c41a99c3452ed", + "instance_gravitational__teleport-3fa6904377c006497169945428e8197158667910-v626ec2a48416b10a88641359a169d99e935ff037", + "instance_gravitational__teleport-53814a2d600ccd74c1e9810a567563432b98386e-vce94f93ad1030e3136852817f2423c1b3ac37bc4", + "instance_gravitational__teleport-59d39dee5a8a66e5b8a18a9085a199d369b1fba8-v626ec2a48416b10a88641359a169d99e935ff037", + "instance_gravitational__teleport-89f0432ad5dc70f1f6a30ec3a8363d548371a718", + "instance_gravitational__teleport-db89206db6c2969266e664c7c0fb51b70e958b64", + "instance_gravitational__teleport-eefac60a350930e5f295f94a2d55b94c1988c04e-vee9b09fb20c43af7e520f57e9239bbcf46b7113d", + "instance_gravitational__teleport-f432a71a13e698b6e1c4672a2e9e9c1f32d35c12", + "instance_internetarchive__openlibrary-1351c59fd43689753de1fca32c78d539a116ffc1-v29f82c9cf21d57b242f8d8b0e541525d259e2d63", + "instance_internetarchive__openlibrary-30bc73a1395fba2300087c7f307e54bb5372b60a-v76304ecdb3a5954fcf13feb710e8c40fcf24b73c", + "instance_internetarchive__openlibrary-6a117fab6c963b74dc1ba907d838e74f76d34a4b-v13642507b4fc1f8d234172bf8129942da2c2ca26", + "instance_internetarchive__openlibrary-798055d1a19b8fa0983153b709f460be97e33064-v13642507b4fc1f8d234172bf8129942da2c2ca26", + "instance_internetarchive__openlibrary-91efee627df01e32007abf2d6ebf73f9d9053076-vbee42ad1b72fb23c6a1c874868a720b370983ed2", + "instance_internetarchive__openlibrary-dbbd9d539c6d4fd45d5be9662aa19b6d664b5137-v08d8e8889ec945ab821fb156c04c7d2e2810debb", + "instance_internetarchive__openlibrary-de6ae10512f1b5ef585c8341b451bc49c9fd4996-vfa6ff903cb27f336e17654595dd900fa943dcd91", + "instance_internetarchive__openlibrary-e1e502986a3b003899a8347ac8a7ff7b08cbfc39-v08d8e8889ec945ab821fb156c04c7d2e2810debb", + "instance_navidrome__navidrome-0488fb92cb02a82924fb1181bf1642f2e87096db", + "instance_navidrome__navidrome-66b74c81f115c78cb69910b0472eeb376750efc4", + "instance_navidrome__navidrome-d21932bd1b2379b0ebca2d19e5d8bae91040268a", + "instance_navidrome__navidrome-d613b1930688422122796b43acb3caf2538c8fd1", + "instance_navidrome__navidrome-f7d4fcdcc1a59d1b4f835519efb402897757e371", + "instance_nodebb__nodebb-0c81642997ea1d827dbd02c311db9d4976112cd4-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e", + "instance_nodebb__nodebb-84e065752f6d7fbe5c08cbf50cb173ffb866b8fa-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e", + "instance_nodebb__nodebb-bad15643013ca15affe408b75eba9e47cc604bb2-vd59a5728dfc977f44533186ace531248c2917516", + "instance_nodebb__nodebb-eb49a64974ca844bca061744fb3383f5d13b02ad-vnan", + "instance_protonmail__webclients-8142704f447df6e108d53cab25451c8a94976b92", + "instance_protonmail__webclients-b9387af4cdf79c2cb2a221dea33d665ef789512e", + "instance_protonmail__webclients-c6f65d205c401350a226bb005f42fac1754b0b5b", + "instance_protonmail__webclients-c8117f446c3d1d7e117adc6e0e46b0ece9b0b90e", + "instance_protonmail__webclients-cba6ebbd0707caa524ffee51c62b197f6122c902", + "instance_protonmail__webclients-da91f084c0f532d9cc8ca385a701274d598057b8", + "instance_qutebrowser__qutebrowser-0b621cb0ce2b54d3f93d8d41d8ff4257888a87e5-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-21b426b6a20ec1cc5ecad770730641750699757b-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d", + "instance_qutebrowser__qutebrowser-305e7c96d5e2fdb3b248b27dfb21042fb2b7e0b8-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-479aa075ac79dc975e2e949e188a328e95bf78ff-vc2f56a753b62a190ddb23cd330c257b9cf560d12", + "instance_qutebrowser__qutebrowser-70248f256f93ed9b1984494d0a1a919ddd774892-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-77c3557995704a683cdb67e2a3055f7547fa22c3-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d", + "instance_qutebrowser__qutebrowser-8d05f0282a271bfd45e614238bd1b555c58b3fc1-v35616345bb8052ea303186706cec663146f0f184", + "instance_tutao__tutanota-fb32e5f9d9fc152a00144d56dd0af01760a2d4dc-vc4e41fd0029957297843cb9dec4a25c7c756f029", + "instance_tutao__tutanota-fbdb72a2bd39b05131ff905780d9d4a2a074de26-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf" +] diff --git a/harness-engineering-bench/swe-bench-pro/partitions/sample/validation.json b/harness-engineering-bench/swe-bench-pro/partitions/sample/validation.json new file mode 100644 index 00000000..647fbf20 --- /dev/null +++ b/harness-engineering-bench/swe-bench-pro/partitions/sample/validation.json @@ -0,0 +1,68 @@ +[ + "instance_ansible__ansible-11c1777d56664b1acb56b387a1ad6aeadef1391d-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_ansible__ansible-1b70260d5aa2f6c9782fd2b848e8d16566e50d85-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5", + "instance_ansible__ansible-1ee70fc272aff6bf3415357c6e13c5de5b928d9b-v1055803c3a812189a1133297f7f5468579283f86", + "instance_ansible__ansible-5f4e332e3762999d94af27746db29ff1729252c1-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_ansible__ansible-77658704217d5f166404fc67997203c25381cb6e-v390e508d27db7a51eece36bb6d9698b63a5b638a", + "instance_ansible__ansible-9142be2f6cabbe6597c9254c5bb9186d17036d55-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_ansible__ansible-d6d2251929c84c3aa883bad7db0f19cc9ff0339e-v30a923fb5c164d6cd18280c02422f75e611e8fb2", + "instance_ansible__ansible-e64c6c1ca50d7d26a8e7747d8eb87642e767cd74-v0f01c69f1e2528b935359cfe578530722bca2c59", + "instance_element-hq__element-web-1077729a19c0ce902e713cf6fab42c91fb7907f1-vnan", + "instance_element-hq__element-web-459df4583e01e4744a52d45446e34183385442d6-vnan", + "instance_element-hq__element-web-ad26925bb6628260cfe0fcf90ec0a8cba381f4a4-vnan", + "instance_element-hq__element-web-aec454dd6feeb93000380523cbb0b3681c0275fd-vnan", + "instance_element-hq__element-web-dae13ac8522fc6d41e64d1ac6e3174486fdcce0c-vnan", + "instance_flipt-io__flipt-0fd09def402258834b9d6c0eaa6d3b4ab93b4446", + "instance_flipt-io__flipt-40007b9d97e3862bcef8c20ae6c87b22ea0627f0", + "instance_flipt-io__flipt-86906cbfc3a5d3629a583f98e6301142f5f14bdb-v6bea0cc3a6fc532d7da914314f2944fc1cd04dee", + "instance_flipt-io__flipt-9f8127f225a86245fa35dca4885c2daef824ee55", + "instance_flipt-io__flipt-c12967bc73fdf02054cf3ef8498c05e25f0a18c0", + "instance_flipt-io__flipt-c1728053367c753688f114ec26e703c8fdeda125", + "instance_flipt-io__flipt-c6a7b1fd933e763b1675281b30077e161fa115a1", + "instance_flipt-io__flipt-c8d71ad7ea98d97546f01cce4ccb451dbcf37d3b", + "instance_future-architect__vuls-36456cb151894964ba1683ce7da5c35ada789970", + "instance_future-architect__vuls-4a72295de7b91faa59d90a5bee91535bbe76755d", + "instance_future-architect__vuls-50580f6e98eeb36f53f27222f7f4fdfea0b21e8d", + "instance_future-architect__vuls-ad2edbb8448e2c41a097f1c0b52696c0f6c5924d", + "instance_future-architect__vuls-e52fa8d6ed1d23e36f2a86e5d3efe9aa057a1b0d", + "instance_future-architect__vuls-ef2be3d6ea4c0a13674aaab08b182eca4e2b9a17-v264a82e2f4818e30f5a25e4da53b27ba119f62b5", + "instance_gravitational__teleport-10123c046e21e1826098e485a4c2212865a49d9f", + "instance_gravitational__teleport-3a5c1e26394df2cb4fb3f01147fb9979662972c5-vee9b09fb20c43af7e520f57e9239bbcf46b7113d", + "instance_gravitational__teleport-73cc189b0e9636d418c4470ecce0d9af5dae2f02-vee9b09fb20c43af7e520f57e9239bbcf46b7113d", + "instance_gravitational__teleport-a95b3ae0667f9e4b2404bf61f51113e6d83f01cd", + "instance_gravitational__teleport-b4e7cd3a5e246736d3fe8d6886af55030b232277", + "instance_gravitational__teleport-d873ea4fa67d3132eccba39213c1ca2f52064dcc-vce94f93ad1030e3136852817f2423c1b3ac37bc4", + "instance_gravitational__teleport-e6895d8934f6e484341034869901145fbc025e72-vce94f93ad1030e3136852817f2423c1b3ac37bc4", + "instance_internetarchive__openlibrary-09865f5fb549694d969f0a8e49b9d204ef1853ca-ve8c8d62a2b60610a3c4631f5f23ed866bada9818", + "instance_internetarchive__openlibrary-1be7de788a444f6255e89c10ef6aa608550604a8-v29f82c9cf21d57b242f8d8b0e541525d259e2d63", + "instance_internetarchive__openlibrary-2abe28b472ffed563a87cfe83685b161b35263b0-v13642507b4fc1f8d234172bf8129942da2c2ca26", + "instance_internetarchive__openlibrary-427f1f4eddfc54735ca451779d4f95bf683d1b0e-v0f5aece3601a5b4419f7ccec1dbda2071be28ee4", + "instance_internetarchive__openlibrary-89e4b4431fe7506c365a6f6eb6f6d048d04c044c-v08d8e8889ec945ab821fb156c04c7d2e2810debb", + "instance_internetarchive__openlibrary-9c392b60e2c6fa1d68cb68084b4b4ff04d0cb35c-v2d9a6c849c60ed19fd0858ce9e40b7cc8e097e59", + "instance_internetarchive__openlibrary-9cd47f4dc21e273320d9e30d889c864f8cb20ccf-v0f5aece3601a5b4419f7ccec1dbda2071be28ee4", + "instance_internetarchive__openlibrary-c506c1b0b678892af5cb22c1c1dbc35d96787a0a-v0f5aece3601a5b4419f7ccec1dbda2071be28ee4", + "instance_navidrome__navidrome-09ae41a2da66264c60ef307882362d2e2d8d8b89", + "instance_navidrome__navidrome-0a650de357babdcc8ce910fe37fee84acf4ed2fe", + "instance_navidrome__navidrome-3977ef6e0f287f598b6e4009876239d6f13b686d", + "instance_navidrome__navidrome-69e0a266f48bae24a11312e9efbe495a337e4c84", + "instance_navidrome__navidrome-e12a14a87d392ac70ee4cc8079e3c3e0103dbcb2", + "instance_nodebb__nodebb-3c85b944e30a0ba8b3ec9e1f441c74f383625a15-v4fbcfae8b15e4ce5d132c408bca69ebb9cf146ed", + "instance_nodebb__nodebb-51d8f3b195bddb13a13ddc0de110722774d9bb1b-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e", + "instance_nodebb__nodebb-b1f9ad5534bb3a44dab5364f659876a4b7fe34c1-vnan", + "instance_nodebb__nodebb-f083cd559d69c16481376868c8da65172729c0ca-vnan", + "instance_protonmail__webclients-01b519cd49e6a24d9a05d2eb97f54e420740072e", + "instance_protonmail__webclients-01ea5214d11e0df8b7170d91bafd34f23cb0f2b1", + "instance_protonmail__webclients-6dcf0d0b0f7965ad94be3f84971afeb437f25b02", + "instance_protonmail__webclients-708ed4a299711f0fa79a907cc5847cfd39c0fc71", + "instance_protonmail__webclients-815695401137dac2975400fc610149a16db8214b", + "instance_protonmail__webclients-944adbfe06644be0789f59b78395bdd8567d8547", + "instance_qutebrowser__qutebrowser-1af602b258b97aaba69d2585ed499d95e2303ac2-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-394bfaed6544c952c6b3463751abab3176ad4997-vafb3e8e01b31319c66c4e666b8a3b1d8ba55db24", + "instance_qutebrowser__qutebrowser-54bcdc1eefa86cc20790973d6997b60c3bba884c-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-8f46ba3f6dc7b18375f7aa63c48a1fe461190430-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-bedc9f7fadf93f83d8dee95feeecb9922b6f063f-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-c09e1439f145c66ee3af574386e277dd2388d094-v2ef375ac784985212b1805e1d0431dc8f1b3c171", + "instance_qutebrowser__qutebrowser-ec2dcfce9eee9f808efc17a1b99e227fc4421dea-v5149fcda2a9a6fe1d35dfed1bade1444a11ef271", + "instance_tutao__tutanota-219bc8f05d7b980e038bc1524cb021bf56397a1b-vee878bb72091875e912c52fc32bc60ec3760227b", + "instance_tutao__tutanota-f3ffe17af6e8ab007e8d461355057ad237846d9d-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf" +] diff --git a/harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py b/harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py index 9be9ab02..89e13028 100644 --- a/harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py +++ b/harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py @@ -41,14 +41,33 @@ TOTAL_TASKS = 731 TARGET_COUNTS = {"development": 146, "validation": 292, "test": 293} +# `--sample` writes a second, smaller split into partitions/sample/ for the +# optimizer variant (baseline/build.sample.yaml). The canonical split above is +# never touched by it. 33/66/66 matches gaia and browsecomp-plus, the suite's +# tightest split, which is the right reference for its most expensive benchmark: +# a case here builds a real repository and runs its test suite, so the full-split +# held-out baseline alone cost ~7.8h of wall clock for three rounds. +SAMPLE_COUNTS = {"development": 33, "validation": 66, "test": 66} + def _parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument( "--tasks-dir", type=Path, - required=True, - help="Directory containing the exported SWE-bench-Pro task directories", + help=( + "Directory containing the exported SWE-bench-Pro task directories. " + "Omit to re-derive the split from the committed manifest, which " + "already records every instance's name and stratum." + ), + ) + parser.add_argument( + "--sample", + action="store_true", + help=( + "Write the smaller SAMPLE_COUNTS split to partitions/sample/ for the " + "optimizer variant. Leaves the canonical split untouched." + ), ) parser.add_argument( "--output-dir", @@ -97,6 +116,27 @@ def _read_tasks(path: Path) -> list[dict[str, str]]: return tasks +def _tasks_from_manifest(output_dir: Path) -> list[dict[str, str]]: + """Re-read the (name, repository) pairs the committed manifest already pins. + + Deriving the sample should not require re-exporting 731 task directories: + the manifest records every instance and its stratum verbatim, which is the + entire input `_allocate` needs. `--tasks-dir` remains the path for re-pinning + against a moved dataset. + """ + manifest_path = output_dir / "manifest.json" + if not manifest_path.is_file(): + raise ValueError("no existing manifest; pass --tasks-dir") + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + tasks = [ + {"name": item["name"], "repository": item["repository"]} + for item in manifest["tasks"] + ] + if len(tasks) != TOTAL_TASKS: + raise ValueError(f"expected {TOTAL_TASKS} tasks, manifest has {len(tasks)}") + return tasks + + async def _fetch_registry_refs() -> tuple[str, dict[str, str]]: try: from harbor.registry.client.factory import RegistryClientFactory @@ -187,6 +227,63 @@ def _allocate(tasks: list[dict[str, str]]) -> dict[str, list[str]]: return result +def _subsample( + full: dict[str, list[str]], tasks: list[dict[str, str]] +) -> dict[str, list[str]]: + """Shrink each partition to SAMPLE_COUNTS without letting tasks cross partitions. + + Sampling happens strictly inside a partition, so every sampled set is a + subset of the corresponding full set. Two properties follow, and both are + why it is done this way rather than by re-running the split at a smaller + size: an instance can never move from test into development (a flat + re-split does move them, which would silently destroy the held-out + guarantee), and a reward already measured over the full partition can be + recomputed over the sample from the existing per-case trials. + + Within a partition the same repository stratification and sha256 ordering + apply, so the sample keeps the partition's repo mix and stays reproducible. + """ + repository = {task["name"]: task["repository"] for task in tasks} + sampled: dict[str, list[str]] = {} + for partition in PARTITIONS: + target = SAMPLE_COUNTS[partition] + members = full[partition] + if target > len(members): + raise ValueError( + f"{partition} sample of {target} exceeds its {len(members)} tasks" + ) + strata: dict[str, list[str]] = defaultdict(list) + for name in members: + strata[repository[name]].append(name) + + quota = { + stratum: target * len(names) // len(members) + for stratum, names in strata.items() + } + remainders = sorted( + strata, + key=lambda stratum: ( + -(target * len(strata[stratum]) % len(members)), + stratum, + ), + ) + cursor = 0 + while sum(quota.values()) < target: + stratum = remainders[cursor % len(remainders)] + if quota[stratum] < len(strata[stratum]): + quota[stratum] += 1 + cursor += 1 + + chosen: list[str] = [] + for stratum in sorted(strata): + chosen.extend(sorted(strata[stratum], key=_stable_key)[: quota[stratum]]) + chosen.sort() + if len(chosen) != target: + raise RuntimeError(f"wrong sampled {partition} size: {len(chosen)}") + sampled[partition] = chosen + return sampled + + def _render( tasks: list[dict[str, str]], partitions: dict[str, list[str]], @@ -235,15 +332,37 @@ def _render( return rendered +def _render_sample(partitions: dict[str, list[str]]) -> dict[str, str]: + """Just the three partition files. The manifest is shared with the full split. + + Nothing benchmark-level changes for the sample: the same manifest pins the + same 731 refs, and the build loader reads only task names out of it. Writing + a second manifest would duplicate that pinning and let the two drift. + """ + return { + f"{partition}.json": json.dumps(partitions[partition], indent=2) + "\n" + for partition in PARTITIONS + } + + def main() -> None: args = _parse_args() - tasks = _read_tasks(args.tasks_dir) args.output_dir = args.output_dir.expanduser().resolve() - if args.fetch_registry: - dataset_version, refs = asyncio.run(_fetch_registry_refs()) + tasks = ( + _read_tasks(args.tasks_dir) + if args.tasks_dir is not None + else _tasks_from_manifest(args.output_dir) + ) + if args.sample: + # Derived from the canonical split, then narrowed inside each partition. + rendered = _render_sample(_subsample(_allocate(tasks), tasks)) + args.output_dir = args.output_dir / "sample" else: - dataset_version, refs = _existing_refs(args.output_dir) - rendered = _render(tasks, _allocate(tasks), dataset_version, refs) + if args.fetch_registry: + dataset_version, refs = asyncio.run(_fetch_registry_refs()) + else: + dataset_version, refs = _existing_refs(args.output_dir) + rendered = _render(tasks, _allocate(tasks), dataset_version, refs) if args.check: changed = [