feat: swe bench scorer#342
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for SWE-bench accuracy evaluation by adding a new accuracy-only SWEBench dataset, a SWEBenchScorer that runs evaluations using mini-swe-agent in an isolated environment, and associated configuration templates, tests, and runbooks. Feedback on the changes focuses on improving the robustness of the SWEBenchScorer implementation, specifically by safely handling missing or null values when parsing configuration templates, benchmark configurations, and evaluation results, as well as gracefully handling cases where the Docker binary is missing from the system's PATH during preflight checks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
2 times, most recently
from
aa7f559 to
bb9b307
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class SWE-bench accuracy support to the benchmarking system by introducing an external (subprocess-driven) scorer and a predefined SWE-bench dataset, plus wiring in config/schema handling and examples to run the workflow.
Changes:
- Introduces
SWEBenchScorer(external evaluation viauv run --project ... mini-extra+swebenchharness) and integrates it into the scoring/benchmark lifecycle (preflight, phase skipping, reporting). - Adds predefined
swe_benchdataset with caching + anACCURACY_ONLYguard to prevent using it as a performance dataset. - Updates schema/templates/examples and adds unit tests for the scorer + dataset + benchmark wiring.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks pyyaml as a direct dependency. |
| pyproject.toml | Adds pyyaml==6.0.3 to runtime dependencies. |
| src/inference_endpoint/evaluation/scoring.py | Adds scorer preflight hooks, shared subprocess logging helper, and new SWEBenchScorer implementation. |
| src/inference_endpoint/dataset_manager/predefined/swe_bench/init.py | Adds SWE-bench predefined dataset with HF download + parquet cache and ACCURACY_ONLY=True. |
| src/inference_endpoint/dataset_manager/dataset.py | Introduces Dataset.ACCURACY_ONLY class flag. |
| src/inference_endpoint/dataset_manager/init.py | Exposes/imports SWEBench dataset. |
| src/inference_endpoint/config/schema.py | Adds swe_bench_scorer to ScorerMethod and injects concurrency → SWE-bench workers default. |
| src/inference_endpoint/commands/benchmark/execute.py | Runs scorer preflight(), prevents accuracy-only datasets as perf, skips endpoint phases for external scorers, and improves sample counting/reporting. |
| src/inference_endpoint/config/templates/online_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/offline_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/concurrency_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| examples/10_Agentic_Inference/swebench_template.yaml | Adds a SWE-bench/mini-swe-agent config template used by the external scorer. |
| examples/10_Agentic_Inference/swe_bench_accuracy.yaml | Adds a runnable example config for SWE-bench external accuracy evaluation. |
| examples/10_Agentic_Inference/README.md | Documents how to sync/run the SWE-bench accuracy subproject. |
| examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/accuracy/RUNBOOK.md | Adds an ops runbook for validating the SWE-bench accuracy pipeline. |
| examples/10_Agentic_Inference/accuracy/pyproject.toml | Adds isolated uv subproject deps for SWE-bench evaluation tooling. |
| tests/unit/evaluation/test_swe_bench_scorer.py | Adds unit coverage for scorer behavior, config patching, and preflight. |
| tests/unit/dataset_manager/test_swe_bench_dataset.py | Adds unit coverage for dataset registration, caching, and subset mapping. |
| tests/unit/commands/test_benchmark.py | Adds tests for concurrency→workers injection, ACCURACY_ONLY enforcement, preflight propagation, and phase skipping behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
8d66f43 to
ae04494
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #342 +/- ##
=======================================
Coverage ? 80.52%
=======================================
Files ? 130
Lines ? 17852
Branches ? 0
=======================================
Hits ? 14375
Misses ? 3477
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
da45a99 to
fe442f8
Compare
litellm>=1.92.0 imports fastapi (proxy-only extra, not installed here) as soon as a completion() call passes tools, breaking every tool-calling request with ImportError/ModuleNotFoundError retries and producing empty SWE-bench patches for all submitted instances.
Move swebench templates into accuracy/ so they aren't accidentally edited by participants, move the SWE-specific module-level helper functions into SWEBenchScorer as staticmethods (hvagadia review feedback), close the accuracy-only dataset guard bypass via `name::preset` suffixes in execute.py, and treat an explicit YAML `null` as a missing value in _get_extra_int/_get_extra_bool instead of raising. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
5fffd61 to
32eceaa
Compare
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
32eceaa to
7224b2b
Compare
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewed by Codex + Claude · Depth thorough
Posting 10 inline findings on the new SWE-bench scorer. See the summary comment for the tiered breakdown. Findings already covered by existing Copilot/Gemini/human threads (many now fixed) were deduplicated out.
Review Council — Multi-AI Code ReviewReviewed by: Codex + Claude | Depth: thorough Found 13 issues; posted 10 inline (3 filtered as duplicates of existing Copilot/Gemini/human threads — 🔴 Must Fix (high)Silently discards a real result after the full (expensive) agent + eval run.
🟡 Should Fix (medium)Real correctness/integrity issues that trigger under supported configurations.
🔵 Consider (low)Valid hardening / coverage; could be follow-ups.
Reviewer coverage. Codex focused on functional path/submission bugs in the newly added workflow (findings 1–3); Claude covered scoring semantics, crash-safety, the
Posted with |
| bash_cmd = ( | ||
| f"awk 'NR>={start} && NR<={end} " | ||
| f'{{printf "%6d\\t%s\\n", NR, $0}}\' {shlex.quote(path)}' | ||
| ) |
| command = actions[0]["command"] | ||
| assert ( | ||
| command == "awk 'NR>=2 && NR<=4 " | ||
| '{printf "%6d\\t%s\\n", NR, $0}\' /testbed/pkg/file.py' | ||
| ) | ||
| assert r"\'" not in command |
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Can we remove docker as a dependency - ideally we would like to have the container be launched/orchestrated from outside the client and only a port/path be passed to the scorer like we do for LCB.
What does this PR do?
Adds SWE-bench accuracy evaluation to the agentic inference workflow (per #310). A new
SWEBenchdataset loadsSWE-bench_Verified/_Lite, andSWEBenchScorerdrives the eval by shelling out tomini-swe-agentand theswebenchharness in an isolated uv subproject, bypassing the built-in accuracy phase.Agent-run parallelism (
extras.workers) defaults to the load pattern'starget_concurrencywhen unset. Includes example configs, an optional Qwen tool-call patch, and docs.Type of change
Related issues
Closes #310
Testing
Checklist