CodeScaleBench

CodeScaleBench measures how external code-retrieval tooling changes what an AI coding agent produces, on realistic developer tasks in large, enterprise-scale codebases. Retrieval is a controlled variable: the same agent and model run on each task with the source available locally and with it reachable only through a retrieval server, and the paired difference is the reported quantity.

• Dataset: sgjarmak/CodeScaleBench on Hugging Face — the 370-task problem layer (instruction, repo, base commit, metadata; repository contents and verifiers are materialized by the eval kit).
• Reproduce the published results: git clone --branch v1-mixed371 https://github.com/sourcegraph/CodeScaleBench.git yields the exact 370-task analysis set, the frozen suite definition, and the result snapshot.

This repository contains:

• 370 benchmark tasks (csb-v1-mixed371) — the analysis set behind the published results, spanning software-lifecycle and organization-scale work across more than forty repositories and nine languages
• Auditable result snapshots with per-task traces, scores, timing, and cost (runs/snapshots/)
• The eval kit — the Harbor-based harness, deterministic gated verifiers, and metrics/reporting pipelines for running and scoring submissions
• Task indexes by complexity, language, repo size, and multi-repository scope (benchmarks/indexes/)

The corpus is under active curation: the default branch tracks the evolving maintained set, while csb-v1-mixed371 is the frozen set the paper reports, retained under a tag. Cite the suite id with any result.

Tasks are executed via the Harbor runner with the Claude Code agent harness.

---

Results

Official results are available as frozen, auditable snapshots in runs/snapshots/.

Snapshot	Tasks	Model	Configs	Mean Reward
csb-v1-mixed371--haiku45--030326	371	Haiku 4.5	Baseline, Sourcegraph MCP	0.536 / 0.565

Each snapshot includes:

• browse.html — Interactive results browser (open in any web browser)
• SNAPSHOT.json — Machine-readable manifest
• export/traces/ — Per-task result metadata, reward scores, agent instructions
• export/summary/ — Aggregate scores, timing, cost breakdowns

Full agent trajectories are available as compressed archives in the corresponding GitHub Release.

Baseline vs MCP side-by-side browser

docs/analysis/compare/ contains a pre-rendered gallery of side-by-side compare pages — for every task where Sourcegraph MCP outperformed the local-filesystem baseline, the full agent tool-call sequence on both sides is laid out in two columns (search queries, file reads, edits, bash commands, conversation events).

To browse it on any machine:

git clone --branch public --single-branch \
    https://github.com/sourcegraph/CodeScaleBench.git
cd CodeScaleBench
python3 -m http.server 8000 -d docs/analysis/compare
# then open http://localhost:8000

The committed snapshot is the top 20 MCP wins by reward delta plus a full manifest of all 59 wins (wins_manifest.json). To regenerate the rest locally from your own runs/analysis/<suite>/<model>/{baseline,mcp}/ trees:

python3 scripts/analysis/browse_compare.py \
    --threshold 0.20 --render-all --out browse/compare

The generator has no external dependencies (Python stdlib only). See docs/analysis/compare/README.md for pairing rules, filter semantics, and source-run requirements.

---

Benchmark Suites

Suite definitions in benchmarks/suites/ specify exactly which tasks are in each benchmark version:

Suite	Tasks	Description
csb-v1-mixed371	371	Initial report: mixed verifiers, 151 SDLC + 220 org tasks
csb-v2-dual264	264	Canonical: all dual-verifier tasks
csb-v2-full-validated	275	Complete: all validated tasks including Daytona-excluded

Task indexes in benchmarks/indexes/ provide views by:

• Complexity — scored by repo size, repo count, GT files, difficulty
• Language — 13 languages (Go, C++, Java, Python, Rust, C#, JS, TS, ...)
• Repo size — small (<100MB) through xlarge (>5GB)
• Multi-repo — 136 multi-repo tasks across 28 repo sets

---

Task Taxonomy

All tasks represent realistic developer work in large, often multi-repo, enterprise codebases. Tasks are organized by developer work type.

Work Type	Tasks	Description	Repo Scope
crossrepo	47	Cross-repo navigation, dependency tracing, org-wide discovery	18 single, 9 dual, 20 multi
understand	44	Codebase comprehension, architecture, onboarding, domain knowledge	36 single, 4 dual, 4 multi
refactor	43	Code transformation, migration, dependency updates	26 single, 2 dual, 15 multi
security	39	Security review, vulnerability remediation, compliance audit	26 single, 2 dual, 11 multi
feature	34	Feature implementation, org-wide feature work	24 single, 2 dual, 8 multi
debug	26	Debugging, root cause analysis, incident triage	15 single, 8 dual, 3 multi
fix	19	Bug repair from issue reports	19 single
test	12	Test generation, code review, QA	12 single
document	11	API docs, architecture docs, migration guides	10 single, 1 dual
Total	275		186 single, 28 dual, 61 multi

---

Evaluation Configurations

Tasks are evaluated across multiple MCP configurations to measure the impact of external code retrieval tools:

Config	Code Access	MCP Tools
baseline-local-direct	Full local code	None (built-in tools only)
mcp-remote-direct	Remote only (Sourcegraph)	keyword_search, nls_search, read_file, find_references, go_to_definition, ...
augment-local-direct	Full local code	Augment Context Engine (codebase-retrieval)
github-remote-direct	Remote only (GitHub API)	search_code, get_file_contents, get_repository_tree, ...

---

Repository Structure

benchmarks/
  suites/                # Versioned benchmark suite definitions (JSON)
  indexes/               # Task indexes by complexity, language, repo size, multi-repo
  tasks/                 # 275 task definitions across 20 suite directories
    csb_sdlc_feature/    #   Feature implementation (23 tasks)
    csb_sdlc_fix/        #   Bug repair (19 tasks)
    csb_org_migration/   #   Framework migration (25 tasks)
    ...                  #   (20 suites total)
runs/
  snapshots/             # Frozen, auditable result snapshots
    {snapshot_id}/
      SNAPSHOT.json      #   Manifest: suite, model, configs, aggregates
      browse.html        #   Interactive results browser
      export/            #   Sanitized traces for public consumption
scripts/                 # Metrics extraction, evaluation, and operational tooling
  publishing/            #   Snapshot creation, verification, export
  evaluation/            #   IR metrics, retrieval analysis, judge pipelines
  csb_metrics/           #   Python package: models, extractors, discovery
docs/
  technical_reports/     # Published technical reports
  official_results/      # Pointer to runs/snapshots/

---

Quickstart

git clone https://github.com/sourcegraph/CodeScaleBench.git
cd CodeScaleBench

# Browse benchmark suites
cat benchmarks/suites/csb-v2-dual264.json | python3 -m json.tool | head -20

# Explore task indexes
cat benchmarks/indexes/by-complexity.json | python3 -m json.tool | head -30

# Open results browser
open runs/snapshots/csb-v1-mixed371--haiku45--030326/browse.html

Running benchmarks (requires Harbor)

# Pre-flight checks
python3 scripts/infra/check_infra.py

# Run canonical suite
./configs/harnesses/run_selected_tasks.sh \
  --suite-file benchmarks/suites/csb-v2-dual264.json

---

Quality Assurance

CodeScaleBench includes a multi-stage QA pipeline:

Phase	Script	Purpose
Pre-flight	scripts/authoring/validate_tasks_preflight.py	Task integrity checks
Calibration triad	scripts/authoring/calibrate_task.py	Null/golden/adversarial gates on the artifact verifier
Infra check	scripts/infra/check_infra.py	OAuth tokens, Docker, disk
Post-run	scripts/authoring/validate_task_run.py	Scoring anomalies, MCP usage
Snapshot verify	scripts/publishing/verify_snapshot.py	Symlink integrity, completeness

Calibration triad (upstream push gate)

Before a task is pushed to the public sourcegraph/CodeScaleBench mirror, its artifact verifier must pass three calibration gates:

• null — empty answer scores ≤ 0.1
• golden — canonical oracle_answer.json (or synthesized perfect input) scores ≥ 0.9
• adversarial — keyword dump with empty structured arrays scores ≤ 0.5

The triad gates the upstream push, not the public corpus's existence. Failing tasks are not removed; per-cluster fix beads are filed in CSB scope and resolved independently. See docs/qa/calibration_run_2026_04_30.md for the most recent corpus run and cluster breakdown.

---

License

See LICENSE.