feat(bench): add depth axis to sweep matrix; aggregator parses L<depth>

Workflow `sweep-full` matrix now includes `depth: [-1, 0, 1, 2, 3, 4]`
with EGO running on {0,1,2,3,4} and non-EGO methods (PPR, BM25, Aider)
running only on the sentinel depth=-1. Excludes are explicit so the
matrix doesn't accidentally multiply 5x for modes that ignore depth.

Per-cell wiring:
- CELL_TAG / CELL_DIR include `L${depth}` so EGO cells with the same
  (method, budget, test_set) but different depth land in distinct dirs.
- The Run cell step exports `DIFFCTX_OP_GRAPH_DEPTH=${matrix.depth}`
  before invoking `run_final_eval` (skipped when depth=-1 sentinel,
  so non-EGO methods inherit the default).
- metadata.json gains `cell.depth` for reproducibility.
- upload-artifact name is `cell-<method>-b<budget>-L<depth>-<test_set>`,
  so 5 EGO cells under the same (budget, test_set) no longer collide.

aggregate_sweep.py:
- Regex split into _ARTIFACT_RE_WITH_DEPTH and _ARTIFACT_RE_LEGACY so
  the new dirs parse with depth and legacy dirs (no L segment) parse
  with depth=-1 sentinel. `_parse_artifact` returns a 4-tuple
  (method, budget, depth, test_set); legacy callers continue to use
  metadata.json values when present.
- collect_cells captures depth into the cell record.
- CSV gains a `depth` column so the L sweep is queryable downstream.

Sanity checks performed before this commit (all PASS, no blockers):
- source_benchmark adapter strings == manifest names
  (swebench_verified, polybench500, contextbench_verified)
- Aider/BM25 do not crash on budget=0 (existing run_25288507618
  shows 100% ok|empty_query|clone_fail status, no crashes)
- Aider PolyBench 0.284 is real, not an integration bug; root cause
  documented in paper Limitations section.

Run-time projection at full matrix (162 cells, 4 self-hosted runners):
~9-15h optimistic, ~50-70h pessimistic depending on per-cell variance.
The multi-budget reuse path (`pool_eval_all_cells`) is implemented but
not used by this workflow — each cell runs heavy phase from scratch
because budget remains a matrix axis. Trade-off is explicit; collapse
to per-(method, depth, test_set) cells with --budgets is a future
optimization.

Author: nikolay-e

.github/workflows/bench-sweep.yml

@@ -434,10 +434,32 @@ jobs:
         # unlimited-budget ceiling. The 5 paying budgets {8k, 16k, 32k, 64k, 128k}
         # form the budget curve referenced in STATS_PLAN.md.
         budget: [-1, 0, 8000, 16000, 32000, 64000, 128000]
+        # `depth` is the EGO graph traversal radius; only meaningful for
+        # method=ego. PPR uses alpha (not depth), BM25 has no graph,
+        # Aider is its own subprocess. Sentinel -1 means "depth does not
+        # apply"; non-EGO methods run only with depth=-1.
+        depth: [-1, 0, 1, 2, 3, 4]
         test_set: [contextbench_verified, polybench500, swebench_verified]
         exclude:
           - {method: bm25, budget: -1}
           - {method: aider, budget: -1}
+          # Non-EGO methods: only the depth=-1 cell.
+          - {method: ppr, depth: 0}
+          - {method: ppr, depth: 1}
+          - {method: ppr, depth: 2}
+          - {method: ppr, depth: 3}
+          - {method: ppr, depth: 4}
+          - {method: bm25, depth: 0}
+          - {method: bm25, depth: 1}
+          - {method: bm25, depth: 2}
+          - {method: bm25, depth: 3}
+          - {method: bm25, depth: 4}
+          - {method: aider, depth: 0}
+          - {method: aider, depth: 1}
+          - {method: aider, depth: 2}
+          - {method: aider, depth: 3}
+          - {method: aider, depth: 4}
+          - {method: ego, depth: -1}
     runs-on: [self-hosted, large]
     timeout-minutes: 350
     container:
@@ -455,8 +477,8 @@ jobs:
         --ulimit nproc=65535:65535
         -v /data/bench_repos:/cache/contextbench_repos
     env:
-      CELL_TAG: ${{ matrix.method }}_b${{ matrix.budget }}_${{ matrix.test_set }}
-      CELL_DIR: results/sweep/${{ matrix.method }}_b${{ matrix.budget }}_${{ matrix.test_set }}
+      CELL_TAG: ${{ matrix.method }}_b${{ matrix.budget }}_L${{ matrix.depth }}_${{ matrix.test_set }}
+      CELL_DIR: results/sweep/${{ matrix.method }}_b${{ matrix.budget }}_L${{ matrix.depth }}_${{ matrix.test_set }}
       MANIFESTS_DIR: benchmarks/manifests/v1
       INPUT_TAU: '0.12'
       INPUT_CBF: '0.5'
@@ -475,6 +497,7 @@ jobs:
             "cell": {
               "method": "${{ matrix.method }}",
               "budget": ${{ matrix.budget }},
+              "depth": ${{ matrix.depth }},
               "test_set": "${{ matrix.test_set }}",
               "tau": ${INPUT_TAU},
               "core_budget_fraction": ${INPUT_CBF}
@@ -653,6 +676,12 @@ jobs:
           set -o pipefail
           python3 /tmp/heartbeat.py >> "${CELL_DIR}/heartbeat.log" 2>&1 &
           HEARTBEAT_PID=$!
+          # Apply EGO graph depth as an env override only when matrix.depth >= 0
+          # (the sentinel -1 means "depth doesn't apply to this method"; in that
+          # case we leave DIFFCTX_OP_GRAPH_DEPTH unset and the default kicks in).
+          if [ "${MATRIX_DEPTH}" != "-1" ]; then
+            export DIFFCTX_OP_GRAPH_DEPTH="${MATRIX_DEPTH}"
+          fi
           python -m benchmarks.run_final_eval \
             --baseline "${BASELINE}" \
             --winner /tmp/winner.json \
@@ -665,6 +694,7 @@ jobs:
           kill "${HEARTBEAT_PID}" 2>/dev/null || true
         env:
           BASELINE: ${{ steps.cfg.outputs.baseline }}
+          MATRIX_DEPTH: ${{ matrix.depth }}
 
       - name: Per-instance metric summary (always)
         if: always()
@@ -764,7 +794,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@v7
         with:
-          name: cell-${{ matrix.method }}-b${{ matrix.budget }}-${{ matrix.test_set }}
+          name: cell-${{ matrix.method }}-b${{ matrix.budget }}-L${{ matrix.depth }}-${{ matrix.test_set }}
           path: ${{ env.CELL_DIR }}/
           retention-days: 90
           if-no-files-found: warn

benchmarks/aggregate_sweep.py

@@ -1,14 +1,18 @@
 """Aggregate per-cell sweep artifacts into one summary JSON + markdown table.
 
 Input layout (from `actions/download-artifact@v4` with pattern=cell-*):
-    <cells-dir>/cell-<method>-b<budget>-<test_set>/
+    <cells-dir>/cell-<method>-b<budget>-L<depth>-<test_set>/
         metadata.json
         cell_summary.json
         <test_set>.checkpoint.jsonl
         <test_set>.json
         run.log
         system_info.log
 
+Legacy layout `cell-<method>-b<budget>-<test_set>/` is still parsed (depth=-1
+sentinel meaning "method does not consume depth") so old artifact dumps
+remain readable.
+
 Output:
     <out>/grand_summary.json   — every cell's metadata + summary in one file
     <out>/SWEEP_TABLE.md       — markdown matrix of mean recall per cell
@@ -63,12 +67,14 @@ def collect_cells(cells_dir: Path) -> list[dict]:
         ckpts = sorted(cell_root.glob("*.checkpoint.jsonl"))
         rows = _load_jsonl(ckpts[0]) if ckpts else []
         cell_info = meta.get("cell") or {}
+        parsed = _parse_artifact(cell_root.name)
         cells.append(
             {
                 "artifact_dir": cell_root.name,
-                "method": cell_info.get("method") or _parse_artifact(cell_root.name)[0],
-                "budget": cell_info.get("budget") if cell_info.get("budget") is not None else _parse_artifact(cell_root.name)[1],
-                "test_set": cell_info.get("test_set") or _parse_artifact(cell_root.name)[2],
+                "method": cell_info.get("method") or parsed[0],
+                "budget": cell_info.get("budget") if cell_info.get("budget") is not None else parsed[1],
+                "depth": cell_info.get("depth") if cell_info.get("depth") is not None else parsed[2],
+                "test_set": cell_info.get("test_set") or parsed[3],
                 "metadata": meta,
                 "summary": summary,
                 "n_instances": len(rows),
@@ -78,23 +84,42 @@ def collect_cells(cells_dir: Path) -> list[dict]:
     return cells
 
 
-_ARTIFACT_RE = __import__("re").compile(r"^cell-(?P<method>[a-zA-Z0-9_]+)-b(?P<budget>-?\d+)-(?P<test_set>.+)$")
+# New artifact layout: cell-<method>-b<budget>-L<depth>-<test_set>
+_ARTIFACT_RE_WITH_DEPTH = __import__("re").compile(
+    r"^cell-(?P<method>[a-zA-Z0-9_]+)-b(?P<budget>-?\d+)-L(?P<depth>-?\d+)-(?P<test_set>.+)$"
+)
+# Legacy artifact layout: cell-<method>-b<budget>-<test_set> (no depth segment)
+_ARTIFACT_RE_LEGACY = __import__("re").compile(r"^cell-(?P<method>[a-zA-Z0-9_]+)-b(?P<budget>-?\d+)-(?P<test_set>.+)$")
 
 
-def _parse_artifact(name: str) -> tuple[str | None, int | None, str | None]:
-    """Best-effort parse of a cell-<method>-b<budget>-<test_set> directory name.
+def _parse_artifact(name: str) -> tuple[str | None, int | None, int | None, str | None]:
+    """Parse a `cell-<method>-b<budget>-L<depth>-<test_set>` directory name.
 
-    Used as a fallback when metadata.json was not produced (e.g., the cell
-    crashed before the metadata step ran).
+    Returns (method, budget, depth, test_set). For legacy artifacts that
+    predate the depth axis, depth resolves to -1 (the sentinel meaning
+    "method does not consume depth"). Used as a fallback when
+    `metadata.json` was not produced (e.g., the cell crashed before the
+    metadata step ran).
     """
-    m = _ARTIFACT_RE.match(name)
+    m = _ARTIFACT_RE_WITH_DEPTH.match(name)
+    if m:
+        try:
+            budget = int(m.group("budget"))
+        except ValueError:
+            budget = None
+        try:
+            depth: int | None = int(m.group("depth"))
+        except ValueError:
+            depth = None
+        return (m.group("method"), budget, depth, m.group("test_set"))
+    m = _ARTIFACT_RE_LEGACY.match(name)
     if not m:
-        return (None, None, None)
+        return (None, None, None, None)
     try:
         budget = int(m.group("budget"))
     except ValueError:
         budget = None
-    return (m.group("method"), budget, m.group("test_set"))
+    return (m.group("method"), budget, -1, m.group("test_set"))
 
 
 _METHOD_ORDER = ["ppr", "ego", "bm25", "aider"]
@@ -147,6 +172,7 @@ def write_csv(cells: list[dict], path: Path) -> None:
     fields = [
         "method",
         "budget",
+        "depth",
         "test_set",
         "n_instances",
         "n_ok",
@@ -170,6 +196,7 @@ def write_csv(cells: list[dict], path: Path) -> None:
             row = {
                 "method": c["method"],
                 "budget": c["budget"],
+                "depth": c.get("depth"),
                 "test_set": c["test_set"],
                 "n_instances": s.get("n", c["n_instances"]),
                 "n_ok": s.get("ok", 0),

benchmarks/run_final_eval.py

@@ -193,7 +193,8 @@ def main() -> int:
         "because rel_scores depend on the traversal radius; budgets within a "
         "depth share scored state. Output: <name>_budget_sweep/L<depth>/b<budget>.checkpoint.jsonl. "
         "Empty (default): single depth from MODE.ego_depth_extended (= 2 unless "
-        "DIFFCTX_OP_GRAPH_DEPTH is set in the calling shell).",
+        "DIFFCTX_OP_GRAPH_DEPTH is set in the calling shell). Non-EGO scoring "
+        "modes ignore --depths (PPR uses alpha; BM25 has no graph traversal).",
     )
     args = p.parse_args()