feat(bench)+docs: --depths flag + editorial paper fixes (D)

nikolay-e · nikolay-e · commit 73678ac34344 · 2026-05-04T08:52:33.000+02:00
(1) Multi-depth orchestrator (E from previous review):

`run_final_eval.py --depths L1,L2,...` adds an outer loop over EGO graph
traversal radius. Heavy phase (graph build + scoring) re-runs per depth
because rel_scores depend on radius; budgets within a depth still share
scored state via the existing multi-budget reuse path. Output:

  &lt;name&gt;_budget_sweep/L&lt;depth&gt;/b&lt;budget&gt;.checkpoint.jsonl

When --depths is empty, behavior is unchanged from the previous commit
(single depth from MODE.ego_depth_extended). The flag is a no-op for
non-diffctx baselines and non-EGO scoring modes (warning printed).

Per-instance EvalResult.extra now records `ego_depth` when the run
explicitly set the depth, so the aggregator can stratify by L.

The per-manifest dispatcher was extracted into `_process_manifest` /
`_sweep_dir` helpers to keep `main` readable through the depth × manifest
× budgets nested loop. Behavior is bit-identical to the previous
single-budget and single-depth single-budget paths when those flags are
empty.

(2) Editorial paper fixes (D):

a. Post-hoc diff-understanding framing in §1. The intro now states
   explicitly that diffctx solves "post-hoc diff understanding" — the
   patch is given as input, recall measures coverage of the files the
   patch touches plus the dependency neighborhood needed to interpret
   them. Distinct from pre-solution issue resolution (the standard
   SWE-bench setup); we use those benchmarks' gold patches as known
   diffs against which retrieval is scored, never as a target an agent
   must generate. Closes the open framing item from external review.

b. "42 languages" downsize. Contributions list and Appendix
   tab:language-support now group languages by empirical-validation tier:
     Tier 1 (n&gt;=10 in eval): Python, TS, JS, Java, Go, C, Rust, C++.
     Tier 2 (semantic edge builders, no large-scale validation):
        ~30 languages including C#, Ruby, PHP, Swift, Haskell, etc.
     Tier 3 (parser only, no semantic edges): YAML, JSON, TOML.
     Generic fallback: Markdown, plain text, dotfiles, unparsed.
   The original undifferentiated "42 languages" claim conflated three
   different levels of support; the tiered table makes the empirical
   contribution and the implementation upper bound separately legible.
diff --git a/benchmarks/run_final_eval.py b/benchmarks/run_final_eval.py
@@ -67,6 +67,93 @@ def _load_winner(path: Path) -> RunParams:
     )
 
 
+def _sweep_dir(out: Path, name: str, depth: int | None) -> Path:
+    """Per-(manifest, depth) sweep subdirectory for budget-sharded checkpoints."""
+    base = out / f"{name}_budget_sweep"
+    if depth is not None:
+        base = base / f"L{depth}"
+    base.mkdir(parents=True, exist_ok=True)
+    return base
+
+
+def _process_manifest(
+    manifest_path: Path,
+    adapters: Any,
+    args: argparse.Namespace,
+    params: RunParams,
+    budgets_list: list[int],
+    eval_fn: Any,
+    eval_all_cells_fn: Any,
+    depth: int | None,
+) -> list[Any]:
+    name = manifest_path.stem.removeprefix("test_")
+    ids = read_manifest(manifest_path)
+    instances = [i for i in filter_instances_by_manifest(adapters, ids) if i.source_benchmark == name]
+    if args.limit:
+        instances = instances[: args.limit]
+    depth_label = f" L={depth}" if depth is not None else ""
+    print(f"\n[{name}{depth_label}] {len(instances)} instances")
+
+    if eval_all_cells_fn is not None:
+        params_list = [
+            RunParams(
+                tau=params.tau,
+                core_budget_fraction=params.core_budget_fraction,
+                budget=b,
+                scoring=params.scoring,
+            )
+            for b in budgets_list
+        ]
+        ckpt_dir = _sweep_dir(args.out, name, depth)
+        results_by_budget = run_eval_set_multi_budget(
+            instances,
+            eval_all_cells_fn,
+            params_list,
+            workers=args.workers,
+            timeout_per_instance=args.timeout_per_instance,
+            resume_dir=ckpt_dir,
+            checkpoint_dir=ckpt_dir,
+        )
+        headline_budget = params.budget if params.budget in results_by_budget else budgets_list[-1]
+        return results_by_budget[headline_budget]
+
+    if len(budgets_list) > 1:
+        ckpt_dir = _sweep_dir(args.out, name, depth)
+        results_by_budget: dict[int, list[Any]] = {}
+        for b in budgets_list:
+            cell_params = RunParams(
+                tau=params.tau,
+                core_budget_fraction=params.core_budget_fraction,
+                budget=b,
+                scoring=params.scoring,
+            )
+            ckpt_b = ckpt_dir / f"b{b}.checkpoint.jsonl"
+            rs = run_eval_set(
+                instances,
+                eval_fn,
+                cell_params,
+                workers=args.workers,
+                timeout_per_instance=args.timeout_per_instance,
+                resume_from=ckpt_b,
+                checkpoint_path=ckpt_b,
+            )
+            results_by_budget[b] = rs
+        headline_budget = params.budget if params.budget in results_by_budget else budgets_list[-1]
+        return results_by_budget[headline_budget]
+
+    depth_suffix = f"_L{depth}" if depth is not None else ""
+    ckpt = args.out / f"{name}{depth_suffix}.checkpoint.jsonl"
+    return run_eval_set(
+        instances,
+        eval_fn,
+        params,
+        workers=args.workers,
+        timeout_per_instance=args.timeout_per_instance,
+        resume_from=ckpt,
+        checkpoint_path=ckpt,
+    )
+
+
 def main() -> int:
     p = argparse.ArgumentParser(description=__doc__)
     p.add_argument("--winner", type=Path, required=True)
@@ -95,6 +182,19 @@ def main() -> int:
         "each budget as a separate process). Output: <name>__b<budget>.checkpoint.jsonl. "
         "Empty (default): use winner.budget as a single cell with the legacy path.",
     )
+    p.add_argument(
+        "--depths",
+        type=str,
+        default="",
+        help="Comma-separated EGO graph traversal depths (e.g. '0,1,2,3,4'). "
+        "When set with --baseline=diffctx and --scoring=ego, the orchestrator "
+        "loops over each depth as the outer axis and reuses --budgets within "
+        "each depth. Heavy phase (graph build + scoring) is re-run per depth "
+        "because rel_scores depend on the traversal radius; budgets within a "
+        "depth share scored state. Output: <name>_budget_sweep/L<depth>/b<budget>.checkpoint.jsonl. "
+        "Empty (default): single depth from MODE.ego_depth_extended (= 2 unless "
+        "DIFFCTX_OP_GRAPH_DEPTH is set in the calling shell).",
+    )
     args = p.parse_args()
 
     repo_root = args.repos_dir or default_repos_dir()
@@ -125,6 +225,13 @@ def main() -> int:
                 "looping per budget without compute_scored_state reuse."
             )
 
+    depths_list: list[int] = []
+    if args.depths.strip():
+        depths_list = [int(x.strip()) for x in args.depths.split(",") if x.strip()]
+        if args.baseline != "diffctx" or params.scoring != "ego":
+            print(f"--depths set but baseline={args.baseline} scoring={params.scoring}; " "depths only affect EGO. Ignoring.")
+            depths_list = []
+
     eval_fn = _make_eval_fn(args.baseline, repo_root, request_timeout=args.timeout_per_instance)
     eval_all_cells_fn = (
         make_diffctx_eval_all_cells_fn(repo_root) if args.baseline == "diffctx" and len(budgets_list) > 1 else None
@@ -133,88 +240,38 @@ def main() -> int:
     args.out.mkdir(parents=True, exist_ok=True)
     reports = []
     all_results = []
-    for manifest_path in manifests:
-        name = manifest_path.stem.removeprefix("test_")
-        ids = read_manifest(manifest_path)
-        instances = [i for i in filter_instances_by_manifest(adapters, ids) if i.source_benchmark == name]
-        if args.limit:
-            instances = instances[: args.limit]
-        print(f"\n[{name}] {len(instances)} instances")
-
-        if eval_all_cells_fn is not None:
-            # Multi-budget reuse path: compute_scored_state runs once per
-            # instance, select_with_params runs once per budget. The
-            # checkpoint files are sharded as <name>__b<budget>.checkpoint.jsonl
-            # so the existing aggregator picks them up unchanged.
-            params_list = [
-                RunParams(
-                    tau=params.tau,
-                    core_budget_fraction=params.core_budget_fraction,
-                    budget=b,
-                    scoring=params.scoring,
-                )
-                for b in budgets_list
-            ]
-            ckpt_dir = args.out / f"{name}_budget_sweep"
-            ckpt_dir.mkdir(parents=True, exist_ok=True)
-            results_by_budget = run_eval_set_multi_budget(
-                instances,
-                eval_all_cells_fn,
-                params_list,
-                workers=args.workers,
-                timeout_per_instance=args.timeout_per_instance,
-                resume_dir=ckpt_dir,
-                checkpoint_dir=ckpt_dir,
-            )
-            # Pick the single "headline" budget (winner) for the per-manifest
-            # aggregation; per-budget JSONLs land in the sweep subdir.
-            headline_budget = params.budget if params.budget in results_by_budget else budgets_list[-1]
-            results = results_by_budget[headline_budget]
-        elif len(budgets_list) > 1:
-            # Multi-budget loop for non-diffctx baselines. No reuse, but
-            # uniform output schema with the diffctx path.
-            results = []
-            ckpt_dir = args.out / f"{name}_budget_sweep"
-            ckpt_dir.mkdir(parents=True, exist_ok=True)
-            results_by_budget: dict[int, list[Any]] = {}
-            for b in budgets_list:
-                cell_params = RunParams(
-                    tau=params.tau,
-                    core_budget_fraction=params.core_budget_fraction,
-                    budget=b,
-                    scoring=params.scoring,
-                )
-                ckpt_b = ckpt_dir / f"b{b}.checkpoint.jsonl"
-                rs = run_eval_set(
-                    instances,
-                    eval_fn,
-                    cell_params,
-                    workers=args.workers,
-                    timeout_per_instance=args.timeout_per_instance,
-                    resume_from=ckpt_b,
-                    checkpoint_path=ckpt_b,
-                )
-                results_by_budget[b] = rs
-            headline_budget = params.budget if params.budget in results_by_budget else budgets_list[-1]
-            results = results_by_budget[headline_budget]
-        else:
-            ckpt = args.out / f"{name}.checkpoint.jsonl"
-            results = run_eval_set(
-                instances,
-                eval_fn,
-                params,
-                workers=args.workers,
-                timeout_per_instance=args.timeout_per_instance,
-                resume_from=ckpt,
-                checkpoint_path=ckpt,
-            )
 
-        for r in results:
-            r.extra.setdefault("benchmark_manifest", name)
-        all_results.extend(results)
-        report = aggregate_test_set(name, results)
-        reports.append(report)
-        (args.out / f"{name}.json").write_text(json.dumps([asdict(r) for r in results], indent=2, default=str))
+    # When --depths is set, loop EGO traversal radius as the outer axis.
+    # Heavy phase (graph build + scoring) re-runs per depth because
+    # rel_scores depend on radius; budgets within a depth share scored state.
+    # When --depths is empty, the depth is whatever DIFFCTX_OP_GRAPH_DEPTH
+    # the parent shell set (default 2 from MODE.ego_depth_extended).
+    loop_depths: list[int | None] = list(depths_list) if depths_list else [None]
+    for depth in loop_depths:
+        if depth is not None:
+            _os.environ["DIFFCTX_OP_GRAPH_DEPTH"] = str(depth)
+            print(f"\n=== Sweep depth L={depth} (DIFFCTX_OP_GRAPH_DEPTH={depth}) ===")
+        for manifest_path in manifests:
+            results = _process_manifest(
+                manifest_path=manifest_path,
+                adapters=adapters,
+                args=args,
+                params=params,
+                budgets_list=budgets_list,
+                eval_fn=eval_fn,
+                eval_all_cells_fn=eval_all_cells_fn,
+                depth=depth,
+            )
+            name = manifest_path.stem.removeprefix("test_")
+            for r in results:
+                r.extra.setdefault("benchmark_manifest", name)
+                if depth is not None:
+                    r.extra.setdefault("ego_depth", depth)
+            all_results.extend(results)
+            report = aggregate_test_set(name, results)
+            reports.append(report)
+            depth_suffix = f"_L{depth}" if depth is not None else ""
+            (args.out / f"{name}{depth_suffix}.json").write_text(json.dumps([asdict(r) for r in results], indent=2, default=str))
 
     paper_table = render_paper_table(reports)
     lang_agg = aggregate_by_language(all_results)
diff --git a/docs/Context-Selection-for-Git-Diff/v2/main.pdf b/docs/Context-Selection-for-Git-Diff/v2/main.pdf
diff --git a/docs/Context-Selection-for-Git-Diff/v2/main.tex b/docs/Context-Selection-for-Git-Diff/v2/main.tex
@@ -45,13 +45,15 @@ \section{Introduction}
 
 We address this gap by formalizing diff context selection as \textbf{budgeted submodular maximization} over Code Property Graphs. Our key insight is that ``understanding a change'' can be operationalized as covering explanatory concepts---definitions, call targets, type signatures, impacted tests---that propagate through structural dependencies with diminishing returns.
 
+\paragraph{Task scope: post-hoc diff understanding.} We address \emph{post-hoc diff understanding}: a patch $\Delta$ is given as input, and the task is to retrieve a compact set of explanatory code fragments needed to review or reason about that patch within a token budget. This is distinct from \emph{pre-solution issue resolution} (the standard SWE-bench setup), where an agent must produce $\Delta$ from a problem statement without seeing the gold patch. The benchmarks in Section~\ref{sec:eval} are repurposed from issue-resolution datasets: we use their gold patches as known diffs against which file-level retrieval is scored, never as a target an agent must generate. The patch is on the input side; the recall metric measures whether the retrieved context covers the files the patch touches and the dependency neighborhood needed to interpret them.
+
 \paragraph{Contributions.}
 \begin{enumerate}
     \item A formal problem statement casting context selection as budgeted utility maximization over a partition matroid on multi-resolution fragments. We do not claim a new approximation algorithm; we use known optimization structure to make context selection explicit, analyzable, and extensible. An ``algorithm $\times$ constraint $\times$ guarantee'' table (Table~\ref{tab:algo-constraint-guarantee}) separates the deployed heuristic from analyzable variants and from the submodular concept-coverage extension
     \item A typed-edge dependency-graph framework with per-category extraction and treatment (e.g., hub-suppression exemption), generalizing flat code-graph approaches
     \item A pluggable relevance-scoring abstraction $\hat{w}(f, \Delta)$ where three algorithm families (bounded ego-network expansion, Personalized PageRank, BM25) instantiate the abstract signal under a single retrieval framework
     \item An evaluation protocol using file-level recall against annotated golden contexts at a fixed token budget on multi-language software-engineering benchmarks, with paired-bootstrap statistical comparison against external baselines (BM25 over patch identifiers; Aider repo-map)
-    \item A reference implementation supporting 42 programming languages and configuration formats (Appendix~\ref{sec:impl-inventory})
+    \item A reference implementation with tree-sitter parsing for 42 source and configuration formats and dedicated semantic edge builders for 36 of them (Appendix~\ref{sec:impl-inventory}). Of these, eight (Python, JavaScript, TypeScript, Java, Go, Rust, C, C++) appear in the empirical evaluation with $n{\geq}10$ instances; the rest are supported but unvalidated at scale (Section~\ref{sec:impl-inventory}, Table~\ref{tab:language-support})
 \end{enumerate}
 
 \newpage
@@ -792,24 +794,34 @@ \section{Heuristics Annex}
 \section{Language Support}
 \label{sec:impl-inventory}
 
-The reference implementation provides dedicated tree-sitter extraction for 42 languages, grouped by domain. Languages without dedicated extractors fall back to a generic strategy (paragraph-level fragmentation for prose-like content, statement-level for code-like content).
+The reference implementation includes tree-sitter parsing for 42 source and configuration formats and dedicated semantic edge builders for 36 of them. Coverage is uneven: a handful of languages have been used extensively in the evaluation; most others are present at the parser/edge-builder level but unvalidated against a benchmark of meaningful size. Table~\ref{tab:language-support} groups languages by tier of empirical exercise so the contribution and the open work are not conflated.
 
 \begin{table}[h]
 \centering
 \small
-\begin{tabular}{lp{8.5cm}}
+\begin{tabular}{lp{9.5cm}}
 \toprule
-\textbf{Group} & \textbf{Languages} \\
+\textbf{Tier} & \textbf{Languages} \\
+\midrule
+Tier 1 — empirically validated ($n{\geq}10$ in evaluation) &
+Python ($n{=}891$), TypeScript ($n{=}191$), JavaScript ($n{=}185$),
+Java ($n{=}140$), Go ($n{=}40$), C ($n{=}23$), Rust ($n{=}20$), C++ ($n{=}10$) \\
+\midrule
+Tier 2 — semantic edge builders, no large-scale empirical validation &
+C\#, Kotlin, Ruby, PHP, Swift, Scala (general-purpose);
+Haskell, Elixir, Erlang, OCaml, Clojure, Julia (functional);
+Lua, R, Perl, Bash/Zsh/Fish/Ksh, Dart, Groovy, Objective-C (scripting);
+HTML, CSS/SCSS/LESS, GraphQL, HCL/Terraform, CMake, Make, Nix (markup/build);
+Cargo, Bazel, dbt, OpenAPI, Prisma, Protobuf, SQL, Ansible (domain) \\
+\midrule
+Tier 3 — parser only, structural fragmentation without semantic edges &
+YAML, JSON, TOML \\
 \midrule
-General-purpose & Python, JavaScript, TypeScript (incl.\ JSX/TSX), Go, Rust, Java, Kotlin, C, C++, C\#, Ruby, PHP, Swift, Scala \\
-Functional      & Haskell, Elixir, Erlang, OCaml, Clojure, Julia, F\# \\
-Scripting       & Lua, R, Perl, Bash/Zsh/Fish/Ksh, Dart, Groovy, Objective-C \\
-Markup/Data     & HTML, CSS/SCSS/LESS, YAML, JSON, GraphQL, HCL/Terraform, CMake, Make, Nix \\
-Domain-specific & Cargo (Rust), Bazel, dbt, OpenAPI, Prisma, Protobuf, SQL, Ansible \\
-Generic fallback & Markdown, plain text, TOML, dotfiles, unparsed sources \\
+Generic fallback &
+Markdown, plain text, dotfiles, unparsed sources (paragraph or statement fragmentation) \\
 \bottomrule
 \end{tabular}
-\caption{Languages supported by the reference implementation. General-purpose and functional languages have full semantic edge builders (call/import resolution, type references); markup and data formats use schema-aware extractors where applicable.}
+\caption{Tiered coverage of the reference implementation. Tier 1 has both edge builders and per-language recall numbers in Table~\ref{tab:prelim-lang}. Tier 2 implements the same edge builders but is not exercised against a benchmark of meaningful size in this paper; its inclusion is a code-level claim, not an empirical one. Tier 3 supplies parsing only — sufficient to fragment files for selection but not to draw structural edges. Treat the count of supported formats as an implementation upper bound; only Tier 1 carries empirical evidence.}
 \label{tab:language-support}
 \end{table}
 
