Add scoped impact ranking benchmark results

* Add impact analysis benchmark harness

* Keep benchmark harness separate from results

* Add scoped impact ranking benchmark results

---------

Co-authored-by: jonathanpopham <jonathan.popham@gmail.com>

Author: greynewell

benchmark/agent-impact/results/real-impact-ranking-opt-in-50c6ae58/README.md

@@ -0,0 +1,95 @@
+# Real Impact Ranking Result: API 50c6ae58
+
+This result records the 10-repository scoped impact-ranking run against the public API implementation at `50c6ae58e408fabcd8a4b36f960b0988f5804432`.
+
+It is intentionally separate from the benchmark harness PR. The harness lives in `benchmark/agent-impact`; this directory is the evidence bundle for one run.
+
+## Run Identity
+
+- Generated: `2026-05-08T20:09:31.845Z`
+- CLI branch: `impact-analysis-benchmark`
+- CLI commit used for this artifact branch base: `3738a6381c156f3aef19261a88b38337653296ab`
+- API branch: `impact-analysis-api-fixes`
+- API commit: `50c6ae58e408fabcd8a4b36f960b0988f5804432`
+- Node: `v24.3.0`
+- npm: `11.4.2`
+- Manifest: `benchmark/agent-impact/post-cutoff-prs.json`
+- Scope: `replay-dirs`
+- Cases: 10
+
+The ranking runner imports the local public API data-plane implementation directly through `SUPERMODEL_PUBLIC_API_REPO`. It does not call the hosted API. Public endpoint compatibility is covered by the API PR tests and smoke scripts.
+
+## Reproduce
+
+Check out the API branch and install data-plane dependencies:
+
+```bash
+git clone git@github.com:supermodeltools/supermodel-public-api.git
+cd supermodel-public-api
+git checkout impact-analysis-api-fixes
+git rev-parse HEAD
+
+cd src/data-plane
+npm install
+```
+
+Check out the CLI benchmark harness:
+
+```bash
+git clone git@github.com:supermodeltools/cli.git
+cd cli
+git checkout impact-analysis-benchmark
+git rev-parse HEAD
+```
+
+Run the benchmark:
+
+```bash
+export SUPERMODEL_PUBLIC_API_REPO=/absolute/path/to/supermodel-public-api
+
+node benchmark/agent-impact/run-real-impact-ranking.mjs \
+  --out-dir target/real-impact-ranking-opt-in-50c6ae58 \
+  --scope replay-dirs
+```
+
+The runner clones the pinned public repositories from `post-cutoff-prs.json`, checks out each PR base SHA, applies the PR regression files from the merge commit, withholds the production fix, builds a scoped graph, and scores the generated `validationFiles` against the hidden PR regression-file labels.
+
+The full local output includes cloned repositories under `target/`, which is intentionally not checked in. This artifact keeps only:
+
+- `report.md`
+- sanitized `summary.json`
+- per-case `IMPACT_ANALYSIS_SCOPED.md`
+- per-case `impact-analysis.scoped.json`
+- per-case `reference-production.diff`
+
+## Headline Result
+
+The headline comparison uses the same 10 post-cutoff PR cases and the same 21 labeled regression files.
+
+| Method | Precision | Recall | F1 | Correct / Expected | Predicted |
+|---|---:|---:|---:|---:|---:|
+| Baseline path/name matcher | 0.060 | 0.286 | 0.099 | 6 / 21 | 100 |
+| Supermodel best scoped packet | 0.274 | 0.952 | 0.426 | 20 / 21 | 73 |
+
+Interpretation: Supermodel recovered 20 of 21 labeled validation files versus 6 of 21 for the naive path/name baseline, while returning fewer candidates. The strongest claim here is recall: scoped impact analysis finds relevant validation files that simple proximity misses. Precision still needs work.
+
+## Per-Repo Summary
+
+| Repo / PR | Expected | Baseline F1 | Supermodel F1 | Supermodel Correct | Supermodel Candidates |
+|---|---:|---:|---:|---:|---:|
+| Next.js #93417 | 4 | 0.000 | 0.615 | 4 / 4 | 9 |
+| VS Code #314217 | 1 | 0.182 | 0.333 | 1 / 1 | 5 |
+| MUI #48472 | 1 | 0.182 | 1.000 | 1 / 1 | 1 |
+| Grafana #123935 | 1 | 0.182 | 0.200 | 1 / 1 | 9 |
+| React #36047 | 1 | 0.000 | 0.200 | 1 / 1 | 9 |
+| Angular #68512 | 1 | 0.000 | 0.400 | 1 / 1 | 4 |
+| Prisma #29512 | 5 | 0.133 | 0.714 | 5 / 5 | 9 |
+| Payload #16465 | 5 | 0.000 | 0.571 | 4 / 5 | 9 |
+| Superset #39504 | 1 | 0.182 | 0.200 | 1 / 1 | 9 |
+| Terraform #38338 | 1 | 0.182 | 0.200 | 1 / 1 | 9 |
+
+## Notes
+
+The benchmark is scoped by design. It asks: "if this production file or function changes, which validation files should be inspected or run first?" It is not trying to return the impact of every file in the repository.
+
+The checked-in `summary.json` has local absolute paths replaced with `<cli-repo>` and `<public-api-repo>`. The raw cloned repositories and local caches are excluded from the artifact.

benchmark/agent-impact/results/real-impact-ranking-opt-in-50c6ae58/cases/angular-68512-async-validator-pending/IMPACT_ANALYSIS_SCOPED.md

@@ -0,0 +1,81 @@
+diff --git a/packages/core/src/core_private_export.ts b/packages/core/src/core_private_export.ts
+index c2237d9ec0..00692f1630 100644
+--- a/packages/core/src/core_private_export.ts
++++ b/packages/core/src/core_private_export.ts
+@@ -153,6 +153,7 @@ export {
+   SharedStylesHost as ɵSharedStylesHost,
+ } from './render3/interfaces/shared_styles_host';
+ export {
++  chain as ɵchain,
+   encapsulateResourceError as ɵencapsulateResourceError,
+   ResourceImpl as ɵResourceImpl,
+ } from './resource/resource';
+diff --git a/packages/core/src/resource/resource.ts b/packages/core/src/resource/resource.ts
+index dad83f1412..3655ea3881 100644
+--- a/packages/core/src/resource/resource.ts
++++ b/packages/core/src/resource/resource.ts
+@@ -598,19 +598,21 @@ class ResourceWrappedError extends Error {
+  * of the other resource if it is available, or propagating the status to the current resource if it
+  * is not.
+  */
++export function chain<T>(resource: Resource<T>): T {
++  switch (resource.status()) {
++    case 'idle':
++      throw ResourceParamsStatus.IDLE;
++    case 'error':
++      throw new ResourceDependencyError(resource);
++    case 'loading':
++    case 'reloading':
++      throw ResourceParamsStatus.LOADING;
++  }
++  return resource.value();
++}
++
+ export const paramsContext: ResourceParamsContext = {
+-  chain<T>(resource: Resource<T>): T {
+-    switch (resource.status()) {
+-      case 'idle':
+-        throw ResourceParamsStatus.IDLE;
+-      case 'error':
+-        throw new ResourceDependencyError(resource);
+-      case 'loading':
+-      case 'reloading':
+-        throw ResourceParamsStatus.LOADING;
+-    }
+-    return resource.value();
+-  },
++  chain,
+ };
+ 
+ let inParamsFunction = false;
+diff --git a/packages/forms/signals/src/api/rules/validation/validate_async.ts b/packages/forms/signals/src/api/rules/validation/validate_async.ts
+index 8785386475..ad3feb196c 100644
+--- a/packages/forms/signals/src/api/rules/validation/validate_async.ts
++++ b/packages/forms/signals/src/api/rules/validation/validate_async.ts
+@@ -6,7 +6,15 @@
+  * found in the LICENSE file at https://angular.dev/license
+  */
+ 
+-import {DebounceTimer, ResourceRef, ResourceSnapshot, Signal, debounced} from '@angular/core';
++import {
++  computed,
++  DebounceTimer,
++  ResourceRef,
++  ResourceSnapshot,
++  Signal,
++  debounced,
++  ɵchain,
++} from '@angular/core';
+ import {FieldNode} from '../../../field/node';
+ import {addDefaultField} from '../../../field/validation';
+ import {FieldPathNode} from '../../../schema/path_node';
+@@ -127,7 +135,8 @@ export function validateAsync<TValue, TParams, TResult, TPathKind extends PathKi
+     (_state, params) => {
+       if (opts.debounce !== undefined) {
+         const debouncedResource = debounced(() => params(), opts.debounce);
+-        return opts.factory(debouncedResource.value);
++        const wrappedParams = computed(() => ɵchain(debouncedResource));
++        return opts.factory(wrappedParams);
+       }
+       return opts.factory(params);
+     },

benchmark/agent-impact/results/real-impact-ranking-opt-in-50c6ae58/cases/angular-68512-async-validator-pending/impact-analysis.scoped.json

@@ -0,0 +1,79 @@
+# Scoped Impact Analysis
+
+Generated by: `supermodel-real-scoped-impact-ranking`
+
+Repository: `grafana/grafana`
+
+Scope: `replay-dirs`
+
+Generated by the local data-plane impact implementation for replay analysis. If diff-derived function targets are present, this benchmark used reference-production.diff; production use should pass user-supplied targets/diffs.
+
+## single-production-file-target->tests:pkg/services/ngalert/store/alert_rule.go
+
+Targets:
+`pkg/services/ngalert/store/alert_rule.go`
+
+Structurally affected files:
+`pkg/services/ngalert/store/provisioning_store_test.go`, `pkg/services/ngalert/store/image_test.go`
+
+Validation files to inspect or run first:
+- `pkg/services/ngalert/store/alert_rule_test.go` (high, score 302) — paired test filename; same directory; mentions changed symbol; shares change terms: optimistic, lock, delete, alert, rule, uid, version, increase; filename shares diff terms: alert, rule
+- `pkg/services/ngalert/store/image_test.go` (high, score 267) — structural reverse dependency; same directory; mentions changed symbol; shares change terms: delete, interval, dbstore, sess, session, int64, where, same; path contains compound diff terms: alert
+- `pkg/services/ngalert/store/provisioning_store_test.go` (high, score 253) — structural reverse dependency; same directory; mentions changed symbol; shares change terms: lock, delete, alert, rule, uid, update, group, uids; path contains compound diff terms: alert
+- `pkg/services/ngalert/store/alert_rule_backfill_test.go` (high, score 216) — same directory; mentions changed symbol; shares change terms: alert, rule, uid, namespace, folder, fullpath, update, list; filename shares diff terms: alert, rule; content contains compound diff terms: note, here
+- `pkg/services/ngalert/store/deltas_test.go` (high, score 207) — same directory; mentions changed symbol; shares change terms: delete, alert, rule, uid, version, namespace, folder, update; path contains compound diff terms: alert; content contains compound diff terms: ngmodel, sess, note, here
+- `pkg/services/ngalert/store/alert_rule_labels_test.go` (high, score 203) — same directory; mentions changed symbol; shares change terms: alert, build, condition, matcher, label, prometheu, origin, sql; filename shares diff terms: alert, rule; content contains compound diff terms: note
+- `pkg/services/ngalert/store/namespace_test.go` (high, score 203) — same directory; mentions changed symbol; shares change terms: uid, namespace, folder, list, uids, org, title, existing; filename shares diff terms: namespace; path contains compound diff terms: alert
+- `pkg/services/ngalert/store/alertmanager_test.go` (high, score 191) — same directory; mentions changed symbol; shares change terms: delete, alert, version, latest, update, deleted, org, existing; path contains compound diff terms: alert; content contains compound diff terms: note, here
+- `pkg/services/ngalert/store/range_to_instant_test.go` (high, score 191) — same directory; mentions changed symbol; shares change terms: alert, rule, uid, query, interval, point, prometheu, invalid; path contains compound diff terms: alert; content contains compound diff terms: ngmodel, here
+- `pkg/services/ngalert/store/admin_configuration_test.go` (high, score 183) — same directory; mentions changed symbol; shares change terms: alert, update, org, insert, existing, sql, dbstore, ngmodel; path contains compound diff terms: alert; content contains compound diff terms: here
+
+Primary validation files:
+- none
+
+## single-production-function-target->tests:pkg/services/ngalert/store/alert_rule.go:ListAlertRulesByGroup
+
+Targets:
+`pkg/services/ngalert/store/alert_rule.go:ListAlertRulesByGroup`
+
+Structurally affected files:
+_none_
+
+Validation files to inspect or run first:
+- `pkg/services/ngalert/store/alert_rule_test.go` (high, score 302) — paired test filename; same directory; mentions changed symbol; shares change terms: list, alert, rule, group, dbstore, query, cursor, condition; filename shares diff terms: alert, rule
+- `pkg/services/ngalert/store/alert_rule_backfill_test.go` (high, score 156) — same directory; shares change terms: list, alert, rule, org, sess, session, int64, uid; filename shares diff terms: alert, rule; content contains compound diff terms: note, here
+- `pkg/services/ngalert/store/deltas_test.go` (high, score 147) — same directory; shares change terms: list, alert, rule, group, query, condition, org, int64; path contains compound diff terms: alert; content contains compound diff terms: ngmodel, sess, note, here
+- `pkg/services/ngalert/store/namespace_test.go` (high, score 143) — same directory; shares change terms: list, dbstore, org, int64, uid, folder, namespace, sql; filename shares diff terms: namespace; path contains compound diff terms: alert; content contains compound diff terms: alert
+- `pkg/services/ngalert/store/compat_test.go` (high, score 123) — same directory; shares change terms: alert, rule, group, query, ngmodel, condition, org, uid; path contains compound diff terms: alert; content contains compound diff terms: note
+- `pkg/services/ngalert/store/instance_database_test.go` (high, score 123) — same directory; shares change terms: list, alert, rule, group, dbstore, query, condition, org; path contains compound diff terms: alert; content contains compound diff terms: ngmodel
+- `pkg/services/ngalert/store/proto_instance_database_test.go` (high, score 123) — same directory; shares change terms: alert, rule, group, condition, org, int64, uid, append; path contains compound diff terms: alert; content contains compound diff terms: note
+- `pkg/services/ngalert/store/image_test.go` (high, score 117) — same directory; shares change terms: dbstore, sess, session, int64, where, same; path contains compound diff terms: alert; content contains compound diff terms: alert, query, note, here
+- `pkg/services/ngalert/store/instance_database_bench_test.go` (high, score 113) — same directory; shares change terms: alert, rule, group, dbstore, org, int64, uid, append; path contains compound diff terms: alert; content contains compound diff terms: ngmodel
+- `pkg/services/ngalert/store/provisioning_store_test.go` (high, score 113) — same directory; shares change terms: alert, rule, group, dbstore, org, int64, uid, same; path contains compound diff terms: alert; content contains compound diff terms: sess
+
+Primary validation files:
+- none
+
+## single-production-function-target->tests:pkg/services/ngalert/store/alert_rule.go:buildGroupCursorCondition
+
+Targets:
+`pkg/services/ngalert/store/alert_rule.go:buildGroupCursorCondition`
+
+Structurally affected files:
+`pkg/services/ngalert/store/alert_rule.go`
+
+Validation files to inspect or run first:
+- `pkg/services/ngalert/store/alert_rule_test.go` (high, score 242) — paired test filename; same directory; shares change terms: group, cursor, condition, dbstore, list, alert, rule, query; filename shares diff terms: alert, rule; content contains compound diff terms: ngmodel, select, note, here
+- `pkg/services/ngalert/store/alert_rule_backfill_test.go` (high, score 156) — same directory; shares change terms: list, alert, rule, org, sess, session, int64, uid; filename shares diff terms: alert, rule; content contains compound diff terms: note, here
+- `pkg/services/ngalert/store/deltas_test.go` (high, score 147) — same directory; shares change terms: group, condition, list, alert, rule, query, org, int64; path contains compound diff terms: alert; content contains compound diff terms: ngmodel, sess, note, here
+- `pkg/services/ngalert/store/namespace_test.go` (high, score 143) — same directory; shares change terms: dbstore, list, org, int64, uid, folder, namespace, sql; filename shares diff terms: namespace; path contains compound diff terms: alert; content contains compound diff terms: alert
+- `pkg/services/ngalert/store/compat_test.go` (high, score 123) — same directory; shares change terms: group, condition, alert, rule, query, ngmodel, org, uid; path contains compound diff terms: alert; content contains compound diff terms: note
+- `pkg/services/ngalert/store/instance_database_test.go` (high, score 123) — same directory; shares change terms: group, condition, dbstore, list, alert, rule, query, org; path contains compound diff terms: alert; content contains compound diff terms: ngmodel
+- `pkg/services/ngalert/store/proto_instance_database_test.go` (high, score 123) — same directory; shares change terms: group, condition, alert, rule, org, int64, uid, append; path contains compound diff terms: alert; content contains compound diff terms: note
+- `pkg/services/ngalert/store/image_test.go` (high, score 117) — same directory; shares change terms: dbstore, sess, session, int64, where, same; path contains compound diff terms: alert; content contains compound diff terms: alert, query, note, here
+- `pkg/services/ngalert/store/instance_database_bench_test.go` (high, score 113) — same directory; shares change terms: group, dbstore, alert, rule, org, int64, uid, append; path contains compound diff terms: alert; content contains compound diff terms: ngmodel
+- `pkg/services/ngalert/store/provisioning_store_test.go` (high, score 113) — same directory; shares change terms: group, dbstore, alert, rule, org, int64, uid, same; path contains compound diff terms: alert; content contains compound diff terms: sess
+
+Primary validation files:
+- none
+