feat(bench): agent-effectiveness benchmark pilot, A0 vs A2 on S1/S3 with seeded ground truth and audit-derived metrics

[cjimti]

Closes #942 (phase 1 of #930).

What this adds

bench/ is the agent-effectiveness benchmark harness: it measures whether an agent connected to this platform answers real data questions more correctly and more efficiently than the same agent connected to bare data tools. It ablates the PLATFORM, not the model: every run holds the model, prompt scaffold, seed data, and task set constant and varies only the platform configuration. It is the "how well" sibling of the test/load "how much" harness and follows the same separate-Go-module convention (excluded from the root ./... gates, own .golangci.yml, checks via make bench-test), and like load and mutation testing it is deliberately not part of make verify.

Arms as config profiles

The ablation mechanism is the platform's own config surface (bench/config/), not code forks:

Arm	What the agent gets
a0	Raw toolkit tools only (trino_*, s3_*); no semantic provider, no search tool, all cross-enrichment off, search-first gate off (the gate is not persona-aware, so a persona without a discovery tool would deadlock on SEARCH_REQUIRED). Equivalent to wiring the standalone toolkit libraries.
a2	The shipped semantic-first platform: DataHub enrichment, the search tool, the search-first gate, seeded metadata and knowledge pages.

Each profile defines a 32-key identity pool alongside the base admin key: the search-first gate keys discovery on the authenticated USER, so every task attempt authenticates as its own pool identity, and a run refuses to start when tasks x k exceeds the pool. make bench-run also resets the persisted search_gate_discovery state so repeated runs start clean. Arms a1 (enrichment-only) and a3 (memory lifecycle) join in phases 2 and 3.

Seeded ground truth

bench/seedgen generates every artifact from one fixed-seed dataset model (seed 930): Trino DDL/DML (memory catalog), DataHub metadata proposals (descriptions, column units, deprecation), knowledge-page SQL, the task YAML, and the smoke script. Ground truths are computed from the generated rows, never hand-typed, and each numeric task records the reference SQL it mirrors. A determinism test (TestCommittedArtifactsMatch) fails when the committed artifacts drift from regeneration (make bench-gen refreshes them).

The pilot task set is 10 tasks: 5 S1 discovery tasks (entity-graded) and 5 S3 knowledge traps across two seeded trap classes:

• units_cents: monetary columns are integers in US cents; the fact lives only in the metadata layer (column and dataset descriptions).
• net_revenue: policy revenue = amount - discount over completed orders only; the fact lives in a knowledge page and dataset description, and the gross-revenue leader and the policy net-revenue leader differ by construction (the generator asserts it).

Rubric items (required caveats) are recorded into per-attempt transcripts for manual review; LLM-judged scoring is phase 2 (#943).

Measurement through the platform's audit log

The harness mints the dps_ session handle via platform_info, strips the injected session_id property from every tool schema shown to the model, and threads the handle itself: measurement plumbing invisible to the agent and uniform across arms. Efficiency metrics come from GET /api/v1/admin/audit/events?session_id=... with audit.delivery: sync. A run fails loudly when a session's rows fall outside the client-side accounting bounds: confirmed calls must all have rows; platform refusals (authz, the gates, the per-user limiter) short-circuit outer to the audit middleware and have none; transport-level failures count as indeterminate (the platform may have audited before the error surfaced). Harness-level failures are excluded from accuracy and pass^k and reported separately, so an infrastructure hiccup can never masquerade as a platform quality signal.

Model adapters

The LLM adapter is provider-pluggable:

• anthropic (official Go SDK): real runs, model pinned in the run manifest, no sampling parameters; k-repeats and pass^k carry run-to-run variance.
• scripted: plays a deterministic script in which SQL-backed tasks run their reference SQL through the live trino_query and answer with the live result. One smoke run therefore validates the seed data, stored ground truths, handle threading, audit read-back, and both graders against the real platform, with no API key and no model variance.
• A claude -p adapter for subscription-funded runs is filed as bench: claude-cli adapter — run benchmark episodes via claude -p (subscription, no API key) #949.

Grading is deterministic over the FINAL ANSWER line only: numeric answers prefer the first decimal-bearing number (a restated year is a bare integer and is skipped when a decimal candidate exists); entity answers must name a correct alias and none of the task's generated wrong_aliases trap answers.

Lifecycle and CI

make bench-up BENCH_ARM=a0|a2 (compose stack + seeded warehouse + release platform build, with stale-instance and port-collision guards), make bench-smoke, make bench-run, make bench-report, make bench-down, make bench-gen, make bench-test. For a2, make bench-seed-datahub pushes the generated metadata into a DataHub quickstart (same external convention as e2e and load). A bench.yml workflow runs the module checks (build/vet/test/lint) when bench/ changes; the benchmark smoke workflow and published results page are phase 4 (#945).

Verification

• make verify green; root module runtime unchanged (bench is a nested module outside the root ./... gates by design).
• Bench module gates green: build, vet, go test ./... (unit tests plus an integration test that assembles a real streamable-HTTP MCP server and the admin audit surface in-process and drives the full pipeline through genuine protocol wiring, including identity-pool rotation and the pool-refusal path), golangci-lint with the same complexity ceilings as the root.
• Live end-to-end smoke on the a0 arm against the compose stack: all 10 tasks pass with the scripted adapter (sessions minted, handles threaded, audit rows converge exactly, both graders validated against live Trino results). This smoke is what pinned the reference SQL to DOUBLE division: Trino evaluates BIGINT / DECIMAL(4,1) at scale 1, which loses cents.
• Real-model pilot runs (both arms, k=3) require ANTHROPIC_API_KEY and, for a2, a DataHub quickstart; those runs answer bench(phase 1): pilot — A0 vs A2 on S1/S3 with deterministic seed generator and audit-derived metrics #942's exit criterion (a measurable A2-A0 gap, or the explanation for why there is none).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.03%. Comparing base (3b38471) to head (f860a64).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #950      +/-   ##
==========================================
- Coverage   89.04%   89.03%   -0.02%     
==========================================
  Files         424      424              
  Lines       48581    48581              
==========================================
- Hits        43260    43255       -5     
- Misses       3547     3550       +3     
- Partials     1774     1776       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.