feat(80.6-12): bench runner + bundled goldens (D-46) + auto_goldens

Plan 80.6-12 — supamem eval --regress, the SC-9 regression gate.

src/supamem/eval/runner.py (NEW):
- run_bench(*, regress, goldens_path, config) -> int
- _load_goldens loads bundled JSONL via importlib.resources.files
  ('supamem.eval.goldens') / 'phase_80_1_tuned_hybrid.jsonl'.
- Per-query: backend.query(q, k=5) → recall@5 substring match against
  required_substrings (parity with softchat scripts/eval/goldens.py
  recall_at_5_substring).
- Aggregates: mean_recall_at_5, p95_latency_ms, total_tokens.
- BASELINE thresholds locked from Phase 80.1 D-19:
    mean_recall_at_5 >= 0.60, total_tokens <= 4000, p95_latency_ms <= 500
- regress=True compares aggregates to baseline; exit 1 + REGRESSION
  reason printed if any threshold breached.

src/supamem/eval/auto_goldens.py (NEW):
- D-07 invariant enforcement: no SaaS LLM calls.
- assert_no_saas_llm_env() raises if OPENAI_API_KEY / ANTHROPIC_API_KEY /
  AZURE_OPENAI_API_KEY / TOGETHER_API_KEY / OPENROUTER_API_KEY is set.
- derive_required_substrings(text) — deterministic local extractor for
  identifiers / dotted names; pure function, no I/O.

src/supamem/eval/goldens/phase_80_1_tuned_hybrid.jsonl (NEW):
- 33 representative records (id + query + required_substrings).
- v0.1 deviation: real Phase 80.1 goldens were per-session sidecars, not a
  single 33-query JSONL. Plan 80.6-14 will migrate the live SoftChat
  goldens into this format. Current bundled set exercises the runner
  contract end-to-end.

src/supamem/eval/__init__.py (NEW): public API surface.

src/supamem/cli.py: cmd_evalbench wired to run_bench with --regress /
--goldens flags.

pyproject.toml: force-include the goldens dir into the wheel so
importlib.resources finds it after pip install.

Tests (9 added, 148/148 total pass):
- bundled goldens load (33 records, schema check)
- external --goldens path overrides bundled
- regress passes when recall >= baseline (perfect-recall mock)
- regress fails when recall < baseline (empty-chunks mock) → exit 1
- report emits 'mean recall@5' + 'total tokens' lines
- auto_goldens raises if SaaS LLM env var is set (D-07)
- auto_goldens passes when no SaaS env vars present
- derive_required_substrings is deterministic + finds identifiers
- BUNDLED_GOLDENS constant points at .jsonl

ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: dzmitrys-dev

pyproject.toml

@@ -50,6 +50,7 @@ packages = ["src/supamem"]
 
 [tool.hatch.build.targets.wheel.force-include]
 "src/supamem/share" = "supamem/share"
+"src/supamem/eval/goldens" = "supamem/eval/goldens"
 
 [tool.ruff]
 line-length = 100

src/supamem/cli.py

@@ -153,7 +153,11 @@ def cmd_evalbench(
     goldens: Optional[str] = typer.Option(None, "--goldens", help="Custom goldens JSONL path."),
 ) -> None:
     """Run the regression harness against the Phase 80.1 golden corpus."""
-    _stub("eval")
+    from supamem.config import load_config
+    from supamem.eval.runner import run_bench
+
+    cfg, _chain = load_config()
+    raise typer.Exit(run_bench(regress=regress, goldens_path=goldens, config=cfg))
 
 
 @app.command("install")

src/supamem/eval/__init__.py

@@ -0,0 +1,22 @@
+"""Bench harness for supamem retrieval — recall@5 + p95 latency + token totals.
+
+Public API:
+
+- :func:`run_bench` — executes the bench against bundled or external goldens
+- :func:`derive_required_substrings` — D-07-safe local extractor
+- :func:`assert_no_saas_llm_env` — D-07 invariant guard
+"""
+from __future__ import annotations
+
+from supamem.eval.auto_goldens import (
+    assert_no_saas_llm_env,
+    derive_required_substrings,
+)
+from supamem.eval.runner import BASELINE, run_bench
+
+__all__ = [
+    "BASELINE",
+    "assert_no_saas_llm_env",
+    "derive_required_substrings",
+    "run_bench",
+]

src/supamem/eval/auto_goldens.py

@@ -0,0 +1,70 @@
+"""Auto-goldens generator — D-07 invariant: no SaaS LLM calls.
+
+Auto-generates ``required_substrings`` for golden records by extracting
+identifiers / tokens from the answer text using a deterministic local
+algorithm. NEVER calls OpenAI / Anthropic / any external API — D-07 is the
+load-bearing invariant: goldens must be reproducible offline.
+"""
+from __future__ import annotations
+
+import os
+import re
+
+# Env vars whose presence indicates a SaaS LLM SDK is configured. If any is
+# set, the user is in a "could call cloud" state — we refuse to run auto-
+# goldens to make the D-07 invariant obvious at the boundary.
+_SAAS_ENV_VARS: tuple[str, ...] = (
+    "OPENAI_API_KEY",
+    "ANTHROPIC_API_KEY",
+    "AZURE_OPENAI_API_KEY",
+    "TOGETHER_API_KEY",
+    "OPENROUTER_API_KEY",
+)
+
+
+def _identifier_tokens(text: str, *, min_len: int = 4) -> list[str]:
+    """Extract camelCase / snake_case / dotted-name tokens that look code-shaped.
+
+    Heuristic: any run of [A-Za-z_][A-Za-z0-9_]+ at least ``min_len`` chars,
+    plus dotted names like ``module.func``. Returns deduped list, order
+    preserved.
+    """
+    pat = re.compile(r"[A-Za-z_][A-Za-z0-9_\.]+[A-Za-z0-9_]")
+    seen: set[str] = set()
+    out: list[str] = []
+    for tok in pat.findall(text):
+        if len(tok) < min_len:
+            continue
+        if tok in seen:
+            continue
+        seen.add(tok)
+        out.append(tok)
+    return out
+
+
+def derive_required_substrings(
+    answer_text: str,
+    *,
+    max_subs: int = 5,
+    min_len: int = 4,
+) -> list[str]:
+    """Deterministic substring extraction. Pure function, no I/O."""
+    tokens = _identifier_tokens(answer_text, min_len=min_len)
+    return tokens[:max_subs]
+
+
+def assert_no_saas_llm_env() -> None:
+    """Raise RuntimeError if any SaaS LLM env var is set (D-07 enforcement)."""
+    found = [name for name in _SAAS_ENV_VARS if os.environ.get(name, "").strip()]
+    if found:
+        raise RuntimeError(
+            "supamem auto_goldens: D-07 invariant breach — refused to run "
+            "with SaaS LLM env vars set ({}). Auto-goldens MUST stay offline."
+            .format(", ".join(found))
+        )
+
+
+__all__ = [
+    "assert_no_saas_llm_env",
+    "derive_required_substrings",
+]

src/supamem/eval/goldens/__init__.py

@@ -0,0 +1,33 @@
+{"id": "g01", "query": "how does the indexer chunk markdown documents", "required_substrings": ["chunk", "markdown"]}
+{"id": "g02", "query": "what is the locked schema for tuned_hybrid retrieval", "required_substrings": ["dense", "sparse"]}
+{"id": "g03", "query": "fail-soft contract for the MCP server tool", "required_substrings": ["fail-soft", "exit"]}
+{"id": "g04", "query": "where does supamem store its config file", "required_substrings": [".supamem/config.toml"]}
+{"id": "g05", "query": "which embedder powers the dense vector arm", "required_substrings": ["MiniLM"]}
+{"id": "g06", "query": "what BM25 model does sparse retrieval use", "required_substrings": ["Qdrant/bm25"]}
+{"id": "g07", "query": "how is the Welford counter schema versioned", "required_substrings": ["sum", "sumsq", "count"]}
+{"id": "g08", "query": "atomic write strategy for client config patches", "required_substrings": ["atomic", "replace"]}
+{"id": "g09", "query": "managed-block fence markers in CLAUDE.md", "required_substrings": ["BEGIN SUPAMEM", "END SUPAMEM"]}
+{"id": "g10", "query": "what is the RRF fusion algorithm parameter", "required_substrings": ["RRF", "fusion"]}
+{"id": "g11", "query": "default chunk soft max token cap", "required_substrings": ["250"]}
+{"id": "g12", "query": "supamem doctor exit codes", "required_substrings": ["exit"]}
+{"id": "g13", "query": "how does init detect Qdrant reachability", "required_substrings": ["healthz", "probe"]}
+{"id": "g14", "query": "brownfield migration paths supported", "required_substrings": ["coexist", "migrate", "adopt-as-is"]}
+{"id": "g15", "query": "stdio transport stdout discipline rule", "required_substrings": ["stdout", "stderr"]}
+{"id": "g16", "query": "Streamable HTTP transport kwarg", "required_substrings": ["streamable-http"]}
+{"id": "g17", "query": "snapshot before destructive migrate path", "required_substrings": ["snapshot"]}
+{"id": "g18", "query": "PreToolUse hook payload shape", "required_substrings": ["hookSpecificOutput", "additionalContext"]}
+{"id": "g19", "query": "cursor session-start snapshot regen target", "required_substrings": ["dual-memory-snapshot.mdc"]}
+{"id": "g20", "query": "share dir reference-not-copy contract", "required_substrings": ["share"]}
+{"id": "g21", "query": "qdrant api key redaction in doctor output", "required_substrings": ["redact"]}
+{"id": "g22", "query": "is_code_target gate file suffix rejection list", "required_substrings": [".md", ".lock"]}
+{"id": "g23", "query": "marker file for daily-rolled hook coordination", "required_substrings": ["queried"]}
+{"id": "g24", "query": "where do auto-goldens prevent SaaS LLM calls", "required_substrings": ["RuntimeError"]}
+{"id": "g25", "query": "what does derive_query strip drop_tokens for", "required_substrings": ["drop_tokens"]}
+{"id": "g26", "query": "search tool max query length DoS guard", "required_substrings": ["MAX_QUERY_LEN"]}
+{"id": "g27", "query": "summary_md field rendered in tool card", "required_substrings": ["summary_md"]}
+{"id": "g28", "query": "config precedence ladder rungs", "required_substrings": ["env", "supamem_toml"]}
+{"id": "g29", "query": "T-1 markdown header chunker headers list", "required_substrings": ["h1", "h2", "h3"]}
+{"id": "g30", "query": "T-5 cosine dedup threshold value", "required_substrings": ["0.97"]}
+{"id": "g31", "query": "T-8 token budget truncation cap", "required_substrings": ["1500"]}
+{"id": "g32", "query": "forbidden collection guard names list", "required_substrings": ["dev_memory"]}
+{"id": "g33", "query": "manifest legacy v1 to v2 schema upgrade", "required_substrings": ["legacy", "v2"]}

src/supamem/eval/goldens/phase_80_1_tuned_hybrid.jsonl

@@ -0,0 +1,159 @@
+"""Bench runner for ``supamem eval --regress``.
+
+Loads a JSONL golden set (bundled or external), runs each query against
+:class:`supamem.retrieval.tuned_hybrid.TunedHybridBackend`, computes
+recall@5 via substring matching against each record's
+``required_substrings`` list, and aggregates to mean recall + p95 latency
++ total tokens. ``--regress`` mode compares the aggregate to Phase 80.1
+locked thresholds and exits non-zero on any breach (SC-9 regression gate).
+"""
+from __future__ import annotations
+
+import json
+import logging
+import time
+from importlib import resources
+from pathlib import Path
+from typing import Any
+
+from supamem.config import ResolvedConfig
+from supamem.retrieval.tuned_hybrid import TunedHybridBackend
+from supamem.retrieval.types import RetrievedChunk
+
+log = logging.getLogger("supamem.eval.runner")
+
+# Phase 80.1 locked thresholds (D-19).
+BASELINE = {
+    "mean_recall_at_5": 0.60,
+    "total_tokens": 4000,
+    "p95_latency_ms": 500,
+}
+
+BUNDLED_GOLDENS = "phase_80_1_tuned_hybrid.jsonl"
+
+
+def _load_goldens(path: str | None) -> list[dict[str, Any]]:
+    """Load JSONL records from ``path`` or the bundled corpus."""
+    if path:
+        body = Path(path).read_text(encoding="utf-8")
+    else:
+        # The goldens dir is a sub-package; resources.files works because
+        # ``supamem.eval.goldens`` has its own __init__.py.
+        files = resources.files("supamem.eval.goldens")
+        target = files / BUNDLED_GOLDENS
+        body = target.read_text(encoding="utf-8")
+    out: list[dict[str, Any]] = []
+    for line in body.splitlines():
+        if not line.strip():
+            continue
+        out.append(json.loads(line))
+    return out
+
+
+def _recall_at_5(retrieved: list[RetrievedChunk], required: list[str]) -> float:
+    """Substring match: fraction of required substrings present in top-5 blob."""
+    if not required:
+        return 0.0
+    blob = " ".join(c.text or "" for c in retrieved[:5])
+    hits = sum(1 for s in required if s in blob)
+    return hits / len(required)
+
+
+def _percentile(values: list[float], pct: float) -> float:
+    if not values:
+        return 0.0
+    s = sorted(values)
+    k = max(0, min(len(s) - 1, int(round(pct / 100.0 * (len(s) - 1)))))
+    return float(s[k])
+
+
+def _build_backend(config: ResolvedConfig) -> TunedHybridBackend:
+    return TunedHybridBackend(config=config)
+
+
+def run_bench(
+    *,
+    regress: bool = False,
+    goldens_path: str | None = None,
+    config: ResolvedConfig | None = None,
+) -> int:
+    """Run the bench. Returns 0 on pass, 1 on regression / fatal."""
+    cfg = config or ResolvedConfig()
+    try:
+        records = _load_goldens(goldens_path)
+    except (FileNotFoundError, OSError) as exc:
+        log.error("supamem eval: failed to load goldens: %s", exc)
+        return 1
+    if not records:
+        log.warning("supamem eval: no golden records loaded")
+        return 1
+
+    backend = _build_backend(cfg)
+    recalls: list[float] = []
+    latencies: list[float] = []
+    total_tokens = 0
+    rows: list[dict[str, Any]] = []
+
+    for rec in records:
+        query = str(rec.get("query") or "").strip()
+        required = list(rec.get("required_substrings") or [])
+        if not query:
+            continue
+        t0 = time.perf_counter()
+        try:
+            chunks = backend.query(query, k=5)
+        except Exception as exc:  # noqa: BLE001
+            log.warning("supamem eval: query %r failed: %s", query, type(exc).__name__)
+            chunks = []
+        elapsed = (time.perf_counter() - t0) * 1000.0
+        latencies.append(elapsed)
+        recall = _recall_at_5(chunks, required)
+        recalls.append(recall)
+        total_tokens += sum(max(1, len(c.text or "") // 4) for c in chunks)
+        rows.append({"id": rec.get("id"), "recall": recall, "latency_ms": elapsed})
+
+    mean_recall = sum(recalls) / len(recalls) if recalls else 0.0
+    p95 = _percentile(latencies, 95.0)
+    summary = {
+        "queries": len(records),
+        "mean_recall_at_5": round(mean_recall, 4),
+        "p95_latency_ms": round(p95, 2),
+        "total_tokens": total_tokens,
+    }
+
+    print("supamem eval — bench summary")
+    print(f"  queries          : {summary['queries']}")
+    print(f"  mean recall@5    : {summary['mean_recall_at_5']}")
+    print(f"  p95 latency (ms) : {summary['p95_latency_ms']}")
+    print(f"  total tokens     : {summary['total_tokens']}")
+
+    if not regress:
+        return 0
+
+    breaches: list[str] = []
+    if mean_recall < BASELINE["mean_recall_at_5"]:
+        breaches.append(
+            f"mean_recall_at_5={mean_recall:.4f} < baseline {BASELINE['mean_recall_at_5']}"
+        )
+    if total_tokens > BASELINE["total_tokens"]:
+        breaches.append(
+            f"total_tokens={total_tokens} > baseline {BASELINE['total_tokens']}"
+        )
+    if p95 > BASELINE["p95_latency_ms"]:
+        breaches.append(
+            f"p95_latency_ms={p95:.2f} > baseline {BASELINE['p95_latency_ms']}"
+        )
+
+    if breaches:
+        print()
+        print("supamem eval — REGRESSION:")
+        for line in breaches:
+            print(f"  - {line}")
+        return 1
+
+    print()
+    print("supamem eval — regress: PASS")
+    return 0
+
+
+__all__ = ["BASELINE", "run_bench"]