week_1-2: Module C — eval harness skeleton + TRACT hub-firewall + scoring

[PRAteek-singHWY]

What this PR is

This is the second of three PRs for Module C, Week 1. PR 1 (week_1-1) landed the RFC-aligned Pydantic contracts and config. This PR lands the regression scaffolding — the eval harness, the TRACT hub-firewall (W1's correctness deliverable), the provisional Q-D scoring rule, the abstract knowledge-source interface with a fixture stub, and a small seed golden dataset so the harness actually runs end-to-end. PR 3 (week_1-3) will replace the seed with the real 277-row dataset derived from the OpenCRE DB.

This is a stacked PR — its base is gsocmodule_C_week_1_1 (not main), so reviewers see only the W1-2 changes. Once week_1-1 merges, GitHub will re-target this to main automatically and the diff stays clean.

Why this PR exists

The W1 contract was contracts + tests before any pipeline code. PR 1 nailed the contracts. This PR nails the ruler — the thing every future C PR (W4 cross-encoder, W6 decision engine, W8 live integration) will be measured against in CI. Without it, accuracy claims can drift silently. The hub-firewall part is the specific deliverable our internal notes call out as W1-critical (see module-c-weekly-deliverables.md, tract-analysis-module-c.md §3.4, master guide §5.7 + §9.3).

What's in this PR, step by step

1. The TRACT hub-firewall (hub_firewall.py + hub_firewall_test.py)

The problem: the OpenCRE graph already links most ASVS/WSTG content to CREs. If we evaluate retrieval on those same standards, the "CRE hub" representation contains the exact text under test → retrieval echoes the text back → reported accuracy is inflated.

The fix: before scoring any golden-set row, strip that row's text from the hub. Whitespace and case insensitive. The CRE vector hub arrives W3 — for now the hub is modeled as a list of (cre_id, text) reps so the methodology and the test exist before any number is claimed.

The harness has the firewall ON by default; --no_hub_firewall is an explicit escape hatch for ablation. The test asserts the row's text is absent from the hub after firewalling — the W1 correctness assertion.

2. The Q-D scoring rule (scoring.py + scoring_test.py)

A chunk can map to several CREs. What counts as "correct" in CI? The provisional rule, isolated in one function and marked # TODO(Q-D):

Jaccard(expected, predicted) ≥ 0.5 AND top-1 in expected.

Strict enough to penalise broad over-linking, lenient enough to not punish near-miss ordering. Tests cover the Jaccard boundary, top-1 outside the expected set, empty predictions, exact match. Confirmation needed at the Friday call before this PR merges — see the "Open question" section.

3. The abstract knowledge source (knowledge_source.py)

Defines the KnowledgeSource interface + a FixtureKnowledgeSource stub that reads JSONL rows in the KnowledgeQueueItem shape (master guide §1.2) — the same shape the real DB-backed source at W8 will yield. Future weeks swap the implementation; the interface stays.

4. The eval harness skeleton (scripts/evaluate_librarian.py)

CLI: --dataset --slice --limit --threshold --top_k_retrieval --top_k_rerank --dry_run --no_hub_firewall. Wires together the dataset loader, the scorer, the firewall, and a predict() stub. Output:

loaded 10 golden rows from .../golden_dataset.json
  ambiguous      2
  explicit       2
  hard_negative  2
  positive       2
  update         2
hub-firewall: ON; stripped 10 leaking hub entries
correct (W1 stub, no predictions): 2/10

The correct 2/10 baseline is right: the 2 review-decision rows expect no CRE ids, so empty predictions match; the 8 linked-decision rows correctly show as not-yet-predicted. Once a real retriever lands, the same harness measures it.

5. Seed golden dataset (fixtures/golden_dataset.json)

10 rows, 2 per slice (explicit, positive, hard_negative, update, ambiguous), schema-valid (round-trips against GoldenDatasetRow from week_1-1). CRE ids are placeholders; provenance.ground_truth_source honestly records that. The full 277-row DB-derived dataset replaces this in week_1-3.

What is deliberately NOT in this PR

• The real 277-row golden_dataset.json — lands in week_1-3.
• The retriever / cross-encoder / SafetyGuard / decision engine — W3 onward.
• CLI wiring into cre.py — W3 per master-guide §8.2.
• DB models or migration — W8.

How to verify locally

python3 -m unittest discover -s application/tests/librarian -p '*_test.py' -t .
# expected: 40 passed (20 from week_1-1, 20 added here)

python3 scripts/evaluate_librarian.py --dataset application/tests/librarian/fixtures/golden_dataset.json
# expected: per-slice counts + "hub-firewall: ON; stripped 10 leaking hub entries" + exit 0

python3 scripts/evaluate_librarian.py --dataset application/tests/librarian/fixtures/golden_dataset.json --no_hub_firewall
# expected: "stripped 0 leaking hub entries"

Black-clean. No existing repo file touched beyond what week_1-1 already pinned (pydantic).

Open question for mentors — blocks this PR's merge

Q-D — multi-link scoring rule. Default proposed: Jaccard ≥ 0.5 of expected CRE set AND top-1 in set. Other reasonable defaults exist (exact-set match — too harsh; top-1 only — ignores multi-link reality). Confirmation appreciated at the Friday call. Until then, the default is in place and TODO-flagged in scoring.py so a rule change touches one function.

---

Stacked on gsocmodule_C_week_1_1. Next: week_1-3 (real 277-row golden dataset derived from standards_cache.sqlite's cre_node_links).