0.3 — Harvest 20 conformance-harness-recall fixtures (Eval 3)

[bb-connor]

Phase 0 task 0.3 (PLAN.md, Phase 0 task 0.3).

Harvest and curate 20 conformance-harness-recall fixtures (Eval 3).

Deliverables

• Run make kb-harvest-fixtures to auto-harvest candidates from arc git history.
• Curate per PHASE-0.md "Manual curation REQUIRED":
  • Drop fixtures with confidence=low.
  • Replace each canonical_fix[].section TODO: human-curated with a real anchor.
  • Verify each failure_message reads like a real test failure (not a commit subject).
  • Delete the _review field once curated.
• Final fixture set: ≥ 20 curated fixtures committed to chio-pack/eval/fixtures/conformance-recall/.
• Implement chio-pack/chio_pack/eval/runners/conformance_recall.py (recall@3 over union of kb_search_code + kb_search_docs top-10).
• Run baseline; commit number to ADR-0002.

Done when

• make kb-eval-outcomes reports conformance-harness-recall as ok.
• ADR-0002's row has a baseline number, sample size, and date.

Reference

• Fixture template
• Harvest script
• PHASE-0.md Eval 3

[bb-connor]

Progress: 6 / 20 fixtures committed

Six hand-curated fixtures landed in 5c735e3 (and predecessor b949309). Each maps to one of arc's six native conformance scenarios on origin/codex/chio-kb-a-grade-dogfood:

Scenario	Fixture
capability-validation	capability-validation-2026-05-07.yml
delegation-attenuation	delegation-attenuation-2026-05-07.yml
receipt-integrity	receipt-integrity-2026-05-07.yml
revocation-propagation	revocation-propagation-2026-05-07.yml
dpop-verification	dpop-verification-2026-05-07.yml
governed-transaction-enforcement	governed-transaction-enforcement-2026-05-07.yml

Each fixture has a real failing_test (scenario path + id), a plausible failure_message modeled on Rust-driver assertion output, and ranked canonical_fix files with real anchors (no TODO: human-curated leftovers).

Eval-outcomes report now shows:

| `conformance-harness-recall` | 6 | BLOCKED — fixtures | have 6, need ≥ 20 (PHASE-0.md) |

What's left for ADR-0002 sign-off

14 more fixtures. Candidate sources:

• arc's tests/conformance/fixtures/guard/*.yaml (e.g., enriched-fields.yaml, tool-gate.yaml) — guard-specific scenarios.
• merged arc PRs tagged feat(conformance): or test(chio-conformance): — e.g., #548 "tighten threat-coverage pin assertions", #347 "verdict matrix rust driver", #334 "scaffold verdict matrix".
• Edge cases per PHASE-0.md: spec-only fixes (no code change), multi-fix fixtures (capped at recall@3).

Harvester gap (follow-up)

The harvester (ops/scripts/harvest-conformance-fixtures.py) was meaningfully improved in b949309: broader CONFORMANCE_PATHS, Rust + JSON signature extraction, --branch flag. Running with --branch origin/codex/chio-kb-a-grade-dogfood --search-limit 800 emitted 9 medium-confidence candidates — but 6 of 9 were SDK-test noise (api dashboard token tests, sweeping refactor commits, SDK serialization unit tests).

The remaining gap: is_test_file() matches *.test.ts filenames anywhere in a commit's diff, not just inside conformance directories. A commit that touches crates/chio-conformance/ AND a sdks/typescript/.../middleware.test.ts ends up with the SDK test file as the "failing_test."

Tightening fix (deferred to a sub-issue): require the test file's own path to contain conformance/ or be a known-conformance file (e.g., the verdict-matrix peer suite). Hand-curation was faster than another harvester iteration for v0; the broader fix should land before ADR-0002 sign-off.

Status

Issue stays open. 6/20 fixtures committed, 14 more needed. Will pick up alongside the other Phase 0 issues (#1, #2, #4, #5).

[bb-connor]

11 / 20 fixtures committed (was 6/20)

Five more fixtures landed in 0f275da:

• verdict-matrix-wasm-synthesis-2026-04-30 — curated from harvester output (arc commit 1f8935589a "fix(conformance): stop synthesizing ts wasm verdicts").
• enriched-fields-block-etc-write-2026-05-07 — hand-crafted from arc's tests/conformance/fixtures/guard/enriched-fields.yaml.
• tool-gate-deny-rm-rf-2026-05-07 — hand-crafted from tests/conformance/fixtures/guard/tool-gate.yaml.
• tool-gate-cross-language-divergence-2026-05-07 — targets the verdict-matrix cross-language consistency test.
• threat-coverage-pin-2026-05-07 — hand-crafted from arc PR #548 ("tighten threat-coverage pin assertions").

Harvester improvements (also 0f275da)

is_test_file() now requires the test file's own path to contain a conformance directory marker. Without this gate, the regex matched any *.test.ts in a commit's diff — including SDK unit tests in commits that happen to also touch conformance paths. Re-running with the tightened filter on a 2000-commit walk produced 5 medium-confidence candidates (down from 9 with the previous filter); 1 was a real focused fix, 4 were sweeping refactors per PHASE-0.md drop policy.

Status

| conformance-harness-recall | 11 | BLOCKED — fixtures | have 11, need ≥ 20 (PHASE-0.md) |

9 more needed for ADR-0002 sign-off. Likely sources for the rest:

• Additional arc fix(conformance): PRs not yet examined.
• More guard fixtures from tests/conformance/fixtures/.
• Conformance-recall-specific edge cases per PHASE-0.md "Conformance-harness-recall edge cases" (spec-only fixes, multi-fix fixtures).

Issue stays open. Progress on track.