|
| 1 | +# Example conformance-harness-recall fixture (Eval 3, retrieval-augmented). |
| 2 | +# |
| 3 | +# Copy this file to a real fixture name (e.g., js-peer-revoke-checkpoint-2026-02-18.yml) |
| 4 | +# and replace placeholder values. The runner ignores files with `_example: true`. |
| 5 | +# |
| 6 | +# See chio-pack/eval/PHASE-0.md "Eval 3" for the field reference and |
| 7 | +# `ops/scripts/harvest-conformance-fixtures.py` for an auto-harvest starting point. |
| 8 | + |
| 9 | +_example: true |
| 10 | + |
| 11 | +# === Identity === |
| 12 | + |
| 13 | +# Stable kebab-case ID. Convention: <peer>-<area>-<short-slug>-<YYYY-MM-DD>. |
| 14 | +id: example-conformance-id-YYYY-MM-DD |
| 15 | + |
| 16 | +# Free-form curator note. The auto-harvester writes this with confidence level; |
| 17 | +# delete it once you've curated the fixture and confirmed it's good ground truth. |
| 18 | +# (`_review:` is also acceptable as a stronger marker — fixtures with `_review:` |
| 19 | +# are flagged in eval-outcomes reports as not-yet-curated.) |
| 20 | +# _review: "confidence=medium; auto-harvested 2026-02-18; manually curated 2026-02-20" |
| 21 | + |
| 22 | +# === The failure === |
| 23 | + |
| 24 | +# Test signature in the language's standard form: |
| 25 | +# Python: path/to/test.py::test_function_name |
| 26 | +# JS/TS: path/to/test.test.ts:test_name_string |
| 27 | +# Rust: crate-name::module::test_function_name |
| 28 | +failing_test: "tests/conformance/peers/js/test_receipts.test.ts:checkpoint_inclusion_proof" |
| 29 | + |
| 30 | +# The actual failure output as it appeared in CI. This is the retrieval query |
| 31 | +# input — the eval issues `kb_search_code(failure_message)` and |
| 32 | +# `kb_search_docs(failure_message)` and computes recall@3 over their union. |
| 33 | +# |
| 34 | +# Keep it realistic. Don't trim stack traces or normalize variable names — |
| 35 | +# real failures aren't normalized, and retrieval quality is judged on real input. |
| 36 | +failure_message: | |
| 37 | + Expected receipt to include checkpoint root, got undefined. |
| 38 | + At RevokeReceipt.serialize() line 47. |
| 39 | +
|
| 40 | +# === The canonical fix === |
| 41 | +# |
| 42 | +# The list of files that authoritative-ly resolve this failure. Recall@3 is |
| 43 | +# computed as |canonical_fix ∩ top_3| / |canonical_fix|. |
| 44 | +# |
| 45 | +# Multi-fix fixtures cap at recall@3 = 3/N — the eval is intentionally biased |
| 46 | +# toward retrieval surfacing the MOST authoritative file first, not all of them. |
| 47 | +# |
| 48 | +# `section` is a free-form anchor that helps a curator verify the right slice |
| 49 | +# of a long file is what the canonical fix touches. For now, the eval doesn't |
| 50 | +# inspect `section`; it's documentation. Phase 2 may bump anchor-level matching. |
| 51 | +canonical_fix: |
| 52 | + - file: "docs/standards/CHIO_RECEIPTS_PROFILE.md" |
| 53 | + section: "Checkpoint inclusion" |
| 54 | + - file: "spec/schemas/chio-wire/v1/receipt/inclusion-proof.schema.json" |
| 55 | + section: "full" |
| 56 | + - file: "crates/chio-receipts/src/checkpoint.rs" |
| 57 | + section: "checkpoint_root" |
| 58 | + |
| 59 | +# === Provenance === |
| 60 | + |
| 61 | +# Where the fix actually landed in arc. Required so a future curator can |
| 62 | +# re-derive `canonical_fix` if the file structure shifts. |
| 63 | +relevant_arc_pr: "arc#NNN" |
| 64 | +relevant_arc_commit: "0000000000000000000000000000000000000000" |
| 65 | + |
| 66 | +# === Optional fields === |
| 67 | + |
| 68 | +# Free-form curator notes. |
| 69 | +notes: | |
| 70 | + Why this is in the fixture set: |
| 71 | + - Spec + JSON schema + Rust impl all touched — a strong test of cross-section |
| 72 | + retrieval (docs-spec-retrieval + code-retrieval categories together). |
| 73 | + - The failure message is generic ("got undefined") — a hard test of whether |
| 74 | + retrieval can disambiguate from the surrounding test name. |
| 75 | + - Fix straddles docs/ and crates/, the kind of split that often confuses |
| 76 | + naive rerankers that prefer either prose or code, not both. |
| 77 | +
|
| 78 | +# Edge-case behaviors per PHASE-0.md "Conformance-harness-recall edge cases": |
| 79 | +# |
| 80 | +# - For **spec-only fixes** (no code change), include the spec/doc files only. |
| 81 | +# Recall scoring still applies; the eval pressures retrieval to surface the |
| 82 | +# canonical doc in top-3. |
| 83 | +# |
| 84 | +# - For **inconclusive PRs** where canonical_fix can't be cleanly identified |
| 85 | +# (sweeping refactor, multiple unrelated bugs in one PR), DROP the fixture |
| 86 | +# rather than guess. PHASE-0.md is explicit on this. |
0 commit comments