Skip to content

Commit 5c735e3

Browse files
bb-connorclaude
andcommitted
Restore conformance-recall/example.yml (accidentally deleted in b949309)
The wildcard `rm *.yml` in the b949309 prep step deleted the template along with the harvester output. example.yml is the schema reference contributors copy when authoring new fixtures — restoring from b949309~1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent b949309 commit 5c735e3

1 file changed

Lines changed: 86 additions & 0 deletions

File tree

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Example conformance-harness-recall fixture (Eval 3, retrieval-augmented).
2+
#
3+
# Copy this file to a real fixture name (e.g., js-peer-revoke-checkpoint-2026-02-18.yml)
4+
# and replace placeholder values. The runner ignores files with `_example: true`.
5+
#
6+
# See chio-pack/eval/PHASE-0.md "Eval 3" for the field reference and
7+
# `ops/scripts/harvest-conformance-fixtures.py` for an auto-harvest starting point.
8+
9+
_example: true
10+
11+
# === Identity ===
12+
13+
# Stable kebab-case ID. Convention: <peer>-<area>-<short-slug>-<YYYY-MM-DD>.
14+
id: example-conformance-id-YYYY-MM-DD
15+
16+
# Free-form curator note. The auto-harvester writes this with confidence level;
17+
# delete it once you've curated the fixture and confirmed it's good ground truth.
18+
# (`_review:` is also acceptable as a stronger marker — fixtures with `_review:`
19+
# are flagged in eval-outcomes reports as not-yet-curated.)
20+
# _review: "confidence=medium; auto-harvested 2026-02-18; manually curated 2026-02-20"
21+
22+
# === The failure ===
23+
24+
# Test signature in the language's standard form:
25+
# Python: path/to/test.py::test_function_name
26+
# JS/TS: path/to/test.test.ts:test_name_string
27+
# Rust: crate-name::module::test_function_name
28+
failing_test: "tests/conformance/peers/js/test_receipts.test.ts:checkpoint_inclusion_proof"
29+
30+
# The actual failure output as it appeared in CI. This is the retrieval query
31+
# input — the eval issues `kb_search_code(failure_message)` and
32+
# `kb_search_docs(failure_message)` and computes recall@3 over their union.
33+
#
34+
# Keep it realistic. Don't trim stack traces or normalize variable names —
35+
# real failures aren't normalized, and retrieval quality is judged on real input.
36+
failure_message: |
37+
Expected receipt to include checkpoint root, got undefined.
38+
At RevokeReceipt.serialize() line 47.
39+
40+
# === The canonical fix ===
41+
#
42+
# The list of files that authoritative-ly resolve this failure. Recall@3 is
43+
# computed as |canonical_fix ∩ top_3| / |canonical_fix|.
44+
#
45+
# Multi-fix fixtures cap at recall@3 = 3/N — the eval is intentionally biased
46+
# toward retrieval surfacing the MOST authoritative file first, not all of them.
47+
#
48+
# `section` is a free-form anchor that helps a curator verify the right slice
49+
# of a long file is what the canonical fix touches. For now, the eval doesn't
50+
# inspect `section`; it's documentation. Phase 2 may bump anchor-level matching.
51+
canonical_fix:
52+
- file: "docs/standards/CHIO_RECEIPTS_PROFILE.md"
53+
section: "Checkpoint inclusion"
54+
- file: "spec/schemas/chio-wire/v1/receipt/inclusion-proof.schema.json"
55+
section: "full"
56+
- file: "crates/chio-receipts/src/checkpoint.rs"
57+
section: "checkpoint_root"
58+
59+
# === Provenance ===
60+
61+
# Where the fix actually landed in arc. Required so a future curator can
62+
# re-derive `canonical_fix` if the file structure shifts.
63+
relevant_arc_pr: "arc#NNN"
64+
relevant_arc_commit: "0000000000000000000000000000000000000000"
65+
66+
# === Optional fields ===
67+
68+
# Free-form curator notes.
69+
notes: |
70+
Why this is in the fixture set:
71+
- Spec + JSON schema + Rust impl all touched — a strong test of cross-section
72+
retrieval (docs-spec-retrieval + code-retrieval categories together).
73+
- The failure message is generic ("got undefined") — a hard test of whether
74+
retrieval can disambiguate from the surrounding test name.
75+
- Fix straddles docs/ and crates/, the kind of split that often confuses
76+
naive rerankers that prefer either prose or code, not both.
77+
78+
# Edge-case behaviors per PHASE-0.md "Conformance-harness-recall edge cases":
79+
#
80+
# - For **spec-only fixes** (no code change), include the spec/doc files only.
81+
# Recall scoring still applies; the eval pressures retrieval to surface the
82+
# canonical doc in top-3.
83+
#
84+
# - For **inconclusive PRs** where canonical_fix can't be cleanly identified
85+
# (sweeping refactor, multiple unrelated bugs in one PR), DROP the fixture
86+
# rather than guess. PHASE-0.md is explicit on this.

0 commit comments

Comments
 (0)