Skip to content

Commit 59fea06

Browse files
committed
chore(governance): tighten benchmark fixture gate and decision index
- Raise MIN_TASK_COUNT to 24 and require all nine workflows in validate_benchmark_fixture - Record Round 68 and post-panel maintenance in PROJECT_KNOWLEDGE Decision Index - Document benchmark upkeep in CONTRIBUTING, GLOSSARY playbook, and anti-drift tests
1 parent 6c5d38a commit 59fea06

6 files changed

Lines changed: 25 additions & 1 deletion

File tree

CONTRIBUTING.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ Before tuning `route_paraphrase_eval.py`:
3131
3. When fixing routing boundaries, add holdout phrases **before** editing `route_paraphrase_eval.py` (see `ROUTING_CONTRACT.md` R8–R12).
3232
4. When cheatsheet cues change, update exported probe tuples in `plans/tests/test_validate_route_fixture.py` and run `test_cheatsheet_*_parity.py` plus `test_cheatsheet_boundary_quick_cues.py`.
3333
5. Record governance-surface decisions in `PROJECT_KNOWLEDGE.md` Decision Index when routing contract changes.
34+
6. When adding benchmark golden tasks in `benchmark_tasks.py`, keep all nine workflows covered and update `MIN_TASK_COUNT` in `validate_benchmark_fixture.py`.
3435

3536
See [ROUTING_CONTRACT](reflective-prompt-library/plans/ROUTING_CONTRACT.md) (R8–R12) and the [Governance Maintenance Playbook](reflective-prompt-library/GLOSSARY.md).
3637

reflective-prompt-library/GLOSSARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,4 +348,5 @@ Ongoing upkeep after panel close (Rounds 1–68). Not agent instructions — ope
348348
5. Reject tenth skill / full `SKILL.md` i18n unless promotion gate met (panel A, Q, W, AJ).
349349
6. When cheatsheet boundary quick cues change, update `BOUNDARY_QUICK_CUE_*` markers and probe snippets in `test_validate_route_fixture.py`; run `test_cheatsheet_boundary_quick_cues.py`.
350350
7. Keep `CONTRIBUTING.md` Routing Maintenance aligned with `ROUTING_CONTRACT.md` R8–R12 when boundaries or cheatsheet parity steps change.
351+
8. When adding benchmark golden tasks, keep `test_benchmark_covers_all_nine_workflows` green and bump `MIN_TASK_COUNT` in `validate_benchmark_fixture.py` if the floor rises.
351352

reflective-prompt-library/PROJECT_KNOWLEDGE.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,8 @@ deferred promotions are recurrence-gated — see [panel backlog](plans/multi-age
8181
- 2026-06-18 Adopt STORM's source-grounded perspective discovery as an optional `reflective-research` method, not a new skill → [record](plans/storm-perspective-discovery-reflection-2026-06-18.md)
8282
- 2026-06-20 Synthesize external-adoption case studies, then correct the evidence and promotion-gate scope after the Test Plan routing counterexample → [record](plans/external-adoption-case-studies-2026-06-20.md)
8383
- 2026-06-21 Evaluate Hyperplan / multi-agent adversarial planning — no change (runtime = non-goal; methodology overlaps; possible gaps not promoted) → [record](plans/external-adoption-case-studies-2026-06-20.md)
84+
- 2026-06-25 Round 68 panel — full-doc drift sync, README/methodology-map anti-drift, reject holdout expansion at 100% → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
85+
- 2026-06-25 Post-Round 68 maintenance — nine-skill wording parity (METHODOLOGY_MAP, skill-map, QUALITY_GATES), benchmark B024 dispatch (24 tasks, 9/9 workflows) → [QUALITY_GATES_SUMMARY.md](plans/QUALITY_GATES_SUMMARY.md)
8486
- 2026-06-25 Round 67 panel — CONTRIBUTING R8–R12 sync, ROUTING_CONTRACT related artifacts, contract anti-drift tests → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
8587
- 2026-06-25 Round 66 panel — R12 boundary quick-cue contract, probe-snippet anti-drift, reject quick-cue expansion → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
8688
- 2026-06-25 R11 zh-TW approved-spec delivery holdout — mixed-language ROUTE-003 phrases + router fairness → [ROUTING_CONTRACT.md](plans/ROUTING_CONTRACT.md#r11-approved-spec-delivery)

reflective-prompt-library/plans/tests/test_quality_gates_summary.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
sys.path.insert(0, str(Path(__file__).parent.parent))
1010

11+
from validate_benchmark_fixture import MIN_TASK_COUNT # noqa: E402
1112
from validate_route_fixture import ( # noqa: E402
1213
ROUTE_001_MIN_PHRASES,
1314
ROUTE_002_MIN_HOLDOUT_GROUPS,
@@ -110,3 +111,7 @@ def test_benchmark_section_covers_nine_workflows(summary_text: str):
110111
assert "8 different skills" not in section
111112
assert "24 golden tasks" in section or "24 benchmark tasks" in section
112113

114+
def test_key_metrics_benchmark_task_count_matches_fixture(summary_text: str):
115+
metrics = _section_between(summary_text, "## Key Metrics", "### Routing Consistency Tracking")
116+
assert f"| Benchmark tasks | {MIN_TASK_COUNT} |" in metrics
117+

reflective-prompt-library/plans/tests/test_validate_benchmark_fixture.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,10 @@ def test_benchmark_covers_all_nine_workflows():
3131
f"unexpected: {sorted(workflows - VALID_WORKFLOWS)}"
3232
)
3333

34+
def test_benchmark_minimum_matches_actual_count():
35+
tasks = BenchmarkSet().tasks
36+
assert len(tasks) == MIN_TASK_COUNT, (
37+
f"expected MIN_TASK_COUNT {MIN_TASK_COUNT} to match actual {len(tasks)}; "
38+
"bump MIN_TASK_COUNT when adding golden tasks"
39+
)
40+

reflective-prompt-library/plans/validate_benchmark_fixture.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
}
2424

2525
VALID_DIFFICULTIES = {"easy", "medium", "hard"}
26-
MIN_TASK_COUNT = 20
26+
MIN_TASK_COUNT = 24
2727

2828

2929
def main() -> int:
@@ -44,6 +44,14 @@ def main() -> int:
4444
f"Expected at least {MIN_TASK_COUNT} benchmark tasks; found {len(benchmark.tasks)}"
4545
)
4646

47+
workflows = {task.expected_workflow for task in benchmark.tasks}
48+
missing_workflows = VALID_WORKFLOWS - workflows
49+
if missing_workflows:
50+
errors.append(
51+
"Benchmark tasks missing workflows: "
52+
+ ", ".join(sorted(missing_workflows))
53+
)
54+
4755
seen_ids = set()
4856
for task in benchmark.tasks:
4957
if not task.id:

0 commit comments

Comments
 (0)