Skip to content

Commit 2b9ad3a

Browse files
committed
chore(governance): expand ROUTE-002/003 holdout for R11 boundaries
Add approved-spec plan-only adversarial trap, dispatch routing probe, and design-comparison holdout phrase. Bump fixture minimums (101/35) with anti-drift tests; no router changes.
1 parent f08250f commit 2b9ad3a

6 files changed

Lines changed: 37 additions & 9 deletions

File tree

reflective-prompt-library/PROJECT_KNOWLEDGE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ deferred promotions are recurrence-gated — see [panel backlog](plans/multi-age
8181
- 2026-06-18 Adopt STORM's source-grounded perspective discovery as an optional `reflective-research` method, not a new skill → [record](plans/storm-perspective-discovery-reflection-2026-06-18.md)
8282
- 2026-06-20 Synthesize external-adoption case studies, then correct the evidence and promotion-gate scope after the Test Plan routing counterexample → [record](plans/external-adoption-case-studies-2026-06-20.md)
8383
- 2026-06-21 Evaluate Hyperplan / multi-agent adversarial planning — no change (runtime = non-goal; methodology overlaps; possible gaps not promoted) → [record](plans/external-adoption-case-studies-2026-06-20.md)
84+
- 2026-06-25 Post-Round 68 holdout expansion — ROUTE-002 design comparison phrase; ROUTE-003 approved-spec plan-only trap + dispatch routing probe → [ROUTING_CONTRACT.md](plans/ROUTING_CONTRACT.md#r11-approved-spec-delivery)
8485
- 2026-06-25 Round 68 panel — full-doc drift sync, README/methodology-map anti-drift, reject holdout expansion at 100% → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
8586
- 2026-06-25 Post-Round 68 maintenance — nine-skill wording parity (METHODOLOGY_MAP, skill-map, QUALITY_GATES), benchmark B024 dispatch (24 tasks, 9/9 workflows) → [QUALITY_GATES_SUMMARY.md](plans/QUALITY_GATES_SUMMARY.md)
8687
- 2026-06-25 Round 67 panel — CONTRIBUTING R8–R12 sync, ROUTING_CONTRACT related artifacts, contract anti-drift tests → [record](plans/multi-agent-panel-consensus-2026-06-25.md)

reflective-prompt-library/plans/QUALITY_GATES_SUMMARY.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ python3 reflective-prompt-library/plans/route_paraphrase_eval.py
111111
- Keeps a lower Phase-1 bar and separate aspirational target to avoid over-claiming
112112

113113
**Results:**
114-
- Tested 36 holdout groups with 100 paraphrases
114+
- Tested 36 holdout groups with 101 paraphrases
115115
- Overall consistency: 100.0% (passes Phase-1 threshold >=80% and aspirational target >=90%)
116116
- Low-confidence route trace coverage: 100.0%
117117
- Review, clarification, workflow-design, workflow-implementation, research, and orchestration-selection phrasing now pass through boundary concepts
@@ -208,7 +208,7 @@ python3 reflective-prompt-library/plans/validate_skill_examples.py
208208
- Post-panel maintenance: ROUTING_CONTRACT **R11** approved-spec delivery (`implement_not_plan_trap`) at 100%
209209

210210
**Results:**
211-
- Tested 13 adversarial groups with 31 paraphrases
211+
- Tested 14 adversarial groups with 35 paraphrases
212212
- Overall consistency: 100.0% (passes Phase-1 threshold >=80% and aspirational target >=90%)
213213
- Low-confidence route trace coverage: 100.0%
214214
- Output: `plans/route-003-results.json`
@@ -224,7 +224,7 @@ python3 reflective-prompt-library/plans/route_paraphrase_eval.py reflective-prom
224224

225225
**What it does:**
226226
- Enforces minimum ROUTE-001/002/003 group and phrase counts before paraphrase eval runs
227-
- Current minimums: ROUTE-001 (12 intent + 4 adversarial groups, 128 phrases); ROUTE-002 (36 holdout groups, 100 phrases); ROUTE-003 (13 adversarial groups, 31 phrases)
227+
- Current minimums: ROUTE-001 (12 intent + 4 adversarial groups, 128 phrases); ROUTE-002 (36 holdout groups, 101 phrases); ROUTE-003 (14 adversarial groups, 35 phrases)
228228
- Round 22 panel compromise: deterministic hygiene without YAML dependency explosion
229229
- Integrated in `make validate` after skill examples gate; mirrored by pytest in `test_validate_route_fixture.py`
230230
- Cheatsheet parity anti-drift: `test_cheatsheet_boundary_quick_cues.py` (R12 quick-cue summary), `test_cheatsheet_*_parity.py` (holdout probe cues in EN/zh-TW cheatsheets)
@@ -279,8 +279,8 @@ The implementation aligns with research findings:
279279
| Skills with governance | 9/9 | ✅ Complete |
280280
| Benchmark tasks | 24 | ✅ Ready |
281281
| Routing consistency | 100.0% | ✅ Passes ROUTE-001 expanded boundary eval |
282-
| Holdout routing consistency | 100.0% | ✅ ROUTE-002 (36 groups, 100 paraphrases) |
283-
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (13 groups, 31 paraphrases) |
282+
| Holdout routing consistency | 100.0% | ✅ ROUTE-002 (36 groups, 101 paraphrases) |
283+
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (14 groups, 35 paraphrases) |
284284
| Skill example coverage | 9/9 | ✅ validate_skill_examples.py |
285285
| Linting errors | 0 | ✅ Clean |
286286

@@ -311,7 +311,7 @@ ROUTE-002 measures unseen phrasing separately from ROUTE-001. Round 7 (2026-06-2
311311
### Done ✅
312312

313313
1. **CI/CD**`.github/workflows/python-tools.yml` runs `make all` on push/PR
314-
2. **ROUTE-001/002/003 in CI** — 128 + 100 + 31 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
314+
2. **ROUTE-001/002/003 in CI** — 128 + 101 + 35 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
315315
3. **Governance validators** — links, lint, governance metadata, PROJECT_KNOWLEDGE, benchmark fixture, skill examples
316316
4. **Harness policy docs** — CONTRIBUTING, AGENTS, SKILL_INSTALLATION, maintenance playbook
317317
5. **Doc anti-drift**`test_routing_contract.py`, cheatsheet parity tests, `test_readme_governance.py` (170+ pytest in CI)

reflective-prompt-library/plans/route-002-holdout-eval.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -362,6 +362,7 @@ holdout_sets:
362362
- compare two API designs on paper only no implementation
363363
- design comparison on paper without repo changes
364364
- 比較兩個 API 設計方案但不要寫 code
365+
- compare API designs on paper without touching the repository
365366

366367
- name: readme_plain_review_holdout
367368
expected_workflow: reflective-review

reflective-prompt-library/plans/route-003-adversarial-eval.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,16 @@ adversarial_sets:
124124

125125

126126

127+
- name: approved_spec_plan_not_implement_trap
128+
expected_workflow: reflective-spec-plan
129+
expected_rigor: medium
130+
expected_enhancements:
131+
- acceptance_criteria
132+
phrases:
133+
- write tickets from the approved spec without implementing
134+
- plan the approved spec without repo changes
135+
- plan 已核准 spec without repo changes
136+
127137
- name: dispatch_meta_skill_trap
128138
expected_workflow: reflective-dispatch
129139
expected_rigor: low
@@ -132,6 +142,7 @@ adversarial_sets:
132142
phrases:
133143
- which skill handles session handoff in this library
134144
- which reflective workflow skill covers handoff retro
145+
- which reflective workflow skill fits a routing-only mixed intent
135146

136147
- name: brief_not_plan_trap
137148
expected_workflow: reflective-brief

reflective-prompt-library/plans/tests/test_validate_route_fixture.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ def test_round_51_boundary_probes():
109109
("lessons learned retro after this sprint", "reflective-handoff-retro"),
110110
("compare official docs for both libraries before deciding", "reflective-research"),
111111
("which reflective workflow skill covers handoff retro", "reflective-dispatch"),
112+
("plan the approved spec without repo changes", "reflective-spec-plan"),
112113
("align stakeholders on goals before writing tickets", "reflective-brief"),
113114
("釐清目標再拆工單", "reflective-brief"),
114115
)
@@ -207,3 +208,17 @@ def test_implement_approved_spec_not_plan_boundary():
207208
workflow, _, _, _ = router.route(text)
208209
assert workflow == expected, f"{text!r} -> {workflow}, want {expected}"
209210

211+
def _approved_spec_plan_not_implement_trap_phrases() -> set[str]:
212+
config = load_route_eval_config(PLANS / "route-003-adversarial-eval.yaml")
213+
for group in config["adversarial_sets"]:
214+
if group["name"] == "approved_spec_plan_not_implement_trap":
215+
return set(group.get("phrases", []))
216+
raise AssertionError("missing ROUTE-003 approved_spec_plan_not_implement_trap group")
217+
218+
219+
def test_approved_spec_plan_not_implement_trap_covers_probes():
220+
"""Anti-drift: ROUTE-003 plan-only approved-spec probes live in dedicated trap group."""
221+
fixture_phrases = _approved_spec_plan_not_implement_trap_phrases()
222+
missing = [p for p in IMPLEMENT_NOT_PLAN_SPEC_PLAN_PROBES if p not in fixture_phrases]
223+
assert not missing, f"approved_spec_plan_not_implement_trap missing probes: {missing}"
224+

reflective-prompt-library/plans/validate_route_fixture.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,10 @@
2929
ROUTE_001_MIN_PHRASES = 128
3030

3131
ROUTE_002_MIN_HOLDOUT_GROUPS = 36
32-
ROUTE_002_MIN_PHRASES = 100
32+
ROUTE_002_MIN_PHRASES = 101
3333

34-
ROUTE_003_MIN_ADVERSARIAL_GROUPS = 13
35-
ROUTE_003_MIN_PHRASES = 31
34+
ROUTE_003_MIN_ADVERSARIAL_GROUPS = 14
35+
ROUTE_003_MIN_PHRASES = 35
3636

3737
REQUIRED_TRACE_FIELDS = {
3838
"canonical_intent",

0 commit comments

Comments
 (0)