Skip to content

Commit 533613d

Browse files
committed
chore(governance): add ROUTE-003 minimality trap and expand adversarial holdout
Close the reflective-minimality gap in ROUTE-003 with a dedicated minimality_not_implement_trap group (8 phrases), thicken thin adversarial groups (+8 phrases), sync fixture minimums to 15 groups / 53 phrases, and add cheatsheet parity + anti-drift probe coverage. Router unchanged.
1 parent e573654 commit 533613d

8 files changed

Lines changed: 53 additions & 7 deletions

reflective-prompt-library/PROJECT_KNOWLEDGE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ deferred promotions are recurrence-gated — see [panel backlog](plans/multi-age
7272

7373
## Decision Index
7474

75+
- 2026-06-25 Post-Round 68 holdout expansion — ROUTE-003 minimality trap + thin adversarial group refresh (15 groups / 53 phrases) → [QUALITY_GATES_SUMMARY.md](plans/QUALITY_GATES_SUMMARY.md)
7576
> Pointers to the causal trail — plans, reflections, tests, commits. Detail is
7677
> not duplicated here; this is a map, not an archive.
7778

reflective-prompt-library/plans/QUALITY_GATES_SUMMARY.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,7 @@ python3 reflective-prompt-library/plans/validate_skill_examples.py
208208
- Post-panel maintenance: ROUTING_CONTRACT **R11** approved-spec delivery (`implement_not_plan_trap`) at 100%
209209

210210
**Results:**
211-
- Tested 14 adversarial groups with 37 paraphrases
211+
- Tested 15 adversarial groups with 53 paraphrases
212212
- Overall consistency: 100.0% (passes Phase-1 threshold >=80% and aspirational target >=90%)
213213
- Low-confidence route trace coverage: 100.0%
214214
- Output: `plans/route-003-results.json`
@@ -224,7 +224,7 @@ python3 reflective-prompt-library/plans/route_paraphrase_eval.py reflective-prom
224224

225225
**What it does:**
226226
- Enforces minimum ROUTE-001/002/003 group and phrase counts before paraphrase eval runs
227-
- Current minimums: ROUTE-001 (12 intent + 4 adversarial groups, 128 phrases); ROUTE-002 (36 holdout groups, 102 phrases); ROUTE-003 (14 adversarial groups, 37 phrases)
227+
- Current minimums: ROUTE-001 (12 intent + 4 adversarial groups, 128 phrases); ROUTE-002 (36 holdout groups, 102 phrases); ROUTE-003 (15 adversarial groups, 53 phrases)
228228
- Round 22 panel compromise: deterministic hygiene without YAML dependency explosion
229229
- Integrated in `make validate` after skill examples gate; mirrored by pytest in `test_validate_route_fixture.py`
230230
- Cheatsheet parity anti-drift: `test_cheatsheet_boundary_quick_cues.py` (R12 quick-cue summary), `test_cheatsheet_*_parity.py` (holdout probe cues in EN/zh-TW cheatsheets)
@@ -280,7 +280,7 @@ The implementation aligns with research findings:
280280
| Benchmark tasks | 24 | ✅ Ready |
281281
| Routing consistency | 100.0% | ✅ Passes ROUTE-001 expanded boundary eval |
282282
| Holdout routing consistency | 100.0% | ✅ ROUTE-002 (36 groups, 102 paraphrases) |
283-
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (14 groups, 37 paraphrases) |
283+
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (15 groups, 53 paraphrases) |
284284
| Skill example coverage | 9/9 | ✅ validate_skill_examples.py |
285285
| Linting errors | 0 | ✅ Clean |
286286

@@ -304,14 +304,14 @@ The latest router improvement uses concept-level boundary rules for these cases
304304

305305
### Holdout Tracking
306306

307-
ROUTE-002 measures unseen phrasing separately from ROUTE-001. Round 7 (2026-06-25) added Traditional Chinese holdout groups with matching router intent keywords — fairness test without full `SKILL.md` translation. Round 65 expanded to 36 holdout groups; post-Round 68 maintenance added ROUTING_CONTRACT **R11** boundaries (102 ROUTE-002 phrases, 14 ROUTE-003 adversarial groups / 37 phrases including `implement_not_plan_trap`, `approved_spec_plan_not_implement_trap`, and `dispatch_meta_skill_trap`) at 100%. Treat this as a seeded holdout, not proof of broad semantic routing; add fresh cases before further router tuning.
307+
ROUTE-002 measures unseen phrasing separately from ROUTE-001. Round 7 (2026-06-25) added Traditional Chinese holdout groups with matching router intent keywords — fairness test without full `SKILL.md` translation. Round 65 expanded to 36 holdout groups; post-Round 68 maintenance added ROUTING_CONTRACT **R11** boundaries (102 ROUTE-002 phrases, 15 ROUTE-003 adversarial groups / 53 phrases including `implement_not_plan_trap`, `approved_spec_plan_not_implement_trap`, `dispatch_meta_skill_trap`, and `minimality_not_implement_trap`) at 100%. Treat this as a seeded holdout, not proof of broad semantic routing; add fresh cases before further router tuning.
308308

309309
## Phase 2 Status (post-Round 68 maintenance)
310310

311311
### Done ✅
312312

313313
1. **CI/CD**`.github/workflows/python-tools.yml` runs `make all` on push/PR
314-
2. **ROUTE-001/002/003 in CI** — 128 + 102 + 37 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
314+
2. **ROUTE-001/002/003 in CI** — 128 + 102 + 53 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
315315
3. **Governance validators** — links, lint, governance metadata, PROJECT_KNOWLEDGE, benchmark fixture, skill examples
316316
4. **Harness policy docs** — CONTRIBUTING, AGENTS, SKILL_INSTALLATION, maintenance playbook
317317
5. **Doc anti-drift**`test_routing_contract.py`, cheatsheet parity tests, `test_readme_governance.py` (190+ pytest anti-drift suite in CI)

reflective-prompt-library/plans/route-003-adversarial-eval.yaml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@ adversarial_sets:
4646
phrases:
4747
- verify production auth change will not expose secrets
4848
- check rollback before touching production billing data
49+
- verify production deploy will not leak secrets
50+
- production auth change rollback plan before merge
4951

5052
- name: plain_review_not_risk
5153
expected_workflow: reflective-review
@@ -55,6 +57,8 @@ adversarial_sets:
5557
phrases:
5658
- inspect this patch for regressions before merge
5759
- review the behavior changes carefully in the diff
60+
- check this patch for readability not security
61+
- inspect patch for regressions not production deploy risk
5862

5963
- name: multi_voice_not_dispatch
6064
expected_workflow: reflective-research
@@ -64,6 +68,8 @@ adversarial_sets:
6468
phrases:
6569
- six-lens debate on whether to merge these skills
6670
- multi-perspective strategic rethink with explicit persona lenses
71+
- six-lens panel debate with explicit persona lenses
72+
- source-backed library comparison before choosing
6773

6874
- name: mixed_language_implement
6975
expected_workflow: reflective-implement
@@ -99,6 +105,8 @@ adversarial_sets:
99105
phrases:
100106
- lessons learned retro after this sprint
101107
- sprint retrospective with lessons learned
108+
- capture lessons learned and promotion candidates from this sprint
109+
- session handoff summary with reusable rules for next agent
102110

103111
- name: implement_not_plan_trap
104112
expected_workflow: reflective-implement
@@ -146,6 +154,21 @@ adversarial_sets:
146154
- which reflective workflow skill fits a routing-only mixed intent
147155
- which reflective workflow skill should handle routing-only mixed intent
148156

157+
- name: minimality_not_implement_trap
158+
expected_workflow: reflective-minimality
159+
expected_rigor: medium
160+
expected_enhancements:
161+
- complexity_cut_list
162+
phrases:
163+
- challenge whether this abstraction layer is needed
164+
- run a minimality gate before adding another helper
165+
- can we delete this wrapper and use stdlib instead
166+
- avoid overengineering this small formatting helper
167+
- which dependencies can we remove without breaking tests
168+
- cut unnecessary abstraction before shipping the feature
169+
- dependency removal review for this module
170+
- 少寫程式 surface area in this module
171+
149172
- name: brief_not_plan_trap
150173
expected_workflow: reflective-brief
151174
expected_rigor: low

reflective-prompt-library/plans/tests/test_quality_gates_summary.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ def test_holdout_tracking_paragraph_matches_fixture_minimums(summary_text: str):
9898
) in section
9999
assert "approved_spec_plan_not_implement_trap" in section
100100
assert "dispatch_meta_skill_trap" in section
101+
assert "minimality_not_implement_trap" in section
101102
assert "100 phrases" not in section
102103

103104

reflective-prompt-library/plans/tests/test_validate_route_fixture.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,12 @@ def test_round_51_boundary_probes():
108108
"which reflective workflow skill should handle routing-only mixed intent",
109109
)
110110

111+
MINIMALITY_NOT_IMPLEMENT_TRAP_PROBES = (
112+
"dependency removal review for this module",
113+
"which dependencies can we remove without breaking tests",
114+
"cut unnecessary abstraction before shipping the feature",
115+
)
116+
111117
ROUTE_003_ADVERSARIAL_BOUNDARY_PROBES = (
112118
("design a handoff workflow specification without runtime code", "reflective-spec-plan"),
113119
("patch the trivial null check in code", "reflective-implement"),
@@ -247,3 +253,16 @@ def test_dispatch_meta_skill_trap_covers_probes():
247253
missing = [p for p in DISPATCH_META_SKILL_TRAP_PROBES if p not in fixture_phrases]
248254
assert not missing, f"dispatch_meta_skill_trap missing probes: {missing}"
249255

256+
def _minimality_not_implement_trap_phrases() -> set[str]:
257+
config = load_route_eval_config(PLANS / "route-003-adversarial-eval.yaml")
258+
for group in config["adversarial_sets"]:
259+
if group["name"] == "minimality_not_implement_trap":
260+
return set(group.get("phrases", []))
261+
raise AssertionError("missing ROUTE-003 minimality_not_implement_trap group")
262+
263+
264+
def test_minimality_not_implement_trap_covers_probes():
265+
"""Anti-drift: ROUTE-003 minimality boundary probes live in dedicated trap group."""
266+
fixture_phrases = _minimality_not_implement_trap_phrases()
267+
missing = [p for p in MINIMALITY_NOT_IMPLEMENT_TRAP_PROBES if p not in fixture_phrases]
268+
assert not missing, f"minimality_not_implement_trap missing probes: {missing}"

reflective-prompt-library/plans/validate_route_fixture.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@
3131
ROUTE_002_MIN_HOLDOUT_GROUPS = 36
3232
ROUTE_002_MIN_PHRASES = 102
3333

34-
ROUTE_003_MIN_ADVERSARIAL_GROUPS = 14
35-
ROUTE_003_MIN_PHRASES = 37
34+
ROUTE_003_MIN_ADVERSARIAL_GROUPS = 15
35+
ROUTE_003_MIN_PHRASES = 53
3636

3737
REQUIRED_TRACE_FIELDS = {
3838
"canonical_intent",

reflective-prompt-library/skills/SKILL_TRIGGER_CHEATSHEET.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,7 @@ Trigger cues:
137137
- "Review this diff only for complexity we can cut."
138138

139139
- "What dependencies can we remove from this module?"
140+
- "Dependency removal review for this module."
140141

141142
Do not use when:
142143

reflective-prompt-library/skills/SKILL_TRIGGER_CHEATSHEET.zh-TW.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,7 @@ Minimality 訊號掃描(skill 內建):
135135
- 只想從 diff 裡找可刪的複雜度
136136

137137
- 「What dependencies can we remove from this module?」
138+
- 「Dependency removal review for this module。」
138139

139140
不要用在:
140141

0 commit comments

Comments
 (0)