Skip to content

Commit 728b4c4

Browse files
johnteeeclaude
andcommitted
Add no-code Test Plan and Workflow Design modes to reflective-spec-plan
Implement the bounded, in-place repair licensed by the corrected evidence/promotion gate (cbb0b6c): operationalize test design and workflow design as modes *inside* reflective-spec-plan, not as a new skill and not as Prompt Sources pointers (install copies only SKILL.md, so a pointer would not reach installed users). - reflective-spec-plan: Test Plan Mode (requirement-to-test matrix, negative/edge/regression/adversarial, false-positive guards) and Workflow Design Mode (formalization ladder, control model, state/ artifacts, transition ownership, recovery, verification) + an Implementation Decision Gate (Claim/Evidence/Unknowns/Counterargument/ Decision/Falsifier). Description and body now match. - reflective-dispatch: route table + explicit test-design and workflow-design boundaries (executable work routes to reflective-implement; comparison to research; review to review). - README, skill-map, trigger cheatsheets: reflect the two new modes. - benchmark + route eval (route-001/002) updated for the new routing. - Regenerated stale index.json. Guardrails preserve the runtime non-goal: a workflow spec is not proof of persistence, replay, recovery, cancellation, or idempotency. Verified: project-knowledge, governance, links, lint, 52 tests, and the routing paraphrase eval (100% consistency) all pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent cbb0b6c commit 728b4c4

13 files changed

Lines changed: 239 additions & 45 deletions

reflective-prompt-library/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ The prompt files are the source material. The `skills/` directory is the operati
5151
| --- | --- |
5252
| Route a task to the smallest useful reflective workflow | `skills/reflective-dispatch/SKILL.md` |
5353
| Clarify goal, assumptions, scope, acceptance, falsifiability | `skills/reflective-brief/SKILL.md` |
54-
| Write spec, usage-first design, task slices, or a no-code Test Plan | `skills/reflective-spec-plan/SKILL.md` |
54+
| Write spec, usage-first design, task slices, a no-code Test Plan, or a no-code agent workflow design | `skills/reflective-spec-plan/SKILL.md` |
5555
| Implement code with verification and traceability | `skills/reflective-implement/SKILL.md` |
5656
| Challenge unnecessary code, dependencies, abstractions, or scope before implementation | `skills/reflective-minimality/SKILL.md` |
5757
| Review code, plans, specs, and AI outputs | `skills/reflective-review/SKILL.md` |

reflective-prompt-library/README.zh-TW.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ Doing the right thing > doing things right.
3434
| --- | --- |
3535
| 分流任務到最小適用 workflow | `skills/reflective-dispatch/SKILL.md` |
3636
| 釐清目標、假設、範圍、驗收、可證偽 | `skills/reflective-brief/SKILL.md` |
37-
| 寫 spec、usage-first 設計、task slices,或不寫程式碼的 Test Plan | `skills/reflective-spec-plan/SKILL.md` |
37+
| 寫 spec、usage-first 設計、task slices、不寫程式碼的 Test Plan,或不寫 runtime code 的 agent workflow design | `skills/reflective-spec-plan/SKILL.md` |
3838
| 實作程式碼,附驗證與可追蹤性 | `skills/reflective-implement/SKILL.md` |
3939
| 實作前挑戰不必要 code、dependency、抽象層或 scope | `skills/reflective-minimality/SKILL.md` |
4040
| 審查 code / plan / spec / AI output | `skills/reflective-review/SKILL.md` |

reflective-prompt-library/index.json

Lines changed: 45 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"version": "1.0",
3-
"generated_at": "2026-06-20T01:31:58.655885",
3+
"generated_at": "2026-06-20T02:33:36.899079",
44
"repo_root": ".",
55
"categories": {
66
"prompt-library": {
@@ -1421,8 +1421,8 @@
14211421
"text": "Conclusion"
14221422
}
14231423
],
1424-
"line_count": 300,
1425-
"char_count": 11962
1424+
"line_count": 303,
1425+
"char_count": 12363
14261426
},
14271427
{
14281428
"path": "reflective-prompt-library/plans/external-adoption-case-studies-2026-06-20.md",
@@ -1442,7 +1442,6 @@
14421442
"agentic-sop-workflow-reflection-2026-06-13.md",
14431443
"../04-agent/sop-compiler.md",
14441444
"../skills/reflective-review/SKILL.md",
1445-
"../PROJECT_KNOWLEDGE.md",
14461445
"../skills/reflective-research/SKILL.md",
14471446
"../skills/reflective-minimality/SKILL.md",
14481447
"../skills/reflective-dispatch/SKILL.md"
@@ -1473,6 +1472,14 @@
14731472
"level": 2,
14741473
"text": "Signal Accounting (do not miscount)"
14751474
},
1475+
{
1476+
"level": 2,
1477+
"text": "Decision-Rule Correction"
1478+
},
1479+
{
1480+
"level": 3,
1481+
"text": "Applied Counterexample: Test Plan Routing"
1482+
},
14761483
{
14771484
"level": 2,
14781485
"text": "Methodology Layer vs Operationalization Layer"
@@ -1494,8 +1501,8 @@
14941501
"text": "State Ledger"
14951502
}
14961503
],
1497-
"line_count": 116,
1498-
"char_count": 7057
1504+
"line_count": 161,
1505+
"char_count": 9690
14991506
},
15001507
{
15011508
"path": "reflective-prompt-library/plans/ROUTING_CONTRACT.md",
@@ -2848,7 +2855,7 @@
28482855
}
28492856
],
28502857
"line_count": 78,
2851-
"char_count": 3928
2858+
"char_count": 4051
28522859
},
28532860
{
28542861
"path": "reflective-prompt-library/skills/SKILL_TRIGGER_CHEATSHEET.md",
@@ -2911,8 +2918,8 @@
29112918
"text": "Fast Routing Rule"
29122919
}
29132920
],
2914-
"line_count": 154,
2915-
"char_count": 5032
2921+
"line_count": 158,
2922+
"char_count": 5367
29162923
},
29172924
{
29182925
"path": "reflective-prompt-library/skills/SKILL_TRIGGER_CHEATSHEET.zh-TW.md",
@@ -2977,8 +2984,8 @@
29772984
"text": "快速順序"
29782985
}
29792986
],
2980-
"line_count": 138,
2981-
"char_count": 2296
2987+
"line_count": 142,
2988+
"char_count": 2495
29822989
},
29832990
{
29842991
"path": "reflective-prompt-library/05-domain/creative-template.md",
@@ -3426,8 +3433,8 @@
34263433
"text": "Guardrails"
34273434
}
34283435
],
3429-
"line_count": 111,
3430-
"char_count": 4685
3436+
"line_count": 115,
3437+
"char_count": 5356
34313438
},
34323439
{
34333440
"path": "reflective-prompt-library/skills/reflective-minimality/SKILL.md",
@@ -3975,8 +3982,8 @@
39753982
"text": "Prompt Sources"
39763983
}
39773984
],
3978-
"line_count": 148,
3979-
"char_count": 5173
3985+
"line_count": 156,
3986+
"char_count": 6005
39803987
},
39813988
{
39823989
"path": "reflective-prompt-library/skills/reflective-handoff-retro/SKILL.md",
@@ -4289,11 +4296,11 @@
42894296
"category": "skills",
42904297
"subcategory": "reflective-spec-plan",
42914298
"name": "reflective-spec-plan",
4292-
"description": "Use this to turn a requirement into a practical spec, usage-first design, and small reviewable task plan. Trigger when the user asks for a spec, PRD, tickets, implementation plan, workflow plan, or wants to avoid building the wrong thing.",
4299+
"description": "Use this to turn a requirement into a practical spec, usage-first design, reviewable task plan, no-code Test Plan, or agent workflow design. Trigger when the user asks for a spec, PRD, tickets, implementation plan, workflow plan, stateful or resumable workflow specification, rigorous test design before implementation, or wants to avoid building the wrong thing.",
42934300
"license": "MIT",
42944301
"frontmatter": {
42954302
"name": "reflective-spec-plan",
4296-
"description": "Use this to turn a requirement into a practical spec, usage-first design, and small reviewable task plan. Trigger when the user asks for a spec, PRD, tickets, implementation plan, workflow plan, or wants to avoid building the wrong thing.",
4303+
"description": "Use this to turn a requirement into a practical spec, usage-first design, reviewable task plan, no-code Test Plan, or agent workflow design. Trigger when the user asks for a spec, PRD, tickets, implementation plan, workflow plan, stateful or resumable workflow specification, rigorous test design before implementation, or wants to avoid building the wrong thing.",
42974304
"license": "MIT",
42984305
"risk_level": "low",
42994306
"human_review_required": "false",
@@ -4308,7 +4315,10 @@
43084315
"spec.md",
43094316
"usage.md",
43104317
"task-plan.md",
4318+
"test-plan.md",
4319+
"workflow-spec.md",
43114320
"02-engineering/spec-writer.md",
4321+
"02-engineering/test-designer.md",
43124322
"02-engineering/usage-first.md",
43134323
"02-engineering/task-slicer.md",
43144324
"04-agent/workflow-engine.md",
@@ -4328,10 +4338,26 @@
43284338
"level": 2,
43294339
"text": "Module Contract"
43304340
},
4341+
{
4342+
"level": 2,
4343+
"text": "Implementation Decision Gate"
4344+
},
43314345
{
43324346
"level": 2,
43334347
"text": "Workflow"
43344348
},
4349+
{
4350+
"level": 2,
4351+
"text": "Test Plan Mode"
4352+
},
4353+
{
4354+
"level": 3,
4355+
"text": "TEST-001: <name>"
4356+
},
4357+
{
4358+
"level": 2,
4359+
"text": "Workflow Design Mode"
4360+
},
43354361
{
43364362
"level": 2,
43374363
"text": "Artifact Output"
@@ -4353,8 +4379,8 @@
43534379
"text": "Prompt Sources"
43544380
}
43554381
],
4356-
"line_count": 135,
4357-
"char_count": 3909
4382+
"line_count": 230,
4383+
"char_count": 10292
43584384
}
43594385
],
43604386
"total_files": 79

reflective-prompt-library/plans/QUALITY_GATES_SUMMARY.md

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ This document summarizes the Phase 1 quality gates implemented for TeaPrompt bas
1818
- Checks for required frontmatter fields (name, description)
1919

2020
**Results:**
21-
- Scanned 96 markdown files
21+
- Scanned 103 files
2222
- 0 errors found
2323
- All links and references are valid
2424

@@ -63,9 +63,9 @@ python3 reflective-prompt-library/plans/generate_index.py
6363
- Provides actionable suggestions
6464

6565
**Results:**
66-
- Scanned 96 files
66+
- Scanned 103 files
6767
- 0 errors, 0 warnings
68-
- 75 files with suggestions (mostly non-critical)
68+
- 81 files with suggestions (mostly non-critical)
6969
- All 9 skills pass validation
7070

7171
**Usage:**
@@ -80,14 +80,14 @@ python3 reflective-prompt-library/plans/lint_skills.py
8080
**What it does:**
8181
- Implements ROUTE-001 from code-followups-plan.md
8282
- Loads intent groups, adversarial sets, thresholds, trace fields, and expected workflows from `route-001-paraphrase-eval.yaml`
83-
- Applies a small deterministic boundary classifier for ambiguous review/risk/planning phrasing
83+
- Applies a small deterministic boundary classifier for ambiguous review/risk/planning and workflow-design phrasing
8484
- Tests routing consistency across paraphrased intents
8585
- Validates that same intent routes to same workflow
8686
- Measures confidence, fallback behavior, low-confidence trace coverage, and silent downgrade incidents
8787
- Generates consistency reports
8888

8989
**Results:**
90-
- Tested 15 groups with 113 paraphrases, including 4 adversarial boundary groups, the minimality gate, and both sides of the Test Plan / executable-test boundary
90+
- Tested 16 groups with 128 paraphrases, including 4 adversarial boundary groups, the minimality gate, and the Test Plan, executable-test, and workflow-design boundaries
9191
- Overall consistency: 100.0% (passes Phase-1 threshold >=70% and aspirational target >=95%)
9292
- Low-confidence route trace coverage: 100.0%
9393
- Adversarial groups now pass through boundary rules rather than isolated synonym matching
@@ -111,10 +111,10 @@ python3 reflective-prompt-library/plans/route_paraphrase_eval.py
111111
- Keeps a lower Phase-1 bar and separate aspirational target to avoid over-claiming
112112

113113
**Results:**
114-
- Tested 9 holdout groups with 37 paraphrases
114+
- Tested 11 holdout groups with 46 paraphrases
115115
- Overall consistency: 100.0% (passes Phase-1 threshold >=80% and aspirational target >=90%)
116116
- Low-confidence route trace coverage: 100.0%
117-
- Review phrasing around patches/regressions and clarification phrasing around unknown desired outcome now pass through boundary concepts
117+
- Review, clarification, workflow-design, workflow-implementation, research, and orchestration-selection phrasing now pass through boundary concepts
118118
- Output: `plans/route-002-results.json`
119119

120120
**Usage:**
@@ -176,16 +176,16 @@ python3 reflective-prompt-library/plans/validate_governance.py
176176
**File:** `reflective-prompt-library/plans/benchmark_tasks.py`
177177

178178
**What it does:**
179-
- Defines 21 golden tasks for validation
180-
- Covers all major workflows (7 different skills)
181-
- Balanced difficulty distribution (5 easy, 9 medium, 6 hard)
182-
- Diverse categories (12 different categories)
179+
- Defines 23 golden tasks for validation
180+
- Covers all major benchmarked workflows (8 different skills)
181+
- Balanced difficulty distribution (5 easy, 11 medium, 7 hard)
182+
- Diverse categories (15 different categories)
183183
- Clear acceptance criteria for each task
184184
- Expected workflow mapping
185185

186186
**Results:**
187-
- 21 benchmark tasks created
188-
- Tasks cover: implementation, planning, review, research, risk, handoff, debugging, refactoring, retrospective, runtime governance, scaffold provenance, SOP compiler planning, and minimality
187+
- 23 benchmark tasks created
188+
- Tasks cover: implementation, planning, review, research, risk, handoff, debugging, refactoring, retrospective, runtime governance, scaffold provenance, SOP compiler planning, minimality, test planning, and workflow design
189189
- Output: `plans/benchmark-tasks.json`
190190

191191
**Usage:**
@@ -207,17 +207,17 @@ The implementation aligns with research findings:
207207

208208
| Metric | Value | Status |
209209
|--------|-------|--------|
210-
| Files validated | 96 | ✅ All pass |
210+
| Files validated | 103 | ✅ All pass |
211211
| Links validated | 0 broken | ✅ Perfect |
212212
| Skills with governance | 9/9 | ✅ Complete |
213-
| Benchmark tasks | 21 | ✅ Ready |
213+
| Benchmark tasks | 23 | ✅ Ready |
214214
| Routing consistency | 100.0% | ✅ Passes ROUTE-001 expanded boundary eval |
215215
| Holdout routing consistency | 100.0% | ✅ Passes ROUTE-002 holdout eval |
216216
| Linting errors | 0 | ✅ Clean |
217217

218218
### Routing Consistency Tracking
219219

220-
The current routing consistency measurement is 100.0% on ROUTE-001 across 15 groups and 113 paraphrases. The eval fixture is now the single source of truth for thresholds, expected workflows, trace fields, and adversarial sets, while `route_paraphrase_eval.py` is responsible for loading the fixture and measuring the current router. This passes the Phase-1 threshold (>=70%) and aspirational target (>=95%) for the expanded seeded boundary set, including the minimality gate and the Test Plan / executable-test boundary. The ROUTING_CONTRACT.md keeps both bars explicit so future evals can distinguish minimum acceptance from aspirational quality.
220+
The current routing consistency measurement is 100.0% on ROUTE-001 across 16 groups and 128 paraphrases. The eval fixture is now the single source of truth for thresholds, expected workflows, trace fields, and adversarial sets, while `route_paraphrase_eval.py` is responsible for loading the fixture and measuring the current router. This passes the Phase-1 threshold (>=70%) and aspirational target (>=95%) for the expanded seeded boundary set, including the minimality gate and the Test Plan, executable-test, and workflow-design boundaries. The ROUTING_CONTRACT.md keeps both bars explicit so future evals can distinguish minimum acceptance from aspirational quality.
221221

222222
### Critical Reflection
223223

@@ -227,19 +227,22 @@ ROUTE-001 should not be interpreted as proof that routing is solved. The useful
227227
- research vs local review
228228
- clarification vs planning
229229
- handoff vs dispatch
230+
- workflow design vs executable orchestration
231+
- workflow research or review vs first-draft design
232+
- workflow formalization selection vs committing to a workflow
230233

231234
The latest router improvement uses concept-level boundary rules for these cases instead of raw synonym accumulation. Future improvements should add unseen holdout cases or baseline-vs-skill evaluation before claiming general routing quality.
232235

233236
### Holdout Tracking
234237

235-
ROUTE-002 currently measures unseen phrasing separately from ROUTE-001. It passes the holdout aspirational target with 100.0% consistency after adding concept-level review and clarification boundaries. This should still be treated as a seeded holdout, not proof of broad semantic routing; the next useful step is adding fresh holdout cases before further router tuning.
238+
ROUTE-002 currently measures unseen phrasing separately from ROUTE-001. It passes the holdout aspirational target with 100.0% consistency after adding concept-level review, clarification, workflow-design, and orchestration-selection boundaries. This should still be treated as a seeded holdout, not proof of broad semantic routing; the next useful step is adding fresh holdout cases before further router tuning.
236239

237240
## Next Steps (Phase 2)
238241

239242
Based on the research and current implementation, suggested next steps:
240243

241244
1. **Expand ROUTE-002 with fresh holdout cases** - Add unseen cases before further router tuning
242-
2. **Run benchmark tests** - Execute the 20 tasks with/without skills to measure effectiveness
245+
2. **Run benchmark tests** - Execute the 23 tasks with/without skills to measure effectiveness
243246
3. **Add CI/CD** - Integrate validation scripts into automated pipeline
244247
4. **Collect feedback** - Use CONTRIBUTING.md process to gather community input
245248
5. **Iterate on routing** - Use paraphrase eval results to improve keyword matching

reflective-prompt-library/plans/benchmark_tasks.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,21 @@ def define_tasks(self) -> List[BenchmarkTask]:
358358
"Adversarial or anti-cheating checks are included when relevant",
359359
"Unknowns and false-positive guards are recorded without changing production code"
360360
]
361+
),
362+
BenchmarkTask(
363+
id="B023",
364+
name="No-code agent workflow design",
365+
description="Design a resumable human-in-the-loop agent workflow without implementing a runner",
366+
expected_workflow="reflective-spec-plan",
367+
difficulty="hard",
368+
category="workflow-design",
369+
acceptance_criteria=[
370+
"The lowest sufficient formalization level is selected and justified",
371+
"Fixed workflow, dynamic agent, and selected orchestration patterns are distinguished",
372+
"State, artifacts, checkpoints, and transition control ownership are explicit",
373+
"Side effects define authority, idempotency, retry caps, compensation, and hard stops",
374+
"Observability and scenario tests separate specified behavior from unproven runtime guarantees"
375+
]
361376
)
362377
]
363378

reflective-prompt-library/plans/route-001-paraphrase-eval.yaml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ intent_groups:
3434
- Patch the parser so it handles empty input
3535
- Build this feature in the code
3636
- Make the code change directly
37+
- Implement a LangGraph workflow with persistence
38+
- Build the runner from this workflow design
3739

3840
- intent: spec_planning
3941
expected_workflow: reflective-spec-plan
@@ -53,6 +55,23 @@ intent_groups:
5355
- Create a roadmap for the workflow
5456
- Prepare a release plan for this feature
5557

58+
- intent: workflow_design_only
59+
expected_workflow: reflective-spec-plan
60+
expected_rigor: medium
61+
expected_enhancements:
62+
- state_and_artifact_contracts
63+
- transition_ownership
64+
- recovery_and_observability
65+
phrases:
66+
- Design a resumable agent workflow with persisted state, gates, recovery, and human review
67+
- Turn this SOP into a gated workflow specification without implementing a runner
68+
- Create a workflow design with sequential parallel routing and evaluator loops
69+
- Define the state model transitions checkpoints and rollback for this process
70+
- Plan a manager-and-handoff workflow without writing code
71+
- Map this process into stages artifacts gates and stop conditions
72+
- Produce an orchestration plan that separates deterministic and model-decided control
73+
- Specify a human-in-the-loop workflow with observable failure recovery
74+
5675
- intent: test_plan_only
5776
expected_workflow: reflective-spec-plan
5877
expected_rigor: medium
@@ -101,6 +120,7 @@ intent_groups:
101120
- Look over my code in detail
102121
- Make sure it actually works
103122
- Review this like it is going to production
123+
- Review this workflow design for missing recovery paths
104124

105125
- intent: research_task
106126
expected_workflow: reflective-research
@@ -119,6 +139,8 @@ intent_groups:
119139
- Check source-backed guidance for this package
120140
- Find the current official recommendation
121141
- Research whether this approach is still valid
142+
- Research current workflow orchestration patterns
143+
- Compare official framework docs for durable workflow design
122144

123145
- intent: risk_gating
124146
expected_workflow: reflective-risk
@@ -185,6 +207,8 @@ intent_groups:
185207
- Dispatch this to the best skill
186208
- Decide whether this needs a workflow or just a prompt
187209
- What skill is appropriate here?
210+
- Choose between a deterministic workflow and a dynamic agent for this process
211+
- Decide whether this needs a prompt workflow or full agent
188212

189213
- intent: minimality_gate
190214
expected_workflow: reflective-minimality

0 commit comments

Comments
 (0)