Skip to content

Commit 6c5d38a

Browse files
committed
chore(governance): add dispatch benchmark task for nine-skill coverage
- Add B024 golden task for reflective-dispatch routing - Sync QUALITY_GATES_SUMMARY counts and nine-skill wording - Anti-drift: benchmark must cover all VALID_WORKFLOWS
1 parent 1c4d210 commit 6c5d38a

4 files changed

Lines changed: 35 additions & 6 deletions

File tree

reflective-prompt-library/plans/QUALITY_GATES_SUMMARY.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -239,16 +239,16 @@ python3 reflective-prompt-library/plans/validate_route_fixture.py
239239
**File:** `reflective-prompt-library/plans/benchmark_tasks.py`
240240

241241
**What it does:**
242-
- Defines 23 golden tasks for validation
243-
- Covers all major benchmarked workflows (8 different skills)
244-
- Balanced difficulty distribution (5 easy, 11 medium, 7 hard)
242+
- Defines 24 golden tasks for validation
243+
- Covers all nine frozen workflow skills
244+
- Balanced difficulty distribution (6 easy, 11 medium, 7 hard)
245245
- Diverse categories (15 different categories)
246246
- Clear acceptance criteria for each task
247247
- Expected workflow mapping
248248

249249
**Results:**
250-
- 23 benchmark tasks created
251-
- Tasks cover: implementation, planning, review, research, risk, handoff, debugging, refactoring, retrospective, runtime governance, scaffold provenance, SOP compiler planning, minimality, test planning, and workflow design
250+
- 24 benchmark tasks created
251+
- Tasks cover: routing, implementation, planning, review, research, risk, handoff, debugging, refactoring, retrospective, runtime governance, scaffold provenance, SOP compiler planning, minimality, test planning, and workflow design
252252
- Output: `plans/benchmark-tasks.json`
253253

254254
**Usage:**
@@ -277,7 +277,7 @@ The implementation aligns with research findings:
277277
| Files validated | 103 | ✅ All pass |
278278
| Links validated | 0 broken | ✅ Perfect |
279279
| Skills with governance | 9/9 | ✅ Complete |
280-
| Benchmark tasks | 23 | ✅ Ready |
280+
| Benchmark tasks | 24 | ✅ Ready |
281281
| Routing consistency | 100.0% | ✅ Passes ROUTE-001 expanded boundary eval |
282282
| Holdout routing consistency | 100.0% | ✅ ROUTE-002 (36 groups, 100 paraphrases) |
283283
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (13 groups, 31 paraphrases) |

reflective-prompt-library/plans/benchmark_tasks.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,21 @@ def define_tasks(self) -> List[BenchmarkTask]:
379379
"Side effects define authority, idempotency, retry caps, compensation, and hard stops",
380380
"Observability and scenario tests separate specified behavior from unproven runtime guarantees"
381381
]
382+
),
383+
BenchmarkTask(
384+
id="B024",
385+
name="Workflow skill selection for mixed intent",
386+
description="Classify a mixed user request and select the smallest useful reflective workflow skill with explicit route trace",
387+
expected_workflow="reflective-dispatch",
388+
difficulty="easy",
389+
category="routing",
390+
acceptance_criteria=[
391+
"Primary workflow skill is named with rationale tied to intent signals",
392+
"Strictness level is stated with risk and cost calibration",
393+
"Route trace includes confidence and at least one rejected alternate route",
394+
"Smallest sufficient workflow is chosen over plan-orchestration bloat",
395+
"Next action is explicit and matches the selected workflow"
396+
]
382397
)
383398
]
384399

reflective-prompt-library/plans/tests/test_quality_gates_summary.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,3 +104,9 @@ def test_research_alignment_lists_nine_frozen_skills(summary_text: str):
104104
assert "nine frozen workflow skills" in section
105105
assert "8 lifecycle skills" not in section
106106

107+
def test_benchmark_section_covers_nine_workflows(summary_text: str):
108+
section = summary_text.split("### 7. Small Benchmark Set", 1)[1].split("## Research Alignment", 1)[0]
109+
assert "nine frozen workflow skills" in section
110+
assert "8 different skills" not in section
111+
assert "24 golden tasks" in section or "24 benchmark tasks" in section
112+

reflective-prompt-library/plans/tests/test_validate_benchmark_fixture.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,11 @@ def test_every_task_maps_to_known_workflow():
2323
def test_benchmark_ids_are_unique():
2424
ids = [task.id for task in BenchmarkSet().tasks]
2525
assert len(ids) == len(set(ids))
26+
27+
def test_benchmark_covers_all_nine_workflows():
28+
workflows = {task.expected_workflow for task in BenchmarkSet().tasks}
29+
assert workflows == VALID_WORKFLOWS, (
30+
f"missing workflows: {sorted(VALID_WORKFLOWS - workflows)}; "
31+
f"unexpected: {sorted(workflows - VALID_WORKFLOWS)}"
32+
)
33+

0 commit comments

Comments
 (0)