Skip to content

Commit 61ff35a

Browse files
committed
feat(thinking): Round 69 prompt contract standardization
Six-lens panel review agreed to add Purpose/Scope/Acceptance/Falsifiability contracts to all 01-thinking prompts, map lenses to frozen workflow skills, and gate quality with test_thinking_prompts_eval_harness.py (80%+ eval_harness). Also: Decision Index fix, QUALITY_GATES 200+ pytest floor, panel Round 69 record.
1 parent d4cdee2 commit 61ff35a

9 files changed

Lines changed: 223 additions & 7 deletions

reflective-prompt-library/01-thinking/counterargument.md

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,31 @@
22

33
Use this to prevent excessive optimism, overengineering, or AI flattery.
44

5+
## Purpose
6+
7+
Stress-test optimism, overengineering, and AI flattery before committing resources. Primary workflow surfaces: `reflective-review` and `reflective-minimality`; escalate to `reflective-risk` when trust-boundary or blast-radius signals appear.
8+
9+
## Scope
10+
11+
- In scope: attacking a proposed plan, architecture, or feature scope for fatal flaws and cheaper alternatives.
12+
- Out of scope: polite encouragement, implementation work, or net-new research without a plan to attack.
13+
14+
## Acceptance Criteria
15+
16+
- Fatal flaws separated from major and minor concerns.
17+
- At least one lower-cost alternative named.
18+
- Minimum viable version stated.
19+
- Kill criteria defined (when to stop).
20+
21+
## Falsifiability
22+
23+
List kill criteria: what evidence or outcome means the proposal should be abandoned rather than iterated.
24+
25+
## Human Review
26+
27+
Escalate to `reflective-risk` with an explicit Human Review gate when the work implies irreversible or high-blast-radius action.
28+
29+
530
```markdown
631
請扮演最嚴格但公平的反方架構師,攻擊我的方案。
732

@@ -30,4 +55,3 @@ Use this to prevent excessive optimism, overengineering, or AI flattery.
3055
- Minimum viable version:最小可行版本
3156
- Kill criteria:什麼情況應該停止
3257
```
33-

reflective-prompt-library/01-thinking/critical-thinking-check.md

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,31 @@
22

33
Use this to review articles, strategies, product ideas, technical choices, or AI-generated claims.
44

5+
## Purpose
6+
7+
Audit claims, assumptions, and evidence quality before accepting a plan, article, or AI output. Primary workflow surface: `reflective-review`; pair with `reflective-research` when external sources are required.
8+
9+
## Scope
10+
11+
- In scope: articles, strategies, product ideas, technical choices, AI-generated claims, diff/plan review prep.
12+
- Out of scope: writing first-draft specs (`reflective-spec-plan`), repository implementation (`reflective-implement`), formal-environment gating (`reflective-risk`).
13+
14+
## Acceptance Criteria
15+
16+
- Major and minor claims listed separately.
17+
- Assumption audit distinguishes explicit vs implicit risks.
18+
- Strongest counterargument stated (not a strawman).
19+
- Each conclusion tagged: supported, plausible-but-unproven, or unsupported.
20+
21+
## Falsifiability
22+
23+
Name at least one observation, dataset, or experiment that would overturn the strongest supported conclusion.
24+
25+
## Human Review
26+
27+
Escalate to `reflective-risk` with an explicit Human Review gate when the work implies irreversible or high-blast-radius action.
28+
29+
530
```markdown
631
請用 Critical Thinking Review 嚴格審查以下內容:
732

@@ -61,4 +86,3 @@ Use this to review articles, strategies, product ideas, technical choices, or AI
6186
- 需要更多證據部分
6287
- 不建議採用部分
6388
```
64-

reflective-prompt-library/01-thinking/falsifiability.md

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,31 @@
22

33
Use this for scientific thinking, product experiments, AI workflows, and business models.
44

5+
## Purpose
6+
7+
Turn vague ideas into testable hypotheses before experiments or delivery commitments. Primary workflow surfaces: `reflective-brief` for early framing; `reflective-spec-plan` when the hypothesis becomes a test plan.
8+
9+
## Scope
10+
11+
- In scope: scientific thinking, product experiments, AI workflow checks, business model bets.
12+
- Out of scope: full implementation, formal-environment rollout, or external source surveys without a stated hypothesis.
13+
14+
## Acceptance Criteria
15+
16+
- Hypothesis is binary and observable.
17+
- Prediction and falsifier are distinct.
18+
- Minimum test, metrics, and timebox are concrete (no "feels better").
19+
- Decision rule maps outcomes to next action.
20+
21+
## Falsifiability
22+
23+
Every output must include an explicit falsifier and a minimum test; if none can be stated, the idea is not ready for execution.
24+
25+
## Human Review
26+
27+
Escalate to `reflective-risk` with an explicit Human Review gate when the work implies irreversible or high-blast-radius action.
28+
29+
530
```markdown
631
請把以下想法改寫成可證偽假說。
732

@@ -22,4 +47,3 @@ Use this for scientific thinking, product experiments, AI workflows, and busines
2247

2348
請避免空泛指標,例如「感覺更好」。請盡量轉成可觀察、可比較、可重複的證據。
2449
```
25-

reflective-prompt-library/01-thinking/socratic-reviewer.md

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,26 @@
22

33
Suitable for requirements interviews, life decisions, product direction, business models, technical selection, learning strategy, and research question definition.
44

5+
## Purpose
6+
7+
Clarify the real question before choosing a direction. Primary workflow surfaces: `reflective-brief` for goal and assumption clarification; escalate to `reflective-spec-plan` when scope is clear enough to plan.
8+
9+
## Scope
10+
11+
- In scope: requirements interviews, product direction, technical selection, learning strategy, research question definition.
12+
- Out of scope: code implementation (`reflective-implement`), production risk gating (`reflective-risk`), source-backed external research (`reflective-research`).
13+
14+
## Acceptance Criteria
15+
16+
- Core problem stated in one sentence.
17+
- At least one high-risk assumption named.
18+
- Minimal falsifiable test or observation defined.
19+
- Explicit defer list when scope is still open.
20+
21+
## Falsifiability
22+
23+
If the session cannot name what evidence would prove the current framing wrong, stop and return to Clarify instead of recommending action.
24+
525
```markdown
626
你是 Socratic Questioner。你的目標不是立刻給答案,而是幫我逼近真正問題。
727

@@ -52,4 +72,3 @@ Suitable for requirements interviews, life decisions, product direction, busines
5272
- 下一步
5373
- 暫時不該做的事
5474
```
55-

reflective-prompt-library/01-thinking/why-what-how-done.md

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,26 @@
22

33
Use this as the core gate prompt before committing to a direction.
44

5+
## Purpose
6+
7+
Gate a task through Why / What / How / Done before choosing strictness or workflow depth. Primary workflow surfaces: `reflective-brief` then `reflective-dispatch` for orchestration level selection.
8+
9+
## Scope
10+
11+
- In scope: pre-commitment checks on goal, scope, method, and completion evidence for a single task or feature.
12+
- Out of scope: post-hoc code review (`reflective-review`), handoff retros (`reflective-handoff-retro`), or detailed test implementation.
13+
14+
## Acceptance Criteria
15+
16+
- Why gate answers cost of doing nothing and doing wrong.
17+
- What gate lists scope in/out, outputs, and acceptance criteria.
18+
- How gate names minimal method, risks, and human review triggers.
19+
- Done gate ties completion to observable evidence.
20+
21+
## Falsifiability
22+
23+
Done gate must name evidence that would prove the task should not proceed or should be rolled back.
24+
525
```markdown
626
請把任務通過 Why / What / How / Done 四層檢查。
727

@@ -52,4 +72,3 @@ Use this as the core gate prompt before committing to a direction.
5272
- 應該做到哪個層級
5373
- Prompting 即可,還是需要 Agent / Workflow / 程式
5474
```
55-

reflective-prompt-library/PROJECT_KNOWLEDGE.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,10 +72,11 @@ deferred promotions are recurrence-gated — see [panel backlog](plans/multi-age
7272

7373
## Decision Index
7474

75-
- 2026-06-25 Post-Round 68 holdout expansion — ROUTE-003 minimality trap + thin adversarial group refresh (15 groups / 53 phrases) → [QUALITY_GATES_SUMMARY.md](plans/QUALITY_GATES_SUMMARY.md)
7675
> Pointers to the causal trail — plans, reflections, tests, commits. Detail is
7776
> not duplicated here; this is a map, not an archive.
7877
78+
- 2026-06-25 Round 69 panel — standardize `01-thinking/` prompt contracts (Purpose/Scope/Acceptance/Falsifiability) + eval_harness anti-drift → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
79+
- 2026-06-25 Post-Round 68 holdout expansion — ROUTE-003 minimality trap + thin adversarial group refresh (15 groups / 53 phrases) → [QUALITY_GATES_SUMMARY.md](plans/QUALITY_GATES_SUMMARY.md)
7980
- 2026-06-11 Add `runtime-trust-boundary` as a lens, not a ninth skill → [record](plans/project-adjustment-reflection-2026-06-11.md)
8081
- 2026-06-17 Adopt a minimal project-judgment contract instead of importing Knowie → [initial record](plans/knowie-project-knowledge-reflection-2026-06-17.md)
8182
- 2026-06-17 Refine the contract from non-normative to non-authoritative, connect promotion, and publish a reusable scaffold → [decision](plans/project-knowledge-authority-promotion-decision-2026-06-17.md)

reflective-prompt-library/plans/QUALITY_GATES_SUMMARY.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -314,7 +314,7 @@ ROUTE-002 measures unseen phrasing separately from ROUTE-001. Round 7 (2026-06-2
314314
2. **ROUTE-001/002/003 in CI** — 128 + 102 + 53 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
315315
3. **Governance validators** — links, lint, governance metadata, PROJECT_KNOWLEDGE, benchmark fixture, skill examples
316316
4. **Harness policy docs** — CONTRIBUTING, AGENTS, SKILL_INSTALLATION, maintenance playbook
317-
5. **Doc anti-drift**`test_routing_contract.py`, cheatsheet parity tests, `test_readme_governance.py` (190+ pytest anti-drift suite in CI)
317+
5. **Doc anti-drift**`test_routing_contract.py`, cheatsheet parity tests, `test_readme_governance.py`, `test_thinking_prompts_eval_harness.py` (200+ pytest anti-drift suite in CI)
318318

319319
### Ongoing maintenance (not blockers)
320320

reflective-prompt-library/plans/multi-agent-panel-consensus-2026-06-25.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1475,3 +1475,56 @@ User directive: **review all docs** — reopen six-lens panel for cross-surface
14751475

14761476
_Ongoing maintenance: [GLOSSARY.md](../GLOSSARY.md) Governance Maintenance Playbook — expand ROUTE-002/003 holdout before `route_paraphrase_eval.py` tuning (R8–R12)._
14771477

1478+
1479+
1480+
## Round 69 — Thinking prompt contract review (2026-06-25)
1481+
1482+
User directive: review prompts, plans, skills, and Socratic/critical-thinking lenses in parallel until all roles agree, then implement.
1483+
1484+
### DA: Standardize `01-thinking/` prompt contracts?
1485+
1486+
| Lens | Position |
1487+
| --- | --- |
1488+
| Opus | **Agree** — eval_harness fails without document-level Purpose/Scope/Acceptance/Falsifiability |
1489+
| Codex | **Agree** — map each lens to frozen workflow skills; no tenth skill |
1490+
| Gemini | **Agree** — keep zh-TW templates; add English contract outside code block |
1491+
| Composer | **Agree** — bounded to five thinking prompts, not all 39 prompts |
1492+
| Sakana | **Agree** — aligns with evidence-over-confidence in PROJECT_KNOWLEDGE |
1493+
| GLM | **Agree** — skill-map already routes thinking lenses to brief/review |
1494+
1495+
**Socratic Q:** Does this duplicate skills?
1496+
**Answer:** No — prompts are composable lenses; skills are workflow contracts. Purpose sections name routing, not replace SKILL.md.
1497+
1498+
**Consensus:** **Agree** — add Purpose/Scope/Acceptance Criteria/Falsifiability to all five `01-thinking/` prompts; add `test_thinking_prompts_eval_harness.py`.
1499+
1500+
### DB: Expand to all prompt categories now?
1501+
1502+
| Lens | Position |
1503+
| --- | --- |
1504+
| All six | **Reject** — recurrence-gated; 01-thinking is highest-leverage for this review |
1505+
1506+
**Consensus:** **Reject** — defer 00-core/02-engineering Purpose sweep.
1507+
1508+
### Round 69 verdict table
1509+
1510+
| ID | Option | Verdict | Action |
1511+
| --- | --- | --- | --- |
1512+
| DA | Thinking prompt contracts | **Agree** | 5 files + pytest anti-drift |
1513+
| DB | All-category Purpose sweep | **Reject** | backlog |
1514+
1515+
**All roles agree.**
1516+
1517+
## Implemented Changes (Round 69)
1518+
1519+
- `01-thinking/*.md`: Purpose, Scope, Acceptance Criteria, Falsifiability + workflow skill mapping
1520+
- `plans/tests/test_thinking_prompts_eval_harness.py`: structural + score floor anti-drift
1521+
- `QUALITY_GATES_SUMMARY.md`: 200+ pytest floor; thinking prompt test mention
1522+
- `PROJECT_KNOWLEDGE.md`: Decision Index Round 69 entry
1523+
1524+
## Verification (Round 69)
1525+
1526+
- `make all`: pytest + ROUTE-001/002/003 100%
1527+
1528+
## Panel status (updated)
1529+
1530+
**Resealed 2026-06-25** after **Round 69** (options DA–DB). Thinking-prompt contract pass complete; broader prompt-category sweep remains recurrence-gated.
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""Anti-drift: 01-thinking prompts must satisfy eval_harness structural rubric."""
2+
3+
import sys
4+
from pathlib import Path
5+
6+
import pytest
7+
8+
sys.path.insert(0, str(Path(__file__).parent.parent))
9+
10+
from eval_harness import EvalHarness # noqa: E402
11+
12+
THINKING_DIR = Path(__file__).parent.parent.parent / "01-thinking"
13+
REPO_ROOT = str(Path(__file__).parent.parent.parent.parent)
14+
MIN_SCORE = 80.0
15+
16+
REQUIRED_HEADINGS = (
17+
"## Purpose",
18+
"## Scope",
19+
"## Acceptance Criteria",
20+
"## Falsifiability",
21+
)
22+
23+
THINKING_PROMPTS = tuple(sorted(THINKING_DIR.glob("*.md")))
24+
25+
26+
@pytest.fixture(scope="module")
27+
def harness() -> EvalHarness:
28+
return EvalHarness(repo_root=REPO_ROOT)
29+
30+
31+
@pytest.mark.parametrize("prompt_path", THINKING_PROMPTS, ids=lambda p: p.name)
32+
def test_thinking_prompt_has_contract_headings(prompt_path: Path):
33+
text = prompt_path.read_text(encoding="utf-8")
34+
preamble = text.split("```", 1)[0]
35+
for heading in REQUIRED_HEADINGS:
36+
assert heading in preamble, f"{prompt_path.name} missing {heading} outside template block"
37+
38+
39+
@pytest.mark.parametrize("prompt_path", THINKING_PROMPTS, ids=lambda p: p.name)
40+
def test_thinking_prompt_meets_eval_harness_floor(prompt_path: Path, harness: EvalHarness):
41+
rel = str(prompt_path.relative_to(REPO_ROOT))
42+
result = harness.evaluate_file(rel)
43+
assert result["score"] >= MIN_SCORE, (
44+
f"{prompt_path.name} eval_harness score {result['score']}% < {MIN_SCORE}%: "
45+
f"{[(c['id'], c['result']) for c in result['checks']]}"
46+
)
47+
48+
49+
def test_thinking_prompts_reference_workflow_skills():
50+
for prompt_path in THINKING_PROMPTS:
51+
text = prompt_path.read_text(encoding="utf-8")
52+
assert "reflective-" in text, f"{prompt_path.name} should map to at least one workflow skill"

0 commit comments

Comments
 (0)