Skip to content

Commit fb1de94

Browse files
committed
fix(walkthrough): honesty + discoverability fixes across CLI surfaces
Auto interactive-walkthrough of the Atlas CLI surfaces (5 reviewed, no P0/P1). Four evidence-backed fixes under TDD + Second-Check: - validate-prd Check 6: fail-closed on an ABSENT requirements section instead of vacuously passing + claiming 'all requirements are specific' (P2 honesty). - watcher permit reason: surface a cheating-winner discrepancy instead of masking it behind 'no to-be-slashed' (P2 legibility; the permit boolean is unchanged). - set-status --status: enumerate the valid status vocabulary in -h (P2). - engine-preflight/next-task/economy-report: add missing flag help (P3 consistency). RAG-ready runlogs under .i-walkthroughs/entries/. Offline suite 886 passed / 5 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent f2b9f33 commit fb1de94

12 files changed

Lines changed: 373 additions & 24 deletions

.i-walkthroughs/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Interactive-walkthrough runlogs
2+
3+
RAG-ingestion-ready learning units from `/interactive-walkthrough` runs on this repo.
4+
One file per (surface, issue) under `entries/`; `index.jsonl` is the append-only machine index;
5+
`schema.md` is the single source of truth. Headless repo: "route" = a CLI/MCP surface
6+
(e.g. `cli:validate-prd`), evidence = live CLI/test runs.
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
id: 2026-06-17-cli-undocumented-flags
3+
schema_version: 1
4+
repo: anombyte93/prd-taskmaster
5+
branch: walkthrough/atlas-cli-surfaces
6+
commit: pending
7+
date: 2026-06-17
8+
route: "cli:set-status"
9+
screen_purpose: "Expose the engine's CLI flags clearly enough that a cold user can discover what each does — and what values it accepts — from -h alone."
10+
mode: auto
11+
viewport: cli
12+
device_class: desktop
13+
severity: P2
14+
category: interaction
15+
root_cause: "Several argparse flags were added with no help= string; most damaging, set-status --status is a REQUIRED flag whose valid vocabulary (pending/in-progress/done/review/deferred/cancelled/blocked) appeared nowhere in help, SKILL.md, or the MCP tool."
16+
assumption_trap: "the flag name is self-documenting (a required enum flag is not — the accepted values must be shown)."
17+
reusable_rule: "Every CLI flag gets help text; a required enum flag must enumerate its accepted values in that help."
18+
evidence_method: [code]
19+
files_changed: ["prd_taskmaster/cli.py", "tests/core/test_cli.py"]
20+
first_check: {result: fail, metric: "build_parser(): set-status --status, engine-preflight --no-configure, next-task --tag, economy-report --input all have help==None", evidence_ref: "pytest test_cli_flags_have_discoverable_help (pre-fix RED) + live set-status -h showing bare '--status STATUS'"}
21+
second_check: {result: pass, metric: "all four flags have non-empty help; set-status --status help contains 'in-progress'", evidence_ref: "pytest (GREEN) + live: set-status -h now prints 'Task status; one of: pending, in-progress, done, review, deferred, cancelled, blocked'", regression_scan: pass}
22+
regression_checks: ["set-status invalid-status error path unchanged (help-only, no argparse choices added → still returns FR-28 dict, not an argparse usage error)", "all existing CLI subprocess tests", "full offline suite"]
23+
verdict: resolved
24+
status: verified
25+
approval: pending
26+
evidence_dir: evidence/2026-06-17-cli-undocumented-flags
27+
tags: [cli, discoverability, help-text, consistency, set-status]
28+
related: []
29+
title: "Undocumented CLI flags — esp. set-status --status whose valid vocabulary was undiscoverable"
30+
---
31+
32+
## Lesson (TL;DR)
33+
Four CLI flags carried no help text; the worst was `set-status --status`, a REQUIRED flag whose accepted values were findable nowhere a user looks (`-h`, SKILL.md, or the MCP tool). A cold user had to read source to learn the status vocabulary. Every flag gets help; a required enum flag enumerates its values.
34+
35+
## Screen & Purpose
36+
Surfaces `cli:set-status`, `cli:engine-preflight`, `cli:next-task`, `cli:economy-report`. Soul purpose: a self-describing CLI a cold user can drive from `-h`.
37+
38+
## Issue (first check)
39+
`build_parser()` introspection: `set-status --status`, `engine-preflight --no-configure`, `next-task --tag`, `economy-report --input` all had `help is None`. Live `set-status -h` printed `--status STATUS` with no value list — the vocabulary (pending/in-progress/done/review/deferred/cancelled/blocked) was invisible.
40+
41+
## Root Cause
42+
Flags were added via `add_argument(...)` without `help=`. Inconsistent with sibling flags (e.g. `parse-prd --input`) that do carry help.
43+
44+
## Fix
45+
`cli.py`: add help strings to the four flags. For `set-status --status`, the help enumerates the valid status vocabulary. Help-only (NOT argparse `choices=`) so the engine's existing FR-28-safe status validation + error dict is preserved rather than replaced by an argparse usage error.
46+
47+
## Second Check (re-verification, MANDATORY)
48+
Post-fix: introspection shows all four flags with non-empty help; `set-status --status` help contains "in-progress"; live `set-status -h` prints the full status list. Regression scan: the invalid-status path still returns the engine's FR-28 error dict (no `choices=` added, so no behavior change to error handling); all existing CLI subprocess tests pass; full offline suite green. `regression_scan=pass`, `verdict=resolved`.
49+
50+
## Reusable Rule
51+
Every CLI flag gets help text; a required enum flag must enumerate its accepted values in that help.
52+
53+
## Decision Trail
54+
Owner: walkthrough auto-mode; real owner approves via PR. Rejected: adding argparse `choices=` for `--status` (would move validation to the parse layer and emit an argparse usage error instead of the engine's structured FR-28 dict — a contract change). Deferred: also documenting the status vocabulary in SKILL.md + the MCP tool description (follow-up; this entry fixes the CLI surface).
55+
56+
## Revisions
57+
- 2026-06-17: initial fix; second check passed on first attempt.
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
id: 2026-06-17-cli-validate-prd-check6-false-pass
3+
schema_version: 1
4+
repo: anombyte93/prd-taskmaster
5+
branch: walkthrough/atlas-cli-surfaces
6+
commit: pending
7+
date: 2026-06-17
8+
route: "cli:validate-prd"
9+
screen_purpose: "Give a user a fast, honest, point-scored verdict on whether a PRD is complete enough to parse into a trustworthy task graph."
10+
mode: auto
11+
viewport: cli
12+
device_class: desktop
13+
severity: P2
14+
category: trust
15+
root_cause: "Check 6 ran VAGUE_PATTERN.findall over an empty string when no requirements section existed; an empty match list read as 'no vague terms' → passed:true + 'All requirements are specific'."
16+
assumption_trap: "no-vague-terms == requirements-are-testable (true only when requirements actually exist)."
17+
reusable_rule: "A presence/quality check over an ABSENT section must fail-closed, never pass-and-assert-quality on emptiness."
18+
evidence_method: [code]
19+
files_changed: ["prd_taskmaster/validation.py", "tests/core/test_validation.py"]
20+
first_check: {result: fail, metric: "empty PRD → check6.passed=true, detail='All requirements are specific', checks_passed=3, score=13", evidence_ref: "live: python3 script.py validate-prd --input /tmp/wt_empty.md (pre-fix)"}
21+
second_check: {result: pass, metric: "empty PRD → check6.passed=false, detail='No requirements section found', checks_passed=2, score=8, grade=NEEDS_WORK", evidence_ref: "live post-fix run + pytest -k 'validate_empty_prd or check6_fails_closed'", regression_scan: pass}
22+
regression_checks: ["sample_prd (real requirements) still passes check6", "thorough PRD still 57/57 EXCELLENT exit 0", "grade boundary tests", "full offline suite tests/core+tests/plugin"]
23+
verdict: resolved
24+
status: verified
25+
approval: pending
26+
evidence_dir: evidence/2026-06-17-cli-validate-prd-check6-false-pass
27+
tags: [validate-prd, honesty, fail-closed, vacuous-pass]
28+
related: []
29+
title: "validate-prd Check 6 falsely passed + claimed specificity on a PRD with zero requirements"
30+
---
31+
32+
## Lesson (TL;DR)
33+
`validate-prd` Check 6 ("Functional requirements are testable") passed AND printed "All requirements are specific" on a PRD that had no requirements section at all — a positively false claim about content that does not exist. Root cause: a vague-term scan over an empty string returns no matches, which the code read as success. A presence/quality check over an absent section must fail-closed.
34+
35+
## Screen & Purpose
36+
Surface `cli:validate-prd` (13-check PRD quality gate, `validation.py:run_validate_prd`). Soul purpose: an honest, point-scored verdict on PRD completeness. Headless/CLI; evidence = live runs + unit tests.
37+
38+
## Issue (first check)
39+
Live, pre-fix, on a PRD containing only a title + Overview (zero requirements):
40+
`check 6 → passed:true, detail:"All requirements are specific"`; `checks_passed:3, score:13`. The detail makes an affirmative specificity claim about requirements that are absent. (The final grade NEEDS_WORK was already correct; the defect is the per-check honesty + an inflated checks_passed.)
41+
42+
## Root Cause
43+
`reqs_section` was empty; `VAGUE_PATTERN.findall("")==[]``passed = len([])==0 = True` and the else-branch detail "All requirements are specific" fired. The check never distinguished "present and specific" from "absent".
44+
45+
## Fix
46+
`validation.py`: when `reqs_section.strip()` is empty, emit `passed:false, detail:"No requirements section found"` (fail-closed); otherwise the original vague-term logic. Smallest coherent change; checks 5 & 10 (user-stories / NFRs are genuinely OPTIONAL sections) left intentionally untouched.
47+
48+
## Second Check (re-verification, MANDATORY)
49+
Same empty PRD, post-fix: `check 6 → passed:false, detail:"No requirements section found"`, `checks_passed:2, score:8, grade:NEEDS_WORK`. Same-metric assertion now passes. Regression scan: `sample_prd` (real requirements) still passes check 6; a thorough PRD still scores 57/57 EXCELLENT (exit 0); grade-boundary tests green; full offline suite green. `regression_scan=pass`, `verdict=resolved`.
50+
51+
## Reusable Rule
52+
A presence/quality check over an ABSENT section must fail-closed — never pass-and-assert-quality on emptiness.
53+
54+
## Decision Trail
55+
Owner: walkthrough auto-mode (Hayden stand-in); real owner approves via the PR. Rejected: only rewording the detail while keeping `passed:true` (would leave the vacuous pass + inflated count). Deferred: checks 5 & 10 pass-on-absent (P3; those sections are legitimately optional) — recorded for a future decision.
56+
57+
## Revisions
58+
- 2026-06-17: initial fix; second check passed on first attempt.
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
id: 2026-06-17-cli-watcher-permit-reason-mask
3+
schema_version: 1
4+
repo: anombyte93/prd-taskmaster
5+
branch: walkthrough/atlas-cli-surfaces
6+
commit: pending
7+
date: 2026-06-17
8+
route: "cli:watcher-run"
9+
screen_purpose: "Let an operator see the fail-closed real-slash permit for a job and understand WHY it is or isn't permitted."
10+
mode: auto
11+
viewport: cli
12+
device_class: desktop
13+
severity: P2
14+
category: trust
15+
root_cause: "The permit reason ladder tested `not to_slash` before `discrepancies`, so a job with no to-be-slashed submissions but a cheating-winner DISCREPANCY reported the benign 'no to-be-slashed' message instead of the discrepancy."
16+
assumption_trap: "empty to-be-slashed set == nothing interesting happened (ignores discrepancies/abstains on the winners)."
17+
reusable_rule: "Order a fail-closed reason ladder most-serious-finding first; never let a benign branch shadow a security-relevant one."
18+
evidence_method: [code]
19+
files_changed: ["prd_taskmaster/tournament/watcher.py", "tests/core/test_tournament_watcher.py"]
20+
first_check: {result: fail, metric: "permit_enforce_slash(record with empty to_slash + cheating-winner DISCREPANCY).reason == 'blocked: no to-be-slashed submissions to confirm' (discrepancy masked)", evidence_ref: "pytest test_permit_reason_surfaces_winner_discrepancy_over_empty_slash (pre-fix RED)"}
21+
second_check: {result: pass, metric: "same record → reason contains 'discrepancy' and NOT 'no to-be-slashed'; permitted still False; ex-win in discrepancies", evidence_ref: "pytest (post-fix GREEN)", regression_scan: pass}
22+
regression_checks: ["existing permit reason tests (discrepancy-among-slashes, abstained-winner, empty-to_slash, mixed-batch) unchanged", "permitted boolean decision unchanged across all permit tests", "full offline suite"]
23+
verdict: resolved
24+
status: verified
25+
approval: pending
26+
evidence_dir: evidence/2026-06-17-cli-watcher-permit-reason-mask
27+
tags: [watcher, slash-gate, fail-closed, legibility, self-review]
28+
related: [atlas-watcher-built]
29+
title: "watcher permit reason masked a caught cheating-winner discrepancy with a benign 'no to-be-slashed' message"
30+
---
31+
32+
## Lesson (TL;DR)
33+
The watcher's real-slash permit correctly BLOCKED a job where a winner was an independent discrepancy, but its human-readable `reason` said "no to-be-slashed submissions to confirm" — hiding the very thing the operator most needs to see. The boolean was right; the explanation lied by omission. Order the reason ladder most-serious-first.
34+
35+
## Screen & Purpose
36+
Surface `cli:watcher-run``watcher.permit_enforce_slash` (the fail-closed gate that decides whether real `--enforce-slash` forfeiture may proceed). Soul purpose: an operator-legible, honest permit verdict.
37+
38+
## Issue (first check)
39+
A record with zero to-be-slashed submissions but a cheating winner (recorded PASS, watcher DISCREPANCY) returned `permitted:false` (correct) with `reason:"blocked: no to-be-slashed submissions to confirm"` (wrong — the discrepancy was in `out["discrepancies"]` but never surfaced in the headline reason). `to_slash` is empty because the winner recorded PASS, so the `elif not to_slash` branch fired before `elif discrepancies`.
40+
41+
## Root Cause
42+
Reason-ladder ordering: `permitted → not to_slash → discrepancies → abstained → track_record`. The benign empty-set branch shadowed the security-relevant discrepancy branch.
43+
44+
## Fix
45+
`watcher.py`: reorder the ladder to `permitted → discrepancies → abstained → not to_slash → track_record`. The `permitted` boolean is unchanged (already `not discrepancies and not abstained and ...`); only the explanatory string ordering changed.
46+
47+
## Second Check (re-verification, MANDATORY)
48+
Post-fix, same record: `reason` contains "discrepancy", does NOT contain "no to-be-slashed", `permitted` still False, `ex-win` in `discrepancies`. Regression scan: all prior permit tests (discrepancy-among-slashes, abstained-winner, empty-to_slash, mixed-batch, concordance boundary) unchanged; every `permitted` boolean identical to before. Full offline suite green. `regression_scan=pass`, `verdict=resolved`.
49+
50+
## Reusable Rule
51+
Order a fail-closed reason ladder most-serious-finding first; a benign branch must never shadow a security-relevant one.
52+
53+
## Decision Trail
54+
Owner: walkthrough auto-mode; real owner approves via PR. This fix touches code authored earlier this same session — a good reminder that the adversarial surface review caught a legibility gap the build-time tests didn't assert. No alternatives rejected; the reorder is the minimal correct change.
55+
56+
## Revisions
57+
- 2026-06-17: initial fix; second check passed on first attempt.

.i-walkthroughs/index.jsonl

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{"id":"2026-06-17-cli-validate-prd-check6-false-pass","repo":"anombyte93/prd-taskmaster","branch":"walkthrough/atlas-cli-surfaces","route":"cli:validate-prd","date":"2026-06-17","severity":"P2","category":"trust","verdict":"resolved","status":"verified","title":"validate-prd Check 6 falsely passed + claimed specificity on a PRD with zero requirements","file":"entries/2026-06-17-cli-validate-prd-check6-false-pass.md"}
2+
{"id":"2026-06-17-cli-watcher-permit-reason-mask","repo":"anombyte93/prd-taskmaster","branch":"walkthrough/atlas-cli-surfaces","route":"cli:watcher-run","date":"2026-06-17","severity":"P2","category":"trust","verdict":"resolved","status":"verified","title":"watcher permit reason masked a caught cheating-winner discrepancy","file":"entries/2026-06-17-cli-watcher-permit-reason-mask.md"}
3+
{"id":"2026-06-17-cli-undocumented-flags","repo":"anombyte93/prd-taskmaster","branch":"walkthrough/atlas-cli-surfaces","route":"cli:set-status","date":"2026-06-17","severity":"P2","category":"interaction","verdict":"resolved","status":"verified","title":"Undocumented CLI flags incl. set-status --status vocabulary","file":"entries/2026-06-17-cli-undocumented-flags.md"}

0 commit comments

Comments
 (0)