You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
refactor(pb-build): replace full test suite with incremental testing strategy
pb-build previously ran the full test suite after every task, which became
a bottleneck as the codebase grew. This change switches to incremental
testing: each task runs only the tests affected by its changes, with a
single full suite run at the end for regression verification.
Changes:
- Unit Test Verification step now runs only affected tests
- TDD-only task RED/GREEN/REFACTOR cycles target specific test files
- ALWAYS constraint updated to mandate incremental tests
- Added "Incremental Test Strategy" section with:
- git diff to identify changed source files
- Convention-based test file mapping
- Import dependency tracing
- Affected test set execution commands
- Evaluator Check D now explicitly references incremental strategy
- Added Step 5a "Full Test Suite Run" after all tasks complete
- Removed redundant "Run full test suite" from completion summary
Copy file name to clipboardExpand all lines: skills/pb-build/SKILL.md
+67-13Lines changed: 67 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -207,7 +207,7 @@ The subagent operates as **Generator** — its sole objective is to make tests p
207
207
208
208
5. **[REFACTOR] Clean Up** — If needed, refactor for clarity/performance. Re-run the scenario to confirm still passing.
209
209
210
-
6. **Unit Test Verification** — Run unit tests: `[Unit test command]`. Ensure no regressions.
210
+
6. **Unit Test Verification (Incremental)** — Run ONLY tests affected by this task's changes (see "Incremental Test Strategy" below). Ensure no regressions.
211
211
212
212
7. **Runtime Verification (when applicable)** — Run runtime checks from task Verification and capture outputs.
213
213
@@ -218,10 +218,10 @@ The subagent operates as **Generator** — its sole objective is to make tests p
218
218
##### For TDD-Only Tasks (Non-Scenario)
219
219
220
220
1. **RED** — Write a failing unit test. **STOP after this step.**
221
-
2. **Confirm RED** — Run test suite. New test must fail for the right reason.
221
+
2. **Confirm RED** — Run the new test file (or targeted test). New test must fail for the right reason.
@@ -289,11 +289,11 @@ After the Generator signals `READY_FOR_EVAL`, the orchestrator must perform an *
289
289
- The exact BDD command run (same as RED)
290
290
- All steps passing
291
291
- The scenario name that passed
292
-
-**Independent Re-run:** Evaluator MUST independently run the scenario to verify it passes:
292
+
-**Independent Re-run:** Evaluator MUST independently run the scenario to verify it passes, plus related unit tests per the Incremental Test Strategy:
293
293
294
-
```text
295
-
[BDD command] --tags@[scenario_tag]
296
-
```
294
+
```text
295
+
[BDD command] --tags@[scenario_tag]
296
+
```
297
297
298
298
- If independent re-run FAILS, evaluation FAILS regardless of Generator's claims.
299
299
@@ -328,7 +328,7 @@ After the Generator signals `READY_FOR_EVAL`, the orchestrator must perform an *
328
328
**Light Review Process (for simple `TDD-only` tasks):**
329
329
330
330
1. Read the `git diff` to confirm scope.
331
-
2.Verify test suite passes.
331
+
2.Run affected tests only (follow Incremental Test Strategy).
332
332
3. Check for architecture violations in the diff.
333
333
4. If clean, emit PASS immediately.
334
334
@@ -446,7 +446,18 @@ If during implementation a subagent discovers that the design is **infeasible or
446
446
447
447
### Step 5: Final State Verification & Output Completion Summary
448
448
449
-
#### 5a. Final State Sweep (Mandatory)
449
+
#### 5a. Full Test Suite Run (Mandatory)
450
+
451
+
After ALL tasks are `🟢 DONE`, run the full test suite ONCE to verify no regressions across the entire build:
452
+
453
+
```bash
454
+
# Run the project's full test suite
455
+
uv run pytest tests/ -v
456
+
```
457
+
458
+
If the full suite fails, investigate the failing tests and determine whether they are related to the tasks just completed or pre-existing failures. Only address failures caused by the build.
459
+
460
+
#### 5b. Final State Sweep (Mandatory)
450
461
451
462
Before outputting the summary, perform a **full reconciliation** of `tasks.md` against the build session's execution log.
452
463
@@ -457,7 +468,7 @@ Before outputting the summary, perform a **full reconciliation** of `tasks.md` a
457
468
- If a task triggered a DCR but not marked → auto-fix to `🔄 DCR`.
458
469
3. Report any auto-fixes applied.
459
470
460
-
#### 5b. Output Completion Summary
471
+
#### 5c. Output Completion Summary
461
472
462
473
```text
463
474
📊 pb-build Summary: specs/<spec-dir>/
@@ -476,7 +487,6 @@ Files changed:
476
487
477
488
Next steps:
478
489
- Review changes: git diff
479
-
- Run full test suite: [project test command]
480
490
- If tasks were skipped, fix and re-run: /pb-build <feature-name>
481
491
```
482
492
@@ -540,7 +550,7 @@ Use `- [ ]` and `- [x]` inside the task block as evidence checkboxes, not as a s
540
550
-**ALWAYS** mark completed tasks in `tasks.md` only after Evaluator PASS.
541
551
-**ALWAYS** capture a pre-task workspace snapshot before spawning a subagent.
542
552
-**ALWAYS** perform adversarial evaluation before marking any `BDD+TDD` task as done.
543
-
-**ALWAYS** run the full test suite after each task.
553
+
-**ALWAYS** run incremental tests (affected files only) after each task — see "Incremental Test Strategy" below.
544
554
-**ALWAYS** run runtime verification for runtime-facing tasks.
545
555
-**ALWAYS** report failures with retry/skip/abort options.
546
556
-**ALWAYS** apply the ponytail ladder — (1) Does this need to exist? (2) Stdlib? (3) Native? (4) Existing dep? (5) One line? (6) Minimum code. Never simplify away: validation, error handling, security.
@@ -551,6 +561,50 @@ Use `- [ ]` and `- [x]` inside the task block as evidence checkboxes, not as a s
551
561
552
562
---
553
563
564
+
## Incremental Test Strategy
565
+
566
+
After each task, run ONLY tests affected by the task's changes. Full test suite is reserved for final verification after ALL tasks complete.
567
+
568
+
### How to Determine Affected Tests
569
+
570
+
1.**Identify changed source files:**
571
+
```bash
572
+
git diff --name-only HEAD
573
+
# or vs pre-task snapshot
574
+
```
575
+
576
+
2.**Find corresponding test files** using project conventions:
577
+
-`src/foo/bar.py` → `tests/test_bar.py` or `tests/test_foo_bar.py`
0 commit comments