Skip to content

Commit 6ae437e

Browse files
committed
refactor(pb-build): replace full test suite with incremental testing strategy
pb-build previously ran the full test suite after every task, which became a bottleneck as the codebase grew. This change switches to incremental testing: each task runs only the tests affected by its changes, with a single full suite run at the end for regression verification. Changes: - Unit Test Verification step now runs only affected tests - TDD-only task RED/GREEN/REFACTOR cycles target specific test files - ALWAYS constraint updated to mandate incremental tests - Added "Incremental Test Strategy" section with: - git diff to identify changed source files - Convention-based test file mapping - Import dependency tracing - Affected test set execution commands - Evaluator Check D now explicitly references incremental strategy - Added Step 5a "Full Test Suite Run" after all tasks complete - Removed redundant "Run full test suite" from completion summary
1 parent 29730d7 commit 6ae437e

1 file changed

Lines changed: 67 additions & 13 deletions

File tree

skills/pb-build/SKILL.md

Lines changed: 67 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -207,7 +207,7 @@ The subagent operates as **Generator** — its sole objective is to make tests p
207207
208208
5. **[REFACTOR] Clean Up** — If needed, refactor for clarity/performance. Re-run the scenario to confirm still passing.
209209
210-
6. **Unit Test Verification** — Run unit tests: `[Unit test command]`. Ensure no regressions.
210+
6. **Unit Test Verification (Incremental)** — Run ONLY tests affected by this task's changes (see "Incremental Test Strategy" below). Ensure no regressions.
211211
212212
7. **Runtime Verification (when applicable)** — Run runtime checks from task Verification and capture outputs.
213213
@@ -218,10 +218,10 @@ The subagent operates as **Generator** — its sole objective is to make tests p
218218
##### For TDD-Only Tasks (Non-Scenario)
219219
220220
1. **RED** — Write a failing unit test. **STOP after this step.**
221-
2. **Confirm RED** — Run test suite. New test must fail for the right reason.
221+
2. **Confirm RED** — Run the new test file (or targeted test). New test must fail for the right reason.
222222
3. **GREEN** — Write minimum implementation. **Only proceed after confirming RED.**
223-
4. **Confirm GREEN** — Run test suite. All tests pass.
224-
5. **REFACTOR** — Clean up. Re-run tests.
223+
4. **Confirm GREEN** — Run the new test + affected tests. All must pass.
224+
5. **REFACTOR** — Clean up. Re-run affected tests.
225225
6. **Runtime Verification (when applicable)** — Run runtime checks.
226226
7. **Architecture Conformance Check** — Verify design conformance.
227227
8. **Scope Check** — Verify scope compliance.
@@ -289,11 +289,11 @@ After the Generator signals `READY_FOR_EVAL`, the orchestrator must perform an *
289289
- The exact BDD command run (same as RED)
290290
- All steps passing
291291
- The scenario name that passed
292-
- **Independent Re-run:** Evaluator MUST independently run the scenario to verify it passes:
292+
- **Independent Re-run:** Evaluator MUST independently run the scenario to verify it passes, plus related unit tests per the Incremental Test Strategy:
293293

294-
```text
295-
[BDD command] --tags@[scenario_tag]
296-
```
294+
```text
295+
[BDD command] --tags@[scenario_tag]
296+
```
297297
298298
- If independent re-run FAILS, evaluation FAILS regardless of Generator's claims.
299299
@@ -328,7 +328,7 @@ After the Generator signals `READY_FOR_EVAL`, the orchestrator must perform an *
328328
**Light Review Process (for simple `TDD-only` tasks):**
329329

330330
1. Read the `git diff` to confirm scope.
331-
2. Verify test suite passes.
331+
2. Run affected tests only (follow Incremental Test Strategy).
332332
3. Check for architecture violations in the diff.
333333
4. If clean, emit PASS immediately.
334334

@@ -446,7 +446,18 @@ If during implementation a subagent discovers that the design is **infeasible or
446446

447447
### Step 5: Final State Verification & Output Completion Summary
448448

449-
#### 5a. Final State Sweep (Mandatory)
449+
#### 5a. Full Test Suite Run (Mandatory)
450+
451+
After ALL tasks are `🟢 DONE`, run the full test suite ONCE to verify no regressions across the entire build:
452+
453+
```bash
454+
# Run the project's full test suite
455+
uv run pytest tests/ -v
456+
```
457+
458+
If the full suite fails, investigate the failing tests and determine whether they are related to the tasks just completed or pre-existing failures. Only address failures caused by the build.
459+
460+
#### 5b. Final State Sweep (Mandatory)
450461

451462
Before outputting the summary, perform a **full reconciliation** of `tasks.md` against the build session's execution log.
452463

@@ -457,7 +468,7 @@ Before outputting the summary, perform a **full reconciliation** of `tasks.md` a
457468
- If a task triggered a DCR but not marked → auto-fix to `🔄 DCR`.
458469
3. Report any auto-fixes applied.
459470

460-
#### 5b. Output Completion Summary
471+
#### 5c. Output Completion Summary
461472

462473
```text
463474
📊 pb-build Summary: specs/<spec-dir>/
@@ -476,7 +487,6 @@ Files changed:
476487
477488
Next steps:
478489
- Review changes: git diff
479-
- Run full test suite: [project test command]
480490
- If tasks were skipped, fix and re-run: /pb-build <feature-name>
481491
```
482492

@@ -540,7 +550,7 @@ Use `- [ ]` and `- [x]` inside the task block as evidence checkboxes, not as a s
540550
- **ALWAYS** mark completed tasks in `tasks.md` only after Evaluator PASS.
541551
- **ALWAYS** capture a pre-task workspace snapshot before spawning a subagent.
542552
- **ALWAYS** perform adversarial evaluation before marking any `BDD+TDD` task as done.
543-
- **ALWAYS** run the full test suite after each task.
553+
- **ALWAYS** run incremental tests (affected files only) after each task — see "Incremental Test Strategy" below.
544554
- **ALWAYS** run runtime verification for runtime-facing tasks.
545555
- **ALWAYS** report failures with retry/skip/abort options.
546556
- **ALWAYS** apply the ponytail ladder — (1) Does this need to exist? (2) Stdlib? (3) Native? (4) Existing dep? (5) One line? (6) Minimum code. Never simplify away: validation, error handling, security.
@@ -551,6 +561,50 @@ Use `- [ ]` and `- [x]` inside the task block as evidence checkboxes, not as a s
551561

552562
---
553563

564+
## Incremental Test Strategy
565+
566+
After each task, run ONLY tests affected by the task's changes. Full test suite is reserved for final verification after ALL tasks complete.
567+
568+
### How to Determine Affected Tests
569+
570+
1. **Identify changed source files:**
571+
```bash
572+
git diff --name-only HEAD
573+
# or vs pre-task snapshot
574+
```
575+
576+
2. **Find corresponding test files** using project conventions:
577+
- `src/foo/bar.py``tests/test_bar.py` or `tests/test_foo_bar.py`
578+
- `src/foo/bar.py``tests/foo/test_bar.py` (mirror layout)
579+
- Check `conftest.py` for shared fixtures used by the changed module
580+
581+
3. **Trace import dependencies** (for non-test source changes):
582+
- If `src/foo/bar.py` imports `src/foo/baz.py`, also run tests for `baz`
583+
- If `src/foo/bar.py` is imported by `src/foo/qux.py`, run tests for `qux` too
584+
- Use `grep -r "from.*bar import\|import.*bar" src/ tests/` to find dependents
585+
586+
4. **Run the affected test set:**
587+
```bash
588+
# Specific test files
589+
uv run pytest tests/test_bar.py tests/test_baz.py -v
590+
591+
# Or by keyword/module
592+
uv run pytest -v -k "bar or baz"
593+
594+
# Or by test path pattern
595+
uv run pytest tests/ -v --co -q # list collected tests first
596+
```
597+
598+
### Rules
599+
600+
- **BDD+TDD tasks:** BDD scenarios already target specific scenarios — run those + related unit tests only.
601+
- **TDD-only tasks:** Run the new test file + any test file that imports from the changed module.
602+
- **Cross-module changes:** If a task touches 3+ modules, still run only affected tests — do not escalate to full suite.
603+
- **Final verification:** After ALL tasks in `tasks.md` are `🟢 DONE`, run the full test suite once as the last step before marking the build complete.
604+
- **Evaluator re-run:** The Evaluator independently runs the same affected tests — not the full suite.
605+
606+
---
607+
554608
## Anti-Pattern Quick Reference
555609

556610
The Evaluator should watch for these common agent mistakes:

0 commit comments

Comments
 (0)