Skip to content

1.3-sequential-design.md

Latest commit

 

History

History
86 lines (53 loc) · 6.32 KB

1.3-sequential-design.md

File metadata and controls

86 lines (53 loc) · 6.32 KB
Raw

Pattern 1.3 — Sequential / Additive Test Design

Sequential Test Ladder

Problem

Tests often run in unpredictable order, making failures hard to diagnose. If a complex order processing test fails, is the bug in checkout logic, authentication, or the fact that the server never started? Unordered tests waste time — agents debug symptoms instead of root causes.

Solution

Sequential/additive test design orders tests by dependency: each test assumes all previous tests pass. The sequence acts as a diagnostic ladder — if test N fails, the agent knows tests 1 through N-1 passed, narrowing the search space.

Standard ordering — each step is a user journey that builds on previous journeys:

  1. Startup: System comes online and reports healthy — the health endpoint runs in test mode, decorating responses with real service health (email relay connectivity, message queue status, secrets manager verification) without requiring synthetic endpoints
  2. Bootstrap: Test fixture data is loaded — seed users, pre-configured resources, reference data required for subsequent journeys
  3. Authentication: User registers, logs in, receives a valid session token
  4. Basic flows: User creates, reads, updates, and deletes resources
  5. Domain operations: User completes a domain-specific workflow (checkout, refund, export)
  6. Advanced features: User exercises edge cases and complex multi-step workflows

In Practice

Use filenames to enforce order:

tests/stack/
  01-app-startup.stack.test.ts
  02-bootstrap-test-data.stack.test.ts
  03-authentication.stack.test.ts
  04-user-crud.stack.test.ts
  05-checkout-complete.stack.test.ts
  06-refund-and-reconciliation.stack.test.ts
  07-rate-limiting.stack.test.ts

Each test is one atomic user journey. The sequence builds from infrastructure to domain logic:

Example: If 04-checkout-basic.stack.test.ts fails:

  • Agent knows: Server starts (01 pass), auth works (02 pass), CRUD works (03 pass)
  • Agent focuses: Checkout logic specifically, not auth or persistence
  • Agent skips: Advanced checkout tests (05), rate limiting (06) — they'd fail anyway

Stack Tests as Vertical Slices

Each stack test is a vertical slice through the system — analogous to a user story in agile development. It cuts across every layer from API to database to external services, validating that the entire stack works together for a specific user journey. This is not a component test that checks one service in isolation; it is a slice that confirms the system delivers value end-to-end.

This framing has practical implications for how tests are run during development:

  • Individual test runs are essential during feature development. When building a checkout flow, running 05-checkout-complete.stack.test.ts in isolation is the fastest way to iterate — the agent brings up the stack, exercises the journey, and gets feedback immediately without waiting for unrelated tests.
  • Full suite runs are for validation, not iteration. An agent executing a plan should run the full suite before claiming completion. A human orchestrator deciding whether a feature is ready may run the full suite less frequently — the individual journey test provides sufficient signal during active development.
  • A test that passes in isolation but fails in the suite reveals a dependency bug — this is valuable information when it surfaces, not a reason to forbid individual runs.

Additive Tests as Comparative Ground Truth

Even with guardrails, skills, and stack tests, agents will sometimes produce undesirable changes that slip through. A skill might have a gap, a hook might not cover a specific mutation, or the agent might introduce subtly wrong internals that pass on the surface but cause problems later. The additive test structure provides a recovery mechanism: use a known-good, passing foundational stack test as executable ground truth to diagnose a failing new test.

Example scenario:

  1. A user-order-completion stack test exists and passes. It verifies: user logs in -> navigates to item -> places in basket -> pays -> stack test asserts order state in /user/orders list API, email notifications sent, payment processor debit for correct amount.

  2. Some time later, you instruct the agent to implement a label-printing capability for order returns. It writes code and a new stack test. The test runs and fails — for odd reasons, and things don't look quite right.

  3. Instead of visual debugging, task the agent to do a comparative analysis: diff the passing user-order-completion test against the failing new test. Focus on logs, code, and tests as they relate to order creation and terminal state assertion expectations.

  4. The comparison surfaces convention misalignments: the new test used a different bootstrap user than expected, used the wrong service to create the order, or the terminal state assertions only verify label existence — not label content legibility.

  5. The agent fixes the in-progress solution to align with the conventions established by the foundational test.

This pattern — employing a foundational, fully passing stack test against a formative, broken one — gives the agent a structured way to discover what's wrong by comparing against what's right. The additive structure means earlier tests represent stable conventions; later tests inherit and extend those conventions. When inheritance breaks, the diff against the parent test reveals where and why.

Anti-Pattern

Don't make tests depend on shared state from previous tests. Each test should still be independently valid — the sequence is about diagnostic ordering, not data dependencies. Use beforeEach to set up fresh state.

Don't put unrelated tests in the middle of the sequence. If a new test doesn't fit the dependency ladder, it probably belongs in a different suite or should be a unit test.

Cross-References

Back to L1 Overview | Previous: Pattern 1.2 | Next: Pattern 1.4