Skip to content

feat: AI-friendly run output and actionable error diagnostics#10

Open
kevinccbsg wants to merge 9 commits into
mainfrom
feat/ai-friendly-output
Open

feat: AI-friendly run output and actionable error diagnostics#10
kevinccbsg wants to merge 9 commits into
mainfrom
feat/ai-friendly-output

Conversation

@kevinccbsg

@kevinccbsg kevinccbsg commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Replaces twd-cli's verbose output with the AI-agent-friendly format used by twd-relay's run subcommand, and adds actionable diagnostics for known infrastructure failures. Motivation: AI TDD loops (run → read failures → fix → re-run) need failure details in one place and minimal noise per iteration.

Output changes

  • New relay-style --- Run complete --- block, always the last output: passed/failed/skipped counts, duration, failed tests with full suite > test paths and error messages together, plus retried tests
  • Removed: config JSON dump, per-test tree (reportResults), and status chatter — a green run is now ~5 lines instead of 100+
  • No ANSI codes in the block; exit codes unchanged (0 pass / 1 any failure)

Error diagnostics (src/diagnostics.js)

Known failures now print cause + remediation instead of a raw stack trace:

  • Dev server unreachable (ERR_CONNECTION_REFUSED / DNS / unreachable) → "Is your dev server running? …"
  • TWD sidebar never mounts → "Ensure twd-js is initialized …"
  • Puppeteer protocolTimeout (existing hint, moved into the module)
  • Chrome launch failure → "Run npx puppeteer browsers install chrome …"

Unknown errors print message first, then stack. bin/twd-cli.js can no longer exit 1 silently.

Dependency cleanup

twd-js removed from dependencies (its only import, reportResults, is gone). Lockfile regenerated on macOS and via npm run lock:linux.

Docs

Design spec and implementation plan under docs/superpowers/; CLAUDE.md updated.

Test plan

  • Full vitest suite: 225/225 passing (new unit tests for the formatter and every diagnostic branch; run-flow tests updated)
  • Manual e2e against test-example-app (real Chrome): green run, deliberately-broken test, and stopped dev server — all matched the spec'd output, verified exit codes
  • Final whole-branch code review: no Critical/Important findings

Release note for the next version bump: old Tests: N passed… / tree output is gone — log-parsers need updating; exit codes are unchanged. Suggest a minor bump (1.3.0).

🤖 Generated with Claude Code

kevinccbsg and others added 8 commits July 6, 2026 16:49
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

TWD Contract Validation

Spec Passed Failed Warnings Mode
./contracts/users-3.0.json 2 3 1 warn
./contracts/posts-3.1.json 2 2 0 warn
./contracts/products-3.0.json 13 23 2 warn
./contracts/events-3.1.json 6 13 0 warn

23 passed · 41 failed · 3 warnings · 1 skipped

Failed validations

./contracts/users-3.0.json

  • GET /users/{userId} (200) — mock getUserNoAddress — in "Contract Validation - Mismatches > should fail: missing nested address field"
    • response.address: missing required property "address"
  • GET /users/{userId} (200) — mock getUserBadAddress — in "Contract Validation - Mismatches > should fail: nested address missing required city"
    • response.address.city: missing required property "city"
    • response.address.country: missing required property "country"
  • GET /users/{userId} (200) — mock getUserBadRole — in "Contract Validation - Mismatches > should fail: oneOf role with invalid variant"
    • response.role: oneOf best match (branch 2 of 2) failed: must be one of: "viewer"

./contracts/posts-3.1.json

  • GET /posts/{postId} (200) — mock getPostNoAuthor — in "Contract Validation - Mismatches > should fail: post missing nested author object"
    • response.author: missing required property "author"
  • GET /posts/{postId} (200) — mock getPostBadMeta — in "Contract Validation - Mismatches > should fail: post oneOf metadata matches neither variant"
    • response.metadata: oneOf best match (branch 1 of 2) failed: missing required property "category", unexpected property "duration", must be one of: "article"

./contracts/products-3.0.json

  • GET /products (200) — mock getProductEmptyName — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: empty name violates minLength"
    • response[0].name: must NOT have fewer than 1 characters
  • GET /products (200) — mock getProductBadSku — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid SKU pattern"
    • response[0].sku: must match pattern "^[A-Z]{2,4}-\d{4,8}$"
  • GET /products (200) — mock getProductBadUuid — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uuid format for id"
    • response[0].id: must match format "uuid"
  • GET /products (200) — mock getProductBadDateTime — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date-time format"
    • response[0].createdAt: must match format "date-time"
  • GET /products (200) — mock getProductBadDate — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date format"
    • response[0].releaseDate: must match format "date"
  • GET /products (200) — mock getProductBadEmail — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid email format"
    • response[0].contactEmail: must match format "email"
  • GET /products (200) — mock getProductBadUri — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uri format"
    • response[0].website: must match format "uri"
  • GET /products (200) — mock getProductBadIp — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv4 format"
    • response[0].serverIp: must match format "ipv4"
  • GET /products (200) — mock getProductBadIpV6 — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv6 format"
    • response[0].serverIpV6: must match format "ipv6"
  • GET /products (200) — mock getProductZeroPrice — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: price of 0 violates exclusiveMinimum"
    • response[0].price: must be > 0
  • GET /products (200) — mock getProductNegQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: negative quantity violates minimum"
    • response[0].quantity: must be >= 0
  • GET /products (200) — mock getProductOverQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: quantity exceeds maximum"
    • response[0].quantity: must be <= 999999
  • GET /products (200) — mock getProductBadWeight — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: weight not multipleOf 0.01"
    • response[0].weight: must be multiple of 0.01
  • GET /products (200) — mock getProductBadRating — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: rating above maximum (5)"
    • response[0].rating: must be <= 5
  • GET /products (200) — mock getProductBadCurrency — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for currency"
    • response[0].currency: must be one of: "USD", "EUR", "GBP", "JPY"
  • GET /products (200) — mock getProductBadCategory — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for category"
    • response[0].category: must be one of: "electronics", "clothing", "food", "books", "toys"
  • GET /products (200) — mock getProductBadBool — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: string value for boolean inStock"
    • response[0].inStock: expected boolean, got string
  • GET /products (200) — mock getProductDupTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: duplicate tags violates uniqueItems"
    • response[0].tags: must NOT have duplicate items (items ## 1 and 0 are identical)
  • GET /products (200) — mock getProductTooManyTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: tags exceeds maxItems (10)"
    • response[0].tags: must NOT have more than 10 items
  • GET /products (200) — mock getProductBadMeta — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: non-string value in metadata additionalProperties"
    • response[0].metadata.count: expected string, got number
  • GET /settings (200) — mock getSettingsBadExtra — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: extra property on Settings (additionalProperties: false)"
    • response.extraField: unexpected property "extraField"
  • GET /settings (200) — mock getSettingsBadLang — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid language pattern in Settings"
    • response.language: must match pattern "^[a-z]{2}(-[A-Z]{2})?$"
  • GET /products (200) — mock getProductBadNullable — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
    • response[0].description: expected string,null, got number

./contracts/events-3.1.json

  • GET /events (200) — mock getEventsEmpty — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty events array violates minItems (1)"
    • response: must NOT have fewer than 1 items
  • GET /events (200) — mock getEventShortName — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: event name too short (minLength: 3)"
    • response[0].name: must NOT have fewer than 3 characters
  • GET /events (200) — mock getEventBadDate — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid date-time format for startDate"
    • response[0].startDate: must match format "date-time"
  • GET /events (200) — mock getEventFloatId — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: float value for integer id"
    • response[0].id: expected integer, got number
    • response[0].id: must match format "int64"
  • GET /events (200) — mock getEventBadBool — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: number value for boolean active"
    • response[0].active: expected boolean, got number
  • GET /events (200) — mock getEventBadStatus — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid enum value for status"
    • response[0].status: must be one of: "draft", "published", "archived"
  • GET /events (200) — mock getEventScoreMax — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: score at exclusiveMaximum boundary (100)"
    • response[0].score: must be < 100
  • GET /events (200) — mock getEventLowPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority below minimum (1)"
    • response[0].priority: must be >= 1
  • GET /events (200) — mock getEventHighPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority above maximum (5)"
    • response[0].priority: must be <= 5
  • GET /events (200) — mock getEventDupAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: duplicate attendees violates uniqueItems"
    • response[0].attendees: must NOT have duplicate items (items ## 1 and 0 are identical)
  • GET /events (200) — mock getEventNoAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty attendees array violates minItems (1)"
    • response[0].attendees: must NOT have fewer than 1 items
  • GET /events (200) — mock getEventBadAttendee — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid email format in attendees"
    • response[0].attendees[0]: must match format "email"
  • GET /events/{eventId} (200) — mock getEventBadNullable — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
    • response.description: expected string,null, got number

View full report →

@kevinccbsg @claude
chore: bump version to 1.3.0-beta.1 for pre-merge beta testing
5851a71 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

TWD Contract Validation

Spec Passed Failed Warnings Mode
./contracts/users-3.0.json 2 3 1 warn
./contracts/posts-3.1.json 2 2 0 warn
./contracts/products-3.0.json 13 23 2 warn
./contracts/events-3.1.json 6 13 0 warn

23 passed · 41 failed · 3 warnings · 1 skipped

Failed validations

./contracts/users-3.0.json

  • GET /users/{userId} (200) — mock getUserNoAddress — in "Contract Validation - Mismatches > should fail: missing nested address field"
    • response.address: missing required property "address"
  • GET /users/{userId} (200) — mock getUserBadAddress — in "Contract Validation - Mismatches > should fail: nested address missing required city"
    • response.address.city: missing required property "city"
    • response.address.country: missing required property "country"
  • GET /users/{userId} (200) — mock getUserBadRole — in "Contract Validation - Mismatches > should fail: oneOf role with invalid variant"
    • response.role: oneOf best match (branch 2 of 2) failed: must be one of: "viewer"

./contracts/posts-3.1.json

  • GET /posts/{postId} (200) — mock getPostNoAuthor — in "Contract Validation - Mismatches > should fail: post missing nested author object"
    • response.author: missing required property "author"
  • GET /posts/{postId} (200) — mock getPostBadMeta — in "Contract Validation - Mismatches > should fail: post oneOf metadata matches neither variant"
    • response.metadata: oneOf best match (branch 1 of 2) failed: missing required property "category", unexpected property "duration", must be one of: "article"

./contracts/products-3.0.json

  • GET /products (200) — mock getProductEmptyName — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: empty name violates minLength"
    • response[0].name: must NOT have fewer than 1 characters
  • GET /products (200) — mock getProductBadSku — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid SKU pattern"
    • response[0].sku: must match pattern "^[A-Z]{2,4}-\d{4,8}$"
  • GET /products (200) — mock getProductBadUuid — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uuid format for id"
    • response[0].id: must match format "uuid"
  • GET /products (200) — mock getProductBadDateTime — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date-time format"
    • response[0].createdAt: must match format "date-time"
  • GET /products (200) — mock getProductBadDate — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date format"
    • response[0].releaseDate: must match format "date"
  • GET /products (200) — mock getProductBadEmail — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid email format"
    • response[0].contactEmail: must match format "email"
  • GET /products (200) — mock getProductBadUri — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uri format"
    • response[0].website: must match format "uri"
  • GET /products (200) — mock getProductBadIp — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv4 format"
    • response[0].serverIp: must match format "ipv4"
  • GET /products (200) — mock getProductBadIpV6 — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv6 format"
    • response[0].serverIpV6: must match format "ipv6"
  • GET /products (200) — mock getProductZeroPrice — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: price of 0 violates exclusiveMinimum"
    • response[0].price: must be > 0
  • GET /products (200) — mock getProductNegQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: negative quantity violates minimum"
    • response[0].quantity: must be >= 0
  • GET /products (200) — mock getProductOverQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: quantity exceeds maximum"
    • response[0].quantity: must be <= 999999
  • GET /products (200) — mock getProductBadWeight — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: weight not multipleOf 0.01"
    • response[0].weight: must be multiple of 0.01
  • GET /products (200) — mock getProductBadRating — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: rating above maximum (5)"
    • response[0].rating: must be <= 5
  • GET /products (200) — mock getProductBadCurrency — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for currency"
    • response[0].currency: must be one of: "USD", "EUR", "GBP", "JPY"
  • GET /products (200) — mock getProductBadCategory — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for category"
    • response[0].category: must be one of: "electronics", "clothing", "food", "books", "toys"
  • GET /products (200) — mock getProductBadBool — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: string value for boolean inStock"
    • response[0].inStock: expected boolean, got string
  • GET /products (200) — mock getProductDupTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: duplicate tags violates uniqueItems"
    • response[0].tags: must NOT have duplicate items (items ## 1 and 0 are identical)
  • GET /products (200) — mock getProductTooManyTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: tags exceeds maxItems (10)"
    • response[0].tags: must NOT have more than 10 items
  • GET /products (200) — mock getProductBadMeta — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: non-string value in metadata additionalProperties"
    • response[0].metadata.count: expected string, got number
  • GET /settings (200) — mock getSettingsBadExtra — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: extra property on Settings (additionalProperties: false)"
    • response.extraField: unexpected property "extraField"
  • GET /settings (200) — mock getSettingsBadLang — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid language pattern in Settings"
    • response.language: must match pattern "^[a-z]{2}(-[A-Z]{2})?$"
  • GET /products (200) — mock getProductBadNullable — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
    • response[0].description: expected string,null, got number

./contracts/events-3.1.json

  • GET /events (200) — mock getEventsEmpty — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty events array violates minItems (1)"
    • response: must NOT have fewer than 1 items
  • GET /events (200) — mock getEventShortName — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: event name too short (minLength: 3)"
    • response[0].name: must NOT have fewer than 3 characters
  • GET /events (200) — mock getEventBadDate — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid date-time format for startDate"
    • response[0].startDate: must match format "date-time"
  • GET /events (200) — mock getEventFloatId — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: float value for integer id"
    • response[0].id: expected integer, got number
    • response[0].id: must match format "int64"
  • GET /events (200) — mock getEventBadBool — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: number value for boolean active"
    • response[0].active: expected boolean, got number
  • GET /events (200) — mock getEventBadStatus — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid enum value for status"
    • response[0].status: must be one of: "draft", "published", "archived"
  • GET /events (200) — mock getEventScoreMax — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: score at exclusiveMaximum boundary (100)"
    • response[0].score: must be < 100
  • GET /events (200) — mock getEventLowPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority below minimum (1)"
    • response[0].priority: must be >= 1
  • GET /events (200) — mock getEventHighPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority above maximum (5)"
    • response[0].priority: must be <= 5
  • GET /events (200) — mock getEventDupAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: duplicate attendees violates uniqueItems"
    • response[0].attendees: must NOT have duplicate items (items ## 1 and 0 are identical)
  • GET /events (200) — mock getEventNoAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty attendees array violates minItems (1)"
    • response[0].attendees: must NOT have fewer than 1 items
  • GET /events (200) — mock getEventBadAttendee — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid email format in attendees"
    • response[0].attendees[0]: must match format "email"
  • GET /events/{eventId} (200) — mock getEventBadNullable — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
    • response.description: expected string,null, got number

View full report →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kevinccbsg