|
| 1 | +# Changelog |
| 2 | + |
| 3 | +All notable changes to `@testsprite/testsprite-cli` are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). |
| 4 | + |
| 5 | +## [Unreleased] |
| 6 | + |
| 7 | +## [0.1.0] - 2026-06-10 |
| 8 | + |
| 9 | +### Added |
| 10 | + |
| 11 | +- **`testsprite init`** — one-shot onboarding command that chains `auth configure` → `auth whoami` → `agent install` in a single interactive invocation. Accepts `--from-env`, `--yes`, and `--agent <target>` for non-interactive and CI use. |
| 12 | + |
| 13 | +- **`agent install` / `agent list`** — write a ready-made TestSprite verification-loop skill file into your project so your coding agent knows the commands, the exit codes, and the failure-bundle layout. Pure-local command: no network, no credentials. Supported targets: `claude` (GA), `codex`, `cursor`, `cline`, `antigravity` (experimental). The `codex` target uses managed-section mode that writes a sentinel-delimited block inside `AGENTS.md` without clobbering surrounding content. `--force` backs up existing own-file targets before overwriting. |
| 14 | + |
| 15 | +- **`auth configure` / `auth whoami` / `auth logout`** — API-key management. `--from-env` reads `TESTSPRITE_API_KEY` for non-interactive setup. Credentials stored at `~/.testsprite/credentials` (INI, mode `0600`). |
| 16 | + |
| 17 | +- **`project list` / `project get`** — cursor-paginated project listing and single-project lookup. |
| 18 | + |
| 19 | +- **`test list` / `test get`** — cursor-paginated test listing under a project (with `--status`, `--type`, `--created-from` filters) and single-test lookup. |
| 20 | + |
| 21 | +- **`test create`** — create a frontend or backend test. Backend tests supply a code file directly (`--code-file`); frontend tests use `--code-file` or generate from a plan-steps document (`--plan-from`). The `--run --wait` flags chain create → trigger → poll in one invocation. Dependency metadata flags for backend tests: `--produces <var>` (repeatable), `--needs <var>` (repeatable), `--category <str>`. |
| 22 | + |
| 23 | +- **`test create-batch`** — bulk-create frontend tests from a JSONL plan file (`--plans`) or a directory of plan files (`--plan-from-dir`). Optional `--run --max-concurrency <N>` fans out triggers. |
| 24 | + |
| 25 | +- **`test update <test-id>` / `test delete <test-id>` / `test delete-batch`** — metadata update (name, description) and permanent hard-delete of one or many tests. `--confirm` is required for destructive operations. `test delete-batch` supports `--all --project <id>` and `--status <list>` for bulk targeted deletes. |
| 26 | + |
| 27 | +- **`test code get <test-id>` / `test code put <test-id>`** — read the generated test source and replace it with etag-guarded optimistic concurrency (`--expected-version`, or `--force` to skip the guard). |
| 28 | + |
| 29 | +- **`test plan put <test-id>`** — replace a frontend test's plan-steps with a refined plan. Optional `--expected-step-count` drift guard. |
| 30 | + |
| 31 | +- **`project create` / `project update`** — manage projects from the CLI. Both commands pre-flight `--target-url` against local addresses for fast feedback. |
| 32 | + |
| 33 | +- **`test steps <test-id>`** — list a test's run steps with screenshot and DOM-snapshot pointers. `--run-id <id>` filters to the steps of one specific run. Without the flag, returns the cumulative step log across all runs with an advisory when steps span multiple runs. |
| 34 | + |
| 35 | +- **`test result <test-id>`** — latest result: status, started/finished timestamps, video URL, step summary counts (`passed / failed / skipped`), and correlation fields (`snapshotId`, `runId`, `codeVersion`). `--include-analysis` adds an inline root-cause hypothesis, recommended fix target, and failure kind. |
| 36 | + |
| 37 | +- **`test result <test-id> --history`** — list a test's prior runs (newest-first). Filters: `--source cli|portal|mcp|schedule|github_action`, `--since 24h|7d|ISO`, `--page-size`, `--cursor`. Each row carries `runId`, `status`, `source`, `isRerun`, timestamps, `codeVersion`, and `failureKind`. A note is shown in place of a blank table for tests created before run-history tracking began. |
| 38 | + |
| 39 | +- **`test failure get <test-id>`** — the agent entry point. Returns one self-consistent failure bundle: the failing step and its immediate neighbors with screenshots and DOM snapshots, the test source, the video pointer, a root-cause hypothesis, a recommended fix target, and correlation metadata. Every artifact in the bundle shares a single `snapshotId`; the CLI refuses to stitch data from different runs or code versions. `--out <dir>` writes the bundle atomically to disk. `--failed-only` keeps only the failing step and its neighbors. |
| 40 | + |
| 41 | +- **`test failure summary <test-id>`** — one-screen triage card (status, failure kind, root-cause hypothesis, recommended fix target) without downloading media. |
| 42 | + |
| 43 | +- **`test run <test-id>`** — trigger a fresh run. Without `--wait`, prints `{ runId, status: "queued", … }` and exits 0. With `--wait`, polls until terminal; exit 0 on `passed`, exit 1 on `failed | blocked | cancelled`, exit 7 on timeout with a `nextAction` pointing at `test wait <runId>`. Accepts `--target-url`, `--timeout`, `--idempotency-key`. |
| 44 | + |
| 45 | +- **`test run --all --project <id>`** — wave-ordered fresh batch run for all (or filtered) backend tests in a project. Routes to a batch endpoint; response enumerates `accepted[]`, `conflicts[]`, `deferred[]`, `skippedFrontend[]`, and `skippedIntegration[]` so a machine consumer reading `accepted` alone can't silently undercount. `--wait` polls all dispatched run IDs concurrently. |
| 46 | + |
| 47 | +- **`test rerun [test-id…]`** — cheap replay of one or more tests. Frontend reruns replay the saved script verbatim (AI heal-on-drift is **on by default**, opt out with `--no-auto-heal`). Backend reruns expand the producer/teardown dependency closure; use `--skip-dependencies` for just the named test. `--all --project <id>` reruns every test in a project. Returns `accepted[]` plus `deferred[]` for any tests shed by the per-key run-rate limit; under `--wait`, a non-empty `deferred[]` exits 7 with a retry hint. |
| 48 | + |
| 49 | +- **`test wait <run-id>`** — block until a run reaches a terminal status. Resumes polling after a timed-out `test run --wait`, or when an agent already has a `runId`. Uses server-driven long-poll where supported; exponential backoff with `Retry-After` otherwise. |
| 50 | + |
| 51 | +- **`test artifact get <run-id>`** — download the failure bundle for a specific run, addressed by `runId` instead of `testId`. Enforces `meta.runId === <run-id>` as an integrity check; exits 5 on mismatch. Default output directory: `./.testsprite/runs/<run-id>/`. |
| 52 | + |
| 53 | +- **`--dry-run` (global)** — every command runs end-to-end without touching the network, credentials, or the local filesystem; emits canned data matching the API contract. |
| 54 | + |
| 55 | +- **Global flags**: `--profile`, `--output json|text`, `--endpoint-url`, `--request-timeout <seconds>`, `--verbose`, `--debug`. |
| 56 | + |
| 57 | +- **Pagination flags on every list**: `--page-size`, `--starting-token`, `--max-items`. |
| 58 | + |
| 59 | +- **`--debug` HTTP tracing** to stderr (method, URL, request-id, latency, retry decisions). The API key is never included. |
| 60 | + |
| 61 | +- **Dashboard URL in outputs** — commands that know both `projectId` and `testId` include a `dashboardUrl` deep-link to the TestSprite web portal in JSON output. Text mode: create paths print a `Dashboard:` line on stderr; run-completion output (`test run --wait`, `test wait`, `test rerun --wait`, `test run --all`) ends the run card with a `dashboard` line on stdout. The portal domain is resolved from the configured API endpoint per environment. |
| 62 | + |
| 63 | +- **TTY-gated progress ticker** — single-line in-place `\x1b[2K\r` updates during polling on TTY; completely silent on non-TTY (CI) and when `--output json` is set. |
| 64 | + |
| 65 | +- **AWS-CLI-style exit-code taxonomy** — see [Exit codes](#exit-codes) in README. |
| 66 | + |
| 67 | +### Changed |
| 68 | + |
| 69 | +- `blocked` is a distinct top-level status alongside `failed` (was collapsed into `failed` in earlier previews). Triage routes: `blocked` → infra (stale seed, login failure, unreachable target), not bug. |
| 70 | + |
| 71 | +- `test failure get` / `test steps` now synthesize a terminal `assertion` step row when no individual step is in error but the test failed at the assertion or overall-outcome layer. Previously the bundle shipped `steps: []` for these tests. Synthetic rows have no screenshot or DOM snapshot. |
| 72 | + |
| 73 | +- `outcomeContributesToFailure` boolean on every step row (`null` when unclassified). The text renderer prefixes contributing rows with `*` so a 50-step list highlights which rows the failure landed on. |
| 74 | + |
| 75 | +- `failureKind` enum widened: adds `assertion_blocked`, `routing_404`, and `network_timeout` (previously these collapsed into `unknown`). The CLI accepts unrecognized values from the wire as `unknown` so new enum values are non-breaking. |
| 76 | + |
| 77 | +- `recommendedFixTarget` returns `null` (not an `unknown` wrapper) when the analysis pipeline produced no fill. Applied uniformly to `/result?includeAnalysis`, `/failure`, and `/failure summary`. |
| 78 | + |
| 79 | +- `Test.details` debug block ships a structured `processingStatus` / `testStatus` pair alongside the previous `rawStatus` string (deprecated but preserved for the transition window). |
| 80 | + |
| 81 | +- `test create --run/--wait/--timeout/--target-url` fully wired: chains `POST /tests` → `POST /tests/{testId}/runs` → `GET /runs/{runId}` in one invocation. `--target-url` pre-flighted against local addresses on the client (exit 5) before the request is sent. |
| 82 | + |
| 83 | +- Auto-minted idempotency keys and request IDs are suppressed by default; exposed under `--verbose` for retry and support use. |
| 84 | + |
| 85 | +- Per-request wall-clock timeout (`--request-timeout`, default 120 s) applied to every outgoing fetch. Under `--wait`, the per-request timeout is auto-raised to cover `--timeout` + 5 s so a large batch under load is never cut at the default. |
| 86 | + |
| 87 | +- `test run --wait` auto-resumes on 409 `run_in_flight` by polling the existing run instead of exiting 6. An advisory is printed to stderr. Other conflict reasons and body-mismatch conflicts still propagate as exit 6. |
| 88 | + |
| 89 | +- Backend test `test run --wait` / `test rerun --wait` include a fallback path that reads `GET /tests/{id}/result` when the run row is not yet finalized server-side, so the verdict is reachable without waiting for a timeout. |
| 90 | + |
| 91 | +- `test rerun` batch `--wait` summary enumerates `deferred` and `conflicts` counts alongside `total` (dispatched) so machine readers can't silently undercount. |
| 92 | + |
| 93 | +### Fixed |
| 94 | + |
| 95 | +- `parseEnvelopeBody` now recognizes NestJS raw 404 shape, surfacing the original `Cannot POST /api/cli/v1/…` message so the user sees which endpoint isn't deployed on the current backend rather than a generic "Server error." message. |
| 96 | + |
| 97 | +- `test run --wait` CONFLICT auto-resume is gated on `details.reason === 'run_in_flight'` only. When `--target-url` is supplied and the in-flight run's URL differs, the CLI fetches the existing run's URL and reports a descriptive conflict (exit 6) with `nextAction: testsprite test wait <runId>`. |
| 98 | + |
| 99 | +- `test steps` now surfaces the synthetic terminal `assertion` step row for assertion-only failures (previously this row was wired only for `test failure get` and `test failure summary`). |
| 100 | + |
| 101 | +- `test create-batch --plan-from-dir`: the `MAX_BATCH_SPECS` (50) cap is enforced on valid specs after non-plan JSON files are skipped, not on the raw directory entry count. The duplicate-name advisory lookup uses a bounded 5-second deadline so a stalled listing endpoint delays `test create` by at most 5 s. |
| 102 | + |
| 103 | +- `localValidationError` and `ApiError.getDetail<T>()` are shared library helpers; redundant inline cast patterns removed from call sites. |
| 104 | + |
| 105 | +- `engine-strict=true` in `.npmrc` so `npm install` hard-fails on Node < 20 instead of warning and proceeding. |
| 106 | + |
| 107 | +- Commander `help [command]` exits 0 (previously exited 5 on `test help` / `project help`). |
| 108 | + |
| 109 | +[Unreleased]: https://github.com/TestSprite/testsprite-cli/compare/v0.1.0...HEAD |
| 110 | +[0.1.0]: https://github.com/TestSprite/testsprite-cli/releases/tag/v0.1.0 |
0 commit comments