Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ All notable changes to `@testsprite/testsprite-cli` are documented here. The for

### Added

- **Declarative backend Suitefiles.** New `suite validate`, `suite graph`, `suite plan`, and `suite apply` commands validate versioned manifests and Python sources, compile dependency declarations into deterministic execution waves, detect remote metadata/code drift, and reconcile creates/updates with explicit confirmation. Adjacent atomic lockfiles preserve stable remote identities and interrupted-create idempotency; name collisions and changed pending creates fail as conflicts, while unmanaged remote tests are reported and never deleted.
- **`test cancel <run-id...>`** — user-initiated cancel of in-flight runs (the real stop button; Ctrl-C only detaches). A single id renders the run card with status `cancelled` (plus an advisory when it was already cancelled); multiple ids print a `{cancelled, alreadyCancelled, conflicts, notFound}` summary. Exit codes: 4 when any id is not found, else 6 on conflicts, else 0. `--dry-run` supported.
- **Graceful Ctrl-C during `--wait`** — SIGINT/SIGTERM now detaches cleanly instead of killing the process mid-poll: the in-flight request aborts immediately, stdout gets the same partial `{runId, status: "running"}` envelope as the request-timeout path, and stderr states the truth — the server-side run keeps executing (and billing) — with a re-attach hint and a `test cancel` pointer. Exit 130/143/129 per the documented signal contract; a second signal forces a hard exit. Interrupting never cancels the server-side run — that's what `test cancel` is for.
- **`project delete <project-id> --confirm`** — permanently delete a project and everything under it (its frontend/backend sub-projects, all their tests, and backend fixtures), mirroring the Portal's cascade delete. Requires `--confirm` (the CLI never prompts); `--dry-run` previews the response shape without a network call. Standard exit codes: 0 success, 3 auth, 4 not-found (or already-deleted), 5 validation (e.g. missing `--confirm`).
Expand Down
64 changes: 63 additions & 1 deletion DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The full reference for the TestSprite CLI: install verification, manual setup, e
- [Manual setup](#manual-setup)
- [The complete agent loop](#the-complete-agent-loop)
- [Agent onboarding (`agent install`)](#agent-onboarding-agent-install)
- [Declarative backend suites (`suite`)](#declarative-backend-suites-suite)
- [Command reference](#command-reference)
- [Read commands](#read-commands)
- [Write commands](#write-commands)
Expand Down Expand Up @@ -45,7 +46,7 @@ testsprite --version
testsprite project list --dry-run --output json
```

`--dry-run` is a global flag that skips the network, credentials, and the local filesystem and emits a canned sample matching the API contract. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the data is fake.
`--dry-run` is a global flag that skips the network, credentials, and filesystem writes and emits a canned sample matching the API contract. File-backed plan inputs and Suitefile manifests/code are still read and validated locally. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the remote data is fake.

## Manual setup

Expand Down Expand Up @@ -131,6 +132,67 @@ The `codex` target uses **managed-section mode** — it writes only a sentinel-d

Re-running with `--force` on **own-file targets** (claude, cursor, cline, antigravity, kiro, windsurf, copilot) backs up the existing file to `<path>.bak` first.

## Declarative backend suites (`suite`)

A Suitefile keeps a backend test suite in source control: test identity, metadata, Python source paths, and data dependencies. The CLI validates the complete local definition, compiles `produces`/`consumes` into deterministic execution waves, compares it with the remote project, then performs only the reviewed creates and updates.

```json
{
"schemaVersion": 1,
"projectId": "proj_xxxxxxxx",
"tests": [
{
"key": "auth/session",
"name": "Create API session",
"codeFile": "tests/auth_session.py",
"priority": "p0",
"produces": ["session_token"],
"category": "setup"
},
{
"key": "orders/create",
"name": "Create order",
"codeFile": "tests/create_order.py",
"consumes": ["session_token"]
},
{
"key": "orders/cleanup",
"name": "Clean up order data",
"codeFile": "tests/cleanup.py",
"category": "teardown"
}
]
}
```

Paths are relative to the Suitefile and must resolve to contained `.py` files. Unknown fields, duplicate keys/test ids, missing or ambiguous producers, self-dependencies, cycles, lexical or symlink path escapes, and oversized code files fail locally before credentials or network are touched. `teardown` and `cleanup` tests are forced behind non-teardown tests.

```bash
# Pure-local checks
testsprite suite validate ./testsprite.suite.json
testsprite suite graph ./testsprite.suite.json --output json

# Live comparison; makes no changes
testsprite suite plan ./testsprite.suite.json

# Fully validate locally without credentials/network
testsprite suite plan ./testsprite.suite.json --dry-run --output json

# Apply exactly the create/update plan
testsprite suite apply ./testsprite.suite.json --confirm
```

The adjacent `testsprite.suite.lock.json` records stable suite-key → remote-test identity, code version, and desired-state hash. Check it into source control with the manifest. You can override its location with `--lock-file <path>`.

Safety rules are deliberately strict:

- Existing tests are adopted only through an explicit manifest `testId` or the lockfile. A same-name remote test is a conflict, not an implicit match.
- Unmanaged remote tests are reported by `plan` and are **never deleted** by `apply`.
- Backend tests only are supported in schema version 1; an adopted frontend test is a conflict.
- Creates and updates use deterministic idempotency keys. The lockfile is written atomically, and interrupted creates retain their request identity for safe replay; changing the definition while a create is pending produces a conflict.
- `name`, Python code, `produces`, and `consumes` are authoritative. Omitted `produces`/`consumes` mean empty lists. `priority` and `category` are changed only when declared.
- `apply` requires `--confirm` whenever the plan contains a mutation. It refuses any plan containing conflicts.

## Command reference

Every command supports the [global flags](#global-flags), and every example below pairs a real call with a `--dry-run` companion that works on a fresh install with no auth.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ Prefer to configure each step by hand (or learn the surface offline with `--dry-
| | `test failure summary` | One-screen triage card (no media download) |
| | `test diff` | Compare two runs — verdict, failure kind, per-step status flips, code-version drift |
| **Write** | `test scaffold` / `test lint` | Author plans locally: emit a schema-correct starter, validate plan files offline — no network, no credentials |
| | `suite validate` / `suite graph` / `suite plan` | Compile a declarative backend Suitefile into dependency waves and preview drift against the remote project |
| | `suite apply` | Reconcile backend test metadata and Python code with guarded, idempotent writes; unmanaged remote tests are never deleted |
| | `test create` / `test create-batch` | Create a test (or bulk-create from a plan file); `--produces` / `--needs` / `--category` wire BE dependency metadata |
| | `test update` / `test delete` / `test delete-batch` | Edit metadata and BE dependency declarations (`--produces` / `--needs` / `--category`) / permanently delete (no restore window; `--confirm` required) |
| | `test code put` | Replace generated code (etag-guarded) |
Expand Down
Loading
Loading