Skip to content

Commit 51d7c47

Browse files
committed
feat(suite): add declarative backend suite sync
1 parent 2708a40 commit 51d7c47

8 files changed

Lines changed: 1852 additions & 5 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ All notable changes to `@testsprite/testsprite-cli` are documented here. The for
88

99
### Added
1010

11+
- **Declarative backend Suitefiles.** New `suite validate`, `suite graph`, `suite plan`, and `suite apply` commands validate versioned manifests and Python sources, compile dependency declarations into deterministic execution waves, detect remote metadata/code drift, and reconcile creates/updates with explicit confirmation. Adjacent atomic lockfiles preserve stable remote identities and interrupted-create idempotency; name collisions and changed pending creates fail as conflicts, while unmanaged remote tests are reported and never deleted.
1112
- **`test cancel <run-id...>`** — user-initiated cancel of in-flight runs (the real stop button; Ctrl-C only detaches). A single id renders the run card with status `cancelled` (plus an advisory when it was already cancelled); multiple ids print a `{cancelled, alreadyCancelled, conflicts, notFound}` summary. Exit codes: 4 when any id is not found, else 6 on conflicts, else 0. `--dry-run` supported.
1213
- **Graceful Ctrl-C during `--wait`** — SIGINT/SIGTERM now detaches cleanly instead of killing the process mid-poll: the in-flight request aborts immediately, stdout gets the same partial `{runId, status: "running"}` envelope as the request-timeout path, and stderr states the truth — the server-side run keeps executing (and billing) — with a re-attach hint and a `test cancel` pointer. Exit 130/143/129 per the documented signal contract; a second signal forces a hard exit. Interrupting never cancels the server-side run — that's what `test cancel` is for.
1314
- **`project delete <project-id> --confirm`** — permanently delete a project and everything under it (its frontend/backend sub-projects, all their tests, and backend fixtures), mirroring the Portal's cascade delete. Requires `--confirm` (the CLI never prompts); `--dry-run` previews the response shape without a network call. Standard exit codes: 0 success, 3 auth, 4 not-found (or already-deleted), 5 validation (e.g. missing `--confirm`).

DOCUMENTATION.md

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ The full reference for the TestSprite CLI: install verification, manual setup, e
1111
- [Manual setup](#manual-setup)
1212
- [The complete agent loop](#the-complete-agent-loop)
1313
- [Agent onboarding (`agent install`)](#agent-onboarding-agent-install)
14+
- [Declarative backend suites (`suite`)](#declarative-backend-suites-suite)
1415
- [Command reference](#command-reference)
1516
- [Read commands](#read-commands)
1617
- [Write commands](#write-commands)
@@ -45,7 +46,7 @@ testsprite --version
4546
testsprite project list --dry-run --output json
4647
```
4748

48-
`--dry-run` is a global flag that skips the network, credentials, and the local filesystem and emits a canned sample matching the API contract. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the data is fake.
49+
`--dry-run` is a global flag that skips the network, credentials, and filesystem writes and emits a canned sample matching the API contract. File-backed plan inputs and Suitefile manifests/code are still read and validated locally. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the remote data is fake.
4950

5051
## Manual setup
5152

@@ -131,6 +132,67 @@ The `codex` target uses **managed-section mode** — it writes only a sentinel-d
131132

132133
Re-running with `--force` on **own-file targets** (claude, cursor, cline, antigravity, kiro, windsurf, copilot) backs up the existing file to `<path>.bak` first.
133134

135+
## Declarative backend suites (`suite`)
136+
137+
A Suitefile keeps a backend test suite in source control: test identity, metadata, Python source paths, and data dependencies. The CLI validates the complete local definition, compiles `produces`/`consumes` into deterministic execution waves, compares it with the remote project, then performs only the reviewed creates and updates.
138+
139+
```json
140+
{
141+
"schemaVersion": 1,
142+
"projectId": "proj_xxxxxxxx",
143+
"tests": [
144+
{
145+
"key": "auth/session",
146+
"name": "Create API session",
147+
"codeFile": "tests/auth_session.py",
148+
"priority": "p0",
149+
"produces": ["session_token"],
150+
"category": "setup"
151+
},
152+
{
153+
"key": "orders/create",
154+
"name": "Create order",
155+
"codeFile": "tests/create_order.py",
156+
"consumes": ["session_token"]
157+
},
158+
{
159+
"key": "orders/cleanup",
160+
"name": "Clean up order data",
161+
"codeFile": "tests/cleanup.py",
162+
"category": "teardown"
163+
}
164+
]
165+
}
166+
```
167+
168+
Paths are relative to the Suitefile and must resolve to contained `.py` files. Unknown fields, duplicate keys/test ids, missing or ambiguous producers, self-dependencies, cycles, lexical or symlink path escapes, and oversized code files fail locally before credentials or network are touched. `teardown` and `cleanup` tests are forced behind non-teardown tests.
169+
170+
```bash
171+
# Pure-local checks
172+
testsprite suite validate ./testsprite.suite.json
173+
testsprite suite graph ./testsprite.suite.json --output json
174+
175+
# Live comparison; makes no changes
176+
testsprite suite plan ./testsprite.suite.json
177+
178+
# Fully validate locally without credentials/network
179+
testsprite suite plan ./testsprite.suite.json --dry-run --output json
180+
181+
# Apply exactly the create/update plan
182+
testsprite suite apply ./testsprite.suite.json --confirm
183+
```
184+
185+
The adjacent `testsprite.suite.lock.json` records stable suite-key → remote-test identity, code version, and desired-state hash. Check it into source control with the manifest. You can override its location with `--lock-file <path>`.
186+
187+
Safety rules are deliberately strict:
188+
189+
- Existing tests are adopted only through an explicit manifest `testId` or the lockfile. A same-name remote test is a conflict, not an implicit match.
190+
- Unmanaged remote tests are reported by `plan` and are **never deleted** by `apply`.
191+
- Backend tests only are supported in schema version 1; an adopted frontend test is a conflict.
192+
- Creates and updates use deterministic idempotency keys. The lockfile is written atomically, and interrupted creates retain their request identity for safe replay; changing the definition while a create is pending produces a conflict.
193+
- `name`, Python code, `produces`, and `consumes` are authoritative. Omitted `produces`/`consumes` mean empty lists. `priority` and `category` are changed only when declared.
194+
- `apply` requires `--confirm` whenever the plan contains a mutation. It refuses any plan containing conflicts.
195+
134196
## Command reference
135197

136198
Every command supports the [global flags](#global-flags), and every example below pairs a real call with a `--dry-run` companion that works on a fresh install with no auth.

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,8 @@ Prefer to configure each step by hand (or learn the surface offline with `--dry-
107107
| | `test failure summary` | One-screen triage card (no media download) |
108108
| | `test diff` | Compare two runs — verdict, failure kind, per-step status flips, code-version drift |
109109
| **Write** | `test scaffold` / `test lint` | Author plans locally: emit a schema-correct starter, validate plan files offline — no network, no credentials |
110+
| | `suite validate` / `suite graph` / `suite plan` | Compile a declarative backend Suitefile into dependency waves and preview drift against the remote project |
111+
| | `suite apply` | Reconcile backend test metadata and Python code with guarded, idempotent writes; unmanaged remote tests are never deleted |
110112
| | `test create` / `test create-batch` | Create a test (or bulk-create from a plan file); `--produces` / `--needs` / `--category` wire BE dependency metadata |
111113
| | `test update` / `test delete` / `test delete-batch` | Edit metadata and BE dependency declarations (`--produces` / `--needs` / `--category`) / permanently delete (no restore window; `--confirm` required) |
112114
| | `test code put` | Replace generated code (etag-guarded) |

0 commit comments

Comments
 (0)