You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,7 @@ All notable changes to `@testsprite/testsprite-cli` are documented here. The for
8
8
9
9
### Added
10
10
11
+
-**Declarative backend Suitefiles.** New `suite validate`, `suite graph`, `suite plan`, and `suite apply` commands validate versioned manifests and Python sources, compile dependency declarations into deterministic execution waves, detect remote metadata/code drift, and reconcile creates/updates with explicit confirmation. Adjacent atomic lockfiles preserve stable remote identities and interrupted-create idempotency; name collisions and changed pending creates fail as conflicts, while unmanaged remote tests are reported and never deleted.
11
12
-**`test cancel <run-id...>`** — user-initiated cancel of in-flight runs (the real stop button; Ctrl-C only detaches). A single id renders the run card with status `cancelled` (plus an advisory when it was already cancelled); multiple ids print a `{cancelled, alreadyCancelled, conflicts, notFound}` summary. Exit codes: 4 when any id is not found, else 6 on conflicts, else 0. `--dry-run` supported.
12
13
-**Graceful Ctrl-C during `--wait`** — SIGINT/SIGTERM now detaches cleanly instead of killing the process mid-poll: the in-flight request aborts immediately, stdout gets the same partial `{runId, status: "running"}` envelope as the request-timeout path, and stderr states the truth — the server-side run keeps executing (and billing) — with a re-attach hint and a `test cancel` pointer. Exit 130/143/129 per the documented signal contract; a second signal forces a hard exit. Interrupting never cancels the server-side run — that's what `test cancel` is for.
13
14
-**`project delete <project-id> --confirm`** — permanently delete a project and everything under it (its frontend/backend sub-projects, all their tests, and backend fixtures), mirroring the Portal's cascade delete. Requires `--confirm` (the CLI never prompts); `--dry-run` previews the response shape without a network call. Standard exit codes: 0 success, 3 auth, 4 not-found (or already-deleted), 5 validation (e.g. missing `--confirm`).
`--dry-run` is a global flag that skips the network, credentials, and the local filesystem and emits a canned sample matching the API contract. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the data is fake.
49
+
`--dry-run` is a global flag that skips the network, credentials, and filesystem writes and emits a canned sample matching the API contract. File-backed plan inputs and Suitefile manifests/code are still read and validated locally. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the remote data is fake.
49
50
50
51
## Manual setup
51
52
@@ -131,6 +132,67 @@ The `codex` target uses **managed-section mode** — it writes only a sentinel-d
131
132
132
133
Re-running with `--force` on **own-file targets** (claude, cursor, cline, antigravity, kiro, windsurf, copilot) backs up the existing file to `<path>.bak` first.
133
134
135
+
## Declarative backend suites (`suite`)
136
+
137
+
A Suitefile keeps a backend test suite in source control: test identity, metadata, Python source paths, and data dependencies. The CLI validates the complete local definition, compiles `produces`/`consumes` into deterministic execution waves, compares it with the remote project, then performs only the reviewed creates and updates.
138
+
139
+
```json
140
+
{
141
+
"schemaVersion": 1,
142
+
"projectId": "proj_xxxxxxxx",
143
+
"tests": [
144
+
{
145
+
"key": "auth/session",
146
+
"name": "Create API session",
147
+
"codeFile": "tests/auth_session.py",
148
+
"priority": "p0",
149
+
"produces": ["session_token"],
150
+
"category": "setup"
151
+
},
152
+
{
153
+
"key": "orders/create",
154
+
"name": "Create order",
155
+
"codeFile": "tests/create_order.py",
156
+
"consumes": ["session_token"]
157
+
},
158
+
{
159
+
"key": "orders/cleanup",
160
+
"name": "Clean up order data",
161
+
"codeFile": "tests/cleanup.py",
162
+
"category": "teardown"
163
+
}
164
+
]
165
+
}
166
+
```
167
+
168
+
Paths are relative to the Suitefile and must resolve to contained `.py` files. Unknown fields, duplicate keys/test ids, missing or ambiguous producers, self-dependencies, cycles, lexical or symlink path escapes, and oversized code files fail locally before credentials or network are touched. `teardown` and `cleanup` tests are forced behind non-teardown tests.
169
+
170
+
```bash
171
+
# Pure-local checks
172
+
testsprite suite validate ./testsprite.suite.json
173
+
testsprite suite graph ./testsprite.suite.json --output json
174
+
175
+
# Live comparison; makes no changes
176
+
testsprite suite plan ./testsprite.suite.json
177
+
178
+
# Fully validate locally without credentials/network
179
+
testsprite suite plan ./testsprite.suite.json --dry-run --output json
180
+
181
+
# Apply exactly the create/update plan
182
+
testsprite suite apply ./testsprite.suite.json --confirm
183
+
```
184
+
185
+
The adjacent `testsprite.suite.lock.json` records stable suite-key → remote-test identity, code version, and desired-state hash. Check it into source control with the manifest. You can override its location with `--lock-file <path>`.
186
+
187
+
Safety rules are deliberately strict:
188
+
189
+
- Existing tests are adopted only through an explicit manifest `testId` or the lockfile. A same-name remote test is a conflict, not an implicit match.
190
+
- Unmanaged remote tests are reported by `plan` and are **never deleted** by `apply`.
191
+
- Backend tests only are supported in schema version 1; an adopted frontend test is a conflict.
192
+
- Creates and updates use deterministic idempotency keys. The lockfile is written atomically, and interrupted creates retain their request identity for safe replay; changing the definition while a create is pending produces a conflict.
193
+
-`name`, Python code, `produces`, and `consumes` are authoritative. Omitted `produces`/`consumes` mean empty lists. `priority` and `category` are changed only when declared.
194
+
-`apply` requires `--confirm` whenever the plan contains a mutation. It refuses any plan containing conflicts.
195
+
134
196
## Command reference
135
197
136
198
Every command supports the [global flags](#global-flags), and every example below pairs a real call with a `--dry-run` companion that works on a fresh install with no auth.
Copy file name to clipboardExpand all lines: README.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -107,6 +107,8 @@ Prefer to configure each step by hand (or learn the surface offline with `--dry-
107
107
||`test failure summary`| One-screen triage card (no media download) |
108
108
||`test diff`| Compare two runs — verdict, failure kind, per-step status flips, code-version drift |
109
109
|**Write**|`test scaffold` / `test lint`| Author plans locally: emit a schema-correct starter, validate plan files offline — no network, no credentials |
110
+
||`suite validate` / `suite graph` / `suite plan`| Compile a declarative backend Suitefile into dependency waves and preview drift against the remote project |
111
+
||`suite apply`| Reconcile backend test metadata and Python code with guarded, idempotent writes; unmanaged remote tests are never deleted |
110
112
||`test create` / `test create-batch`| Create a test (or bulk-create from a plan file); `--produces` / `--needs` / `--category` wire BE dependency metadata |
111
113
||`test update` / `test delete` / `test delete-batch`| Edit metadata and BE dependency declarations (`--produces` / `--needs` / `--category`) / permanently delete (no restore window; `--confirm` required) |
0 commit comments