Skip to content

Commit fcc3929

Browse files
merlinsantiago982-cmdmerlinsantiago982-cmd
committed
feat(agent): refresh Gemini CLI target
1 parent 8898ca6 commit fcc3929

15 files changed

Lines changed: 868 additions & 215 deletions

‎DOCUMENTATION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ testsprite agent install gemini # install into GEMINI.md for Gemini CLI (man
114114
testsprite agent install cursor # .cursor/rules/testsprite-verify.mdc
115115
testsprite agent install cline # .clinerules/testsprite-verify.md
116116
testsprite agent install antigravity # .agents/skills/testsprite-verify/SKILL.md
117-
testsprite agent list # list all targets with status + mode + path
117+
testsprite agent list # list all 6 targets with status + mode + path
118118
```
119119

120120
Supported targets: `claude` (GA), `codex` (experimental), `gemini` (experimental), `cursor` (experimental), `cline` (experimental), `antigravity` (experimental).

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ testsprite setup
6767
TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yes --agent claude
6868
```
6969

70-
> **Pointing a coding agent (Claude Code, Cursor, Codex, Cline, …) at TestSprite?** Have it run `testsprite setup` first — that installs the verification skill, so the agent knows how to create, run, and triage tests on its own (instead of guessing from this README). New here? Start with the **[getting-started overview](https://docs.testsprite.com/cli/getting-started/overview)**.
70+
> **Pointing a coding agent (Claude Code, Gemini CLI, Cursor, Codex, Cline, …) at TestSprite?** Have it run `testsprite setup` first — that installs the verification skill, so the agent knows how to create, run, and triage tests on its own (instead of guessing from this README). New here? Start with the **[getting-started overview](https://docs.testsprite.com/cli/getting-started/overview)**.
7171
7272
From there, the loop runs on its own — an example session, typed by the coding agent:
7373

@@ -110,7 +110,7 @@ Prefer to configure each step by hand (or learn the surface offline with `--dry-
110110
| | `test rerun` | Cheap replay of one/many tests (FE verbatim; BE with deps); `--all --project <id>` reruns all tests |
111111
| | `test wait` | Block on a `runId` until terminal |
112112
| | `test artifact get` | Download the failure bundle for a specific `runId` |
113-
| **Agent** | `agent install` / `agent list` | Add or list coding-agent targets (pure-local): `claude`, `codex`, `gemini`, `cursor`, `cline`, `antigravity` |
113+
| **Agent** | `agent install` / `agent list` | Add or list coding-agent targets (pure-local): `claude`, `codex`, `cursor`, `cline`, `antigravity` |
114114

115115
> The earlier command names — `init`, `auth configure`, `auth whoami`, `auth logout` — still work as hidden, deprecated aliases (each prints a one-line notice pointing at the new name), so existing scripts keep running. `auth configure` now runs the full `setup` (it also installs the skill).
116116

‎docs/cli-v1-agent-install/onboard-skill-template.md‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
---
2+
name: testsprite-onboard
3+
description: Stand up a complete, runnable TestSprite test suite for the current repo at first use — create a project (with a target URL and auth), derive a coherent set of tests from the codebase, batch-create them, and smoke-run a few to a green verdict so the user immediately has something worth running. Use ONLY when a repo has no TestSprite tests yet (a fresh project), right after `testsprite setup`, or when the user asks to "set up / bootstrap / seed tests". This is first-run setup, NOT change verification — once a project already has tests, use the testsprite-verify skill instead.
4+
---
5+
6+
<!--
7+
Canonical reference for the `testsprite-onboard` skill installed by
8+
`testsprite agent install` / `testsprite setup`.
9+
10+
The `description` in the frontmatter above is the SINGLE SOURCE OF TRUTH for
11+
the skill's auto-load description; a unit test asserts byte-identity between it
12+
and `SKILLS['testsprite-onboard'].description` in `src/lib/agent-targets.ts`.
13+
Edit BOTH together.
14+
15+
The body below mirrors `skills/testsprite-onboard.skill.md` (the body-only
16+
asset that ships in the package); this doc is the human-readable reference with
17+
frontmatter attached.
18+
19+
codex (managed-section / AGENTS.md) does NOT get this full body — it gets the
20+
one-line `ONBOARD_CODEX_LINE` from the registry, because AGENTS.md is always-on
21+
with a 32 KiB budget.
22+
-->
23+
24+
# TestSprite: onboard a repo with a seed test suite
25+
26+
Your job is to take a repo that has **no TestSprite tests yet** and leave it with a
27+
**coherent, runnable suite** plus a couple of **already-green** smoke tests — in one pass.
28+
A new user who can immediately run a real, passing test is an activated user; an empty
29+
project is the #1 drop-off.
30+
31+
This skill only uses shipped CLI commands. It works the same whether the user is on the V2
32+
or V3 backend — the CLI routes internally. Do **not** call backend APIs directly.
33+
34+
## When to use
35+
36+
- Right after `testsprite setup`, or any time the active project has 0–1 tests.
37+
- The user says "set up tests", "bootstrap", "seed a suite", "get me started", or similar.
38+
39+
## When NOT to use
40+
41+
- The project already has tests — that's the `testsprite-verify` skill's job, not this one.
42+
- The user only changed code and wants it checked — again, `testsprite-verify`.
43+
44+
## Prerequisites
45+
46+
`testsprite setup` has run (an API key is configured). If `testsprite project list` errors on
47+
auth, stop and tell the user to run `testsprite setup` first — don't try to configure for them.
48+
49+
## Steps
50+
51+
### 1. Understand the app (don't skip — this is where coverage quality comes from)
52+
53+
Read the repo to establish, concretely:
54+
55+
- **Frontend**: the deployed/local **URL**, the 4–8 most important user flows (auth, core
56+
CRUD, checkout, search, settings…), and whether flows need **login**.
57+
- **Backend**: the key **API endpoints** and their success/error contracts.
58+
59+
Prefer **code-derived** routes/handlers over guessing — you have the source; use it. This
60+
beats a blind crawl.
61+
62+
### 2. Create the project (FE must have a URL)
63+
64+
Frontend:
65+
66+
```bash
67+
testsprite project create --type frontend --name "<repo name>" --url <app-url> \
68+
[--username <user> --password-file <path-to-secret>]
69+
```
70+
71+
Backend:
72+
73+
```bash
74+
testsprite project create --type backend --name "<repo name>"
75+
```
76+
77+
Capture the returned `projectId`.
78+
79+
> **Critical for FE**: a frontend test with no resolvable target URL fails immediately with
80+
> `No environment URL configured` — the suite goes all-red. Always pass `--url`. If flows need
81+
> login, pass `--username/--password-file` now so authenticated pages are reachable.
82+
83+
### 3. Author the tests (quality over quantity)
84+
85+
**Frontend** — one JSON plan file per flow, in a directory (e.g. `./testsprite-plans/`).
86+
Each file is a COMPLETE plan and must include `projectId` (from step 2), `type: "frontend"`,
87+
`name`, and `planSteps` — `create-batch` reads the project from each file, not from a flag:
88+
89+
```json
90+
{
91+
"projectId": "<projectId from step 2>",
92+
"type": "frontend",
93+
"name": "Checkout — guest can complete a purchase",
94+
"planSteps": [
95+
{ "type": "action", "description": "Navigate to /products and open the first product" },
96+
{ "type": "action", "description": "Click 'Add to cart', then go to /cart" },
97+
{
98+
"type": "assertion",
99+
"description": "The cart shows exactly 1 line item with the product's name and price"
100+
},
101+
{ "type": "action", "description": "Proceed to checkout as guest and submit the test payment" },
102+
{ "type": "assertion", "description": "A confirmation page appears showing an order number" }
103+
]
104+
}
105+
```
106+
107+
**Backend** — one `.py` file per endpoint, using `requests` with concrete assertions on
108+
status code and response body.
109+
110+
**Assertion rule (this is the whole game for FE):** every `assertion` step must name a
111+
**concrete, observable** outcome — an element, text, URL, count, or status. Never write
112+
`"verify it works"`, `"check the page loads"`, or other narrative that an AI judge can
113+
rubber-stamp. Vague assertions are how false-PASS sneaks in.
114+
115+
Aim for ~8–15 tests covering the core flows. Don't pad.
116+
117+
### 4. Batch-create
118+
119+
Frontend (one call, up to 50 plans from the directory — `create-batch` is FE-only and has
120+
**no `--project` flag**; the project comes from each plan file's `projectId`):
121+
122+
```bash
123+
testsprite test create-batch --plan-from-dir ./testsprite-plans
124+
```
125+
126+
Backend (one call per file — `create-batch` is FE-only; `--name` is required):
127+
128+
```bash
129+
testsprite test create --type backend --name "<behavior being tested>" \
130+
--code-file ./tests/<endpoint>.py --project <projectId>
131+
```
132+
133+
Capture the created `testId`s from the output.
134+
135+
### 5. Smoke-run a few — NOT all (protect credits)
136+
137+
Pick the **2–3 highest-value happy-path** tests (prefer ones you're most confident are green)
138+
and run only those:
139+
140+
```bash
141+
testsprite test run <testId> --wait
142+
```
143+
144+
Do **not** run the whole suite automatically — a 20-test FE suite is ~40 credits and a free
145+
account only has 150. Running the full suite is the user's explicit choice.
146+
147+
### 6. Report
148+
149+
Tell the user, plainly:
150+
151+
- "Your project now has **N** tests covering: <list the flows>."
152+
- "I smoke-ran **M** — here's the result: <pass/fail + the dashboard link from the run output>."
153+
- "To run the rest (≈X credits — state the cost so they choose knowingly):
154+
- frontend — run each remaining test by id: `testsprite test run <testId> --wait` (there is
155+
**no `--all` for frontend**);
156+
- backend — `testsprite test run --all --project <id>` (wave-ordered, runs every BE test)."
157+
158+
## Quality checklist (self-check before reporting done)
159+
160+
- [ ] FE project has a real `--url`; login configured if the app needs it.
161+
- [ ] Every FE assertion names a concrete, observable outcome (no "verify it works").
162+
- [ ] Tests cover the core flows you found in the code, not just one page.
163+
- [ ] Smoke-ran 2–3 happy-path tests, not the whole suite.
164+
- [ ] Reported test count, smoke result + dashboard link, and the cost to run the rest.
165+
166+
## Don'ts
167+
168+
- Don't auto-run the full suite (credit wall / surprise 402).
169+
- Don't write narrative assertions an AI judge can't fail.
170+
- Don't call backend endpoints directly — only the `testsprite` CLI.
171+
- Don't create a FE project without a URL.
172+
- Don't re-seed a project that already has tests — that's not this skill's job.
173+
174+
## Hand off to verify
175+
176+
This skill's job ends once the project has a seeded suite and a first green run. From here on,
177+
the **`testsprite-verify`** skill takes over: after the user changes code, it runs the tests
178+
covering that change before they report the work done. Onboard once; verify continuously.

‎skills/testsprite-onboard.skill.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
<!--
2+
User-facing content for `testsprite agent install` (skill: testsprite-onboard).
3+
Body only — name + description frontmatter is emitted by the install wrapper.
4+
The cursor (.mdc), cline (.clinerules), and antigravity (experimental)
5+
wrappers reuse this body verbatim and swap only the frontmatter/header.
6+
-->
7+
8+
# TestSprite: onboard a repo with a seed test suite
9+
10+
Your job is to take a repo that has **no TestSprite tests yet** and leave it with a
11+
**coherent, runnable suite** plus a couple of **already-green** smoke tests — in one pass.
12+
A new user who can immediately run a real, passing test is an activated user; an empty
13+
project is the #1 drop-off.
14+
15+
This skill only uses shipped CLI commands. It works the same whether the user is on the V2
16+
or V3 backend — the CLI routes internally. Do **not** call backend APIs directly.
17+
18+
## When to use
19+
20+
- Right after `testsprite setup`, or any time the active project has 0–1 tests.
21+
- The user says "set up tests", "bootstrap", "seed a suite", "get me started", or similar.
22+
23+
## When NOT to use
24+
25+
- The project already has tests — that's the `testsprite-verify` skill's job, not this one.
26+
- The user only changed code and wants it checked — again, `testsprite-verify`.
27+
28+
## Prerequisites
29+
30+
`testsprite setup` has run (an API key is configured). If `testsprite project list` errors on
31+
auth, stop and tell the user to run `testsprite setup` first — don't try to configure for them.
32+
33+
## Steps
34+
35+
### 1. Understand the app (don't skip — this is where coverage quality comes from)
36+
37+
Read the repo to establish, concretely:
38+
39+
- **Frontend**: the deployed/local **URL**, the 4–8 most important user flows (auth, core
40+
CRUD, checkout, search, settings…), and whether flows need **login**.
41+
- **Backend**: the key **API endpoints** and their success/error contracts.
42+
43+
Prefer **code-derived** routes/handlers over guessing — you have the source; use it. This
44+
beats a blind crawl.
45+
46+
### 2. Create the project (FE must have a URL)
47+
48+
Frontend:
49+
50+
```bash
51+
testsprite project create --type frontend --name "<repo name>" --url <app-url> \
52+
[--username <user> --password-file <path-to-secret>]
53+
```
54+
55+
Backend:
56+
57+
```bash
58+
testsprite project create --type backend --name "<repo name>"
59+
```
60+
61+
Capture the returned `projectId`.
62+
63+
> **Critical for FE**: a frontend test with no resolvable target URL fails immediately with
64+
> `No environment URL configured` — the suite goes all-red. Always pass `--url`. If flows need
65+
> login, pass `--username/--password-file` now so authenticated pages are reachable.
66+
67+
### 3. Author the tests (quality over quantity)
68+
69+
**Frontend** — one JSON plan file per flow, in a directory (e.g. `./testsprite-plans/`).
70+
Each file is a COMPLETE plan and must include `projectId` (from step 2), `type: "frontend"`,
71+
`name`, and `planSteps` — `create-batch` reads the project from each file, not from a flag:
72+
73+
```json
74+
{
75+
"projectId": "<projectId from step 2>",
76+
"type": "frontend",
77+
"name": "Checkout — guest can complete a purchase",
78+
"planSteps": [
79+
{ "type": "action", "description": "Navigate to /products and open the first product" },
80+
{ "type": "action", "description": "Click 'Add to cart', then go to /cart" },
81+
{
82+
"type": "assertion",
83+
"description": "The cart shows exactly 1 line item with the product's name and price"
84+
},
85+
{ "type": "action", "description": "Proceed to checkout as guest and submit the test payment" },
86+
{ "type": "assertion", "description": "A confirmation page appears showing an order number" }
87+
]
88+
}
89+
```
90+
91+
**Backend** — one `.py` file per endpoint, using `requests` with concrete assertions on
92+
status code and response body.
93+
94+
**Assertion rule (this is the whole game for FE):** every `assertion` step must name a
95+
**concrete, observable** outcome — an element, text, URL, count, or status. Never write
96+
`"verify it works"`, `"check the page loads"`, or other narrative that an AI judge can
97+
rubber-stamp. Vague assertions are how false-PASS sneaks in.
98+
99+
Aim for ~8–15 tests covering the core flows. Don't pad.
100+
101+
### 4. Batch-create
102+
103+
Frontend (one call, up to 50 plans from the directory — `create-batch` is FE-only and has
104+
**no `--project` flag**; the project comes from each plan file's `projectId`):
105+
106+
```bash
107+
testsprite test create-batch --plan-from-dir ./testsprite-plans
108+
```
109+
110+
Backend (one call per file — `create-batch` is FE-only; `--name` is required):
111+
112+
```bash
113+
testsprite test create --type backend --name "<behavior being tested>" \
114+
--code-file ./tests/<endpoint>.py --project <projectId>
115+
```
116+
117+
Capture the created `testId`s from the output.
118+
119+
### 5. Smoke-run a few — NOT all (protect credits)
120+
121+
Pick the **2–3 highest-value happy-path** tests (prefer ones you're most confident are green)
122+
and run only those:
123+
124+
```bash
125+
testsprite test run <testId> --wait
126+
```
127+
128+
Do **not** run the whole suite automatically — a 20-test FE suite is ~40 credits and a free
129+
account only has 150. Running the full suite is the user's explicit choice.
130+
131+
### 6. Report
132+
133+
Tell the user, plainly:
134+
135+
- "Your project now has **N** tests covering: <list the flows>."
136+
- "I smoke-ran **M** — here's the result: <pass/fail + the dashboard link from the run output>."
137+
- "To run the rest (≈X credits — state the cost so they choose knowingly):
138+
- frontend — run each remaining test by id: `testsprite test run <testId> --wait` (there is
139+
**no `--all` for frontend**);
140+
- backend — `testsprite test run --all --project <id>` (wave-ordered, runs every BE test)."
141+
142+
## Quality checklist (self-check before reporting done)
143+
144+
- [ ] FE project has a real `--url`; login configured if the app needs it.
145+
- [ ] Every FE assertion names a concrete, observable outcome (no "verify it works").
146+
- [ ] Tests cover the core flows you found in the code, not just one page.
147+
- [ ] Smoke-ran 2–3 happy-path tests, not the whole suite.
148+
- [ ] Reported test count, smoke result + dashboard link, and the cost to run the rest.
149+
150+
## Don'ts
151+
152+
- Don't auto-run the full suite (credit wall / surprise 402).
153+
- Don't write narrative assertions an AI judge can't fail.
154+
- Don't call backend endpoints directly — only the `testsprite` CLI.
155+
- Don't create a FE project without a URL.
156+
- Don't re-seed a project that already has tests — that's not this skill's job.
157+
158+
## Hand off to verify
159+
160+
This skill's job ends once the project has a seeded suite and a first green run. From here on,
161+
the **`testsprite-verify`** skill takes over: after the user changes code, it runs the tests
162+
covering that change before they report the work done. Onboard once; verify continuously.

‎skills/testsprite-verify.codex.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ testsprite test run --all --project <id> [--filter <substr>] \
6464
already have the change deployed (e.g. a CI preview deploy) — the CLI tests a
6565
deployed URL, it doesn't host your environment. Running earlier verifies the
6666
previous build.
67+
- Backend `--code-file`: the runner executes the file top-to-bottom (not `pytest`), so **call your `test_*` function(s) at the end of the file** — a defined-but-uncalled test silently passes.
68+
- Backend sandbox has only stdlib + `requests` + `pytest` + `numpy` + `scipy`. Test the API over HTTP with `requests`; do **not** `import` the project's own source modules or other packages (e.g. `torch`) — they aren't installed and the test won't run.
6769
- `--wait` long-polls until terminal. Do not wrap it in a retry loop.
6870
- Exit `0` = passed; `1` = failed/blocked; `7` = timeout (resume with `test wait <run-id>`).
6971
- BE dependency flags (`--produces`/`--needs`/`--category`) are backend-only and

0 commit comments

Comments
 (0)