Skip to content

Commit 66793ab

Browse files
fyZhang66claude
andcommitted
docs(skill): backend test authoring rules — call the test fn + sandbox limits
Two authoring gaps surfaced while debugging CLI backend tests that silently passed: 1. The runner executes --code-file top-to-bottom (not pytest), so a defined-but- uncalled test_* function never runs its assertions. Tell agents to call their test function(s) at the end of the file. 2. The execution sandbox only has stdlib + requests + pytest + numpy + scipy. Tests that import the project's own source modules or other packages (torch, pandas, local app modules) fail to run. Tell agents to test the API over HTTP with requests and not import app internals / unavailable packages. Applies to testsprite-verify.skill.md (full note) + testsprite-verify.codex.md (bullets). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 18f6e6e commit 66793ab

2 files changed

Lines changed: 20 additions & 1 deletion

File tree

skills/testsprite-verify.codex.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ testsprite test run --all --project <id> [--filter <substr>] \
6464
already have the change deployed (e.g. a CI preview deploy) — the CLI tests a
6565
deployed URL, it doesn't host your environment. Running earlier verifies the
6666
previous build.
67+
- Backend `--code-file`: the runner executes the file top-to-bottom (not `pytest`), so **call your `test_*` function(s) at the end of the file** — a defined-but-uncalled test silently passes.
68+
- Backend sandbox has only stdlib + `requests` + `pytest` + `numpy` + `scipy`. Test the API over HTTP with `requests`; do **not** `import` the project's own source modules or other packages (e.g. `torch`) — they aren't installed and the test won't run.
6769
- `--wait` long-polls until terminal. Do not wrap it in a retry loop.
6870
- Exit `0` = passed; `1` = failed/blocked; `7` = timeout (resume with `test wait <run-id>`).
6971
- BE dependency flags (`--produces`/`--needs`/`--category`) are backend-only and

skills/testsprite-verify.skill.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,10 @@ language; you don't write browser code.
135135
**Backend — write the Python yourself and use `--code-file`.** There is no
136136
server-side codegen on the CLI. Read the API surface that changed (OpenAPI, the
137137
route handler, request/response shapes) and write a pytest-style assertion script
138-
to a tempfile:
138+
to a tempfile. **End the file by calling your `test_*` function(s)** — the runner
139+
executes the file top-to-bottom and does NOT auto-discover/collect test functions
140+
the way `pytest` does, so a test that is only *defined* (never called) silently
141+
passes regardless of its assertions:
139142

140143
```python
141144
# /tmp/login-empty-password.py — runs against the project's target URL, creds injected.
@@ -145,8 +148,22 @@ def test_login_rejects_empty_password():
145148
r = requests.post(f"{TARGET_URL}/login", json={"email": "a@b.c", "password": ""})
146149
assert r.status_code == 400
147150
assert r.json().get("error") == "invalid password"
151+
152+
# Required: actually invoke the test so its assertions run.
153+
test_login_rejects_empty_password()
148154
```
149155

156+
**Execution environment (backend).** The code runs in a locked-down sandbox with
157+
only the Python **standard library + `requests` + `pytest` + `numpy` + `scipy`**
158+
(plus `requests`' own deps like `urllib3`). So:
159+
- **Test the API over HTTP** with `requests` against the target URL — that's what a
160+
backend test verifies.
161+
- **Do NOT `import` the project's own source modules** (e.g. `from app.services import …`,
162+
`import core`, `import model`) or other third-party/ML packages (e.g. `torch`,
163+
`pandas`, `django`). They are not installed, so the test fails to even run.
164+
- Get values from the API's responses (and captured variables), not by importing and
165+
calling the app's internals.
166+
150167
**Backend tests that share state declare dependencies at create time.** For a
151168
one-off verification, prefer a single self-contained script (log in inside the
152169
same file). But when the coverage set splits naturally into producer → consumer

0 commit comments

Comments
 (0)