|
1 | 1 | --- |
2 | 2 | name: blacksmith-testbox |
3 | 3 | description: > |
4 | | - Validate code changes against real CI. Use for all tests, builds, |
5 | | - migrations, and any command that depends on secrets or services. |
6 | | - Use when testing, validating, checking, verifying, or before any |
7 | | - commit or push. |
| 4 | + Validate code changes against real CI when local execution is not |
| 5 | + enough. Use for CI-parity checks, secrets/services, migrations, or |
| 6 | + builds/tests that cannot run reliably on the local machine. Do not |
| 7 | + replace repo-documented local test/build loops just because this |
| 8 | + skill exists. |
8 | 9 | --- |
9 | 10 |
|
10 | 11 | # Blacksmith Testbox |
11 | 12 |
|
| 13 | +## Scope |
| 14 | + |
| 15 | +Use Testbox when you need remote CI parity, injected secrets, hosted services, |
| 16 | +or an OS/runtime image that your local machine cannot provide cheaply. |
| 17 | + |
| 18 | +Do not default to Testbox for every local test/build loop. If the repo has |
| 19 | +documented local commands for normal iteration, use those first so you keep |
| 20 | +warm caches, local build state, and fast feedback. |
| 21 | + |
| 22 | +Testbox is the expensive path. Reach for it deliberately. |
| 23 | + |
12 | 24 | ## Install the CLI |
13 | 25 |
|
14 | 26 | If `blacksmith` is not installed, install it: |
@@ -55,10 +67,41 @@ Optional flags: |
55 | 67 |
|
56 | 68 | - `--dashboard-url <url>` — Override dashboard URL (e.g. for staging) |
57 | 69 |
|
| 70 | +## Decide first: local or Testbox |
| 71 | + |
| 72 | +Before warming anything up, check the repo's own instructions. |
| 73 | + |
| 74 | +Prefer local commands when: |
| 75 | + |
| 76 | +- the repo documents a supported local test/build workflow |
| 77 | +- you are iterating on unit tests, lint, typecheck, formatting, or other |
| 78 | + local-only validation |
| 79 | +- the value comes from warm local caches and fast repeat runs |
| 80 | +- the command does not need remote secrets, hosted services, or CI-only images |
| 81 | + |
| 82 | +Prefer Testbox when: |
| 83 | + |
| 84 | +- the repo explicitly requires CI-parity or remote validation |
| 85 | +- the command needs secrets, service containers, or provisioned infra |
| 86 | +- you are reproducing CI-only failures |
| 87 | +- you need the exact workflow image/job environment from GitHub Actions |
| 88 | + |
| 89 | +For OpenClaw specifically, normal local iteration should stay local: |
| 90 | + |
| 91 | +- `pnpm check:changed` |
| 92 | +- `pnpm test:changed` |
| 93 | +- `pnpm test <path-or-filter>` |
| 94 | +- `pnpm test:serial` |
| 95 | +- `pnpm build` |
| 96 | + |
| 97 | +Only use Testbox in OpenClaw when the user explicitly wants CI-parity or the |
| 98 | +check truly depends on remote secrets/services that the local repo loop cannot |
| 99 | +provide. |
| 100 | + |
58 | 101 | ## Setup: Warmup before coding |
59 | 102 |
|
60 | | -Before writing any code, warm up a testbox. This returns an ID instantly |
61 | | -and boots the CI environment in the background while you work: |
| 103 | +If you decided Testbox is actually warranted, warm one up early. This returns |
| 104 | +an ID instantly and boots the CI environment in the background while you work: |
62 | 105 |
|
63 | 106 | blacksmith testbox warmup ci-check-testbox.yml |
64 | 107 | # → tbx_01jkz5b3t9... |
@@ -177,39 +220,62 @@ to the remote testbox before executing your command: |
177 | 220 | testbox. The sync model keeps the remote in lockstep with your local managed |
178 | 221 | file set. |
179 | 222 |
|
180 | | -## CRITICAL: Never run tests locally |
| 223 | +## CRITICAL: Do not ban local tests |
| 224 | + |
| 225 | +Do not assume local validation is forbidden. Many repos intentionally invest in |
| 226 | +fast, warm local loops, and forcing every run through Testbox destroys that |
| 227 | +advantage. |
181 | 228 |
|
182 | | -Your local machine does not have the secrets, services, or environment |
183 | | -parity needed to validate changes reliably. Every test run, migration, |
184 | | -build, or integration check MUST go through Testbox. |
| 229 | +Use Testbox for the checks that actually need it: remote parity, secrets, |
| 230 | +services, CI-only runners, or reproducibility against the workflow image. |
| 231 | + |
| 232 | +If the repo says local tests/builds are the normal path, follow the repo. |
185 | 233 |
|
186 | 234 | ## When to use |
187 | 235 |
|
188 | | -ALWAYS use Testbox when: |
| 236 | +Use Testbox when: |
189 | 237 |
|
190 | | -- Running any test (unit, integration, e2e) |
191 | | -- Running database migrations |
192 | | -- Building the project to check for compile errors |
193 | | -- Running any command that depends on secrets or environment variables |
194 | | -- Validating changes before committing |
| 238 | +- running database migrations or destructive environment checks |
| 239 | +- running commands that depend on secrets or environment variables not present locally |
| 240 | +- reproducing CI-only failures or validating against the workflow image |
| 241 | +- validating behavior that needs provisioned services or remote runners |
| 242 | +- doing a final parity check before commit/push when the repo or user wants that |
195 | 243 |
|
196 | | -The ONLY exception is trivial checks with zero external dependencies |
197 | | -(e.g., running a linter or formatter locally). |
| 244 | +Trim that list based on repo guidance. If the repo documents supported local |
| 245 | +tests/builds, prefer local for routine iteration and keep Testbox for the |
| 246 | +checks that need parity or remote state. |
198 | 247 |
|
199 | 248 | ## Workflow |
200 | 249 |
|
201 | | -1. Warm up immediately when you receive a coding task: |
| 250 | +1. Decide whether the repo's local loop is the right default. |
| 251 | +2. Only if Testbox is warranted, warm up early: |
202 | 252 | `blacksmith testbox warmup ci-check-testbox.yml` → save the ID |
203 | | -2. Write code while the testbox boots in the background. |
204 | | -3. Run tests (the CLI auto-waits if the testbox isn't ready yet): |
| 253 | +3. Write code while the testbox boots in the background. |
| 254 | +4. Run the remote command when needed: |
205 | 255 | `blacksmith testbox run --id <ID> "npm test"` |
206 | | -4. If tests fail, fix code and re-run (fast — same warm testbox, only |
207 | | - changed files are synced). |
208 | | -5. If you changed dependency manifests (package.json, etc.), prepend |
| 256 | +5. If tests fail, fix code and re-run against the same warm box. |
| 257 | +6. If you changed dependency manifests (package.json, etc.), prepend |
209 | 258 | the install command: `blacksmith testbox run --id <ID> "npm install && npm test"` |
210 | | -6. If you need artifacts (coverage reports, build outputs, etc.), download them: |
| 259 | +7. If you need artifacts (coverage reports, build outputs, etc.), download them: |
211 | 260 | `blacksmith testbox download --id <ID> coverage/ ./coverage/` |
212 | | -7. Once green, commit and push. |
| 261 | +8. Once green, commit and push. |
| 262 | + |
| 263 | +## OpenClaw full test suite |
| 264 | + |
| 265 | +For OpenClaw, use the repo package manager and the measured stable full-suite |
| 266 | +profile below. It keeps six Vitest project shards active while limiting each |
| 267 | +shard to one worker to avoid worker OOMs on Testbox: |
| 268 | + |
| 269 | + blacksmith testbox run --id <ID> "env NODE_OPTIONS=--max-old-space-size=4096 OPENCLAW_TEST_PROJECTS_PARALLEL=6 OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test" |
| 270 | + |
| 271 | +Observed full-suite time on Blacksmith Testbox is about 3-4 minutes: |
| 272 | + |
| 273 | +- 173-180s on a warmed box |
| 274 | +- 219s on a fresh 32-vCPU box |
| 275 | + |
| 276 | +When validating before commit/push, run `pnpm check:changed` first when |
| 277 | +appropriate, then the full suite with the profile above if broad confidence is |
| 278 | +needed. |
213 | 279 |
|
214 | 280 | ## Examples |
215 | 281 |
|
|
0 commit comments