Skip to content

Commit b16da86

Browse files
authored
feat: tools/test-matrix.sh — local Docker mirror of compat-matrix CI (#2420)
Adds a single command-line entry point for running the cross-engine × cross-database test matrix locally via Docker, mirroring the compat-matrix.yml CI workflow. Drives the existing tools/docker/ Dockerfiles and root compose.yml — no new container images, no changes to CI, no changes to per-engine images. Just a wrapper that: - Forces COMPOSE_PROJECT_NAME=wheels so containers are named wheels-<svc>-1, matching every CI assertion (otherwise Docker Compose names them after the cwd). - Uses portable wait loops (no GNU coreutils `timeout` — works on macOS bash 3.2 out of the box). - Mirrors compat-matrix.yml's engine-restart-between-DB-runs logic so cached model metadata doesn't leak across DB suites. - Parses test JSON the same way tools/ci/run-tests.sh does and prints pass/fail/error summary plus the actual failing spec names. Usage: tools/test-matrix.sh # Lucee 7 + SQLite (happy path) tools/test-matrix.sh lucee7 mysql # Lucee 7 + MySQL tools/test-matrix.sh lucee7 sqlite,mysql # Multiple DBs against one engine tools/test-matrix.sh lucee6,lucee7 sqlite # Multiple engines against one DB tools/test-matrix.sh --all # Full matrix (every engine × every DB) tools/test-matrix.sh --rebuild lucee7 # Force docker compose build tools/test-matrix.sh --down # Tear everything down Source is bind-mounted via compose.yml (./:/wheels-test-suite), so edit-reload-test cycles don't require image rebuilds — only the Wheels application reloads between iterations. Default: containers stay running between invocations for fast iteration. Replaces the stale "Docker — Legacy" section in CLAUDE.md (which referenced a non-existent rig/ directory and a manual ad-hoc curl recipe) with the new wrapper as the primary documented path.
1 parent ef70e59 commit b16da86

2 files changed

Lines changed: 439 additions & 40 deletions

File tree

CLAUDE.md

Lines changed: 48 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -555,60 +555,68 @@ bash tools/test-local.sh dispatch # vendor/wheels/tests/specs/dispatch/
555555
bash tools/test-local.sh migrator # vendor/wheels/tests/specs/migrator/
556556
```
557557

558-
## Running Tests Locally (Docker — Legacy)
558+
## Running Tests Locally (Docker matrix)
559559

560-
Docker is still supported for cross-engine testing (Adobe CF, multiple Lucee versions, multiple databases). For day-to-day development, use the LuCLI method above.
560+
Docker is the authoritative way to reproduce CI's `compat-matrix.yml` workflow
561+
(every engine × every database) before pushing. Source is bind-mounted via
562+
[compose.yml](compose.yml) at `./:/wheels-test-suite`, so edit-reload-test
563+
cycles don't require image rebuilds — only the Wheels application reloads
564+
between iterations.
565+
566+
### `tools/test-matrix.sh` — local mirror of `compat-matrix.yml`
561567

562-
### Minimum: test both Lucee AND Adobe before pushing
563-
Lucee and Adobe CF have different runtime behaviors (struct member functions,
564-
application scope, closure scoping). Always test at least **two engines**:
565568
```bash
566-
cd /path/to/wheels/rig # must be in the repo root with compose.yml
567-
568-
# Start both engines (SQLite is built-in on all engines, no external DB needed)
569-
docker compose up -d lucee6 adobe2025
570-
571-
# Wait ~60s for startup, then run both:
572-
curl -s -o /tmp/lucee6-results.json "http://localhost:60006/wheels/core/tests?db=sqlite&format=json"
573-
curl -s -o /tmp/adobe2025-results.json "http://localhost:62025/wheels/core/tests?db=sqlite&format=json"
574-
575-
# Check results (HTTP 200=pass, 417=failures)
576-
for f in /tmp/lucee6-results.json /tmp/adobe2025-results.json; do
577-
python3 -c "
578-
import json
579-
d = json.load(open('$f'))
580-
engine = '$f'.split('/')[-1].replace('-results.json','')
581-
print(f'{engine}: {d[\"totalPass\"]} pass, {d[\"totalFail\"]} fail, {d[\"totalError\"]} error')
582-
for b in d.get('bundleStats',[]):
583-
for s in b.get('suiteStats',[]):
584-
for sp in s.get('specStats',[]):
585-
if sp.get('status') in ('Failed','Error'):
586-
print(f' {sp[\"status\"]}: {sp[\"name\"]}: {sp.get(\"failMessage\",\"\")[:120]}')
587-
"
588-
done
569+
tools/test-matrix.sh # Lucee 7 + SQLite (happy path, fastest)
570+
tools/test-matrix.sh lucee7 mysql # Lucee 7 + MySQL
571+
tools/test-matrix.sh lucee7 sqlite,mysql # Multiple DBs against one engine
572+
tools/test-matrix.sh lucee6,lucee7 sqlite # Multiple engines against one DB
573+
tools/test-matrix.sh --all # Full matrix (every engine × every DB)
574+
tools/test-matrix.sh --rebuild lucee7 # Force `docker compose build` (image cache stale)
575+
tools/test-matrix.sh --down # Tear everything down
589576
```
590577

591-
### Engine ports
578+
Mirrors CI exactly: engine + DB containers come up under
579+
`COMPOSE_PROJECT_NAME=wheels` (so containers are named `wheels-<service>-1`,
580+
matching every assertion in `compat-matrix.yml`); engine restarts between DB
581+
runs to clear cached model metadata; warmup curl before each test run; same
582+
test URL (`/wheels/core/tests?db=<db>&format=json`); same JSON parsing.
583+
584+
Default behavior: containers stay running between invocations (fast iteration
585+
for repeated runs against the same engine/DB). Edit framework code → `--reload`
586+
isn't needed if you're hitting the test endpoint, since `wheels/core/tests`
587+
re-evaluates each request. For full app reload (model metadata, package
588+
discovery): `curl "http://localhost:<port>/?reload=true&password=wheels"`.
589+
590+
### Engines and ports (mirror `compat-matrix.yml` matrix)
592591
| Engine | Port |
593592
|--------|------|
594-
| lucee5 | 60005 |
595593
| lucee6 | 60006 |
596594
| lucee7 | 60007 |
597-
| adobe2018 | 62018 |
598-
| adobe2021 | 62021 |
599595
| adobe2023 | 62023 |
600596
| adobe2025 | 62025 |
601597
| boxlang | 60001 |
602598

603-
### Test with a specific database
604-
```bash
605-
docker compose up -d lucee6 mysql
606-
curl -sf "http://localhost:60006/wheels/core/tests?db=mysql&format=json" > /tmp/results.json
607-
```
599+
`compose.yml` also defines `lucee5`, `adobe2018`, `adobe2021` services for
600+
historical reasons; they are NOT in the CI matrix and should be considered
601+
unsupported for new development.
602+
603+
### Databases (mirror `compat-matrix.yml` DATABASES env)
604+
605+
`sqlite`, `h2` (Lucee only), `mysql`, `postgres`, `sqlserver`, `cockroachdb`,
606+
`oracle`. SQLite and H2 are file-based (no container needed). The rest spawn
607+
their own service containers.
608+
609+
### Manual ad-hoc invocations (skip the wrapper)
610+
611+
If you want to script something the wrapper doesn't cover, the underlying
612+
moves are documented in [.github/workflows/compat-matrix.yml](.github/workflows/compat-matrix.yml).
613+
Always set `COMPOSE_PROJECT_NAME=wheels` first so container names match CI.
608614

609-
### Run a specific test directory
610615
```bash
611-
curl "http://localhost:60006/wheels/core/tests?db=sqlite&format=json&directory=tests.specs.controller"
616+
export COMPOSE_PROJECT_NAME=wheels
617+
docker compose up -d lucee7 mysql
618+
# wait for ready (see compat-matrix.yml lines 79-124 for canonical readiness check)
619+
curl -s "http://localhost:60007/wheels/core/tests?db=mysql&format=json&directory=tests.specs.controller" > /tmp/results.json
612620
```
613621

614622
### Known cross-engine gotchas
@@ -631,7 +639,7 @@ curl -s "http://localhost:62023/wheels/core/tests?db=mysql&format=json" | \
631639

632640
### Cleanup
633641
```bash
634-
docker compose down # Stop all containers
642+
tools/test-matrix.sh --down # Stop and remove all containers + network
635643
```
636644

637645
## Local Onboarding Harness

0 commit comments

Comments
 (0)