ci(docs): exercise the checkout's CLI module in docs-verify and attest the mode#3170
Conversation
…t the mode
The wheels CLI is the released LuCLI runtime plus this repo's cli/lucli/
module. docs-verify now overlays the checkout's cli/lucli/ (and
vendor/wheels/) onto $HOME/.wheels/modules/wheels before any command
runs, so every {test:cli}/{test:tutorial} block dispatches to the branch
under review instead of the released brew module. cli/lucli/** is added
to the workflow's path filter so CLI PRs run the docs gate.
The attestation line gains a mode segment fed by WHEELS_ATTEST_MODE
(default: 'as-installed (no module overlay declared)'), so every run
states whose CLI code it exercised; CI sets it to the overlay + sha.
Also strengthens six of the wheels --version-only {test:cli} blocks to
command-specific assertions: migrate info / seed / routes now pin the
documented no-running-server refusal (exit 1 + message), and the three
install-verification blocks pin the documented 'Wheels Version:' line.
Closes #3042
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
There was a problem hiding this comment.
Wheels Bot — Reviewer
TL;DR: This PR completes #3042 by overlaying the checkout's cli/lucli/ module onto the brew-installed CLI in docs-verify.yml, adding a mode segment to the attestation line in lib/exec.mjs (with red-first unit tests), and strengthening six version-only {test:cli} blocks into command-specific assertions. The implementation is sound: the overlay runs before any wheels invocation, the .module-version marker copy correctly defuses the wrapper's first-run sync, the ${{ github.sha }} env interpolation is injection-safe, the new asserts-output/asserts-exit attributes are supported by the harness (web/sites/guides/scripts/verify-docs/lib/cli-assert.mjs:14-23), and the asserted refusal string exists in the CLI (cli/lucli/Module.cfc:7135,7153-7154). No correctness, cross-engine, or security findings; two minor non-blocking observations below. Verdict: comment.
Tests
Non-blocking: the new refusal assertions inherit a (low-probability) port-collision race in CI. The three strengthened blocks (web/sites/guides/src/content/docs/v4-0-0/basics/migrations.mdx:30, basics/seeding.mdx:90, basics/routing.mdx:164) assert "No running Wheels server" + asserts-exit=1. Read-side server detection falls through to probing common ports — cli/lucli/Module.cfc ("4. Try common ports (read-side only)" → 8080/60000/3000/8500) — and the CLI's own regression suite documents the precondition explicitly: cli/lucli/tests/test-server-required-exit-codes.sh:27-28 ("No Wheels dev server running on 8080/60000/3000/8500"). Meanwhile the tutorial driver binds an OS-assigned ephemeral port (web/sites/guides/scripts/verify-docs/drivers/tutorial.mjs findFreePort()), and Linux's default ephemeral range (32768–60999) includes 60000. Under the verify step's 4-way concurrency, a tutorial session that lands on 60000 while a cli block runs would make wheels migrate info / wheels routes attach instead of refuse, failing the block. Probability is roughly 1-in-28k per run and the "Verify v4 docs" step is continue-on-error: true, so this can't break CI today — but it's worth a follow-up: have findFreePort() re-roll if it lands on one of the four common probe ports, or note the race in VALIDATION.md. The per-block retry in drivers/cli.mjs won't save it (TRANSIENT_PATTERNS doesn't match this failure shape).
Otherwise test coverage is good: web/sites/guides/scripts/verify-docs/test/exec.test.mjs covers both the env-var-set and unset paths, was written red-first per the PR body, restores process.env correctly in finally, and passes even where the wheels binary is absent (the attestation degrades to --version failed (exit -1...) without throwing, and the assertions only match the mode/path/source segments).
Conventions
Non-blocking: trigger paths include cli/lucli/** but not vendor/wheels/**, though the overlay ships both. The new trigger comment in .github/workflows/docs-verify.yml justifies cli/lucli/** with "the harness exercises the checkout's CLI module" — but the overlay step also copies vendor/wheels/. into the module, so a framework change that breaks wheels new scaffolding (every cli block runs in a fresh fixture: drivers/cli.mjs → createFixture()) is equally exercised yet won't trigger the gate. Likely a deliberate cost trade-off (running the docs gate on every framework PR is expensive, and the verify step is soft-fail anyway), but worth a one-line comment in the paths: block stating that vendor/wheels/** is intentionally excluded, so the asymmetry reads as a decision rather than an oversight.
One observation for the record (already disclosed, no action needed): the three wheels --version blocks (start-here/installing.mdx, start-here/first-15-minutes.mdx, command-line-tools/installation.mdx) attest the brew wrapper, not the overlaid module — the workflow comment and VALIDATION.md both state this caveat, and the CI attestation mode string will technically over-claim for those three blocks. Acceptable as documented.
Docs
VALIDATION.md is updated to describe the new mode segment and the CI overlay, and the prose around each strengthened block was rewritten to describe the refusal behavior the block now pins (e.g. migrations.mdx: "every wheels migrate subcommand refuses with a pointer at wheels start") — the docs and their executable assertions now say the same thing, which is exactly what this harness is for. No changelog fragment is needed: this is a ci-type infrastructure change, not a user-facing fix/feat.
Commits
Single commit ci(docs): exercise the checkout's CLI module in docs-verify and attest the mode — valid type, valid optional scope, subject under 100 chars, describes the why. Clean.
Security
WHEELS_ATTEST_MODE interpolates ${{ github.sha }} at the job-env level — hex-only and therefore injection-safe, as the inline comment itself notes. No PR-controlled strings (title/body/branch) are interpolated into any run: script. The harness continues to spawn without a shell (lib/exec.mjs runExec, shell: false). No findings.
asserts-exit=1 pins a tighter contract than the CLI suite's "non-zero" (cli/lucli/tests/specs/commands/MigrationExitCodeSpec.cfc asserts non-zero only) — I considered flagging this, but it's correct by design: the PR verified exit 1 empirically on both the released 4.0.3 CLI and the branch module, and a future exit-code change flipping these blocks red is precisely the detection behavior #3042 asked for.
What
Completes #3042 (remainder after #3050):
verify:docsCI now exercises the branch's CLI, every run's attestation line states which mode ran, and six of the version-only{test:cli}blocks are strengthened to command-specific assertions.How the branch CLI is wired (
.github/workflows/docs-verify.yml)The
wheelsCLI is the released LuCLI runtime + a CFML module, and that module is this repo'scli/lucli/(the brew formula ships it verbatim aswheels-module-<ver>.tar.gz, withvendor/wheels/staged inside). The brew wrapper syncs the released module to$HOME/.wheels/modules/wheelson the first non-intercepted command, keyed on a.module-versionmarker diff.New Overlay checkout CLI module step writes that directory from the checkout (
cli/lucli/.+vendor/wheels/.) before anything triggers the first-run sync, and copies the formula's marker so the wrapper never overwrites the overlay mid-job. Everywheelsinvocation in the job — fixturewheels new, cli blocks, the tutorial dev server — dispatches to the branch's CLI module. This replaces the previous "Warm up wheels module" step, whose premise was wrong anyway:wheels --versionis intercepted in the wrapper before the sync block, so it never copied anything.cli/lucli/**is added to the workflow'spull_requestpaths so CLI PRs run the docs gate too.Deliberately unconditional (the "cheapest honest variant"): docs always document develop-next, and the checkout's module on a docs-only PR == develop's module. Out of scope: the LuCLI runtime stays at the released pin (runtime changes ship via LuCLI releases, not this repo), and bare
wheels --version/--helpare answered by the wrapper script itself (kept in sync withModule.cfc) — both caveats documented in the workflow comment andVALIDATION.md.Attestation states the mode (acceptance 1)
wheelsBinaryAttestation()(lib/exec.mjs) appends— mode: <WHEELS_ATTEST_MODE>, defaulting toas-installed (no module overlay declared). CI sets the job-level env tocheckout cli/lucli module overlay @ <sha> (released LuCLI runtime via brew); the smoke-test step echoes it too. TDD:test/exec.test.mjsadded red-first (both tests fail on the old line), green after.Strengthened blocks (acceptance 2)
basics/migrations.mdx→wheels migrate infopins the documented no-running-server refusal (asserts-output="No running Wheels server"+asserts-exit=1)basics/seeding.mdx→wheels seed, same refusal contractbasics/routing.mdx→wheels routes, same refusal contractstart-here/installing.mdx,start-here/first-15-minutes.mdx,command-line-tools/installation.mdx→ assert the exact documentedWheels Version:line instead of bareWheelsThe refusal blocks were verified to behave identically on the released 4.0.3 CLI and the branch module, so local contributor runs (no overlay) stay green.
Evidence
Local overlay simulation (fake
$HOME, brew wrapper + worktree module, mirroring the CI step exactly):Acceptance-criterion red-flip demo — rewording the no-server guard text in the overlaid module (simulating a branch CLI behavior change) flips the strengthened block red with a clean pointer:
wheels infounder the overlay reports the checkout module (Wheels CLI v@build.version@,Wheels: v0.0.0-dev), confirming branch dispatch.pnpm test:docs-harness: 58/59 — the 1 failure (tutorial driver walks mini-tutorial end to end) reproduces identically on pristineorigin/developin this environment (verified via stash → baseline → pop; it's the known local blog-tutorial fixture issue the workflow comment already tracks,continue-on-errorin CI).pnpm build(the hard gate): 433 pages, green. Workflow YAML parse-validated.Fixes #3042
🤖 Generated with Claude Code