Skip to content

ci(docs): exercise the checkout's CLI module in docs-verify and attest the mode#3170

Merged
bpamiri merged 1 commit into
developfrom
peter/issue-3042-branch-cli-attestation
Jun 12, 2026
Merged

ci(docs): exercise the checkout's CLI module in docs-verify and attest the mode#3170
bpamiri merged 1 commit into
developfrom
peter/issue-3042-branch-cli-attestation

Conversation

@bpamiri

@bpamiri bpamiri commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

What

Completes #3042 (remainder after #3050): verify:docs CI now exercises the branch's CLI, every run's attestation line states which mode ran, and six of the version-only {test:cli} blocks are strengthened to command-specific assertions.

How the branch CLI is wired (.github/workflows/docs-verify.yml)

The wheels CLI is the released LuCLI runtime + a CFML module, and that module is this repo's cli/lucli/ (the brew formula ships it verbatim as wheels-module-<ver>.tar.gz, with vendor/wheels/ staged inside). The brew wrapper syncs the released module to $HOME/.wheels/modules/wheels on the first non-intercepted command, keyed on a .module-version marker diff.

New Overlay checkout CLI module step writes that directory from the checkout (cli/lucli/. + vendor/wheels/.) before anything triggers the first-run sync, and copies the formula's marker so the wrapper never overwrites the overlay mid-job. Every wheels invocation in the job — fixture wheels new, cli blocks, the tutorial dev server — dispatches to the branch's CLI module. This replaces the previous "Warm up wheels module" step, whose premise was wrong anyway: wheels --version is intercepted in the wrapper before the sync block, so it never copied anything.

cli/lucli/** is added to the workflow's pull_request paths so CLI PRs run the docs gate too.

Deliberately unconditional (the "cheapest honest variant"): docs always document develop-next, and the checkout's module on a docs-only PR == develop's module. Out of scope: the LuCLI runtime stays at the released pin (runtime changes ship via LuCLI releases, not this repo), and bare wheels --version/--help are answered by the wrapper script itself (kept in sync with Module.cfc) — both caveats documented in the workflow comment and VALIDATION.md.

Attestation states the mode (acceptance 1)

wheelsBinaryAttestation() (lib/exec.mjs) appends — mode: <WHEELS_ATTEST_MODE>, defaulting to as-installed (no module overlay declared). CI sets the job-level env to checkout cli/lucli module overlay @ <sha> (released LuCLI runtime via brew); the smoke-test step echoes it too. TDD: test/exec.test.mjs added red-first (both tests fail on the old line), green after.

Strengthened blocks (acceptance 2)

  • basics/migrations.mdxwheels migrate info pins the documented no-running-server refusal (asserts-output="No running Wheels server" + asserts-exit=1)
  • basics/seeding.mdxwheels seed, same refusal contract
  • basics/routing.mdxwheels routes, same refusal contract
  • start-here/installing.mdx, start-here/first-15-minutes.mdx, command-line-tools/installation.mdx → assert the exact documented Wheels Version: line instead of bare Wheels

The refusal blocks were verified to behave identically on the released 4.0.3 CLI and the branch module, so local contributor runs (no overlay) stay green.

Evidence

Local overlay simulation (fake $HOME, brew wrapper + worktree module, mirroring the CI step exactly):

verify-docs: wheels binary: /opt/homebrew/bin/wheels (via PATH discovery) — Wheels Version: 4.0.3 — mode: checkout cli/lucli module overlay @ c3631941f (released LuCLI runtime via brew)
21 passed, 0 failed     # migrations + seeding + routing, overlay mode
21 passed, 0 failed     # same files, released CLI / as-installed mode
5 passed, 0 failed      # installing + first-15-minutes + installation, both modes

Acceptance-criterion red-flip demo — rewording the no-server guard text in the overlaid module (simulating a branch CLI behavior change) flips the strengthened block red with a clean pointer:

[cli] .../basics/migrations.mdx:30 (body-sha256: 9feca930bb1c)
combined output missing expected text "No running Wheels server"
8 passed, 1 failed

wheels info under the overlay reports the checkout module (Wheels CLI v@build.version@, Wheels: v0.0.0-dev), confirming branch dispatch.

pnpm test:docs-harness: 58/59 — the 1 failure (tutorial driver walks mini-tutorial end to end) reproduces identically on pristine origin/develop in this environment (verified via stash → baseline → pop; it's the known local blog-tutorial fixture issue the workflow comment already tracks, continue-on-error in CI). pnpm build (the hard gate): 433 pages, green. Workflow YAML parse-validated.

Fixes #3042

🤖 Generated with Claude Code

…t the mode

The wheels CLI is the released LuCLI runtime plus this repo's cli/lucli/
module. docs-verify now overlays the checkout's cli/lucli/ (and
vendor/wheels/) onto $HOME/.wheels/modules/wheels before any command
runs, so every {test:cli}/{test:tutorial} block dispatches to the branch
under review instead of the released brew module. cli/lucli/** is added
to the workflow's path filter so CLI PRs run the docs gate.

The attestation line gains a mode segment fed by WHEELS_ATTEST_MODE
(default: 'as-installed (no module overlay declared)'), so every run
states whose CLI code it exercised; CI sets it to the overlay + sha.

Also strengthens six of the wheels --version-only {test:cli} blocks to
command-specific assertions: migrate info / seed / routes now pin the
documented no-running-server refusal (exit 1 + message), and the three
install-verification blocks pin the documented 'Wheels Version:' line.

Closes #3042

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri bpamiri enabled auto-merge (squash) June 12, 2026 20:26
@github-actions github-actions Bot added the docs label Jun 12, 2026
@bpamiri bpamiri merged commit 4dbbe22 into develop Jun 12, 2026
14 checks passed
@bpamiri bpamiri deleted the peter/issue-3042-branch-cli-attestation branch June 12, 2026 20:29

@wheels-bot wheels-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wheels Bot — Reviewer

TL;DR: This PR completes #3042 by overlaying the checkout's cli/lucli/ module onto the brew-installed CLI in docs-verify.yml, adding a mode segment to the attestation line in lib/exec.mjs (with red-first unit tests), and strengthening six version-only {test:cli} blocks into command-specific assertions. The implementation is sound: the overlay runs before any wheels invocation, the .module-version marker copy correctly defuses the wrapper's first-run sync, the ${{ github.sha }} env interpolation is injection-safe, the new asserts-output/asserts-exit attributes are supported by the harness (web/sites/guides/scripts/verify-docs/lib/cli-assert.mjs:14-23), and the asserted refusal string exists in the CLI (cli/lucli/Module.cfc:7135,7153-7154). No correctness, cross-engine, or security findings; two minor non-blocking observations below. Verdict: comment.

Tests

Non-blocking: the new refusal assertions inherit a (low-probability) port-collision race in CI. The three strengthened blocks (web/sites/guides/src/content/docs/v4-0-0/basics/migrations.mdx:30, basics/seeding.mdx:90, basics/routing.mdx:164) assert "No running Wheels server" + asserts-exit=1. Read-side server detection falls through to probing common ports — cli/lucli/Module.cfc ("4. Try common ports (read-side only)" → 8080/60000/3000/8500) — and the CLI's own regression suite documents the precondition explicitly: cli/lucli/tests/test-server-required-exit-codes.sh:27-28 ("No Wheels dev server running on 8080/60000/3000/8500"). Meanwhile the tutorial driver binds an OS-assigned ephemeral port (web/sites/guides/scripts/verify-docs/drivers/tutorial.mjs findFreePort()), and Linux's default ephemeral range (32768–60999) includes 60000. Under the verify step's 4-way concurrency, a tutorial session that lands on 60000 while a cli block runs would make wheels migrate info / wheels routes attach instead of refuse, failing the block. Probability is roughly 1-in-28k per run and the "Verify v4 docs" step is continue-on-error: true, so this can't break CI today — but it's worth a follow-up: have findFreePort() re-roll if it lands on one of the four common probe ports, or note the race in VALIDATION.md. The per-block retry in drivers/cli.mjs won't save it (TRANSIENT_PATTERNS doesn't match this failure shape).

Otherwise test coverage is good: web/sites/guides/scripts/verify-docs/test/exec.test.mjs covers both the env-var-set and unset paths, was written red-first per the PR body, restores process.env correctly in finally, and passes even where the wheels binary is absent (the attestation degrades to --version failed (exit -1...) without throwing, and the assertions only match the mode/path/source segments).

Conventions

Non-blocking: trigger paths include cli/lucli/** but not vendor/wheels/**, though the overlay ships both. The new trigger comment in .github/workflows/docs-verify.yml justifies cli/lucli/** with "the harness exercises the checkout's CLI module" — but the overlay step also copies vendor/wheels/. into the module, so a framework change that breaks wheels new scaffolding (every cli block runs in a fresh fixture: drivers/cli.mjscreateFixture()) is equally exercised yet won't trigger the gate. Likely a deliberate cost trade-off (running the docs gate on every framework PR is expensive, and the verify step is soft-fail anyway), but worth a one-line comment in the paths: block stating that vendor/wheels/** is intentionally excluded, so the asymmetry reads as a decision rather than an oversight.

One observation for the record (already disclosed, no action needed): the three wheels --version blocks (start-here/installing.mdx, start-here/first-15-minutes.mdx, command-line-tools/installation.mdx) attest the brew wrapper, not the overlaid module — the workflow comment and VALIDATION.md both state this caveat, and the CI attestation mode string will technically over-claim for those three blocks. Acceptable as documented.

Docs

VALIDATION.md is updated to describe the new mode segment and the CI overlay, and the prose around each strengthened block was rewritten to describe the refusal behavior the block now pins (e.g. migrations.mdx: "every wheels migrate subcommand refuses with a pointer at wheels start") — the docs and their executable assertions now say the same thing, which is exactly what this harness is for. No changelog fragment is needed: this is a ci-type infrastructure change, not a user-facing fix/feat.

Commits

Single commit ci(docs): exercise the checkout's CLI module in docs-verify and attest the mode — valid type, valid optional scope, subject under 100 chars, describes the why. Clean.

Security

WHEELS_ATTEST_MODE interpolates ${{ github.sha }} at the job-env level — hex-only and therefore injection-safe, as the inline comment itself notes. No PR-controlled strings (title/body/branch) are interpolated into any run: script. The harness continues to spawn without a shell (lib/exec.mjs runExec, shell: false). No findings.


asserts-exit=1 pins a tighter contract than the CLI suite's "non-zero" (cli/lucli/tests/specs/commands/MigrationExitCodeSpec.cfc asserts non-zero only) — I considered flagging this, but it's correct by design: the PR verified exit 1 empirically on both the released 4.0.3 CLI and the branch module, and a future exit-code change flipping these blocks red is precisely the detection behavior #3042 asked for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

verify:docs: cli fixtures verify the installed brew CLI, not develop HEAD

1 participant