Add post-build smoke E2E against built .zip artifact#241
Conversation
Runs the @smoke-tagged activation suite against the packaged plugin zip in CI, after build_package_release and before publishing the release. - Tag activation.spec.js describe with @smoke so the bootstrap CLI picks it up via Playwright's --grep. - Bump @gravitykit/e2e-bootstrap to ^1.1.0 (smoke CLI subcommand). - Persist .release and the gf-entries-in-excel-*.zip glob from build_package_release so downstream jobs can install the built artifact. - Add run_post_build_smoke job: unzips the artifact, restores dev deps through gktools' Docker composer (PHP 7.4), boots wp-env, runs the smoke spec against the built plugin. - Extract Create GitHub release + Announce build into a publish_release job gated on smoke success; release publishing now only happens when the built artifact actually activates cleanly.
WalkthroughThis PR integrates post-build E2E smoke testing into the CircleCI pipeline: build collects and persists artifacts to the workspace, a new job runs ChangesPost-Build Smoke Testing
Sequence DiagramsequenceDiagram
participant BuildPackageRelease
participant Workspace
participant RunPostBuildSmoke
participant PublishRelease
participant GitHubRelease
BuildPackageRelease->>Workspace: persist `plugin/.release` and `gf-entries-in-excel-*.zip`
RunPostBuildSmoke->>Workspace: attach workspace and read artifacts
RunPostBuildSmoke->>RunPostBuildSmoke: execute `@gravitykit/e2e-bootstrap smoke`
RunPostBuildSmoke->>RunPostBuildSmoke: publish E2E results and JUnit artifacts
PublishRelease->>Workspace: attach workspace (post-smoke)
PublishRelease->>GitHubRelease: create GitHub release
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.circleci/config.yml (1)
137-137: 💤 Low valueRedundant working_directory declaration.
The
working_directoryis already set to/home/circleci/pluginin thedefault_job_configanchor (line 8). This explicit override is unnecessary.♻️ Simplify by removing redundant declaration
run_post_build_smoke: <<: *default_job_config - working_directory: /home/circleci/plugin machine:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.circleci/config.yml at line 137, Remove the redundant working_directory declaration by deleting the explicit "working_directory: /home/circleci/plugin" entry in the job where you overrode it; the value is already provided by the default_job_config anchor, so simply remove that line (leave the rest of the job block and its indentation intact).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.circleci/config.yml:
- Line 125: The cp command currently swallows errors so missing artifacts don't
fail the job; edit the line that runs cp gf-entries-in-excel-*.zip .release/
2>/dev/null || true in .circleci/config.yml to remove the error suppression
(delete "2>/dev/null || true") so the step fails when no matching .zip is found,
or alternately replace it with an explicit existence check (e.g., test -e
gf-entries-in-excel-*.zip && cp ... ) so the pipeline fails fast if the
gf-entries-in-excel-*.zip artifact from the build step isn't produced.
- Around line 146-152: The comment over the "Post-build smoke" job is misleading
because the npx `@gravitykit/e2e-bootstrap` smoke command does not perform
retries; either implement CI-level retry behavior similar to the existing
run_e2e_tests job or update the comment to remove the "boots wp-env with retry"
claim. To fix, locate the "Post-build smoke" job in .circleci/config.yml where
the command uses npx `@gravitykit/e2e-bootstrap` smoke and either add a shell
retry loop with proper teardown/cleanup that mirrors run_e2e_tests' retry
pattern, or change the job comment to accurately describe that no retry is
performed and retries must be handled at the CI level.
---
Nitpick comments:
In @.circleci/config.yml:
- Line 137: Remove the redundant working_directory declaration by deleting the
explicit "working_directory: /home/circleci/plugin" entry in the job where you
overrode it; the value is already provided by the default_job_config anchor, so
simply remove that line (leave the rest of the job block and its indentation
intact).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1ad9e63d-ce38-4701-b10f-8f0f4273292a
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (3)
.circleci/config.ymlpackage.jsontests/E2E/tests/activation.spec.js
| name: Post-build smoke | ||
| # Unzips .release/*.zip, restores dev deps via gktools' Docker composer, | ||
| # boots wp-env with retry, runs `playwright test --grep=@smoke` against | ||
| # the built artifact. See @gravitykit/e2e-bootstrap README for details. | ||
| command: | | ||
| npx @gravitykit/e2e-bootstrap smoke \ | ||
| --composer-cmd "npx @gravitykit/gktools composer install" |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Check if `@gravitykit/e2e-bootstrap` smoke command handles retries internally
# Check the e2e-bootstrap package README or source for retry logic
gh repo clone gravitykit/e2e-bootstrap /tmp/e2e-bootstrap-check 2>/dev/null || true
if [ -d /tmp/e2e-bootstrap-check ]; then
echo "=== Checking for retry logic in e2e-bootstrap ==="
rg -i "retry|attempt" /tmp/e2e-bootstrap-check --type md --type js -C 3
fiRepository: GravityKit/GravityExport-Lite
Length of output: 55
🏁 Script executed:
# Search for e2e-bootstrap references in the current repo to understand how it's used
rg "e2e-bootstrap" . -A 2 -B 2Repository: GravityKit/GravityExport-Lite
Length of output: 2884
🏁 Script executed:
# Check the CircleCI config for dependency versions or references
cat -n .circleci/config.yml | sed -n '140,160p'Repository: GravityKit/GravityExport-Lite
Length of output: 991
🌐 Web query:
@gravitykit/e2e-bootstrap smoke command retry wp-env setup
💡 Result:
The @gravitykit/e2e-bootstrap package is used in conjunction with wp-env to facilitate end-to-end (E2E) testing workflows, particularly in CI/CD environments like GitHub Actions [1]. There is no native "smoke command" or built-in "retry" flag within @gravitykit/e2e-bootstrap itself for wp-env setup. Instead, retry logic is typically implemented at the CI/CD workflow level (e.g., in.circleci/config.yml or GitHub Actions YAML) [1]. To implement a robust retry mechanism for wp-env setup, the following pattern is recommended: 1. Implement a loop in your CI configuration that attempts to run the setup command (e.g., npm run tests:e2e:setup) multiple times [1]. 2. Ensure that each retry attempt starts from a clean state. Simply deleting the local metadata directory (e.g., /home/circleci/.wp-env) is often insufficient because stale Docker containers or volumes may persist [1]. 3. The retry loop should explicitly invoke a teardown routine (such as wp-env stop or a custom npm run tests:e2e:teardown task) upon failure before attempting to clear the environment and retry [1]. For example, in a CI environment, you might structure your loop to: - Run the setup command. - If it fails, execute the teardown command. - Remove the.wp-env directory. - Sleep briefly. - Retry the setup [1]. This approach ensures that subsequent attempts do not inherit corrupted or stale configurations from failed previous runs [1].
Citations:
The comment claiming "boots wp-env with retry" is inaccurate; the @gravitykit/e2e-bootstrap smoke command does not implement retry logic internally.
Unlike run_e2e_tests (lines 62-75) which has an explicit retry loop, this job lacks retry handling for wp-env setup failures. The e2e-bootstrap package does not provide built-in retry functionality—retry logic must be implemented at the CI/CD level with proper teardown and cleanup on failure.
Consider adding a retry loop similar to run_e2e_tests, or update the comment to remove the misleading claim about retries.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.circleci/config.yml around lines 146 - 152, The comment over the
"Post-build smoke" job is misleading because the npx `@gravitykit/e2e-bootstrap`
smoke command does not perform retries; either implement CI-level retry behavior
similar to the existing run_e2e_tests job or update the comment to remove the
"boots wp-env with retry" claim. To fix, locate the "Post-build smoke" job in
.circleci/config.yml where the command uses npx `@gravitykit/e2e-bootstrap` smoke
and either add a shell retry loop with proper teardown/cleanup that mirrors
run_e2e_tests' retry pattern, or change the job comment to accurately describe
that no retry is performed and retries must be handled at the CI level.
- Remove '2>/dev/null || true' from 'cp gf-entries-in-excel-*.zip .release/' so the step fails loudly when the build produced no zip, instead of swallowing the cp error and surfacing the problem later. - Drop the redundant 'working_directory: /home/circleci/plugin' from run_post_build_smoke. The default_job_config anchor already sets it; the duplicate was a copy-paste artifact from the prompt template.
1 participant
Summary
@smoke-tagged activation suite against the packaged plugin zip in CI, gated betweenbuild_package_releaseand the newpublish_releasejob — so a failed smoke now blocks the GitHub release + announcement.Pipeline shape (workflow shape A)
prepare(single job) →run_e2e_tests(pre-build) /build_package_release→run_post_build_smoke→publish_releasepreparealready installs deps AND writes.env, so the linear ancestor chain inherits the environment without conflict. No workspace surgery needed.Changes
.circleci/config.ymlbuild_package_release: movedCollect build artifactsup, thenpersist_to_workspaceexports bothplugin/.releaseand theplugin/gf-entries-in-excel-*.zipglob (the latter is required becausegktools releasegreps cwd, not.release/).run_post_build_smokejob: machine + DLC, attaches workspace, invokesnpx @gravitykit/e2e-bootstrap smoke --composer-cmd "npx @gravitykit/gktools composer install"(Composer must run inside gktools' PHP 7.4 Docker), uploads Playwright results/report + JUnit.Create GitHub release+Announce buildinto a newpublish_releasejob thatrequires: run_post_build_smoke.run_post_build_smoke→build_package_release,publish_release→run_post_build_smoke.tests/E2E/tests/activation.spec.js— appended@smoketo thedescribetitle so the bootstrap CLI grep picks it up.package.json+package-lock.json— bumped@gravitykit/e2e-bootstrapfrom^1.0.0to^1.1.0(smoke subcommand). Lock diff scoped to this dep only.Test plan
test_and_releaseruns on this branch; verifyrun_post_build_smokejob appears and consumes the built artifact (logs should show unzip ofgf-entries-in-excel-*.zip, composer install via gktools Docker, wp-env boot,playwright test --grep=@smoke).publish_releaseruns; on smoke fail,publish_releaseis blocked.@smoke-tagged describe (the three activation tests), not the full E2E suite.tests/E2E/results/junit.xmlis uploaded as test results.Notes
run_e2e_testsis untouched and continues to run the full suite against the source tree.playwright.config.jsalready uses a single chromium project (nodependencies: ['chromium']split), so no Playwright #22408 grep-bypass fix needed.wp-env.config.jsalready hard-codespluginPath: '../../..'(noBUILT_PLUGIN_PATHconditional); the bootstrap CLI rewrites this internally when unzipping the built artifact.Summary by CodeRabbit
Tests
Chores
💾 Build file (154e40c).