Add E2E smoke tests#236
Conversation
Adds Playwright E2E activation smoke tests using the shared @gravitykit/e2e-bootstrap package. Refactors CircleCI config from a single build job into prepare + e2e + build stages.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughAdds Playwright E2E test infrastructure and CI orchestration: new CircleCI two-phase jobs (prepare -> run E2e -> build), npm/Playwright configs and scripts, wp-env setup files, Changes
Sequence Diagram(s)sequenceDiagram
participant CircleCI as CircleCI Workflow
participant Prep as prepare_test_and_build_environment
participant Workspace as Persisted Workspace (/home/circleci)
participant E2E as run_e2e_tests (xlarge)
participant WPEnv as wp-env setup (npm/@gravitykit/e2e-bootstrap)
participant Playwright as Playwright Test Runner
participant Artifacts as Artifacts & JUnit Store
participant Build as build_package_release
CircleCI ->> Prep: start prepare_test_and_build_environment
Prep ->> Prep: checkout, clone tooling, restore/save caches\ninstall deps, download GF zips, generate `.env`
Prep ->> Workspace: persist workspace to /home/circleci
CircleCI ->> E2E: trigger run_e2e_tests (attach workspace)
E2E ->> Workspace: attach persisted workspace
E2E ->> WPEnv: run `npm run tests:e2e:setup` (retry up to 3, clear .wp-env between attempts)
WPEnv -->> E2E: wp-env started
E2E ->> Playwright: discover specs and execute tests
Playwright -->> Artifacts: upload traces/screenshots and JUnit results
CircleCI ->> Build: run build_package_release (depends on run_e2e_tests)
Build ->> Workspace: attach persisted workspace and continue packaging
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
.circleci/config.yml (2)
71-75: Retrying setup should also tear downwp-envstate.Deleting
/home/circleci/.wp-envonly clears the local metadata directory. Ifnpm run tests:e2e:setupalready started Docker containers or volumes, the next iteration is still reusing stale state, so the retry loop may not actually recover the failure.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.circleci/config.yml around lines 71 - 75, The retry loop that runs npm run tests:e2e:setup should also perform a full wp-env teardown before deleting /home/circleci/.wp-env so stale containers/volumes aren’t reused; update the while loop (the block around npm run tests:e2e:setup) to run the project’s teardown routine (for example invoking the wp-env stop/destroy or npm run tests:e2e:teardown task) on failure, then remove the metadata dir and sleep/retry, ensuring each attempt starts from a clean wp-env state.
53-56: Scope the workspace to only what downstream jobs actually need.Persisting
/home/circleci/.carries hidden home-directory files, temp downloads, and the.npmrc/.envcreated earlier into every downstream job. That makes workspace transfers heavier and keeps token/license material on disk longer than necessary.💡 Keep the workspace narrow
- persist_to_workspace: root: /home/circleci paths: - - . + - plugin + - toolingThen recreate
.npmrcand.envin the job that consumes them so they do not get propagated intobuild_package_release.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.circleci/config.yml around lines 53 - 56, The workspace persistence currently uses "root: /home/circleci" with "paths: - .", which captures the entire home including hidden files; change the persist_to_workspace stanza to only persist the specific repo/build artifact directory(ies) needed by downstream jobs instead of the home root, and remove persisting of dotfiles like ".npmrc" and ".env"; instead, recreate those files in the job that consumes them (do not persist them into build_package_release). Update the "root" and "paths" entries to target the specific project output folder(s) and/or artifact files referenced by downstream jobs rather than the blanket ".".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.circleci/config.yml:
- Around line 40-41: The unzip step can fail if the glob matches multiple
archives; change the pipeline so the download command and unzip use a single
explicit filename variable instead of relying on an unquoted glob: capture the
downloaded archive name (e.g., from the output of
"/home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R
gravityforms/gravityforms --clobber --pattern \"*.zip\" --dir .tmp'"), assert
that exactly one file matches in .tmp (fail the job if more/none), assign that
single path to a variable (e.g., GF_ZIP) and then call "unzip $GF_ZIP -d .tmp"
so shell glob expansion cannot pass multiple args to unzip.
---
Nitpick comments:
In @.circleci/config.yml:
- Around line 71-75: The retry loop that runs npm run tests:e2e:setup should
also perform a full wp-env teardown before deleting /home/circleci/.wp-env so
stale containers/volumes aren’t reused; update the while loop (the block around
npm run tests:e2e:setup) to run the project’s teardown routine (for example
invoking the wp-env stop/destroy or npm run tests:e2e:teardown task) on failure,
then remove the metadata dir and sleep/retry, ensuring each attempt starts from
a clean wp-env state.
- Around line 53-56: The workspace persistence currently uses "root:
/home/circleci" with "paths: - .", which captures the entire home including
hidden files; change the persist_to_workspace stanza to only persist the
specific repo/build artifact directory(ies) needed by downstream jobs instead of
the home root, and remove persisting of dotfiles like ".npmrc" and ".env";
instead, recreate those files in the job that consumes them (do not persist them
into build_package_release). Update the "root" and "paths" entries to target the
specific project output folder(s) and/or artifact files referenced by downstream
jobs rather than the blanket ".".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 79bb1a45-1502-4339-b169-335e9b5acffd
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (9)
.circleci/config.yml.env.sample.gitignorepackage.jsontests/E2E/setup/playwright.config.jstests/E2E/setup/playwright.global.setup.jstests/E2E/setup/playwright.global.teardown.jstests/E2E/setup/wp-env.config.jstests/E2E/tests/activation.spec.js
| /home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R gravityforms/gravityforms --clobber --pattern "*.zip" --dir .tmp' | ||
| unzip .tmp/gravityforms*.zip -d .tmp |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's find and examine the CircleCI config file
find . -name "config.yml" -path "*/.circleci/*" | head -5Repository: GravityKit/GravityExport-Lite
Length of output: 95
🏁 Script executed:
# Read the relevant section of the CircleCI config around lines 40-41
if [ -f ".circleci/config.yml" ]; then
sed -n '35,50p' .circleci/config.yml | cat -n
fiRepository: GravityKit/GravityExport-Lite
Length of output: 1089
🌐 Web query:
GitHub CLI (gh) release download --pattern behavior and how it filters zip files
💡 Result:
The GitHub CLI gh release download command uses the --pattern (or -p) flag to filter release assets based on glob patterns. It downloads only assets whose filenames match any of the provided patterns. Key behavior: - Uses Go's filepath.Match function for matching, which supports standard shell glob patterns: * (any characters), ? (single character), [abc] (character class), [a-z] (range). - Multiple patterns can be specified (e.g., -p '.deb' -p '.rpm'); an asset matches if it fits any pattern. - Patterns are case-sensitive on case-sensitive filesystems (Linux/macOS), case-insensitive on Windows. - Examples from docs: --pattern '.deb' downloads Debian packages; --pattern 'amd64.deb' for specific binaries. - For zip files: --pattern '.zip' matches any zip archive asset (e.g., gh release download --pattern '.zip'). - If no assets match, it errors with "no assets match the file pattern". - Without a tag, targets latest release and requires --pattern or --archive. - Note: Default source code zip/tar.gz (not uploaded assets) are handled separately via --archive=zip, not --pattern. Shell quoting matters (e.g., Windows CMD may need adjustments, works better in PowerShell/Git Bash). Complex regex-like patterns (e.g., alternations with |) do not work; use simple globs.
Citations:
- 1: https://cli.github.com/manual/gh_release_download
- 2: https://fossies.org/linux/gh-cli/pkg/cmd/release/download/download.go
- 3: Pattern Matching for gh release download cli/cli#8809
🌐 Web query:
unzip command glob pattern behavior with multiple matching files
💡 Result:
The unzip command supports glob patterns (wildcards like , ?, []) both for specifying multiple ZIP archive files and for selecting multiple files within a single archive to extract. When specifying the ZIP file(s) with a glob pattern (e.g., unzip '.zip'), quote the pattern to prevent shell expansion. Without quotes, the shell expands .zip to individual filenames, causing unzip to treat the first as the archive and the rest as internal files, resulting in "filename not matched" errors for the others. With quotes (e.g., unzip '.zip'), unzip receives the literal pattern, expands it internally, and processes each matching archive in OS-determined order. For extracting specific files from an archive using a glob (e.g., unzip archive.zip '.txt'), quote the pattern so unzip—not the shell—performs the matching against archive members. Unzip's wildcards match sequences of characters (), single characters (?), or bracketed sets/ranges ([abc], [a-z]). By default, wildcards match '/' directory separators, so patterns can span subdirectories unless modified by options like -W. Multiple matching files are all extracted. Examples from man pages: - unzip source.zip "*.[fch]" Makefile -d /tmp (extracts all *.f, *.c, *.h, and Makefile) - unzip foo *.[ch] -x / (extracts C source files from main directory only) Sources confirm this behavior consistently across Unix-like systems (Ubuntu, Oracle Solaris, Linux man pages).
Citations:
- 1: https://manpages.ubuntu.com/manpages/jammy/man1/unzip.1.html
- 2: https://man.cx/unzip
- 3: https://linux.die.net/man/1/unzip
- 4: http://www.info-zip.org/pub/infozip/UnZip.html
- 5: https://www.manpagez.com/man/1/unzip/
- 6: https://www.unix.com/man_page/php/1/unzip
- 7: https://manpages.opensuse.org/Tumbleweed/unzip-doc/unzip.1.en.html
Select exactly one Gravity Forms archive before extraction.
Line 40 downloads all .zip files from the gravityforms release, but line 41 uses an unquoted glob pattern gravityforms*.zip. If the shell expands this to multiple files, unzip receives them as separate arguments—treating the first as the archive and interpreting the rest as file patterns to extract from it, causing extraction to fail or produce unexpected results.
Narrow the download pattern to gravityforms*.zip, assert a single match, and use an explicit variable to avoid shell glob expansion:
Suggested fix
- /home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R gravityforms/gravityforms --clobber --pattern "*.zip" --dir .tmp'
- unzip .tmp/gravityforms*.zip -d .tmp
+ /home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R gravityforms/gravityforms --clobber --pattern "gravityforms*.zip" --dir .tmp'
+ zip_count=$(find .tmp -maxdepth 1 -type f -name 'gravityforms*.zip' | wc -l)
+ [ "$zip_count" -eq 1 ] || { echo "Expected exactly one Gravity Forms zip, found $zip_count"; exit 1; }
+ zip_file=$(find .tmp -maxdepth 1 -type f -name 'gravityforms*.zip')
+ unzip "$zip_file" -d .tmp📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R gravityforms/gravityforms --clobber --pattern "*.zip" --dir .tmp' | |
| unzip .tmp/gravityforms*.zip -d .tmp | |
| /home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R gravityforms/gravityforms --clobber --pattern "gravityforms*.zip" --dir .tmp' | |
| zip_count=$(find .tmp -maxdepth 1 -type f -name 'gravityforms*.zip' | wc -l) | |
| [ "$zip_count" -eq 1 ] || { echo "Expected exactly one Gravity Forms zip, found $zip_count"; exit 1; } | |
| zip_file=$(find .tmp -maxdepth 1 -type f -name 'gravityforms*.zip') | |
| unzip "$zip_file" -d .tmp |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.circleci/config.yml around lines 40 - 41, The unzip step can fail if the
glob matches multiple archives; change the pipeline so the download command and
unzip use a single explicit filename variable instead of relying on an unquoted
glob: capture the downloaded archive name (e.g., from the output of
"/home/circleci/tooling/build-tools/build_tools.sh gh -o 'release download -R
gravityforms/gravityforms --clobber --pattern \"*.zip\" --dir .tmp'"), assert
that exactly one file matches in .tmp (fail the job if more/none), assign that
single path to a variable (e.g., GF_ZIP) and then call "unzip $GF_ZIP -d .tmp"
so shell glob expansion cannot pass multiple args to unzip.
…ict mode violation WordPress adds a second <tr data-plugin="..."> for the update notification row. The broad [data-plugin*="gfexcel.php"] selector matched both rows, causing a Playwright strict mode violation. Targeting .deactivate a within the plugin row resolves to a single element regardless of pending updates.
2 participants
Summary
@gravitykit/e2e-bootstrappackage.jsonfrom scratch (plugin had no npm setup)prepare → e2e → buildpipeline.env.sampleand E2E/environment entries to.gitignoreWhy
Rolling out activation smoke tests across all GravityKit plugins to catch issues before releases — e.g. vendor namespacing bugs that crash plugins on activation.
Test plan
npm run tests:e2e:setuplocallynpm run tests:e2eand verify all 3 activation tests passSummary by CodeRabbit
💾 Build file (7f30e42).