Skip to content

Commit 9374634

Browse files
ryancbahanclaude
andcommitted
Remove Cucumber test infrastructure
The packages/features Cucumber test suite has been replaced by the Playwright-based e2e tests in packages/e2e. This removes the entire Cucumber infrastructure: - Delete packages/features (feature files, step definitions, world setup, helpers, snapshots, and config) - Remove test:features and test:regenerate-snapshots scripts - Remove acceptance-tests CI job from PR workflow - Remove acceptance test steps from main and manual workflows - Remove features package from knip config and ESLint overrides - Update testing strategy docs to reference e2e tests Active cucumber scenarios to consider migrating to e2e in a followup: - App scaffold + build (react-router and extension-only apps) - CLI command snapshot validation - GitHub Actions pinning check - Shared node dependency version sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent dcd6203 commit 9374634

33 files changed

Lines changed: 63 additions & 1699 deletions

.github/workflows/tests-main.yml

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -59,12 +59,6 @@ jobs:
5959
env:
6060
VITEST_MIN_THREADS: "1"
6161
VITEST_MAX_THREADS: "4"
62-
- name: Acceptance tests
63-
if: ${{ matrix.node == '24.1.0' }}
64-
env:
65-
SHOPIFY_CLI_PARTNERS_TOKEN: ${{ secrets.SHOPIFY_CLI_PARTNERS_TOKEN }}
66-
SHOPIFY_FLAG_CLIENT_ID: ${{ secrets.SHOPIFY_FLAG_CLIENT_ID }}
67-
run: pnpm nx run features:test
6862
- name: Send Slack notification on failure
6963
uses: slackapi/slack-github-action@007b2c3c751a190b6f0f040e47ed024deaa72844 # v1.23.0
7064
if: ${{ failure() && !cancelled() }}

.github/workflows/tests-manual.yml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,6 @@ jobs:
6060
node-version: ${{ inputs.node-version }}
6161
- name: Unit tests
6262
run: pnpm test:unit --output-style=stream
63-
- name: Acceptance tests
64-
env:
65-
SHOPIFY_CLI_PARTNERS_TOKEN: ${{ secrets.SHOPIFY_CLI_PARTNERS_TOKEN }}
66-
run: pnpm test:features --output-style=stream
6763
- name: Setup tmate session
6864
if: ${{ failure() && inputs.debug-enabled }}
6965
uses: mxschmitt/action-tmate@c0afd6f790e3a5564914980036ebf83216678101 # v3

.github/workflows/tests-pr.yml

Lines changed: 1 addition & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -170,30 +170,6 @@ jobs:
170170
VITEST_MIN_THREADS: "1"
171171
VITEST_MAX_THREADS: "4"
172172

173-
acceptance-tests:
174-
name: 'Acceptance tests with Node ${{ matrix.node }} in ${{ matrix.os }}'
175-
runs-on: ${{ matrix.os }}
176-
timeout-minutes: 30
177-
strategy:
178-
matrix:
179-
os: [ 'ubuntu-latest', 'macos-latest', 'windows-latest' ]
180-
node: [ '24.1.0' ]
181-
steps:
182-
- uses: actions/checkout@v3
183-
with:
184-
repository: ${{ github.event.pull_request.head.repo.full_name || github.event.repository.full_name }}
185-
ref: ${{ github.event.pull_request.head.ref || github.event.merge_group.head_ref }}
186-
fetch-depth: 1
187-
- name: Setup deps
188-
uses: ./.github/actions/setup-cli-deps
189-
with:
190-
node-version: ${{ matrix.node }}
191-
- name: Acceptance tests
192-
env:
193-
SHOPIFY_CLI_PARTNERS_TOKEN: ${{ secrets.SHOPIFY_CLI_PARTNERS_TOKEN }}
194-
SHOPIFY_FLAG_CLIENT_ID: ${{ secrets.SHOPIFY_FLAG_CLIENT_ID }}
195-
run: pnpm test:features --output-style=stream
196-
197173
test-coverage:
198174
if: ${{ github.event.pull_request.user.login != 'dependabot[bot]' }}
199175
name: 'Test Coverage'
@@ -282,7 +258,7 @@ jobs:
282258
with:
283259
node-version: ${{ env.DEFAULT_NODE_VERSION }}
284260
- name: Build
285-
run: pnpm nx run-many --all --skip-nx-cache --target=build --exclude=features --output-style=stream
261+
run: pnpm nx run-many --all --skip-nx-cache --target=build --output-style=stream
286262
- name: Type-diff
287263
working-directory: workspace
288264
id: type-diff

docs/cli/testing-strategy.md

Lines changed: 2 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -73,32 +73,9 @@ test("writes", async () => {
7373
- [Vitest API](https://vitest.dev/api/)
7474
- [Examples](https://vitest.dev/guide/#examples)
7575

76-
## Acceptance tests 🥒
76+
## E2E tests
7777

78-
[Acceptance tests](https://en.wikipedia.org/wiki/Acceptance_testing) play an essential role in ensuring all the small pieces fit together to create user experiences.
79-
They are defined as user stories which are a set of steps a user would take and the expectations they'd have while navigating through those steps.
80-
81-
```feature
82-
Scenario: I create a new app with pnpm
83-
Given I have a working directory
84-
When I create an app named MyApp with pnpm as dependency manager
85-
Then I have an app named MyApp with pnpm as dependency manager
86-
```
87-
Acceptance tests live under `packages/features` and implemented using [Cucumber](https://cucumber.io/). We create a working directory for every test that isolates the test from the rest. Moreover, the CLIs are invoked, configuring them to store global states in those temporary directories. That way, we prevent the global state from leaking into other tests and making them fail.
88-
89-
Each `.feature` file under `features/` group of scenarios has something in common (e.g. all scenarios related to app development). Steps are declared in Typescript files under `steps`. All the files in that directory or sub-directories are automatically loaded. We recommend keeping parity between those files and the features to quickly locate the steps.
90-
91-
### How to implement an acceptance test
92-
93-
1. Describe the scenario in a `.feature` file. Create a new one if you can't find a feature file your scenario fits into.
94-
2. Implement the steps of your scenario.
95-
3. Run the scenario with `FEATURE=path/to/scenario.feature:line yarn test`.
96-
97-
**Note** that we don't need to test all the user scenarios. Unlike unit tests, acceptance tests are slow. Focus on the user journeys that are most common and prefer larger but fewer scenarios over smaller but more.
98-
99-
> :bulb: Try to make them as generic as possible by using regular expressions when defining steps. That way, your steps can easily be reused by other scenarios.
100-
101-
> :bulb: If your scenario relies on a global state, for example, storing a file in the user's environment, adjust the implementation to control the state's location from the acceptance tests. This is extremely important to prevent flakiness.
78+
End-to-end tests live under `packages/e2e` and are implemented using [Playwright](https://playwright.dev/). They test full user journeys by invoking the CLI and verifying outputs. Run them with `pnpm test:e2e`.
10279

10380
## Github Actions
10481
Before being able to marge a PR, it must pass all CI checks executed in Github Actions.

package.json

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,8 @@
2929
"shopify:run": "node packages/cli/bin/dev.js",
3030
"shopify": "nx build cli && node packages/cli/bin/dev.js",
3131
"test:e2e": "nx run-many --target=build --projects=cli,create-app --skip-nx-cache && pnpm --filter e2e exec playwright test",
32-
"test:features": "pnpm nx run features:test",
33-
"test:regenerate-snapshots": "nx build cli && packages/features/snapshots/regenerate.sh",
3432
"test:unit": "pnpm vitest run",
35-
"test": "pnpm vitest run && pnpm nx run features:test",
33+
"test": "pnpm vitest run",
3634
"type-check:affected": "nx affected --target=type-check",
3735
"type-check": "nx run-many --target=type-check --all --skip-nx-cache",
3836
"update-bugsnag": "bin/update-bugsnag.js"
@@ -173,7 +171,6 @@
173171
"graphql.config.ts"
174172
],
175173
"ignoreBinaries": [
176-
"packages/features/snapshots/regenerate.sh",
177174
"bin/*",
178175
"shopify",
179176
"shopify-nightly",
@@ -318,15 +315,6 @@
318315
"**/src/index.ts"
319316
],
320317
"project": "**/*.{ts,tsx}"
321-
},
322-
"packages/features": {
323-
"entry": [
324-
"**/cucumber.js",
325-
"**/world/index.ts",
326-
"**/steps/*.ts",
327-
"**/lib/*.ts"
328-
],
329-
"project": "**/*.ts"
330318
}
331319
}
332320
}

packages/eslint-plugin-cli/config.js

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -297,14 +297,6 @@ const config = [
297297
rules: testFileRules,
298298
},
299299

300-
// Features package - allow console for test utilities
301-
{
302-
files: ['**/packages/features/**/*.ts'],
303-
rules: {
304-
'no-console': 'off',
305-
},
306-
},
307-
308300
// JS bin files - different rules apply
309301
{
310302
files: ['**/bin/*.js'],

packages/features/CHANGELOG.md

Lines changed: 0 additions & 140 deletions
This file was deleted.

packages/features/cucumber.js

Lines changed: 0 additions & 22 deletions
This file was deleted.

packages/features/features/app.feature

Lines changed: 0 additions & 26 deletions
This file was deleted.

packages/features/features/commands.feature

Lines changed: 0 additions & 5 deletions
This file was deleted.

0 commit comments

Comments
 (0)