-
Notifications
You must be signed in to change notification settings - Fork 15
ci(e2e): add Playwright-based E2E workflow for Mage-OS #313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
digitalrisedorset
wants to merge
4
commits into
mage-os:main
Choose a base branch
from
digitalrisedorset:ci/tests-e2e-playwright-elgentos-hyva
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| name: E2E Test | ||
|
|
||
| on: | ||
| workflow_dispatch: {} | ||
| push: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - "_test/demo-package/**" | ||
| - ".github/workflows/_internal-e2e.yaml" | ||
| - ".github/workflows/e2e.yaml" | ||
| - "supported-version/**" | ||
| - "!(**/*.md)" | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - "_test/demo-package/**" | ||
| - ".github/workflows/_internal-e2e.yaml" | ||
| - ".github/workflows/e2e.yaml" | ||
| - "supported-version/**" | ||
| - "!(**/*.md)" | ||
|
|
||
| jobs: | ||
| compute_matrix: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| matrix: ${{ steps.supported-version.outputs.matrix }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: ./supported-version | ||
| with: | ||
| kind: all | ||
| id: supported-version | ||
| - run: echo ${{ steps.supported-version.outputs.matrix }} | ||
| e2e-workflow: | ||
| if: ${{ hashFiles('_test/demo-package/**') != '' }} | ||
| needs: compute_matrix | ||
| uses: ./.github/workflows/e2e.yaml | ||
| with: | ||
| package_name: mage-os/magento2-demo-package | ||
| source_folder: $GITHUB_WORKSPACE/_test/demo-package | ||
| matrix: ${{ needs.compute_matrix.outputs.matrix }} | ||
| test_command: | | ||
| npx playwright test --grep "@setup" --trace on | ||
| npx playwright test --grep-invert "@setup" --trace on | ||
| fail-fast: false | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # End-to-End (E2E) Tests for Mage-OS / Magento | ||
|
|
||
| A reusable GitHub Actions workflow that installs Magento (Mage-OS) | ||
| and runs browser-based End-to-End (E2E) tests against a real storefront | ||
| using Playwright or an equivalent E2E runner. | ||
|
|
||
| ## Inputs | ||
|
|
||
| See the [e2e.yaml](./e2e.yaml) | ||
|
|
||
| | Input | Description | Required | Default | | ||
| | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------- | | ||
| | matrix | JSON string of [version matrix for Magento](./#matrix-format) | true | NULL | | ||
| | fail-fast | Same as Github's [fail-fast](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast) | false | true | | ||
| | package_name | The name of the package | true | NULL | | ||
| | source_folder | The source folder of the package | false | $GITHUB_WORKSPACE | | ||
| | magento_directory | The folder where Magento will be installed | false | ../magento2 | | ||
| | magento_repository | Where to install Magento from | false | https://mirror.mage-os.org/ | | ||
| | test_command | The e2e test command to run | false | "../../../vendor/bin/phpunit" | | ||
| | composer_cache_key | A key to version the composer cache. Can be incremented if you need to bust the cache. | false | "" | | ||
|
|
||
| ## Secrets | ||
| | Input | Description | Required | Default | | ||
| | ------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | ||
| | composer_auth | JSON string of [composer credentials](https://devdocs.magento.com/guides/v2.4/install-gde/prereq/connect-auth.html) | false | NULL | | ||
| > The `test_command` is executed from the Magento installation directory. | ||
| > E2E tests are expected to live under `dev/tests/e2e`. | ||
|
|
||
| ### Matrix Format | ||
|
|
||
| The Magento matrix format outlined by the [supported versions action.](https://github.com/mage-os/github-actions/tree/main/supported-version/supported.json) | ||
|
|
||
|
|
||
| ## E2E execution model | ||
|
|
||
| The E2E test suite is executed in two phases: | ||
|
|
||
| ### 1. Setup phase | ||
|
|
||
| The setup phase prepares shared browser state (e.g. authenticated sessions, | ||
| cookies, required entities). This phase is marked using the `@setup` tag. | ||
|
|
||
| ```bash | ||
| npx playwright test --grep "@setup" --trace on | ||
| ``` | ||
|
|
||
| This step is required once per fresh environment. | ||
|
|
||
| ### 2. Test phase | ||
|
|
||
| Once setup has completed successfully, the full E2E test suite can be executed. | ||
| Setup tests are skipped using --grep-invert. | ||
|
|
||
| ```bash | ||
| npx playwright test --grep-invert "@setup" --trace on | ||
| ``` | ||
|
|
||
| This phase validates real user journeys against a live Magento storefront. | ||
|
|
||
| ```yml | ||
| name: E2E Test | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| pull_request: | ||
| branches: | ||
| - main | ||
|
|
||
| jobs: | ||
| compute_matrix: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| matrix: ${{ steps.supported-version.outputs.matrix }} | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: mage-os/github-actions/supported-version@main | ||
| id: supported-version | ||
| - run: echo ${{ steps.supported-version.outputs.matrix }} | ||
| e2e-workflow: | ||
| needs: compute_matrix | ||
| uses: mage-os/github-actions/.github/workflows/e2e.yaml@main | ||
| with: | ||
| package_name: my-vendor/package | ||
| matrix: ${{ needs.compute_matrix.outputs.matrix }} | ||
| test_command: | | ||
| npx playwright test --grep "@setup" --trace on | ||
| npx playwright test --grep-invert "@setup" --trace on | ||
| secrets: | ||
| composer_auth: ${{ secrets.COMPOSER_AUTH }} | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,246 @@ | ||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| use_local_source: | ||
| type: boolean | ||
| required: false | ||
| default: true | ||
| description: "Whether or not you want to test your local package or not." | ||
|
|
||
| source_folder: | ||
| type: string | ||
| required: false | ||
| default: $GITHUB_WORKSPACE | ||
| description: "The source folder of the package" | ||
|
|
||
| package_name: | ||
| type: string | ||
| required: true | ||
| description: "The name of the package" | ||
|
|
||
| magento_directory: | ||
| type: string | ||
| required: false | ||
| default: "../magento2" | ||
| description: "The folder where Magento will be installed" | ||
|
|
||
| magento_repository: | ||
| type: string | ||
| required: false | ||
| default: "https://mirror.mage-os.org/" | ||
| description: "Where to install Magento from" | ||
|
|
||
| matrix: | ||
| type: string | ||
| required: true | ||
| description: "The matrix of Magento versions to test against" | ||
|
|
||
| fail-fast: | ||
| type: boolean | ||
| required: false | ||
| default: true | ||
|
|
||
| test_command: | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| description: "Shell commands used to execute the E2E test phases (e.g. Playwright)" | ||
|
|
||
| composer_cache_key: | ||
| type: string | ||
| required: false | ||
| default: '' | ||
| description: A key to version the composer cache. Can be incremented if you need to bust the cache. | ||
|
|
||
| secrets: | ||
| composer_auth: | ||
| required: false | ||
|
|
||
| jobs: | ||
| e2e_test: | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: ${{ inputs.fail-fast }} | ||
| matrix: ${{ fromJSON(inputs.matrix) }} | ||
| services: | ||
| elasticsearch: | ||
| image: ${{ matrix.elasticsearch || '' }} | ||
| env: | ||
| # By default, ElasticSearch refuses to spawn in single node configuration, as it expects redundancy. | ||
| # This is a dev environment, so redundancy is just wasteful. | ||
| discovery.type: single-node | ||
| # Disable HTTPS and password authentication | ||
| # this is a local dev environment, so the added CA chain complexity is an extreme overkill | ||
| xpack.security.enabled: false | ||
| xpack.security.http.ssl.enabled: false | ||
| xpack.security.transport.ssl.enabled: false | ||
|
|
||
| options: >- | ||
| --health-cmd "curl http://localhost:9200/_cluster/health" | ||
| --health-interval 10s | ||
| --health-timeout 5s | ||
| --health-retries 10 | ||
| ports: | ||
| - 9200:9200 | ||
|
|
||
| opensearch: | ||
| image: ${{ matrix.opensearch || '' }} | ||
| env: | ||
| # By default, ElasticSearch refuses to spawn in single node configuration, as it expects redundancy. | ||
| # This is a dev environment, so redundancy is just wasteful. | ||
| discovery.type: single-node | ||
| # Disable HTTPS and password authentication | ||
| DISABLE_INSTALL_DEMO_CONFIG: true | ||
| DISABLE_SECURITY_PLUGIN: true | ||
|
|
||
| options: >- | ||
| --health-cmd "curl http://localhost:9200/_cluster/health" | ||
| --health-interval 10s | ||
| --health-timeout 5s | ||
| --health-retries 10 | ||
| ports: | ||
| - 9200:9200 | ||
|
|
||
| mysql: | ||
| image: ${{ matrix.mysql }} | ||
| env: | ||
| MYSQL_DATABASE: magento_e2e_tests | ||
| MYSQL_USER: user | ||
| MYSQL_PASSWORD: password | ||
| MYSQL_ROOT_PASSWORD: rootpassword | ||
| ports: | ||
| - 3306:3306 | ||
| options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3 | ||
|
|
||
| rabbitmq: | ||
| image: ${{ matrix.rabbitmq }} | ||
| env: | ||
| RABBITMQ_DEFAULT_USER: guest | ||
| RABBITMQ_DEFAULT_PASS: guest | ||
| ports: | ||
| - 5672:5672 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Set PHP Version | ||
| uses: shivammathur/setup-php@v2 | ||
| with: | ||
| php-version: ${{ matrix.php }} | ||
| tools: composer:v${{ matrix.composer }} | ||
| coverage: none | ||
|
|
||
| - name: Setup Node | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 20 | ||
|
|
||
| - name: Allow SQL triggers | ||
| run: | | ||
| mysql --host 127.0.0.1 --port ${{ job.services.mysql.ports['3306'] }} -u root -prootpassword -e "set global log_bin_trust_function_creators=1;" | ||
|
|
||
| - run: composer create-project --repository-url="${{ inputs.magento_repository }}" "${{ matrix.magento }}" ${{ inputs.magento_directory }} --no-install | ||
| shell: bash | ||
| env: | ||
| COMPOSER_AUTH: ${{ secrets.composer_auth }} | ||
| name: Create Magento ${{ matrix.magento }} Project | ||
|
|
||
| - uses: mage-os/github-actions/get-magento-version@main | ||
| id: magento-version | ||
| with: | ||
| working-directory: ${{ inputs.magento_directory }} | ||
|
|
||
| - name: Get Composer Cache Directory | ||
| shell: bash | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| id: composer-cache | ||
| run: | | ||
| echo "dir=$(composer config cache-files-dir)" >> $GITHUB_OUTPUT | ||
|
|
||
|
|
||
| - name: "Cache Composer Packages" | ||
| uses: actions/cache@v4 | ||
| with: | ||
| key: "composer | v5 | ${{ inputs.composer_cache_key }} | ${{ hashFiles('composer.lock') }} | ${{ matrix.os }} | ${{ matrix.composer }} | ${{ matrix.php }} | ${{ matrix.magento }}" | ||
| path: ${{ steps.composer-cache.outputs.dir }} | ||
|
|
||
| - run: composer config repositories.local path ${{ inputs.source_folder }} | ||
| name: Add Github Repo for Testing | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| shell: bash | ||
| if: ${{ inputs.use_local_source == true }} | ||
|
|
||
| - run: composer require monolog/monolog:"<2.7.0" --no-update | ||
| name: Fixup Monolog (https://github.com/magento/magento2/pull/35596) | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| if: | | ||
| steps.magento-version.outputs.version == '"2.4.4"' | ||
|
|
||
| - run: composer require "dotmailer/dotmailer-magento2-extension-package:4.6.0-p2 as 4.6.0" --no-update | ||
| name: Fixup Dotmailer (https://devdocs.magento.com/guides/v2.4/release-notes/release-notes-2-4-0-commerce.html#dotdigital-1) | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| if: | | ||
| steps.magento-version.outputs.version == '"2.4.0"' | ||
|
|
||
| - run: | | ||
| composer config --no-interaction allow-plugins.dealerdirect/phpcodesniffer-composer-installer true | ||
| composer config --no-interaction allow-plugins.laminas/laminas-dependency-plugin true | ||
| composer config --no-interaction allow-plugins.magento/* true | ||
| name: Fixup Composer Plugins | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| if: ${{ !startsWith(matrix.composer, '1') }} | ||
|
|
||
| - run: | | ||
| composer global require hirak/prestissimo | ||
| name: Install composer plugin for parallel downloads | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| if: ${{ startsWith(matrix.composer, '1') }} | ||
|
|
||
| - run: composer require ${{ inputs.package_name }} "@dev" --no-update && composer install | ||
| name: Require and attempt install | ||
| working-directory: ${{ inputs.magento_directory }} | ||
| shell: bash | ||
| env: | ||
| COMPOSER_CACHE_DIR: ${{ steps.composer-cache.outputs.dir }} | ||
| COMPOSER_AUTH: ${{ secrets.composer_auth }} | ||
|
|
||
| - name: Replace Configuration Settings for env | ||
| working-directory: ${{ inputs.magento_directory }}/dev/tests/e2e | ||
| run: | | ||
| sed -i "s/'db-host' => 'localhost'/'db-host' => '127.0.0.1'/" etc/install-config-mysql.php.dist | ||
| sed -i "s/'db-user' => 'root'/'db-user' => 'user'/" etc/install-config-mysql.php.dist | ||
| sed -i "s/'db-password' => '123123q'/'db-password' => 'password'/" etc/install-config-mysql.php.dist | ||
| sed -i "s/'elasticsearch-host' => 'localhost'/'elasticsearch-host' => '127.0.0.1'/" etc/install-config-mysql.php.dist | ||
| sed -i "s/'amqp-host' => 'localhost'/'amqp-host' => '127.0.0.1'/" etc/install-config-mysql.php.dist | ||
|
|
||
| # mysql server 5.7 doesn't have the column-statistics expected by mysql client 8 (failing 2.3.7-p* builds) | ||
| # ref: https://gist.github.com/tobias-khs/8dcf82f719a2b3a7c3b9604b4df53bbf | ||
| - name: Switch from mysql-client 8 to mysql-client 5.7 | ||
| if: | | ||
| steps.magento-version.outputs.version == '"2.3.7-p3"' || steps.magento-version.outputs.version == '"2.3.7-p4"' | ||
| run: | | ||
| mkdir -p /tmp/mysql-5.7 | ||
| cd /tmp/mysql-5.7 | ||
| sudo apt-get purge mysql-server mysql-client mysql-common mysql-server-core-* mysql-client-core-* | ||
| sudo rm -rf /etc/mysql /var/lib/mysql | ||
| sudo apt-get autoremove | ||
| sudo apt-get autoclean | ||
| wget --quiet https://downloads.mysql.com/archives/get/p/23/file/mysql-server_5.7.30-1ubuntu18.04_amd64.deb-bundle.tar | ||
| tar -xf mysql-server_5.7.30-1ubuntu18.04_amd64.deb-bundle.tar | ||
| sudo dpkg -i mysql-common_5.7.30-1ubuntu18.04_amd64.deb | ||
| sudo dpkg -i libmysqlclient20_5.7.30-1ubuntu18.04_amd64.deb | ||
| sudo dpkg -i mysql-community-client_5.7.30-1ubuntu18.04_amd64.deb | ||
| sudo dpkg -i mysql-client_5.7.30-1ubuntu18.04_amd64.deb | ||
| sudo dpkg -i libmysqlclient20_5.7.30-1ubuntu18.04_amd64.deb | ||
| mysqldump --version | ||
|
|
||
| - run: ${{ inputs.test_command }} | ||
| working-directory: ${{ inputs.magento_directory }}/dev/tests/e2e | ||
| name: Run Playwright E2E phases | ||
|
|
||
| - name: Upload test sandbox dir | ||
| uses: actions/upload-artifact@v4 | ||
| if: failure() | ||
| with: | ||
| name: sandbox-data-${{ steps.magento-version.outputs.version }} | ||
| path: /home/runner/work/infrastructure/magento2/dev/tests/e2e/tmp/sandbox-* | ||
| retention-days: 3 | ||
|
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this added line should test if the folder exist. I have a possible alternative if needed but this is simple and hopefully doing what we want
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That throws an error on execution:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found this method mentioned in https://stackoverflow.com/questions/71336204/github-action-check-if-a-file-already-exists
I'd like to suggest the below but I could be more reliable if I understand to test this workflow end to end locally.
Can I suggest to discuss this after the meeting this afternoon?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@digitalrisedorset what's the purpose of the existence check? Won't this always exist in the
internalversion of this?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The initial commit checks showed CI failures on all Magento versions because no Magento version has the Playwright tests bundled. We need that to not happen. https://github.com/mage-os/github-actions/actions/runs/21867114747
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@damienwebdev like @rhoerr explains, this is very much backward compliance. The hashfile is a github method that does exactly what we are after. But the method needs to be positioned at a place where github can handle it. The comment I am making above is a possible solution whereby I have just moved the hasfile statement higher up in the workflow.
Rather than guessing where to add this statement though, I'd like to know how I can trigger the workflow. I think sansec process prevents me to trigger the workflow.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made a new commit today. This commit is inspired of a working version of the same check made on a dummy repo: https://github.com/digitalrisedorset/dummy-repo
this dummy repo reproduces the same folder structure as our github-actions repo and only performs the step test if the folder demo-package exists. Can you let me know if this new commit helps preventing the e2e workflow to be triggered if the folder is missing? thanks