[pull] main from MetaMask:main#101
Merged
Merged
Conversation
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** This PR is based on Javi's PR here #17706 This PR introduces [Appwright](https://www.npmjs.com/package/appwright), a testing framework we’ll use specifically for performance testing. It integrates with BrowserStack to run tests across multiple device configurations, providing detailed metrics and insights into app performance during key user flows. We also added two new CI jobs to trigger run the tests via workflow dispatch and another job to build the apps and run the tests automatically. ## What Was Added - **Appwright Framework**: A complete performance testing infrastructure built on top of Appwright - **Device Matrix Configuration**: Device coverage including high-end, mid-range, and low-end devices for both Android and iOS - **Performance Test Scenarios**: 9 test scenarios covering critical user journeys (onboarding, account creation, wallet interactions) - **Performance Tracking System**: Custom timer utilities and performance metrics collection - **Multi-Platform Support**: Android and iOS testing with both BrowserStack and local simulator support - **Custom Reporting**: Enhanced reporting with performance metrics visualization and JSON export ## The Tech We Used - [**Appwright**:](https://www.npmjs.com/package/appwright) This is a test framework that builds on top of Appium and uses the Playwright test runner internally - **BrowserStack**: For testing on real devices in the cloud - **Custom Timers**: Our own timing tools to measure user interactions precisely - **HTML Reports**: Easy-to-read performance summaries ## How to Run Tests ### Against BrowserStack Devices ***You can get your browserstack username and access key from the Access key dropdown on the[ app automate screen](https://app-automate.browserstack.com/dashboard/v2/builds) in BrowserStack*** ```bash # Set environment variables for BrowserStack export BROWSERSTACK_USERNAME='' export BROWSERSTACK_ACCESS_KEY='' export BROWSERSTACK_ANDROID_APP_URL="bs://your-app-url.apk" export BROWSERSTACK_IOS_APP_URL="bs://your-app-url.ipa" # Run Android tests on BrowserStack yarn run-appwright:android-bs # Run iOS tests on BrowserStack yarn run-appwright:ios-bs ``` ### Testing Locally (Simulators/Emulators) ```bash # Test on your local Android emulator yarn run-appwright:android # Test on your local iOS simulator yarn run-appwright:ios ``` **Important**: You need to specify where your app build lives in the `appwright.config.ts` file. For example: - **Android**: Set `buildPath` to your `.apk` file location - **iOS**: Set `buildPath` to your `.app` file location The config will look something like this: ```typescript { name: 'android', use: { platform: Platform.ANDROID, device: { provider: 'emulator' }, buildPath: '/path/to/your/app-release.apk', // Your APK location }, } ``` ### Run Performance Tests on CI <img width="1897" height="781" alt="image" src="https://github.com/user-attachments/assets/75659e5f-7ddc-4e44-acb8-edebc77188c3" /> In order to run the test on CI, you would need to use the workflow dispatch job called `trigger-performance-e2e.yml` You’ll need a BrowserStack URL first. To get it: Run create_qa_builds_pipeline on Bitrise. Once done, open the Artifacts tab and find browserstack_uploaded_apps.json (from build_android_qa). For example this [build](https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/7060652a-7ef9-4d0f-a44f-c736ae6a3bbc?tab=artifacts). The first entry in that JSON will include your app’s URL (look for the bs:// prefix). <img width="647" height="343" alt="image" src="https://github.com/user-attachments/assets/d1c4618b-593c-4f4e-8142-cadca4a9ac60" /> There is another job to trigger the builds and tests called `run-performance-e2e.yml` which essentially builds the apps via bitrise, then uses the generated build bs urls and launches the tests. In future iterations, this workflow will no longer depend on bitrise but more so github actions to build the apps. ## Test Reports <img width="1008" height="798" alt="image" src="https://github.com/user-attachments/assets/0d46fc62-8811-4adf-85dc-60f91c225cbc" /> Reports are automatically generated in the `appwright/test-reports/` directory and include: - Individual test performance metrics - Total execution time per scenario - Device-specific performance data - Step-by-step timing breakdowns ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** 1. Go to this page... 2. 3. ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [ ] I’ve followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I’ve included tests if applicable - [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. --------- Co-authored-by: Javier Garcia Vera <javier.vera@consensys.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.3)
Can you help keep this open source service alive? 💖 Please sponsor : )