Skip to content

Commit c847907

Browse files
test: performance e2e suite (MetaMask#17773)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** This PR is based on Javi's PR here MetaMask#17706 This PR introduces [Appwright](https://www.npmjs.com/package/appwright), a testing framework we’ll use specifically for performance testing. It integrates with BrowserStack to run tests across multiple device configurations, providing detailed metrics and insights into app performance during key user flows. We also added two new CI jobs to trigger run the tests via workflow dispatch and another job to build the apps and run the tests automatically. ## What Was Added - **Appwright Framework**: A complete performance testing infrastructure built on top of Appwright - **Device Matrix Configuration**: Device coverage including high-end, mid-range, and low-end devices for both Android and iOS - **Performance Test Scenarios**: 9 test scenarios covering critical user journeys (onboarding, account creation, wallet interactions) - **Performance Tracking System**: Custom timer utilities and performance metrics collection - **Multi-Platform Support**: Android and iOS testing with both BrowserStack and local simulator support - **Custom Reporting**: Enhanced reporting with performance metrics visualization and JSON export ## The Tech We Used - [**Appwright**:](https://www.npmjs.com/package/appwright) This is a test framework that builds on top of Appium and uses the Playwright test runner internally - **BrowserStack**: For testing on real devices in the cloud - **Custom Timers**: Our own timing tools to measure user interactions precisely - **HTML Reports**: Easy-to-read performance summaries ## How to Run Tests ### Against BrowserStack Devices ***You can get your browserstack username and access key from the Access key dropdown on the[ app automate screen](https://app-automate.browserstack.com/dashboard/v2/builds) in BrowserStack*** ```bash # Set environment variables for BrowserStack export BROWSERSTACK_USERNAME='' export BROWSERSTACK_ACCESS_KEY='' export BROWSERSTACK_ANDROID_APP_URL="bs://your-app-url.apk" export BROWSERSTACK_IOS_APP_URL="bs://your-app-url.ipa" # Run Android tests on BrowserStack yarn run-appwright:android-bs # Run iOS tests on BrowserStack yarn run-appwright:ios-bs ``` ### Testing Locally (Simulators/Emulators) ```bash # Test on your local Android emulator yarn run-appwright:android # Test on your local iOS simulator yarn run-appwright:ios ``` **Important**: You need to specify where your app build lives in the `appwright.config.ts` file. For example: - **Android**: Set `buildPath` to your `.apk` file location - **iOS**: Set `buildPath` to your `.app` file location The config will look something like this: ```typescript { name: 'android', use: { platform: Platform.ANDROID, device: { provider: 'emulator' }, buildPath: '/path/to/your/app-release.apk', // Your APK location }, } ``` ### Run Performance Tests on CI <img width="1897" height="781" alt="image" src="https://github.com/user-attachments/assets/75659e5f-7ddc-4e44-acb8-edebc77188c3" /> In order to run the test on CI, you would need to use the workflow dispatch job called `trigger-performance-e2e.yml` You’ll need a BrowserStack URL first. To get it: Run create_qa_builds_pipeline on Bitrise. Once done, open the Artifacts tab and find browserstack_uploaded_apps.json (from build_android_qa). For example this [build](https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/7060652a-7ef9-4d0f-a44f-c736ae6a3bbc?tab=artifacts). The first entry in that JSON will include your app’s URL (look for the bs:// prefix). <img width="647" height="343" alt="image" src="https://github.com/user-attachments/assets/d1c4618b-593c-4f4e-8142-cadca4a9ac60" /> There is another job to trigger the builds and tests called `run-performance-e2e.yml` which essentially builds the apps via bitrise, then uses the generated build bs urls and launches the tests. In future iterations, this workflow will no longer depend on bitrise but more so github actions to build the apps. ## Test Reports <img width="1008" height="798" alt="image" src="https://github.com/user-attachments/assets/0d46fc62-8811-4adf-85dc-60f91c225cbc" /> Reports are automatically generated in the `appwright/test-reports/` directory and include: - Individual test performance metrics - Total execution time per scenario - Device-specific performance data - Step-by-step timing breakdowns ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** 1. Go to this page... 2. 3. ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [ ] I’ve followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I’ve included tests if applicable - [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. --------- Co-authored-by: Javier Garcia Vera <javier.vera@consensys.net>
1 parent 82b2455 commit c847907

53 files changed

Lines changed: 3266 additions & 290 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.e2e.env.example

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,12 @@ export MULTICHAIN_DAPP_URL="https://metamask.github.io/test-dapp-multichain/late
2020
# E2E prebuild paths
2121
export PREBUILT_IOS_APP_PATH='build/MetaMask.app'
2222
export PREBUILT_ANDROID_APK_PATH='build/MetaMask.apk'
23-
export PREBUILT_ANDROID_TEST_APK_PATH='build/MetaMask-Test.apk'
23+
export PREBUILT_ANDROID_TEST_APK_PATH='build/MetaMask-Test.apk'
24+
25+
export TEST_SRP_1=
26+
export TEST_SRP_2=
27+
export TEST_SRP_3=
28+
export BROWSERSTACK_USERNAME=
29+
export BROWSERSTACK_ACCESS_KEY=
30+
export SEEDLESS_ONBOARDING_ENABLED=
31+
export SOLANA_MODAL_ENABLED=

.eslintrc.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ module.exports = {
2121
],
2222
overrides: [
2323
{
24-
files: ['e2e/**/*.{js,ts}'],
24+
files: ['e2e/**/*.{js,ts}', 'appwright/**/*.{js,ts}'],
2525
extends: ['./e2e/framework/.eslintrc.js'],
2626
},
2727
{

.gitignore

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@ buck-out/
6565
# bundle artifact
6666
*.jsbundle
6767

68-
# testing
69-
tests
7068

7169
# app-specific
7270
/scripts/inpage-bridge/dist
@@ -110,6 +108,15 @@ e2e/reports
110108
# Performance test results
111109
e2e/specs/performance/reports/*-performance-results.json
112110

111+
# appwright
112+
appwright/report
113+
playwright-report/*
114+
appwright/test-reports/
115+
test-results/
116+
test-reports/
117+
appwright/reporters/reports/*
118+
*.apk
119+
113120
# anvil binaries
114121
.metamask/*
115122
# ppom

appwright/appwright.config.ts

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
// In appwright.config.ts
2+
import dotenv from 'dotenv';
3+
dotenv.config({ path: '.e2e.env' });
4+
import { defineConfig, Platform } from 'appwright';
5+
export default defineConfig({
6+
testMatch: '**/tests/performance/*.spec.js',
7+
timeout: 7 * 60 * 1000, //7 minutes until we introduce fixtures
8+
expect: {
9+
timeout: 30 * 1000, //30 seconds
10+
},
11+
reporter: [
12+
// The default HTML reporter from Appwright
13+
[
14+
'html',
15+
{ open: 'never', outputFolder: './test-reports/appwright-report' },
16+
],
17+
['./reporters/custom-reporter.js'],
18+
['list'],
19+
],
20+
21+
projects: [
22+
{
23+
name: 'android',
24+
use: {
25+
platform: Platform.ANDROID,
26+
device: {
27+
provider: 'emulator',
28+
name: 'Samsung Galaxy S24 Ultra', // this can be changed to your emulator name
29+
osVersion: '14', // this can be changed to your emulator version
30+
},
31+
buildPath: 'PATH-TO-BUILD', // Path to your .apk file
32+
},
33+
},
34+
{
35+
name: 'ios',
36+
use: {
37+
platform: Platform.IOS,
38+
device: {
39+
provider: 'emulator',
40+
osVersion: '16.0', // this can be changed to your simulator version
41+
},
42+
buildPath: 'PATH-TO-BUILD', // Path to your .app file
43+
},
44+
},
45+
{
46+
name: 'browserstack-android',
47+
use: {
48+
platform: Platform.ANDROID,
49+
device: {
50+
provider: 'browserstack',
51+
name: process.env.BROWSERSTACK_DEVICE || 'Samsung Galaxy S23 Ultra', // this can changed
52+
osVersion: process.env.BROWSERSTACK_OS_VERSION || '13.0', // this can changed
53+
},
54+
buildPath: process.env.BROWSERSTACK_ANDROID_APP_URL, // Path to Browserstack url
55+
},
56+
},
57+
{
58+
name: 'browserstack-ios',
59+
use: {
60+
platform: Platform.IOS,
61+
device: {
62+
provider: 'browserstack',
63+
name: process.env.BROWSERSTACK_DEVICE || 'iPhone 14 Pro Max',
64+
osVersion: process.env.BROWSERSTACK_OS_VERSION || '16.0',
65+
},
66+
buildPath: process.env.BROWSERSTACK_IOS_APP_URL, // Path to Browserstack url
67+
},
68+
},
69+
],
70+
});

appwright/reporters/custom-reporter.js

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
/* eslint-disable */
2-
// import fs from 'fs';
3-
// import path from 'path';
1+
/* eslint-disable import/no-nodejs-modules */
2+
import fs from 'fs';
3+
import path from 'path';
44

55
class CustomReporter {
66
constructor() {
@@ -72,6 +72,7 @@ class CustomReporter {
7272
);
7373
fs.writeFileSync(jsonPath, JSON.stringify(this.metrics, null, 2));
7474
// Generate HTML report
75+
/* eslint-disable */
7576
const html = `
7677
<!DOCTYPE html>
7778
<html>
@@ -82,10 +83,10 @@ class CustomReporter {
8283
<style>
8384
body { font-family: Arial, sans-serif; margin: 40px; }
8485
table { border-collapse: collapse; width: 100%; }
85-
th, td { border: 1px solid #ddd; padding: 12px; text-align: left; }
86-
th { background-color: #4CAF50; color: white; }
87-
tr:nth-child(even) { background-color: #f2f2f2; }
88-
.total { font-weight: bold; background-color: #e7f3e7; }
86+
th, td { border: 1px solid #e0e0e0; padding: 12px; text-align: left; }
87+
th { background-color: #2e7d32; color: white; }
88+
tr:nth-child(even) { background-color: #f5f5f5; }
89+
.total { font-weight: bold; background-color: #e8f5e8; }
8990
</style>
9091
</head>
9192
<body>
@@ -120,6 +121,7 @@ class CustomReporter {
120121
</tr>
121122
`;
122123
}
124+
return ''; // Return empty string for device key
123125
})
124126
.join('')}
125127
</table>
@@ -130,6 +132,7 @@ class CustomReporter {
130132
</body>
131133
</html>
132134
`;
135+
/* eslint-enable */
133136

134137
const reportPath = path.join(
135138
reportsDir,
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
import { test, expect } from 'appwright';
2+
3+
import TimerHelper from '../../utils/TimersHelper.js';
4+
import { PerformanceTracker } from '../../reporters/PerformanceTracker.js';
5+
import WelcomeScreen from '../../../wdio/screen-objects/Onboarding/OnboardingCarousel.js';
6+
import TermOfUseScreen from '../../../wdio/screen-objects/Modals/TermOfUseScreen.js';
7+
import OnboardingScreen from '../../../wdio/screen-objects/Onboarding/OnboardingScreen.js';
8+
import CreateNewWalletScreen from '../../../wdio/screen-objects/Onboarding/CreateNewWalletScreen.js';
9+
import MetaMetricsScreen from '../../../wdio/screen-objects/Onboarding/MetaMetricsScreen.js';
10+
import OnboardingSucessScreen from '../../../wdio/screen-objects/OnboardingSucessScreen.js';
11+
import OnboardingSheet from '../../../wdio/screen-objects/Onboarding/OnboardingSheet.js';
12+
import SolanaFeatureSheet from '../../../wdio/screen-objects/Modals/SolanaFeatureSheet.js';
13+
import WalletAccountModal from '../../../wdio/screen-objects/Modals/WalletAccountModal.js';
14+
import SkipAccountSecurityModal from '../../../wdio/screen-objects/Modals/SkipAccountSecurityModal.js';
15+
import ImportFromSeedScreen from '../../../wdio/screen-objects/Onboarding/ImportFromSeedScreen.js';
16+
import CreatePasswordScreen from '../../../wdio/screen-objects/Onboarding/CreatePasswordScreen.js';
17+
import AccountListComponent from '../../../wdio/screen-objects/AccountListComponent.js';
18+
import AddAccountModal from '../../../wdio/screen-objects/Modals/AddAccountModal.js';
19+
import WalletMainScreen from '../../../wdio/screen-objects/WalletMainScreen.js';
20+
import { importSRPFlow, onboardingFlowImportSRP } from '../../utils/Flows.js';
21+
22+
test('Account creation with 50+ accounts, SRP 1 + SRP 2 + SRP 3', async ({
23+
device,
24+
}, testInfo) => {
25+
WelcomeScreen.device = device;
26+
TermOfUseScreen.device = device;
27+
OnboardingScreen.device = device;
28+
CreateNewWalletScreen.device = device;
29+
MetaMetricsScreen.device = device;
30+
OnboardingSucessScreen.device = device;
31+
OnboardingSheet.device = device;
32+
SolanaFeatureSheet.device = device;
33+
WalletAccountModal.device = device;
34+
SkipAccountSecurityModal.device = device;
35+
ImportFromSeedScreen.device = device;
36+
CreatePasswordScreen.device = device;
37+
WalletMainScreen.device = device;
38+
AccountListComponent.device = device;
39+
AddAccountModal.device = device;
40+
41+
await onboardingFlowImportSRP(device, process.env.TEST_SRP_1);
42+
await importSRPFlow(device, process.env.TEST_SRP_2);
43+
await importSRPFlow(device, process.env.TEST_SRP_3);
44+
45+
const screen1Timer = new TimerHelper(
46+
'Time since the user clicks on "Account list" button until the account list is visible',
47+
);
48+
const screen2Timer = new TimerHelper(
49+
'Time since the user clicks on "Add account" button until the next modal is visible',
50+
);
51+
const screen3Timer = new TimerHelper(
52+
'Time since the user clicks on "Create Ethereum account" button until the Token list is visible',
53+
);
54+
55+
await WalletMainScreen.isTokenVisible('Ethereum');
56+
screen1Timer.start();
57+
await WalletMainScreen.tapIdenticon();
58+
await AccountListComponent.isComponentDisplayed();
59+
screen1Timer.stop();
60+
screen2Timer.start();
61+
await AccountListComponent.tapAddAccountButton();
62+
screen2Timer.stop();
63+
screen3Timer.start();
64+
await AddAccountModal.tapCreateEthereumAccountButton();
65+
await WalletMainScreen.isTokenVisible('Ethereum');
66+
screen3Timer.stop();
67+
68+
const performanceTracker = new PerformanceTracker();
69+
performanceTracker.addTimer(screen1Timer);
70+
performanceTracker.addTimer(screen2Timer);
71+
performanceTracker.addTimer(screen3Timer);
72+
73+
await performanceTracker.attachToTest(testInfo);
74+
});
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
import { test, expect } from 'appwright';
2+
3+
import TimerHelper from '../../utils/TimersHelper.js';
4+
import { PerformanceTracker } from '../../reporters/PerformanceTracker.js';
5+
import WelcomeScreen from '../../../wdio/screen-objects/Onboarding/OnboardingCarousel.js';
6+
import TermOfUseScreen from '../../../wdio/screen-objects/Modals/TermOfUseScreen.js';
7+
import OnboardingScreen from '../../../wdio/screen-objects/Onboarding/OnboardingScreen.js';
8+
import CreateNewWalletScreen from '../../../wdio/screen-objects/Onboarding/CreateNewWalletScreen.js';
9+
import MetaMetricsScreen from '../../../wdio/screen-objects/Onboarding/MetaMetricsScreen.js';
10+
import OnboardingSucessScreen from '../../../wdio/screen-objects/OnboardingSucessScreen.js';
11+
import OnboardingSheet from '../../../wdio/screen-objects/Onboarding/OnboardingSheet.js';
12+
import SolanaFeatureSheet from '../../../wdio/screen-objects/Modals/SolanaFeatureSheet.js';
13+
import WalletAccountModal from '../../../wdio/screen-objects/Modals/WalletAccountModal.js';
14+
import SkipAccountSecurityModal from '../../../wdio/screen-objects/Modals/SkipAccountSecurityModal.js';
15+
import WalletMainScreen from '../../../wdio/screen-objects/WalletMainScreen.js';
16+
17+
const SOLANA_MODAL_ENABLED = process.env.SOLANA_MODAL_ENABLED
18+
? process.env.SOLANA_MODAL_ENABLED
19+
: false;
20+
test('Onboarding new wallet, SRP 1 + SRP 2 + SRP 3', async ({
21+
device,
22+
}, testInfo) => {
23+
const screen1Timer = new TimerHelper(
24+
'Time until the user clicks on the "Get Started" button',
25+
);
26+
screen1Timer.start();
27+
WelcomeScreen.device = device;
28+
TermOfUseScreen.device = device;
29+
OnboardingScreen.device = device;
30+
CreateNewWalletScreen.device = device;
31+
MetaMetricsScreen.device = device;
32+
OnboardingSucessScreen.device = device;
33+
OnboardingSheet.device = device;
34+
SolanaFeatureSheet.device = device;
35+
WalletAccountModal.device = device;
36+
SkipAccountSecurityModal.device = device;
37+
WalletMainScreen.device = device;
38+
39+
const timer1 = new TimerHelper(
40+
'Time since the user clicks on "Get Started" button until the Term of Use screen is visible',
41+
);
42+
const timer2 = new TimerHelper(
43+
'Time since the user clicks on "Aggree to T&C" button until the Onboarding screen is visible',
44+
);
45+
const timer3 = new TimerHelper(
46+
'Time since the user clicks on "Create new wallet" button until "continue with SRP" button is visible',
47+
);
48+
const timer4 = new TimerHelper(
49+
'Time since the user clicks on "Continue with SRP" button until password fields are visible',
50+
);
51+
const timer5 = new TimerHelper(
52+
'Time since the user clicks on "Create Password" button until "Remind me later" shows up',
53+
);
54+
const timer6 = new TimerHelper(
55+
'Time since the user clicks on "Proceed without wallet secure" button until Metrics screen is displayed',
56+
);
57+
const timer7 = new TimerHelper(
58+
'Time since the user clicks on "Aggree" button on Metrics screen until Onboarding Success screen is visible',
59+
);
60+
const timer8 = new TimerHelper(
61+
'Time since the user clicks on "Done" button until Solana feature sheet is visible',
62+
);
63+
64+
timer1.start();
65+
await WelcomeScreen.clickGetStartedButton();
66+
await TermOfUseScreen.isDisplayed();
67+
timer1.stop();
68+
await TermOfUseScreen.tapAgreeCheckBox();
69+
await TermOfUseScreen.tapScrollEndButton();
70+
timer2.start();
71+
await TermOfUseScreen.tapAcceptButton();
72+
await OnboardingScreen.isScreenTitleVisible();
73+
timer2.stop();
74+
timer3.start();
75+
await OnboardingScreen.tapCreateNewWalletButton();
76+
await OnboardingSheet.isVisible();
77+
timer3.stop();
78+
timer4.start();
79+
await OnboardingSheet.tapImportSeedButton();
80+
await CreateNewWalletScreen.isNewAccountScreenFieldsVisible();
81+
timer4.stop();
82+
await CreateNewWalletScreen.inputPasswordInFirstField('123456789');
83+
await CreateNewWalletScreen.inputConfirmPasswordField('123456789');
84+
timer5.start();
85+
await CreateNewWalletScreen.tapSubmitButton();
86+
await CreateNewWalletScreen.tapRemindMeLater();
87+
await SkipAccountSecurityModal.isVisible();
88+
timer5.stop();
89+
timer6.start();
90+
await SkipAccountSecurityModal.proceedWithoutWalletSecure();
91+
await MetaMetricsScreen.isScreenTitleVisible();
92+
timer6.stop();
93+
timer7.start();
94+
await MetaMetricsScreen.tapIAgreeButton();
95+
await OnboardingSucessScreen.isVisible();
96+
timer7.stop();
97+
timer8.start();
98+
await OnboardingSucessScreen.tapDone();
99+
if (SOLANA_MODAL_ENABLED) {
100+
await SolanaFeatureSheet.isVisible();
101+
await SolanaFeatureSheet.tapNotNowButton();
102+
}
103+
timer8.stop();
104+
105+
const performanceTracker = new PerformanceTracker();
106+
performanceTracker.addTimer(timer1);
107+
performanceTracker.addTimer(timer2);
108+
performanceTracker.addTimer(timer3);
109+
performanceTracker.addTimer(timer4);
110+
performanceTracker.addTimer(timer5);
111+
performanceTracker.addTimer(timer6);
112+
performanceTracker.addTimer(timer7);
113+
performanceTracker.addTimer(timer8);
114+
await performanceTracker.attachToTest(testInfo);
115+
});

0 commit comments

Comments
 (0)