Skip to content

Commit 46ab400

Browse files
javiergarciaveracortisiko
javiergarciavera
and
cortisiko
authored
test: added quality gates logic for performance test (MetaMask#24060)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** This PR: - addresses some test refactors to simplify timers - Added quaility gates for tests and defined threshold (10%) for Android and iOS (once merged, we track the metrics and decide about the thresholds) <!-- Write a short description of the changes included in this pull request, also include relevant motivation and context. Have in mind the following questions: 1. What is the reason for the change? 2. What is the improvement/solution? --> ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** ```gherkin Feature: my feature name Scenario: user [verb for user action] Given [describe expected initial app state] When user [verb for user action] Then [describe expected outcome] ``` ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [ ] I’ve followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I’ve included tests if applicable - [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > Introduces platform-specific thresholds with 10% margins and quality gates to performance tests, refactors timers/reporters, updates tests to use measure() + thresholds, and adds aggregated HTML reporting. > > - **Performance framework**: > - Add `QualityGatesValidator` with platform-specific thresholds (+10%) and pass/fail validation. > - Extend `TimersHelper` with threshold resolution (iOS/Android), `measure()` helper, and `hasThreshold()`/`threshold` metadata. > - Update fixture `fixtures/performance-test.js` to auto-attach metrics and assert quality gates post-test. > - Enhance `PerformanceTracker` to emit per-step thresholds/validation and optional total validation; add `addTimers`. > - **Reporter & outputs**: > - Update `custom-reporter.js` to validate quality gates, embed results in HTML/CSV, include BrowserStack video/profiling. > - Improve per-test HTML tables (duration/threshold/status) and CSV; include total threshold status. > - Aggregation script now generates a polished `aggregated-reports/performance-report.html` dashboard. > - **Tests**: > - Refactor performance specs to use `TimerHelper(..., { ios, android }, device)` and `measure()`; consolidate timer adds via `addTimers`. > - Add explicit thresholds across login, swaps, send, perps, predict, and onboarding flows; minor skips/timeout tweaks. > - **Docs**: > - Overhaul `appwright/README.md` with device matrix, categories, performance system, quality gates, reports, aggregation, and best practices. > - **Page objects**: > - `WalletMainScreen.isVisible()` now uses `appwright` expect. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 35f1aa5. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Curtis David <Curtis.David7@gmail.com>
1 parent 0ae171a commit 46ab400

28 files changed

Lines changed: 2434 additions & 388 deletions

‎appwright/README.md‎

Lines changed: 427 additions & 130 deletions
Large diffs are not rendered by default.

‎appwright/fixtures/performance-test.js‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { test as base } from 'appwright';
22
import { PerformanceTracker } from '../reporters/PerformanceTracker.js';
3-
import TimerHelper from '../utils/TimersHelper.js';
3+
import QualityGatesValidator from '../utils/QualityGatesValidator.js';
44

55
// Create a custom test fixture that handles performance tracking and cleanup
66
export const test = base.extend({
@@ -33,6 +33,19 @@ export const test = base.extend({
3333
console.error('❌ Failed to attach performance metrics:', error.message);
3434
}
3535

36+
// Validate quality gates if any timer has thresholds defined
37+
const hasThresholds = performanceTracker.timers.some((t) =>
38+
t.hasThreshold(),
39+
);
40+
if (hasThresholds) {
41+
console.log('🔍 Validating quality gates...');
42+
QualityGatesValidator.assertThresholds(
43+
testInfo.title,
44+
performanceTracker.timers,
45+
);
46+
console.log('✅ Quality gates PASSED');
47+
}
48+
3649
console.log('🔍 Looking for session ID...');
3750

3851
let sessionId = null;

‎appwright/reporters/PerformanceTracker.js‎

Lines changed: 63 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,12 @@ export class PerformanceTracker {
66
this.timers = [];
77
}
88

9+
addTimers(...timers) {
10+
timers.forEach((timer) => {
11+
this.addTimer(timer);
12+
});
13+
}
14+
915
addTimer(timer) {
1016
if (this.timers.find((existingTimer) => existingTimer.id === timer.id)) {
1117
return;
@@ -111,26 +117,80 @@ export class PerformanceTracker {
111117
}
112118

113119
async attachToTest(testInfo) {
120+
const THRESHOLD_MARGIN_PERCENT = 10;
114121
const metrics = {
115122
steps: [],
123+
timestamp: new Date().toISOString(),
124+
thresholdMarginPercent: THRESHOLD_MARGIN_PERCENT,
116125
};
117126
let totalSeconds = 0;
127+
let totalThresholdMs = 0;
128+
let allHaveThresholds = true;
118129

119130
for (const timer of this.timers) {
120131
const duration = timer.getDuration();
121132
const durationInSeconds = timer.getDurationInSeconds();
122133

123134
if (duration !== null && !isNaN(duration) && duration > 0) {
124-
// Create a step object with the timer id as key and duration as value
125-
const stepObject = {};
126-
stepObject[timer.id] = duration;
135+
const hasThreshold = timer.threshold !== null;
136+
const passed = !hasThreshold || duration <= timer.threshold;
137+
const exceeded =
138+
hasThreshold && !passed ? duration - timer.threshold : null;
139+
const percentOver =
140+
exceeded !== null
141+
? ((exceeded / timer.threshold) * 100).toFixed(1)
142+
: null;
143+
144+
// Create a step object with timer info including thresholds and validation
145+
const stepObject = {
146+
name: timer.id,
147+
duration,
148+
baseThreshold: timer.baseThreshold,
149+
threshold: timer.threshold, // Includes +10% margin
150+
validation: hasThreshold
151+
? {
152+
passed,
153+
exceeded,
154+
percentOver: percentOver ? `${percentOver}%` : null,
155+
}
156+
: null,
157+
};
127158
metrics.steps.push(stepObject);
128159

129160
totalSeconds += durationInSeconds;
161+
162+
if (timer.threshold !== null) {
163+
totalThresholdMs += timer.threshold;
164+
} else {
165+
allHaveThresholds = false;
166+
}
130167
}
131168
}
132169

133170
metrics.total = totalSeconds;
171+
metrics.totalThreshold = allHaveThresholds ? totalThresholdMs : null;
172+
metrics.hasThresholds = this.timers.some((t) => t.hasThreshold());
173+
174+
// Add total validation if all steps have thresholds
175+
if (allHaveThresholds && totalThresholdMs > 0) {
176+
const totalDurationMs = totalSeconds * 1000;
177+
const totalPassed = totalDurationMs <= totalThresholdMs;
178+
const totalExceeded = !totalPassed
179+
? totalDurationMs - totalThresholdMs
180+
: null;
181+
const totalPercentOver =
182+
totalExceeded !== null
183+
? ((totalExceeded / totalThresholdMs) * 100).toFixed(1)
184+
: null;
185+
186+
metrics.totalValidation = {
187+
passed: totalPassed,
188+
exceeded: totalExceeded,
189+
percentOver: totalPercentOver ? `${totalPercentOver}%` : null,
190+
};
191+
} else {
192+
metrics.totalValidation = null;
193+
}
134194

135195
// Safely get device info with fallbacks
136196
const deviceInfo = testInfo?.project?.use?.device;

‎appwright/reporters/custom-reporter.js‎

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
/* eslint-disable import/no-nodejs-modules */
22
import { PerformanceTracker } from './PerformanceTracker';
33
import { AppProfilingDataHandler } from './AppProfilingDataHandler';
4+
import QualityGatesValidator from '../utils/QualityGatesValidator';
45
import fs from 'fs';
56
import path from 'path';
67

@@ -9,6 +10,7 @@ class CustomReporter {
910
this.metrics = [];
1011
this.sessions = []; // Array to store all session data
1112
this.processedTests = new Set(); // Track processed tests to avoid duplicates
13+
this.qualityGatesValidator = new QualityGatesValidator();
1214
}
1315

1416
// We'll skip the onStdOut and onStdErr methods since the list reporter will handle those
@@ -117,6 +119,26 @@ class CustomReporter {
117119
}
118120
}
119121

122+
// Validate quality gates using thresholds from timers
123+
if (metricsEntry.steps && metricsEntry.steps.length > 0) {
124+
const qualityGatesResult = this.qualityGatesValidator.validateMetrics(
125+
test.title,
126+
metricsEntry.steps,
127+
metricsEntry.total || 0,
128+
metricsEntry.totalThreshold || null,
129+
);
130+
metricsEntry.qualityGates = qualityGatesResult;
131+
132+
// Log quality gates result to console
133+
if (qualityGatesResult.hasThresholds) {
134+
console.log(
135+
this.qualityGatesValidator.formatConsoleReport(
136+
qualityGatesResult,
137+
),
138+
);
139+
}
140+
}
141+
120142
this.metrics.push(metricsEntry);
121143
} catch (error) {
122144
console.error('Error processing metrics:', error);
@@ -462,19 +484,52 @@ class CustomReporter {
462484
<table>
463485
<tr>
464486
<th>Steps</th>
465-
<th>Time (ms)</th>
487+
<th>Duration</th>
488+
<th>Threshold</th>
489+
<th>Status</th>
466490
</tr>
467491
${
468492
test.steps && Array.isArray(test.steps)
469-
? // New array structure with steps
470-
test.steps
493+
? test.steps
471494
.map((stepObject) => {
495+
// Handle new format with threshold info
496+
if (stepObject.name !== undefined) {
497+
const { name, duration, threshold, baseThreshold } =
498+
stepObject;
499+
const hasThreshold =
500+
threshold !== null && threshold !== undefined;
501+
const passed =
502+
!hasThreshold || duration <= threshold;
503+
const statusIcon = hasThreshold
504+
? passed
505+
? '✅'
506+
: '❌'
507+
: '—';
508+
const rowStyle =
509+
hasThreshold && !passed
510+
? 'background-color: #ffebee;'
511+
: '';
512+
const thresholdStr = hasThreshold
513+
? `${threshold}ms<br><small style="color: #666;">(base: ${baseThreshold}ms)</small>`
514+
: '—';
515+
return `
516+
<tr style="${rowStyle}">
517+
<td>${name}</td>
518+
<td>${duration} ms</td>
519+
<td>${thresholdStr}</td>
520+
<td>${statusIcon}</td>
521+
</tr>
522+
`;
523+
}
524+
// Handle old format {stepName: duration}
472525
const [stepName, duration] =
473526
Object.entries(stepObject)[0];
474527
return `
475528
<tr>
476529
<td>${stepName}</td>
477530
<td>${duration} ms</td>
531+
<td>—</td>
532+
<td>—</td>
478533
</tr>
479534
`;
480535
})
@@ -487,6 +542,8 @@ class CustomReporter {
487542
<tr>
488543
<td>${stepName}</td>
489544
<td>${duration} ms</td>
545+
<td>—</td>
546+
<td>—</td>
490547
</tr>
491548
`,
492549
)
@@ -509,6 +566,8 @@ class CustomReporter {
509566
<tr>
510567
<td>${key}</td>
511568
<td>${value} ms</td>
569+
<td>—</td>
570+
<td>—</td>
512571
</tr>
513572
`,
514573
)
@@ -517,6 +576,8 @@ class CustomReporter {
517576
<tr class="total">
518577
<td>TOTAL TIME</td>
519578
<td>${test.total} s</td>
579+
<td>${test.totalThreshold ? `${(test.totalThreshold / 1000).toFixed(2)} s` : '—'}</td>
580+
<td>${test.totalThreshold ? (test.total * 1000 <= test.totalThreshold ? '✅' : '❌') : '—'}</td>
520581
</tr>
521582
${
522583
test.testFailed
@@ -547,6 +608,13 @@ class CustomReporter {
547608
: ''
548609
}
549610
</table>
611+
${
612+
test.qualityGates
613+
? this.qualityGatesValidator.generateHtmlSection(
614+
test.qualityGates,
615+
)
616+
: ''
617+
}
550618
`,
551619
)
552620
.join('')}
@@ -1035,6 +1103,14 @@ class CustomReporter {
10351103
}
10361104
}
10371105

1106+
// Add quality gates information
1107+
if (test.qualityGates) {
1108+
const qgRows = this.qualityGatesValidator.generateCsvRows(
1109+
test.qualityGates,
1110+
);
1111+
csvRows.push(...qgRows);
1112+
}
1113+
10381114
// Add spacing between tables (3 blank lines to clearly separate tables)
10391115
if (i < this.metrics.length - 1) {
10401116
csvRows.push('');

‎appwright/tests/performance/login/asset-balances.spec.js‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import { login } from '../../../utils/Flows.js';
1010
test('Aggregated Balance Loading Time, SRP 1 + SRP 2 + SRP 3', async ({
1111
device,
1212
performanceTracker,
13-
}, testInfo) => {
13+
}) => {
1414
WalletMainScreen.device = device;
1515
TabBarModal.device = device;
1616
LoginScreen.device = device;
@@ -19,14 +19,14 @@ test('Aggregated Balance Loading Time, SRP 1 + SRP 2 + SRP 3', async ({
1919

2020
const balanceStableTimer = new TimerHelper(
2121
'Time since the user navigates to wallet tab until the balance stabilizes',
22+
{ ios: 25000, android: 25000 },
23+
device,
2224
);
2325

24-
balanceStableTimer.start();
25-
26-
await WalletMainScreen.waitForBalanceToStabilize();
27-
balanceStableTimer.stop();
26+
await balanceStableTimer.measure(() =>
27+
WalletMainScreen.waitForBalanceToStabilize(),
28+
);
2829

2930
performanceTracker.addTimer(balanceStableTimer);
30-
31-
await performanceTracker.attachToTest(testInfo);
31+
// Quality gates validation is performed by the reporter when generating reports
3232
});

‎appwright/tests/performance/login/asset-view.spec.js‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,17 +58,19 @@ test('Asset View, SRP 1 + SRP 2 + SRP 3', async ({
5858

5959
const assetViewScreen = new TimerHelper(
6060
'Time since the user clicks on the asset view button until the user sees the token overview screen',
61+
{ ios: 600, android: 600 },
62+
device,
6163
);
6264

6365
await WalletMainScreen.tapNetworkNavBar();
6466
await NetworksScreen.selectNetwork('Ethereum');
6567

6668
await WalletMainScreen.tapOnToken('USDC');
67-
assetViewScreen.start();
68-
await TokenOverviewScreen.isTokenOverviewVisible();
69-
await TokenOverviewScreen.isTodaysChangeVisible();
70-
await TokenOverviewScreen.isSendButtonVisible();
71-
assetViewScreen.stop();
69+
await assetViewScreen.measure(async () => {
70+
await TokenOverviewScreen.isTokenOverviewVisible();
71+
await TokenOverviewScreen.isTodaysChangeVisible();
72+
await TokenOverviewScreen.isSendButtonVisible();
73+
});
7274

7375
performanceTracker.addTimer(assetViewScreen);
7476

‎appwright/tests/performance/login/cross-chain-swap-flow.spec.js‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -36,24 +36,24 @@ test('Cross-chain swap flow - ETH to SOL - 50+ accounts, SRP 1 + SRP 2 + SRP 3',
3636

3737
const timer1 = new TimerHelper(
3838
'Time since the user clicks on the "Swap" button until the swap page is loaded',
39+
{ ios: 1100, android: 1000 },
40+
device,
3941
);
4042

4143
await WalletMainScreen.tapSwapButton();
42-
timer1.start();
43-
await BridgeScreen.isVisible();
44-
timer1.stop();
44+
await timer1.measure(() => BridgeScreen.isVisible());
4545

4646
await BridgeScreen.selectNetworkAndTokenTo('Solana', 'SOL');
4747
await BridgeScreen.enterSourceTokenAmount('1');
4848

4949
const timer2 = new TimerHelper(
5050
'Time since the user enters the amount until the quote is displayed',
51+
{ ios: 9000, android: 7000 },
52+
device,
5153
);
5254

53-
timer2.start();
54-
await BridgeScreen.isQuoteDisplayed();
55-
timer2.stop();
56-
performanceTracker.addTimer(timer1);
57-
performanceTracker.addTimer(timer2);
55+
await timer2.measure(() => BridgeScreen.isQuoteDisplayed());
56+
57+
performanceTracker.addTimers(timer1, timer2);
5858
await performanceTracker.attachToTest(testInfo);
5959
});

‎appwright/tests/performance/login/eth-swap-flow.spec.js‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -32,22 +32,23 @@ test('Swap flow - ETH to LINK, SRP 1 + SRP 2 + SRP 3', async ({
3232

3333
const swapLoadTimer = new TimerHelper(
3434
'Time since the user clicks on the "Swap" button until the swap page is loaded',
35+
{ ios: 1100, android: 1000 },
36+
device,
3537
);
3638

3739
await WalletMainScreen.tapSwapButton();
38-
swapLoadTimer.start();
39-
await BridgeScreen.isVisible();
40-
swapLoadTimer.stop();
40+
await swapLoadTimer.measure(() => BridgeScreen.isVisible());
41+
4142
const swapTimer = new TimerHelper(
4243
'Time since the user enters the amount until the quote is displayed',
44+
{ ios: 9000, android: 7000 },
45+
device,
4346
);
4447
await BridgeScreen.selectNetworkAndTokenTo('Ethereum', 'LINK');
4548
await BridgeScreen.enterSourceTokenAmount('1');
4649

47-
swapTimer.start();
48-
await BridgeScreen.isQuoteDisplayed();
49-
swapTimer.stop();
50-
performanceTracker.addTimer(swapLoadTimer);
51-
performanceTracker.addTimer(swapTimer);
50+
await swapTimer.measure(() => BridgeScreen.isQuoteDisplayed());
51+
52+
performanceTracker.addTimers(swapLoadTimer, swapTimer);
5253
await performanceTracker.attachToTest(testInfo);
5354
});

0 commit comments

Comments
 (0)