Skip to content

Commit c5c4623

Browse files
test: added new sentry attributes (MetaMask#28138)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** Added new scenario attributes to Sentry: - Browserstack recording link - Github action link - Scenario team owner <!-- Write a short description of the changes included in this pull request, also include relevant motivation and context. Have in mind the following questions: 1. What is the reason for the change? 2. What is the improvement/solution? --> ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** ```gherkin Feature: my feature name Scenario: user [verb for user action] Given [describe expected initial app state] When user [verb for user action] Then [describe expected outcome] ``` ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [ ] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I've included tests if applicable - [ ] I've documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I've applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Adds new external BrowserStack API calls and extra metadata to Sentry transaction payloads during test teardown, which could impact reporting or introduce intermittent failures if env/config changes. Core app logic is unaffected. > > **Overview** > E2E performance runs now publish richer Sentry transactions by **mirroring key scenario attributes** (project/provider/team/status/retry/build variant/device/file path) into both Sentry `tags` and each step `span.data` for easier filtering and correlation. > > The performance fixture now optionally fetches and attaches a **BrowserStack session recording URL** (derived from the test `sessionId` annotation and BrowserStack session details) and the Sentry publisher also includes **GitHub Actions run/job metadata** (from `GITHUB_*` env vars) in `extra` and span data. > > Tests were updated to cover the new payload fields and to manage the additional GitHub env vars during setup/teardown. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 8b2546d. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 6bb954e commit c5c4623

3 files changed

Lines changed: 187 additions & 20 deletions

File tree

tests/framework/fixtures/performance/performance-fixture.ts

Lines changed: 39 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import {
44
type MetricsOutput,
55
} from '../../../reporters/PerformanceTracker';
66
import { publishPerformanceScenarioToSentry } from '../../../reporters/providers/sentry/PerformanceSentryPublisher';
7+
import { BrowserStackAPI } from '../../services/providers/browserstack/BrowserStackAPI';
78
import {
89
QualityGatesValidator,
910
markQualityGateFailure,
@@ -17,6 +18,36 @@ interface PerformanceFixtures {
1718
performanceTracker: PerformanceTracker;
1819
}
1920

21+
async function getBrowserStackRecordingUrl(
22+
sessionId: string | null,
23+
projectName: string,
24+
): Promise<string | null> {
25+
if (!sessionId || !projectName.toLowerCase().includes('browserstack')) {
26+
return null;
27+
}
28+
29+
try {
30+
const api = new BrowserStackAPI();
31+
const sessionDetails = await api.getSessionDetails(sessionId);
32+
if (!sessionDetails?.buildId) {
33+
return null;
34+
}
35+
36+
return api.buildSessionURL(sessionDetails.buildId, sessionId);
37+
} catch {
38+
return null;
39+
}
40+
}
41+
42+
function getSessionIdFromAnnotations(
43+
annotations?: { type: string; description?: string }[],
44+
): string | null {
45+
return (
46+
annotations?.find((annotation) => annotation.type === 'sessionId')
47+
?.description ?? null
48+
);
49+
}
50+
2051
// Create a custom test fixture that handles performance tracking and cleanup
2152
export const test = base.extend<PerformanceFixtures>({
2253
// eslint-disable-next-line no-empty-pattern
@@ -76,13 +107,21 @@ export const test = base.extend<PerformanceFixtures>({
76107
);
77108
}
78109

110+
const sessionId = getSessionIdFromAnnotations(testInfo.annotations);
111+
79112
if (metrics) {
113+
const browserstackRecordingUrl = await getBrowserStackRecordingUrl(
114+
sessionId,
115+
testInfo.project?.name ?? 'unknown',
116+
);
117+
80118
try {
81119
const sentToSentry = await publishPerformanceScenarioToSentry({
82120
metrics,
83121
testTitle: testInfo.title,
84122
projectName: testInfo.project?.name ?? 'unknown',
85123
testFilePath: testInfo.file,
124+
browserstackRecordingUrl,
86125
tags: testTags,
87126
status: testInfo.status,
88127
retry: testInfo.retry,
@@ -128,15 +167,6 @@ export const test = base.extend<PerformanceFixtures>({
128167

129168
console.log('🔍 Looking for session ID...');
130169

131-
let sessionId: string | null = null;
132-
133-
if (testInfo?.annotations) {
134-
sessionId =
135-
testInfo.annotations.find(
136-
(annotation) => annotation.type === 'sessionId',
137-
)?.description ?? null;
138-
}
139-
140170
if (sessionId) {
141171
// Store session data as a test attachment for the reporter to find
142172
// Include team info and tags in session data

tests/reporters/providers/sentry/PerformanceSentryPublisher.test.ts

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,13 +66,21 @@ describe('PerformanceSentryPublisher', () => {
6666
process.env.E2E_PERFORMANCE_SENTRY_SAMPLE_RATE;
6767
const originalSentryEnabled = process.env.E2E_PERFORMANCE_SENTRY_ENABLED;
6868
const originalBuildVariant = process.env.E2E_PERFORMANCE_BUILD_VARIANT;
69+
const originalGithubServerUrl = process.env.GITHUB_SERVER_URL;
70+
const originalGithubRepository = process.env.GITHUB_REPOSITORY;
71+
const originalGithubRunId = process.env.GITHUB_RUN_ID;
72+
const originalGithubJob = process.env.GITHUB_JOB;
6973

7074
beforeEach(() => {
7175
jest.clearAllMocks();
7276
delete process.env.E2E_PERFORMANCE_SENTRY_DSN;
7377
delete process.env.E2E_PERFORMANCE_SENTRY_SAMPLE_RATE;
7478
delete process.env.E2E_PERFORMANCE_SENTRY_ENABLED;
7579
delete process.env.E2E_PERFORMANCE_BUILD_VARIANT;
80+
delete process.env.GITHUB_SERVER_URL;
81+
delete process.env.GITHUB_REPOSITORY;
82+
delete process.env.GITHUB_RUN_ID;
83+
delete process.env.GITHUB_JOB;
7684
fetchMock = jest.spyOn(global, 'fetch');
7785
});
7886

@@ -101,6 +109,30 @@ describe('PerformanceSentryPublisher', () => {
101109
process.env.E2E_PERFORMANCE_BUILD_VARIANT = originalBuildVariant;
102110
}
103111

112+
if (originalGithubServerUrl === undefined) {
113+
delete process.env.GITHUB_SERVER_URL;
114+
} else {
115+
process.env.GITHUB_SERVER_URL = originalGithubServerUrl;
116+
}
117+
118+
if (originalGithubRepository === undefined) {
119+
delete process.env.GITHUB_REPOSITORY;
120+
} else {
121+
process.env.GITHUB_REPOSITORY = originalGithubRepository;
122+
}
123+
124+
if (originalGithubRunId === undefined) {
125+
delete process.env.GITHUB_RUN_ID;
126+
} else {
127+
process.env.GITHUB_RUN_ID = originalGithubRunId;
128+
}
129+
130+
if (originalGithubJob === undefined) {
131+
delete process.env.GITHUB_JOB;
132+
} else {
133+
process.env.GITHUB_JOB = originalGithubJob;
134+
}
135+
104136
fetchMock.mockRestore();
105137
});
106138

@@ -124,6 +156,10 @@ describe('PerformanceSentryPublisher', () => {
124156
process.env.E2E_PERFORMANCE_SENTRY_DSN =
125157
'https://publicKey@o123.ingest.sentry.io/4567';
126158
process.env.E2E_PERFORMANCE_BUILD_VARIANT = 'exp';
159+
process.env.GITHUB_SERVER_URL = 'https://github.com';
160+
process.env.GITHUB_REPOSITORY = 'MetaMask/metamask-mobile';
161+
process.env.GITHUB_RUN_ID = '12345';
162+
process.env.GITHUB_JOB = 'e2e-performance-android';
127163
fetchMock.mockResolvedValue({
128164
ok: true,
129165
status: 200,
@@ -134,6 +170,8 @@ describe('PerformanceSentryPublisher', () => {
134170
testTitle: 'Import wallet flow',
135171
projectName: 'browserstack-android',
136172
testFilePath: 'tests/performance/onboarding/import-wallet.spec.js',
173+
browserstackRecordingUrl:
174+
'https://app-automate.browserstack.com/builds/build-123/sessions/sess-123',
137175
tags: ['@PerformanceOnboarding', '@PerformanceLaunch'],
138176
status: 'passed',
139177
retry: 0,
@@ -168,7 +206,40 @@ describe('PerformanceSentryPublisher', () => {
168206
expect(payload.measurements.scenario_total_time_ms.value).toBe(1300);
169207
expect(payload.tags.project_name).toBe('browserstack-android');
170208
expect(payload.tags.build_variant).toBe('exp');
209+
expect(payload.tags.test_team).toBe('qa-automation');
171210
expect(payload.extra.timer_steps).toHaveLength(2);
211+
expect(payload.extra.recording_url).toBe(
212+
'https://app-automate.browserstack.com/builds/build-123/sessions/sess-123',
213+
);
214+
expect(payload.extra.github_job_url).toBe(
215+
'https://github.com/MetaMask/metamask-mobile/actions/runs/12345',
216+
);
217+
expect(payload.extra.github_job_name).toBe('e2e-performance-android');
218+
expect(payload.spans).toHaveLength(2);
219+
expect(payload.spans[0].op).toBe('e2e.performance.step');
220+
expect(payload.spans[0].data.project_name).toBe('browserstack-android');
221+
expect(payload.spans[0].data.test_team).toBe('qa-automation');
222+
expect(payload.spans[0].data.provider).toBe('browserstack');
223+
expect(payload.spans[0].data.team_id).toBe('qa-automation');
224+
expect(payload.spans[0].data.team_name).toBe('QA Automation');
225+
expect(payload.spans[0].data.test_status).toBe('passed');
226+
expect(payload.spans[0].data.retry).toBe(0);
227+
expect(payload.spans[0].data.worker_index).toBe(3);
228+
expect(payload.spans[0].data.build_variant).toBe('exp');
229+
expect(payload.spans[0].data.device_name).toBe('Samsung Galaxy S23 Ultra');
230+
expect(payload.spans[0].data.device_os_version).toBe('13.0');
231+
expect(payload.spans[0].data.test_file_path).toBe(
232+
'tests/performance/onboarding/import-wallet.spec.js',
233+
);
234+
expect(payload.spans[0].data.recording_url).toBe(
235+
'https://app-automate.browserstack.com/builds/build-123/sessions/sess-123',
236+
);
237+
expect(payload.spans[0].data.github_job_url).toBe(
238+
'https://github.com/MetaMask/metamask-mobile/actions/runs/12345',
239+
);
240+
expect(payload.spans[0].data.github_job_name).toBe(
241+
'e2e-performance-android',
242+
);
172243
});
173244

174245
it('protects reserved aggregate keys from timer-key collisions', async () => {

tests/reporters/providers/sentry/PerformanceSentryPublisher.ts

Lines changed: 77 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@ const ENV_SENTRY_SAMPLE_RATE = 'E2E_PERFORMANCE_SENTRY_SAMPLE_RATE';
1313
const ENV_SENTRY_ENVIRONMENT = 'E2E_PERFORMANCE_SENTRY_ENVIRONMENT';
1414
const ENV_SENTRY_RELEASE = 'E2E_PERFORMANCE_SENTRY_RELEASE';
1515
const ENV_SENTRY_BUILD_VARIANT = 'E2E_PERFORMANCE_BUILD_VARIANT';
16+
const ENV_GITHUB_SERVER_URL = 'GITHUB_SERVER_URL';
17+
const ENV_GITHUB_REPOSITORY = 'GITHUB_REPOSITORY';
18+
const ENV_GITHUB_RUN_ID = 'GITHUB_RUN_ID';
19+
const ENV_GITHUB_JOB = 'GITHUB_JOB';
1620
const MAX_MEASUREMENT_KEY_LENGTH = 64;
1721
const RESERVED_MEASUREMENT_KEYS = [
1822
'scenario_total_time_ms',
@@ -24,6 +28,7 @@ interface PublishPerformanceScenarioOptions {
2428
testTitle: string;
2529
projectName: string;
2630
testFilePath?: string;
31+
browserstackRecordingUrl?: string | null;
2732
tags: string[];
2833
status?: string;
2934
retry?: number;
@@ -53,6 +58,24 @@ interface SentryMeasurement {
5358
unit: 'millisecond';
5459
}
5560

61+
interface MirroredScenarioAttributes {
62+
project_name: string;
63+
test_team: string;
64+
provider: string;
65+
team_id: string;
66+
team_name: string;
67+
test_status: string;
68+
retry: number;
69+
worker_index: number;
70+
build_variant: 'rc' | 'exp' | 'unknown';
71+
device_name: string;
72+
device_os_version: string;
73+
test_file_path: string;
74+
recording_url: string | null;
75+
github_job_url: string | null;
76+
github_job_name: string | null;
77+
}
78+
5679
function getEnvValue(key: string): string | undefined {
5780
return Reflect.get(process.env, key) as string | undefined;
5881
}
@@ -178,6 +201,17 @@ function parseSampleRate(rawSampleRate: string | undefined): number | null {
178201
return sampleRate;
179202
}
180203

204+
function getGithubJobUrl(): string | null {
205+
const serverUrl = getEnvValue(ENV_GITHUB_SERVER_URL);
206+
const repository = getEnvValue(ENV_GITHUB_REPOSITORY);
207+
const runId = getEnvValue(ENV_GITHUB_RUN_ID);
208+
if (!serverUrl || !repository || !runId) {
209+
return null;
210+
}
211+
212+
return `${serverUrl}/${repository}/actions/runs/${runId}`;
213+
}
214+
181215
export async function publishPerformanceScenarioToSentry(
182216
options: PublishPerformanceScenarioOptions,
183217
): Promise<boolean> {
@@ -255,6 +289,35 @@ export async function publishPerformanceScenarioToSentry(
255289
};
256290
}
257291

292+
const provider = options.metrics.device.provider || 'unknown';
293+
const teamId = options.metrics.team?.teamId || 'unknown';
294+
const teamName = options.metrics.team?.teamName || 'unknown';
295+
const testStatus = options.status || 'unknown';
296+
const retry = options.retry ?? 0;
297+
const workerIndex = options.workerIndex ?? 0;
298+
const buildVariant = normalizeBuildVariant(
299+
getEnvValue(ENV_SENTRY_BUILD_VARIANT),
300+
);
301+
const testFilePath = options.testFilePath || '';
302+
303+
const mirroredScenarioAttributes: MirroredScenarioAttributes = {
304+
project_name: options.projectName,
305+
test_team: teamId,
306+
provider,
307+
team_id: teamId,
308+
team_name: teamName,
309+
test_status: testStatus,
310+
retry,
311+
worker_index: workerIndex,
312+
build_variant: buildVariant,
313+
device_name: options.metrics.device.name,
314+
device_os_version: options.metrics.device.osVersion,
315+
test_file_path: testFilePath,
316+
recording_url: options.browserstackRecordingUrl ?? null,
317+
github_job_url: getGithubJobUrl(),
318+
github_job_name: getEnvValue(ENV_GITHUB_JOB) ?? null,
319+
};
320+
258321
let cursor = startTimestamp;
259322
const spans = timerMeasurements.map((timerMeasurement) => {
260323
const spanStart = cursor;
@@ -276,6 +339,7 @@ export async function publishPerformanceScenarioToSentry(
276339
base_threshold_ms: timerMeasurement.baseThreshold,
277340
exceeded_ms: timerMeasurement.exceeded,
278341
percent_over: timerMeasurement.percentOver,
342+
...mirroredScenarioAttributes,
279343
},
280344
};
281345
});
@@ -310,21 +374,23 @@ export async function publishPerformanceScenarioToSentry(
310374
},
311375
tags: {
312376
source: 'appwright-e2e-performance',
313-
project_name: options.projectName,
314-
provider: options.metrics.device.provider || 'unknown',
315-
team_id: options.metrics.team?.teamId || 'unknown',
316-
team_name: options.metrics.team?.teamName || 'unknown',
317-
test_status: options.status || 'unknown',
318-
retry: String(options.retry ?? 0),
319-
worker_index: String(options.workerIndex ?? 0),
320-
build_variant: normalizeBuildVariant(
321-
getEnvValue(ENV_SENTRY_BUILD_VARIANT),
322-
),
377+
project_name: mirroredScenarioAttributes.project_name,
378+
provider: mirroredScenarioAttributes.provider,
379+
team_id: mirroredScenarioAttributes.team_id,
380+
team_name: mirroredScenarioAttributes.team_name,
381+
test_team: mirroredScenarioAttributes.test_team,
382+
test_status: mirroredScenarioAttributes.test_status,
383+
retry: String(mirroredScenarioAttributes.retry),
384+
worker_index: String(mirroredScenarioAttributes.worker_index),
385+
build_variant: mirroredScenarioAttributes.build_variant,
323386
},
324387
measurements,
325388
spans,
326389
extra: {
327-
test_file_path: options.testFilePath || '',
390+
test_file_path: mirroredScenarioAttributes.test_file_path,
391+
recording_url: mirroredScenarioAttributes.recording_url,
392+
github_job_url: mirroredScenarioAttributes.github_job_url,
393+
github_job_name: mirroredScenarioAttributes.github_job_name,
328394
test_tags: options.tags,
329395
threshold_margin_percent: options.metrics.thresholdMarginPercent,
330396
has_thresholds: options.metrics.hasThresholds,

0 commit comments

Comments
 (0)