Skip to content

Commit bfae32d

Browse files
agrawal-ayushclaude
agrawal-ayush
and
claude
authored
feat: add setCustomTag API for custom test metadata (#51)
* feat: add setCustomTag API for custom test metadata Expose browser.setCustomTag(keyName, keyValue, buildLevelCustomTag) to let users tag tests with custom metadata sent to TestHub. Tags with the same key are merged (not overridden). Supports both test-level and build-level metadata, comma-separated values, and pending tag buffering for calls made before the test UUID is assigned. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve custom tag bugs — regex parsing, UUID race condition, and worker IPC - Fix splitValues regex to handle whitespace before quoted strings (e.g. `foo, "bar,baz"` now correctly parses as ["foo", "bar,baz"]) - Fix test-level tags missing for non-first tests in a suite: test body runs before TestRunStarted assigns the new UUID, so setCustomTag was storing tags under the previous test's UUID. Now clear UUID in TestRunFinished and drain pendingTestTags synchronously before the async sendTestRunEvent to avoid race conditions. - Add worker IPC for build-level tags: workers send tags via process.send (Cucumber) and temp files (Nightwatch parallel), parent aggregates before stopBuildUpstream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: scope temp files to run ID, deep-clone metadata getters - Scope build-tag temp files with BROWSERSTACK_TESTHUB_UUID to prevent cross-run contamination from stale files in tmpdir - Deep-clone returns from getTestLevelCustomMetadata and getBuildLevelCustomMetadata to prevent callers mutating internal state - Add tests for deep-clone and temp file scoping Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * eslint comments --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent abe5c4f commit bfae32d

5 files changed

Lines changed: 433 additions & 0 deletions

File tree

‎.github/dev-guide/local-testing.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Local Testing Guide (Nightwatch Plugin)
2+
3+
> Pack, link, and run the local nightwatch-plugin-browserstack against a sample project to verify changes.
4+
5+
---
6+
7+
## Prerequisites
8+
9+
- **Node.js:** 16+ recommended
10+
- **Sample repo:** A Nightwatch project that uses `@nightwatch/browserstack` (e.g., `nightwatch-browserstack`)
11+
- **BrowserStack credentials:** `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` exported in your shell
12+
13+
---
14+
15+
## Build (Pack)
16+
17+
```bash
18+
cd /absolute/path/to/nightwatch-plugin-browserstack
19+
20+
# Remove any previous tarball first — avoids installing a stale version
21+
rm -f nightwatch-browserstack-*.tgz
22+
23+
npm pack
24+
# → nightwatch-browserstack-<version>.tgz (e.g., nightwatch-browserstack-3.8.0.tgz)
25+
26+
# Capture the tarball path
27+
export NW_PLUGIN_TGZ=$(pwd)/$(ls nightwatch-browserstack-*.tgz | head -1)
28+
echo "Tarball: $NW_PLUGIN_TGZ"
29+
```
30+
31+
---
32+
33+
## Link to Sample
34+
35+
Install the local tarball directly into the sample project:
36+
37+
```bash
38+
cd /absolute/path/to/<sample>
39+
40+
npm install "$NW_PLUGIN_TGZ"
41+
```
42+
43+
Verify the linked version has your changes:
44+
45+
```bash
46+
# Check the installed version
47+
cat node_modules/@nightwatch/browserstack/package.json | grep version
48+
```
49+
50+
> **Tip:** When iterating on changes, always delete the old `.tgz` before repacking:
51+
> ```bash
52+
> cd /absolute/path/to/nightwatch-plugin-browserstack
53+
> rm -f nightwatch-browserstack-*.tgz
54+
> npm pack
55+
> cd /absolute/path/to/<sample>
56+
> npm install "$NW_PLUGIN_TGZ"
57+
> ```
58+
59+
---
60+
61+
## Run Sample
62+
63+
> **Working directory:** Run all test commands from the **sample's root directory**, not the plugin directory.
64+
65+
```bash
66+
cd /absolute/path/to/<sample>
67+
68+
# Export credentials if not already set
69+
export BROWSERSTACK_USERNAME=<your-username>
70+
export BROWSERSTACK_ACCESS_KEY=<your-access-key>
71+
72+
# Run a single test
73+
npx nightwatch --test ./tests/single/single_test.js --env browserstack 2>&1 | tee /tmp/nw-test-run.log
74+
```
75+
76+
After the run, verify:
77+
```bash
78+
# Check for BrowserStack signals and errors
79+
grep -i "browserstack\|bstack\|error\|exception" /tmp/nw-test-run.log | head -30
80+
```
81+
82+
---
83+
84+
## Troubleshooting
85+
86+
### "Still using published version, not local"
87+
```bash
88+
# Verify the installed version matches your tarball
89+
cat <sample>/node_modules/@nightwatch/browserstack/package.json | grep version
90+
91+
# If stale, reinstall
92+
cd <sample> && npm install "$NW_PLUGIN_TGZ"
93+
```
94+
95+
### "Cannot convert undefined or null to object" in `helper.js:checkTestEnvironmentForAppAutomate`
96+
The nightwatch version in the sample may be too old. The plugin 3.x requires nightwatch 3.x:
97+
```bash
98+
npx nightwatch --version
99+
# If 2.x, upgrade:
100+
npm install nightwatch@latest
101+
```
102+
103+
### "TypeError: browser.<method> is not a function"
104+
The plugin was not correctly linked. Re-pack and reinstall:
105+
```bash
106+
cd /absolute/path/to/nightwatch-plugin-browserstack
107+
rm -f nightwatch-browserstack-*.tgz
108+
npm pack
109+
cd /absolute/path/to/<sample>
110+
npm install /absolute/path/to/nightwatch-plugin-browserstack/nightwatch-browserstack-*.tgz
111+
```

‎nightwatch/globals.js‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,13 @@ const helper = require('../src/utils/helper');
66
const Logger = require('../src/utils/logger');
77
const {v4: uuidv4} = require('uuid');
88
const path = require('path');
9+
const fs = require('fs');
10+
const os = require('os');
911
const AccessibilityAutomation = require('../src/accessibilityAutomation');
1012
const eventHelper = require('../src/utils/eventHelper');
1113
const OrchestrationUtils = require('../src/testorchestration/orchestrationUtils');
1214
const TestMap = require('../src/utils/testMap');
15+
const CustomTagManager = require('../src/utils/customTagManager');
1316
const localTunnel = new LocalTunnel();
1417
const testObservability = new TestObservability();
1518
const accessibilityAutomation = new AccessibilityAutomation();
@@ -99,6 +102,14 @@ module.exports = {
99102

100103
Object.values(workerList).forEach((worker) => {
101104
worker.process.on('message', async (data) => {
105+
if (data.BUILD_LEVEL_CUSTOM_TAGS) {
106+
try {
107+
const tags = JSON.parse(data.BUILD_LEVEL_CUSTOM_TAGS);
108+
CustomTagManager.mergeBuildLevelTags(tags);
109+
} catch (err) {
110+
Logger.debug(`Error merging build-level tags from worker: ${err}`);
111+
}
112+
}
102113
if (data.POST_SESSION_EVENT) {
103114
helper.storeSessionsData(data);
104115
}
@@ -151,6 +162,7 @@ module.exports = {
151162
if (testMetaData) {
152163
delete _tests[testCaseId];
153164
testMetaData.finishedAt = new Date().toISOString();
165+
CustomTagManager.drainPendingTestTags(testMetaData.uuid);
154166
await testObservability.sendTestRunEventForCucumber(reportData, gherkinDocument, pickleData, 'TestRunFinished', testMetaData, args);
155167
}
156168
} catch (error) {
@@ -279,8 +291,15 @@ module.exports = {
279291
try {
280292
await accessibilityAutomation.afterEachExecution(test, uuid);
281293
if (testRunner !== 'cucumber'){
294+
// Drain pending tags synchronously before the async sendTestRunEvent,
295+
// to avoid races where the next test's tags leak into pendingTestTags
296+
// and get drained into the wrong UUID.
297+
CustomTagManager.drainPendingTestTags(uuid);
282298
testEventPromises.push(testObservability.sendTestRunEvent('TestRunFinished', test, uuid));
283299
TestMap.markTestFinished(uuid);
300+
// Clear UUID so the next test's setCustomTag calls buffer to pendingTestTags
301+
// (test body runs before TestRunStarted assigns the new UUID)
302+
delete process.env.TEST_RUN_UUID;
284303
}
285304

286305
} catch (error) {
@@ -479,6 +498,26 @@ module.exports = {
479498
await Promise.all(testEventPromises);
480499
testEventPromises.length = 0; // Clear the array
481500
}
501+
502+
// Aggregate build-level custom tags from worker processes
503+
try {
504+
const tmpDir = os.tmpdir();
505+
const runId = process.env.BROWSERSTACK_TESTHUB_UUID || process.pid;
506+
const tagFiles = fs.readdirSync(tmpDir).filter(f => f.startsWith(`bstack_build_tags_${runId}_`) && f.endsWith('.json'));
507+
for (const tagFile of tagFiles) {
508+
const filePath = path.join(tmpDir, tagFile);
509+
try {
510+
const tags = JSON.parse(fs.readFileSync(filePath, 'utf8'));
511+
CustomTagManager.mergeBuildLevelTags(tags);
512+
fs.unlinkSync(filePath);
513+
} catch (err) {
514+
Logger.debug(`Error reading build-level tags file ${tagFile}: ${err}`);
515+
}
516+
}
517+
} catch (err) {
518+
Logger.debug(`Error aggregating build-level tags from workers: ${err}`);
519+
}
520+
482521
await testObservability.stopBuildUpstream();
483522
if (process.env.BROWSERSTACK_TESTHUB_UUID) {
484523
Logger.info(`\nVisit https://automation.browserstack.com/builds/${process.env.BROWSERSTACK_TESTHUB_UUID} to view build report, insights, and many more debugging information all at one place!\n`);
@@ -499,6 +538,13 @@ module.exports = {
499538
browser.getAccessibilityResultsSummary = () => { return accessibilityAutomation.getAccessibilityResultsSummary() };
500539
}
501540
// await accessibilityAutomation.beforeEachExecution(browser);
541+
542+
// Clear stale UUID so tags go to pendingTestTags until TestRunStarted assigns the new UUID
543+
delete process.env.TEST_RUN_UUID;
544+
545+
browser.setCustomTag = (keyName, keyValue, buildLevelCustomTag) => {
546+
CustomTagManager.setCustomTag(keyName, keyValue, buildLevelCustomTag || false, process.env.TEST_RUN_UUID);
547+
};
502548
},
503549

504550
// This will be run after each test suite is finished
@@ -552,6 +598,24 @@ module.exports = {
552598

553599
async afterChildProcess() {
554600

601+
// Send build-level custom tags from worker to parent process
602+
try {
603+
const buildTags = CustomTagManager.getBuildLevelCustomMetadata();
604+
if (buildTags && Object.keys(buildTags).length > 0) {
605+
if (process.send) {
606+
// Cucumber parallel: send via IPC directly
607+
process.send({BUILD_LEVEL_CUSTOM_TAGS: JSON.stringify(buildTags)});
608+
}
609+
// Also write to temp file for standard Nightwatch parallel where
610+
// we don't have a direct IPC listener in the parent
611+
const runId = process.env.BROWSERSTACK_TESTHUB_UUID || process.pid;
612+
const tagFile = path.join(os.tmpdir(), `bstack_build_tags_${runId}_${process.pid}.json`);
613+
fs.writeFileSync(tagFile, JSON.stringify(buildTags));
614+
}
615+
} catch (err) {
616+
Logger.debug(`Error sending build-level tags from worker: ${err}`);
617+
}
618+
555619
await helper.shutDownRequestHandler();
556620
if (testEventPromises.length > 0) {
557621
await Promise.all(testEventPromises);

‎src/testObservability.js‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ const OrchestrationUtils = require('./testorchestration/orchestrationUtils');
1212
const AccessibilityAutomation = require('./accessibilityAutomation');
1313
const accessibilityScripts = require('./scripts/accessibilityScripts');
1414
const TestMap = require('./utils/testMap');
15+
const CustomTagManager = require('./utils/customTagManager');
1516
const hooksMap = {};
1617
const accessibilityAutomation = new AccessibilityAutomation();
1718

@@ -265,6 +266,10 @@ class TestObservability {
265266
const data = {
266267
'finished_at': new Date().toISOString()
267268
};
269+
const buildCustomMetadata = CustomTagManager.getBuildLevelCustomMetadata();
270+
if (buildCustomMetadata && Object.keys(buildCustomMetadata).length > 0) {
271+
data.custom_metadata = buildCustomMetadata;
272+
}
268273
const config = {
269274
headers: {
270275
'Authorization': `Bearer ${process.env.BROWSERSTACK_TESTHUB_JWT}`,
@@ -523,6 +528,15 @@ class TestObservability {
523528
await this.processTestRunData (eventData, uuid);
524529
}
525530

531+
const customMetadata = CustomTagManager.getTestLevelCustomMetadata(uuid);
532+
if (customMetadata && Object.keys(customMetadata).length > 0) {
533+
testData.custom_metadata = customMetadata;
534+
}
535+
536+
if (eventType === 'TestRunFinished') {
537+
CustomTagManager.clearTestLevelCustomMetadata(uuid);
538+
}
539+
526540
const uploadData = {
527541
event_type: eventType,
528542
test_run: testData
@@ -685,6 +699,15 @@ class TestObservability {
685699
}
686700
}
687701

702+
const cucumberCustomMetadata = CustomTagManager.getTestLevelCustomMetadata(uuid);
703+
if (cucumberCustomMetadata && Object.keys(cucumberCustomMetadata).length > 0) {
704+
testData.custom_metadata = cucumberCustomMetadata;
705+
}
706+
707+
if (eventType === 'TestRunFinished') {
708+
CustomTagManager.clearTestLevelCustomMetadata(uuid);
709+
}
710+
688711
const uploadData = {
689712
event_type: eventType,
690713
test_run: testData

0 commit comments

Comments
 (0)