Skip to content

Commit c942b47

Browse files
wenytang-msCopilot
andauthored
test: upgrade e2e test framework (#985)
* test: upgrade e2e test framework * test: add screen shot * test: update * ci: update pipeline * ci: update pipeline * test: update * test: update * test: update * test: update * test: update test case * fix: update * test: update * docs: remove agents.md * fix: use context menus for rename/delete, skip native dialog test - Rename/delete commands have 'when: false' in command palette, so executeCommand cannot find them. Use right-click context menu instead, matching the approach used in the old UI test suite. - Skip 'create with no build tools' test because scaffoldSimpleProject() opens a native OS file dialog that Playwright cannot automate. - Add selectContextMenuItem() helper to VscodeOperator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: click menuitem role instead of action-item container The .action-item div is a presentation container; VS Code only handles click events on the inner <a role='menuitem'> anchor. Use getByRole to find and click the correct element, and wait for menu dismissal to confirm the click registered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use keyboard shortcuts instead of context menu clicks Playwright's click() does not reliably trigger actions on VS Code Electron context menu items (the menu stays open). Switch to: 1. Right-click to select tree item and set focusedView 2. Escape to close context menu (returns focus to tree) 3. F2/Delete keyboard shortcut to trigger rename/delete commands Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use dispatchEvent mouseup for context menu items VS Code menus internally handle mouseup events, not click events. Playwright's simulated .click() in Electron does not reliably trigger the mouseup handler. Use dispatchEvent('mouseup') to directly dispatch the event that VS Code's menu system listens for. Previous approaches that failed: - .click() on .action-item container: menu stays open - getByRole('menuitem').click(): menu stays open - keyboard shortcut (F2/Delete): focusedView not set after Escape Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use page.mouse.click with bounding box for context menu Neither .click(), dispatchEvent('mouseup'), nor keyboard shortcuts reliably trigger VS Code Electron context menu items via Playwright. Switch to raw page.mouse.click() at the element's bounding box center coordinates, which sends CDP-level InputDispatchMouseEvent that Electron's native event handling processes correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: context menu click with hover+focused wait, auto-dismiss native dialogs Root causes found and fixed: 1. Context menu click requires hover first to trigger VS Code's menu focus state (.action-item.focused), then click works reliably. 2. redhat.java shows an Electron native dialog (dialog.showMessageBox) for refactoring confirmation on file rename. Playwright Page API cannot interact with native dialogs. Monkey-patch showMessageBox in the Electron main process to auto-return OK. Verified locally: both rename and delete tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: find OK button by label in showMessageBox, handle Refactor Preview On Linux the button order in dialog.showMessageBox differs from Windows, so response:0 selects 'Show Preview' instead of 'OK'. Fix by scanning the buttons array for the 'OK' label. Also handle the Refactor Preview panel (Apply/Discard) as a fallback in case the dialog still enters preview mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: match Delete/Move to Trash in showMessageBox monkey-patch On Linux CI (headless, no recycle bin), the delete confirmation uses 'Delete' or 'Move to Trash' as button labels, not 'OK'. Expand the regex to match all known confirm labels. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: address PR review comments - Remove unused imports in fileOperations.test.ts and libraries.test.ts - Clean up skipped test bodies and dead beforeEach in libraries.test.ts - Fix JSDoc comment in vscodeOperator.ts to reflect actual CSS fallbacks - Increase Playwright timeout from 180s to 240s for LS readiness margin - Remove cliArgs spread from Electron launch (avoid CLI-only flags) - Remove VSIX install from globalSetup (extensionDevelopmentPath suffices) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent abf89fd commit c942b47

File tree

14 files changed

+1031
-77
lines changed

14 files changed

+1031
-77
lines changed

.github/workflows/linuxUI.yml

Lines changed: 9 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -44,35 +44,13 @@ jobs:
4444
- name: Build VSIX file
4545
run: vsce package
4646

47-
- name: UI Test
48-
continue-on-error: true
49-
id: test
50-
run: DISPLAY=:99 npm run test-ui
47+
- name: E2E Test (Playwright)
48+
run: DISPLAY=:99 npm run test-e2e
5149

52-
- name: Retry UI Test 1
53-
continue-on-error: true
54-
if: steps.test.outcome=='failure'
55-
id: retry1
56-
run: |
57-
git reset --hard
58-
git clean -fd
59-
DISPLAY=:99 npm run test-ui
60-
61-
- name: Retry UI Test 2
62-
continue-on-error: true
63-
if: steps.retry1.outcome=='failure'
64-
id: retry2
65-
run: |
66-
git reset --hard
67-
git clean -fd
68-
DISPLAY=:99 npm run test-ui
69-
70-
- name: Set test status
71-
if: ${{ steps.test.outcome=='failure' && steps.retry1.outcome=='failure' && steps.retry2.outcome=='failure' }}
72-
run: |
73-
echo "Tests failed"
74-
exit 1
75-
76-
- name: Print language server Log
77-
if: ${{ failure() }}
78-
run: find ./test-resources/settings/User/workspaceStorage/*/redhat.java/jdt_ws/.metadata/.log -print -exec cat '{}' \;;
50+
- name: Upload test results
51+
if: ${{ always() }}
52+
uses: actions/upload-artifact@v4
53+
with:
54+
name: e2e-results-linux
55+
path: test-results/
56+
retention-days: 7

.github/workflows/windowsUI.yml

Lines changed: 9 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -44,35 +44,13 @@ jobs:
4444
- name: Build VSIX file
4545
run: vsce package
4646

47-
- name: UI Test
48-
continue-on-error: true
49-
id: test
50-
run: npm run test-ui
47+
- name: E2E Test (Playwright)
48+
run: npm run test-e2e
5149

52-
- name: Retry UI Test 1
53-
continue-on-error: true
54-
if: steps.test.outcome=='failure'
55-
id: retry1
56-
run: |
57-
git reset --hard
58-
git clean -fd
59-
npm run test-ui
60-
61-
- name: Retry UI Test 2
62-
continue-on-error: true
63-
if: steps.retry1.outcome=='failure'
64-
id: retry2
65-
run: |
66-
git reset --hard
67-
git clean -fd
68-
npm run test-ui
69-
70-
- name: Set test status
71-
if: ${{ steps.test.outcome=='failure' && steps.retry1.outcome=='failure' && steps.retry2.outcome=='failure' }}
72-
run: |
73-
echo "Tests failed"
74-
exit 1
75-
76-
- name: Print language server Log if job failed
77-
if: ${{ failure() }}
78-
run: Get-ChildItem -Path ./test-resources/settings/User/workspaceStorage/*/redhat.java/jdt_ws/.metadata/.log | cat
50+
- name: Upload test results
51+
if: ${{ always() }}
52+
uses: actions/upload-artifact@v4
53+
with:
54+
name: e2e-results-windows
55+
path: test-results/
56+
retention-days: 7

package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1263,6 +1263,7 @@
12631263
"watch": "webpack --mode development --watch",
12641264
"test": "tsc -p . && webpack --config webpack.config.js --mode development && node ./dist/test/index.js",
12651265
"test-ui": "tsc -p . && webpack --config webpack.config.js --mode development && node ./dist/test/ui/index.js",
1266+
"test-e2e": "npx playwright test --config test/e2e/playwright.config.ts",
12661267
"build-server": "node scripts/buildJdtlsExt.js",
12671268
"vscode:prepublish": "tsc -p ./ && webpack --mode production",
12681269
"tslint": "tslint -t verbose --project tsconfig.json"
@@ -1284,6 +1285,7 @@
12841285
"tslint": "^6.1.3",
12851286
"typescript": "^4.9.4",
12861287
"vscode-extension-tester": "^8.23.0",
1288+
"@playwright/test": "^1.50.0",
12871289
"webpack": "^5.105.0",
12881290
"webpack-cli": "^4.10.0"
12891291
},

test/e2e/fixtures/baseTest.ts

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT license.
3+
4+
/**
5+
* Playwright test fixture that launches VS Code via Electron,
6+
* opens a temporary copy of a test project, and tears everything
7+
* down after the test.
8+
*
9+
* Usage in test files:
10+
*
11+
* import { test, expect } from "../fixtures/baseTest";
12+
*
13+
* test("my test", async ({ page }) => {
14+
* // `page` is a Playwright Page attached to VS Code
15+
* });
16+
*/
17+
18+
import { _electron, test as base, type Page } from "@playwright/test";
19+
import { downloadAndUnzipVSCode } from "@vscode/test-electron";
20+
import * as fs from "fs-extra";
21+
import * as os from "os";
22+
import * as path from "path";
23+
24+
export { expect } from "@playwright/test";
25+
26+
// Root of the extension source tree
27+
const EXTENSION_ROOT = path.join(__dirname, "..", "..", "..");
28+
// Root of the test data projects
29+
const TEST_DATA_ROOT = path.join(EXTENSION_ROOT, "test");
30+
31+
export type TestOptions = {
32+
/** VS Code version to download, default "stable" */
33+
vscodeVersion: string;
34+
/** Relative path under `test/` to the project to open (e.g. "maven") */
35+
testProjectDir: string;
36+
};
37+
38+
type TestFixtures = TestOptions & {
39+
/** Playwright Page connected to the VS Code Electron window */
40+
page: Page;
41+
};
42+
43+
export const test = base.extend<TestFixtures>({
44+
vscodeVersion: [process.env.VSCODE_VERSION || "stable", { option: true }],
45+
testProjectDir: ["maven", { option: true }],
46+
47+
page: async ({ vscodeVersion, testProjectDir }, use, testInfo) => {
48+
// 1. Create a temp directory and copy the test project into it.
49+
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "java-dep-e2e-"));
50+
const projectName = path.basename(testProjectDir);
51+
const projectDir = path.join(tmpDir, projectName);
52+
fs.copySync(path.join(TEST_DATA_ROOT, testProjectDir), projectDir);
53+
54+
// Write VS Code settings to suppress telemetry prompts and notification noise
55+
const vscodeDir = path.join(projectDir, ".vscode");
56+
fs.ensureDirSync(vscodeDir);
57+
const settingsPath = path.join(vscodeDir, "settings.json");
58+
let existingSettings: Record<string, unknown> = {};
59+
if (fs.existsSync(settingsPath)) {
60+
// settings.json may contain JS-style comments (JSONC), strip them before parsing
61+
const raw = fs.readFileSync(settingsPath, "utf-8");
62+
const stripped = raw.replace(/\/\/.*$/gm, "").replace(/\/\*[\s\S]*?\*\//g, "");
63+
try {
64+
existingSettings = JSON.parse(stripped);
65+
} catch {
66+
// If still invalid, start fresh — our injected settings are more important
67+
existingSettings = {};
68+
}
69+
}
70+
const mergedSettings = {
71+
...existingSettings,
72+
"telemetry.telemetryLevel": "off",
73+
"redhat.telemetry.enabled": false,
74+
"workbench.colorTheme": "Default Dark Modern",
75+
"update.mode": "none",
76+
"extensions.ignoreRecommendations": true,
77+
};
78+
fs.writeFileSync(settingsPath, JSON.stringify(mergedSettings, null, 4));
79+
80+
// 2. Resolve VS Code executable.
81+
const vscodePath = await downloadAndUnzipVSCode(vscodeVersion);
82+
// resolveCliArgsFromVSCodeExecutablePath returns CLI-specific args
83+
// (e.g. --ms-enable-electron-run-as-node) that are unsuitable for
84+
// Electron UI launch. Extract only --extensions-dir and --user-data-dir.
85+
const vscodeTestDir = path.join(EXTENSION_ROOT, ".vscode-test");
86+
const extensionsDir = path.join(vscodeTestDir, "extensions");
87+
const userDataDir = path.join(vscodeTestDir, "user-data");
88+
89+
// 3. Launch VS Code as an Electron app.
90+
const electronApp = await _electron.launch({
91+
executablePath: vscodePath,
92+
env: { ...process.env, NODE_ENV: "development" },
93+
args: [
94+
"--no-sandbox",
95+
"--disable-gpu-sandbox",
96+
"--disable-updates",
97+
"--skip-welcome",
98+
"--skip-release-notes",
99+
"--disable-workspace-trust",
100+
"--password-store=basic",
101+
// Suppress notifications that block UI interactions
102+
"--disable-telemetry",
103+
`--extensions-dir=${extensionsDir}`,
104+
`--user-data-dir=${userDataDir}`,
105+
`--extensionDevelopmentPath=${EXTENSION_ROOT}`,
106+
projectDir,
107+
],
108+
});
109+
110+
const page = await electronApp.firstWindow();
111+
112+
// Auto-dismiss Electron native dialogs (e.g. redhat.java refactoring
113+
// confirmation, delete file confirmation). These dialogs are outside
114+
// the renderer DOM and cannot be handled via Playwright Page API.
115+
// Monkey-patch dialog.showMessageBox to find and click the confirm
116+
// button by label, falling back to the first button.
117+
await electronApp.evaluate(({ dialog }) => {
118+
const confirmLabels = /^(OK|Delete|Move to Recycle Bin|Move to Trash)$/i;
119+
dialog.showMessageBox = async (_win: any, opts: any) => {
120+
const options = opts || _win;
121+
const buttons: string[] = options?.buttons || [];
122+
let idx = buttons.findIndex((b: string) => confirmLabels.test(b));
123+
if (idx < 0) idx = 0;
124+
return { response: idx, checkboxChecked: true };
125+
};
126+
dialog.showMessageBoxSync = (_win: any, opts: any) => {
127+
const options = opts || _win;
128+
const buttons: string[] = options?.buttons || [];
129+
let idx = buttons.findIndex((b: string) => confirmLabels.test(b));
130+
if (idx < 0) idx = 0;
131+
return idx;
132+
};
133+
});
134+
135+
// Dismiss any startup notifications/dialogs before handing off to tests
136+
await page.waitForTimeout(3_000);
137+
await dismissAllNotifications(page);
138+
139+
// 4. Optional tracing
140+
if (testInfo.retry > 0 || !process.env.CI) {
141+
await page.context().tracing.start({ screenshots: true, snapshots: true, title: testInfo.title });
142+
}
143+
144+
// ---- hand off to the test ----
145+
await use(page);
146+
147+
// ---- teardown ----
148+
// Save trace on failure/retry
149+
if (testInfo.status !== "passed" || testInfo.retry > 0) {
150+
const tracePath = testInfo.outputPath("trace.zip");
151+
try {
152+
await page.context().tracing.stop({ path: tracePath });
153+
testInfo.attachments.push({ name: "trace", path: tracePath, contentType: "application/zip" });
154+
} catch {
155+
// Tracing may not have been started
156+
}
157+
}
158+
159+
await electronApp.close();
160+
161+
// Clean up temp directory
162+
try {
163+
fs.rmSync(tmpDir, { force: true, recursive: true });
164+
} catch (e) {
165+
console.warn(`Warning: failed to clean up ${tmpDir}: ${e}`);
166+
}
167+
},
168+
});
169+
170+
/**
171+
* Dismiss all VS Code notification toasts (telemetry prompts, theme suggestions, etc.).
172+
* These notifications can steal focus and block Quick Open / Command Palette interactions.
173+
*/
174+
async function dismissAllNotifications(page: Page): Promise<void> {
175+
try {
176+
// Click "Clear All Notifications" if the notification center button is visible
177+
const clearAll = page.locator(".notifications-toasts .codicon-notifications-clear-all, .notification-toast .codicon-close");
178+
let count = await clearAll.count().catch(() => 0);
179+
while (count > 0) {
180+
await clearAll.first().click();
181+
await page.waitForTimeout(500);
182+
count = await clearAll.count().catch(() => 0);
183+
}
184+
185+
// Also try the command palette approach as a fallback
186+
const notificationToasts = page.locator(".notification-toast");
187+
if (await notificationToasts.count().catch(() => 0) > 0) {
188+
// Use keyboard shortcut to clear all notifications
189+
await page.keyboard.press("Control+Shift+P");
190+
const input = page.locator(".quick-input-widget input.input");
191+
if (await input.isVisible({ timeout: 3_000 }).catch(() => false)) {
192+
await input.fill("Notifications: Clear All Notifications");
193+
await page.waitForTimeout(500);
194+
await input.press("Enter");
195+
await page.waitForTimeout(500);
196+
}
197+
}
198+
} catch {
199+
// Best effort
200+
}
201+
}

test/e2e/globalSetup.ts

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT license.
3+
4+
import { downloadAndUnzipVSCode, resolveCliArgsFromVSCodeExecutablePath } from "@vscode/test-electron";
5+
import * as childProcess from "child_process";
6+
7+
/**
8+
* Global setup runs once before all test files.
9+
* It downloads VS Code and installs the redhat.java extension so that
10+
* every test run starts from an identical, pre-provisioned state.
11+
*
12+
* Our own extension is loaded at launch time via --extensionDevelopmentPath
13+
* (see baseTest.ts), so there is no need to install a VSIX here.
14+
*/
15+
export default async function globalSetup(): Promise<void> {
16+
// Download VS Code stable (or the version configured via VSCODE_VERSION env).
17+
const vscodeVersion = process.env.VSCODE_VERSION || "stable";
18+
console.log(`[globalSetup] Downloading VS Code ${vscodeVersion}…`);
19+
const vscodePath = await downloadAndUnzipVSCode(vscodeVersion);
20+
const [cli, ...cliArgs] = resolveCliArgsFromVSCodeExecutablePath(vscodePath);
21+
22+
// On Windows, the CLI is a .cmd batch file which requires shell: true.
23+
const isWindows = process.platform === "win32";
24+
const execOptions: childProcess.ExecFileSyncOptions = {
25+
encoding: "utf-8",
26+
stdio: "inherit",
27+
timeout: 120_000,
28+
shell: isWindows,
29+
};
30+
31+
// Install the Language Support for Java extension from the Marketplace.
32+
console.log("[globalSetup] Installing redhat.java extension…");
33+
childProcess.execFileSync(cli, [...cliArgs, "--install-extension", "redhat.java"], execOptions);
34+
}

test/e2e/playwright.config.ts

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT license.
3+
4+
import { defineConfig } from "@playwright/test";
5+
import * as path from "path";
6+
7+
export default defineConfig({
8+
testDir: path.join(__dirname, "tests"),
9+
reporter: process.env.CI
10+
? [["list"], ["junit", { outputFile: path.join(__dirname, "..", "..", "test-results", "e2e-results.xml") }]]
11+
: "list",
12+
// Java Language Server can take 2-3 minutes to fully index on first run.
13+
timeout: 240_000,
14+
// Run tests sequentially — launching multiple VS Code instances is too resource-heavy.
15+
workers: 1,
16+
// Allow one retry in CI to handle transient environment issues.
17+
retries: process.env.CI ? 1 : 0,
18+
expect: {
19+
timeout: 30_000,
20+
},
21+
globalSetup: path.join(__dirname, "globalSetup.ts"),
22+
use: {
23+
// Automatically take a screenshot when a test fails.
24+
screenshot: "only-on-failure",
25+
// Capture full trace on retry for deep debugging (includes screenshots, DOM snapshots, network).
26+
trace: "on-first-retry",
27+
},
28+
outputDir: path.join(__dirname, "..", "..", "test-results", "e2e"),
29+
});

0 commit comments

Comments
 (0)