Skip to content

Commit 3858fdd

Browse files
wenytang-mswenytang-ms
authored
test(e2e): add Phase C plans — project create, export jar, delete permanently (#1029)
* test(e2e): add Phase C plans — project create, export jar, delete permanently Adds three new E2E test plans that cover the remaining high-value Java Manager commands not yet exercised by Phase A or Phase B: - test/e2e-plans/java-dep-project-creation.yaml Covers java.project.create — the flagship "Java: Create Java Project..." wizard, exercised on the "No build tools" branch (the only project type whose scaffolding lives in this repo; every other type delegates to a third-party extension). Asserts on-disk presence of the scaffolded template files. - test/e2e-plans/java-dep-export-jar.yaml Covers java.view.package.exportJar — the multi-step Export Jar wizard. Pins java.project.exportJar.targetPath via user settings so the showSaveDialog branch is skipped, then asserts the jar file exists at the configured target path after the Jdtls generation step finishes. - test/e2e-plans/java-dep-delete-permanent.yaml Covers java.view.package.deleteFilePermanently — the non-trash branch of file removal. Has no plain-click UI affordance for regular local files (the context menu and delete keybinding are both gated on !explorerResourceMoveableToTrash), so the command is invoked directly by id with tree-selection fallback, same pattern as removeLibrary in the classpath plan. All three plans pass locally (45/45 new steps). Full suite remains green across 11 plans / 308 steps. * test(e2e): drop flaky LLM verify on expandTreeItem steps The LLM screenshot check on expand-existing-pkg / expand-source-root / expand-project is inherently flaky on already-expanded trees — the recent CI run downgraded a mechanically-successful expand from pass to fail, then immediately contradicted itself in the post-mortem analysis ('was already expanded before the action and remains expanded after'). The deterministic verifyTreeItem steps that follow each expand provide authoritative ground truth: if the expand didn't happen, the child item won't be visible. So the LLM verify lines are pure redundancy with no upside and a non-trivial false-negative rate. * test(e2e): drop flaky LLM verify on select-interface Same flake pattern as e633cd0: the LLM screenshot check is unreliable when a quick-pick advances immediately after a selection. On Windows CI the LLM saw the second wizard page (source-folder prompt) and concluded 'Interface not selected', when in fact the wizard had advanced exactly because Interface was selected. The downstream fillQuickInput steps and final file-creation assertion provide deterministic verification of the whole flow — if Interface hadn't been picked, the file would have been created with the wrong extension or template. * test(e2e): address Copilot review on header comments Two header comments were internally inconsistent with the plan body: 1. java-dep-export-jar.yaml line 21-22 said 'we set it via workspaceSettings' but the setup block actually uses user-level \settings:\. The detailed rationale further down already explains the user-level choice; the upstream summary now matches. 2. java-dep-project-creation.yaml line 35-36 said the template ships 'plus an empty lib/ ensure-dir'. The template directory in the repo contains only README.md, .vscode/settings.json and src/App.java — the empty lib/ is created at runtime by scaffoldSimpleProject via \ se.ensureDir(path.join(projectRoot, 'lib'))\ (projectController.ts:185), not by the template copy. Reworded to make the distinction explicit. Doc-only changes; YAML behaviour and assertions unchanged.
1 parent 4db30ab commit 3858fdd

5 files changed

Lines changed: 488 additions & 5 deletions

‎test/e2e-plans/java-dep-delete-permanent.yaml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# Test Plan: Java Dependency — Permanent Delete
2+
#
3+
# Covers java.view.package.deleteFilePermanently (the "shift+delete" /
4+
# non-trash branch of file removal). The companion command
5+
# java.view.package.moveFileToTrash is already covered by
6+
# java-dep-file-operations.yaml.
7+
#
8+
# Why this is a separate plan
9+
# ───────────────────────────
10+
# The permanent-delete command has no plain-click UI affordance on regular
11+
# local files: the JAVA PROJECTS context-menu entry (package.json:812-820)
12+
# and the `delete` keybinding (package.json:485-490) are both gated on
13+
# `!explorerResourceMoveableToTrash`, so files in `test/maven` only ever
14+
# see "Delete" (move-to-trash). We therefore invoke the command directly
15+
# via id and rely on tree selection to supply the target node — the same
16+
# pattern the classpath plan uses for `removeLibrary`.
17+
#
18+
# The confirmation dialog also differs from moveFileToTrash: the prompt
19+
# says "permanently delete" instead of "delete", and the action button is
20+
# labelled "Delete" instead of "Move to Recycle Bin" (see
21+
# src/explorerCommands/delete.ts line 17 + getInformationMessage).
22+
#
23+
# Usage:
24+
# npx autotest run test/e2e-plans/java-dep-delete-permanent.yaml --vsix <path-to-vsix>
25+
26+
name: "Java Dependency — Permanent Delete"
27+
description: |
28+
Tests the java.view.package.deleteFilePermanently command on a regular
29+
Maven-project file. Invokes the command by id (no UI affordance on local
30+
files), confirms the "permanently delete" dialog, and verifies the file
31+
is gone from disk and from the Java Projects tree.
32+
33+
setup:
34+
extension: "redhat.java"
35+
vscodeVersion: "stable"
36+
workspace: "../maven"
37+
timeout: 180
38+
settings:
39+
java.configuration.checkProjectSettingsExclusions: false
40+
workbench.startupEditor: "none"
41+
42+
steps:
43+
# ── Setup: wait for LS, free sidebar space, focus Java Projects ──
44+
- id: "ls-ready"
45+
action: "waitForLanguageServer"
46+
# No `verify:` — `waitForLanguageServer` is itself the deterministic
47+
# readiness check. The AFTER screenshot may transiently show
48+
# "Java: Building - 0%" which a strict LLM mis-reads as a failure.
49+
timeout: 180
50+
51+
- id: "close-aux-bar"
52+
action: "executeVSCodeCommand workbench.action.closeAuxiliaryBar"
53+
verify: "Auxiliary bar (Chat) closed"
54+
55+
- id: "collapse-outline"
56+
action: "collapseSidebarSection OUTLINE"
57+
58+
- id: "collapse-timeline"
59+
action: "collapseSidebarSection TIMELINE"
60+
61+
- id: "collapse-workspace-root"
62+
action: "collapseWorkspaceRoot"
63+
64+
- id: "focus-java-projects"
65+
action: "executeVSCodeCommand javaProjectExplorer.focus"
66+
verify: "Java Projects view is focused"
67+
68+
- id: "wait-tree-load"
69+
action: "wait 3 seconds"
70+
71+
# ── Reveal App1.java via link-with-editor, then select it ──
72+
# Opening the file makes link-with-editor expand the tree path and
73+
# reveal+select App1 deterministically — much more reliable than
74+
# manual expandTreeItem chains on the virtualised tree.
75+
- id: "open-target-file"
76+
action: "open file App1.java"
77+
waitBefore: 2
78+
79+
- id: "collapse-workspace-root-2"
80+
action: "collapseWorkspaceRoot"
81+
82+
- id: "focus-java-projects-2"
83+
action: "executeVSCodeCommand javaProjectExplorer.focus"
84+
waitBefore: 2
85+
86+
# Click the tree item to guarantee it's the active selection — getCmdNode
87+
# (explorerCommands/utility.ts:38) falls back to `selectedNodes[0]` when
88+
# the command is invoked without a node argument. Linking-with-editor
89+
# already auto-selected App1 when the file was opened, but clicking
90+
# re-asserts the selection deterministically after the focus-java-projects
91+
# round trip.
92+
#
93+
# We intentionally do NOT verify the App1 tree row here. On 1024x768 CI
94+
# displays the row gets virtualised out of view in the brief window
95+
# between `open file` and the click, and an inView:"Java Projects"
96+
# verifyTreeItem then times out — even though the underlying selection
97+
# is set correctly (proven by the subsequent delete succeeding). The
98+
# post-delete `verify-file-gone` + `verify-tree-item-gone` block is the
99+
# authoritative ground truth for whether the right node was targeted.
100+
- id: "select-app1"
101+
action: "click App1 tree item"
102+
waitBefore: 1
103+
104+
# ── Invoke java.view.package.deleteFilePermanently ──
105+
# The handler at views/dependencyExplorer.ts:177 takes `node?: DataNode`.
106+
# With no node arg, getCmdNode uses the current selection (set by the
107+
# `click App1 tree item` step above). DataNode is a class instance with
108+
# methods (.getChildren()) so passing a POJO would crash — selection
109+
# fallback is the only viable path from a smoke-test.
110+
- id: "invoke-delete-permanently"
111+
action: "executeVSCodeCommand java.view.package.deleteFilePermanently"
112+
113+
# `expectConfirmDialog` waits for the dialog and clicks the first
114+
# recognized confirm button (autotest knows "Delete" is one of them, see
115+
# dialogOperations.ts:16). It's the strict variant — throws if no dialog
116+
# appears, surfacing a silently-failed command invocation immediately
117+
# instead of 15s later when verifyFile times out.
118+
- id: "confirm-delete"
119+
action: "expectConfirmDialog"
120+
121+
- id: "wait-after-delete"
122+
action: "wait 5 seconds"
123+
124+
# ── Verify deletion on disk AND in the tree ──
125+
# The disk check is the strongest signal: useTrash=false routes through
126+
# workspace.fs.delete with the OS-level unlink, so a passing verifyFile
127+
# exists:false proves the permanent-delete path actually fired (rather
128+
# than silently downgrading to a no-op or moving to trash).
129+
- id: "verify-file-gone"
130+
action: "wait 1 seconds"
131+
verifyFile:
132+
path: "${workspaceFolder}/src/main/java/com/mycompany/app1/App1.java"
133+
exists: false
134+
timeout: 15
135+
136+
- id: "verify-tree-item-gone"
137+
action: "wait 1 seconds"
138+
# No `verify:` — verifyTreeItem is authoritative.
139+
verifyTreeItem:
140+
name: "App1"
141+
exact: true
142+
visible: false
143+
inView: "Java Projects"
144+
timeout: 15

‎test/e2e-plans/java-dep-export-jar.yaml‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# Test Plan: Java Dependency — Export Jar
2+
#
3+
# Covers java.view.package.exportJar — the multi-step wizard that builds a
4+
# runnable jar from a Java project. The command is contributed both as the
5+
# title-bar `$(export)` icon on the JAVA PROJECTS workspace-root node
6+
# (package.json:858-861, group=inline) and as a context-menu entry, but we
7+
# invoke it directly by id: the inline icon is rendered against the
8+
# `java:workspace` viewItem (not the Maven project node `my-app`), which
9+
# is fragile to locate by name, and ResolveJavaProject auto-resolves when
10+
# there's exactly one project in the workspace — so direct invocation
11+
# bypasses one wizard step that's not the command's responsibility.
12+
#
13+
# Wizard step machine (BuildArtifactTaskProvider.ts:284 `createJarFile`):
14+
# 1. ResolveJavaProject → auto when single-project; quick-pick otherwise
15+
# 2. ResolveMainClass → quick-pick of main classes + "<without main class>"
16+
# 3. GenerateJar
17+
# a. generateClasspaths → multi-select quick-pick if >1 dependency item
18+
# b. showSaveDialog → only if outputPath === "" (skipped when
19+
# java.project.exportJar.targetPath is set
20+
# to a non-empty value; we set it via a
21+
# user-level `settings:` block to keep the
22+
# output filename deterministic — see the
23+
# setup section below for why user-level
24+
# and not workspaceSettings)
25+
# c. Jdtls.exportJar → writes the jar file
26+
#
27+
# Verification strategy
28+
# ─────────────────────
29+
# The jar is written to a known absolute path inside the workspace
30+
# (`output.jar`). We assert with `verifyFile exists: true` — the strongest
31+
# possible signal that the full wizard completed end-to-end, not just that
32+
# the command was dispatched. The Jdtls export runs in a hidden Pseudoterminal
33+
# (BuildArtifactTaskProvider line 91: `presentationOptions.reveal = Never`),
34+
# so there is no terminal text to inspect; the file on disk is the only
35+
# unambiguous post-condition.
36+
#
37+
# Usage:
38+
# npx autotest run test/e2e-plans/java-dep-export-jar.yaml --vsix <path-to-vsix>
39+
40+
name: "Java Dependency — Export Jar"
41+
description: |
42+
Exercises the multi-step Export Jar wizard end-to-end on the maven
43+
fixture: triggers the command, picks the main class, accepts the default
44+
classpath element selection, and verifies that the resulting jar file
45+
exists at the configured target path.
46+
47+
setup:
48+
extension: "redhat.java"
49+
vscodeVersion: "stable"
50+
workspace: "../maven"
51+
# Bumped above the standard 180s — the wizard runs a full workspace build
52+
# (await buildWorkspace() in executeExportJarTask) before the first
53+
# quick-pick appears, which on a cold JDT-LS warmup commonly takes 60-90s.
54+
timeout: 360
55+
settings:
56+
java.configuration.checkProjectSettingsExclusions: false
57+
workbench.startupEditor: "none"
58+
# Pinning java.project.exportJar.targetPath to an absolute deterministic
59+
# path bypasses the showSaveDialog branch in GenerateJarExecutor
60+
# (BuildArtifactTaskProvider lines 240-244 short-circuit only when
61+
# outputPath === ""), and lets the verifyFile assertion target a stable
62+
# location regardless of how the fixture's worktree is named.
63+
#
64+
# We use user-level `settings:` (not `workspaceSettings:`) because
65+
# autotest's workspaceSettings merge uses JSON.parse on the existing
66+
# `.vscode/settings.json`, and the maven fixture's settings.json
67+
# contains JSONC `//` comments which fail JSON.parse. User settings
68+
# are written fresh each run (user-data-dir is wiped on launch) so
69+
# there is no JSONC merge hazard. The Settings.getExportJarTargetPath
70+
# config read is unscoped, so user-level setting takes effect identically.
71+
java.project.exportJar.targetPath: "${workspaceFolder}/output.jar"
72+
73+
steps:
74+
# ── Setup: wait for LS, free sidebar space, focus Java Projects ──
75+
- id: "ls-ready"
76+
action: "waitForLanguageServer"
77+
# No `verify:` — `waitForLanguageServer` is itself the deterministic
78+
# readiness check. The AFTER screenshot may transiently show
79+
# "Java: Building - 0%" which a strict LLM mis-reads as a failure.
80+
timeout: 180
81+
82+
- id: "close-aux-bar"
83+
action: "executeVSCodeCommand workbench.action.closeAuxiliaryBar"
84+
verify: "Auxiliary bar (Chat) closed"
85+
86+
- id: "collapse-outline"
87+
action: "collapseSidebarSection OUTLINE"
88+
89+
- id: "collapse-timeline"
90+
action: "collapseSidebarSection TIMELINE"
91+
92+
- id: "collapse-workspace-root"
93+
action: "collapseWorkspaceRoot"
94+
95+
- id: "focus-java-projects"
96+
action: "executeVSCodeCommand javaProjectExplorer.focus"
97+
verify: "Java Projects view is focused"
98+
99+
- id: "wait-tree-load"
100+
action: "wait 3 seconds"
101+
102+
# ── Trigger the Export Jar wizard ──
103+
# Direct command invocation avoids the brittle inline-icon-on-workspace-
104+
# node click path (the `$(export)` icon is contributed against viewItem
105+
# `java:workspace`, not against the Maven project node `my-app`, and
106+
# locating workspace-level inline icons in autotest is fragile).
107+
# ResolveJavaProject auto-resolves for single-project workspaces.
108+
- id: "invoke-export-jar"
109+
action: "executeVSCodeCommand java.view.package.exportJar"
110+
111+
# The wizard first triggers a full workspace build (await buildWorkspace()
112+
# in executeExportJarTask) before any UI appears. On a cold JDT-LS this
113+
# typically takes 30-60s; the Jdtls.getMainClasses fetch then needs
114+
# another 2-5s before the first quick-pick is shown. Wait generously.
115+
- id: "wait-build-complete"
116+
action: "wait 60 seconds"
117+
118+
# ── Step 2: pick the main class ──
119+
# The maven fixture has a single class with a `public static void main`
120+
# entry point: com.mycompany.app.App. App1 has no main, so the only
121+
# quick-pick options are "App" and "<without main class>". We pick "App"
122+
# explicitly so the resulting jar is runnable.
123+
- id: "pick-main-class"
124+
action: "select App option"
125+
# No `verify:` — selectPaletteOption is the action and verification in
126+
# one. The quick-pick closes on selection; the AFTER screenshot may
127+
# already show the next quick-pick (classpath elements), which a
128+
# strict LLM could mis-read as "App selection was lost".
129+
130+
- id: "wait-after-main-class"
131+
action: "wait 5 seconds"
132+
133+
# ── Step 3a: accept the pre-selected classpath elements ──
134+
# GenerateJarExecutor.generateClasspaths builds a list of runtime / test
135+
# output folders + dependency artifacts, pre-selects everything tagged
136+
# `runtime`, and shows a multi-select QuickPick. For the maven fixture
137+
# this includes target/classes plus any resolved JUnit jars under test
138+
# scope. The pre-selection is the sane default — accepting it gives a
139+
# runnable jar without needing to compute which items to check.
140+
#
141+
# `confirmQuickInput` presses Enter on the open quick-pick widget without
142+
# typing anything, which leaves selections untouched and submits.
143+
- id: "accept-classpath-elements"
144+
action: "confirmQuickInput"
145+
146+
# ── Step 3c: wait for Jdtls to generate the jar ──
147+
# Jdtls.exportJar writes the jar via a JDT LSP request. The custom
148+
# pseudoterminal stays hidden (presentationOptions.reveal=Never), so
149+
# there is no terminal-text signal — we just wait long enough for the
150+
# filesystem write to settle.
151+
- id: "wait-jar-generated"
152+
action: "wait 30 seconds"
153+
154+
# ── Verification: jar file exists at the configured target path ──
155+
# This is the strongest possible end-to-end check: the file appearing
156+
# on disk proves ResolveJavaProject + ResolveMainClass + generateClasspaths
157+
# + Jdtls.exportJar all completed in order. A failure here pinpoints
158+
# the wizard breaking; a pass means the full happy path worked.
159+
- id: "verify-jar-created"
160+
action: "wait 1 seconds"
161+
verifyFile:
162+
path: "${workspaceFolder}/output.jar"
163+
exists: true
164+
timeout: 60

‎test/e2e-plans/java-dep-new-types.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,10 @@ steps:
7878

7979
- id: "select-interface"
8080
action: "select Interface option"
81-
verify: "Interface selected"
81+
# No `verify:` — LLM screenshot check is flaky when the quick-pick
82+
# advances immediately after selection (LLM sees the next page and
83+
# mistakes the advance for "selection not visible"). The downstream
84+
# fillQuickInput / file creation provides authoritative verification.
8285

8386
- id: "select-source-folder-1"
8487
action: "select src/main/java option"

0 commit comments

Comments
 (0)