Skip to content

Commit 11a0645

Browse files
wenytang-mswenytang-ms
authored
ci(e2e): unify Linux + Windows UI tests into one parallel workflow (#1027)
* test: phase A coverage — newJavaRecord, clean.workspace, project.update Adds three more vscode-java-dependency commands to the E2E plans, each exercised through its documented user-visible UI surface: * java-dep-new-types.yaml - Adds a 7th cycle for newJavaRecord between Annotation and Abstract Class. The Record option is filtered out for projects below Java 16 (see JavaType.getDisplayNames in src/explorerCommands/new.ts); the fixture pom.xml is Java 17 so the option renders. Plan goes from 64 → 75 steps. * java-dep-build-lifecycle.yaml - Adds Test 6: java.project.update via the Maven submenu on the project context menu (the command's only UI mount — no palette, no toolbar). Uses the new 'contextMenuSubmenu' autotest action that hovers a submenu trigger then clicks a leaf inside the nested menu. - Adds Test 7: java.project.clean.workspace via the view-title overflow menu. The command may or may not raise a confirmation dialog depending on JDT.LS state; we use the tolerant tryClickDialogButton variant so the step passes either way (cancelling the dialog if shown, avoiding the destructive VS Code reload). Plan goes from 18 → 26 steps. Requires @vscjava/vscode-autotest >= 0.7.11 (adds contextMenuSubmenu). Locally verified: java-dep-new-types.yaml 75/75 passing java-dep-build-lifecycle.yaml 26/26 passing * ci(e2e): parallelize UI tests across plans with matrix strategy Refactor linuxUI.yml and windowsUI.yml from a single sequential job running autotest run-all into a 5-job parallel pipeline modelled after vscode-java-pack/.github/workflows/e2e-autotest.yml: 1. build-<os>: produce VSIX once, upload as artifact 2. discover-plans: scan test/e2e-plans/*.yaml -> matrix output 3. e2e-<os> (matrix per plan): download VSIX, run a single plan 4. analyze-<os>: aggregate results.json across plans into summary.md 5. linux-ui / windows-ui: gate job preserving the original required status-check name so existing branch protection keeps working Per-job timeout drops to 25 min and individual plan failures surface independently. Sub-screenshots from each plan are preserved per-plan in their own artifact for easier triage. * ci(e2e): consolidate Linux + Windows UI workflows into one e2eUI.yml Merges the per-OS workflows (linuxUI.yml, windowsUI.yml) into a single e2eUI.yml with a unified entry point and a single shared summary. Structure: - discover-plans (matrix output) - build-linux / build-windows (VSIX artifacts) - e2e-linux / e2e-windows (matrix) (one job per (OS, plan)) - analyze (one combined summary across both OSes, plan dirs prefixed by OS for clarity) - Linux-UI / Windows-UI (gate jobs) (preserve required status-check names) * ci(e2e): fold OS into the matrix; extract Lint/Checkstyle to its own job Reworks the e2e workflow so {os, plan} is a single cartesian matrix instead of two parallel jobs (e2e-linux, e2e-windows): - lint tslint + checkstyle (ubuntu, OS-agnostic) - discover-plans - build (matrix on os) - e2e (matrix on {os, plan}) cartesian 2 x N parallel jobs - analyze one combined summary across both OSes - Linux-UI / Windows-UI gates preserve required-check names Inspired by vscode-java-pack/e2e-autotest.yml extended with an OS dimension. OS-specific steps (Xvfb, shell flavor) are guarded by runner.os conditionals. * ci(e2e): drop redundant Linux-UI / Windows-UI gate jobs Each matrix cell already surfaces as its own PR check, so a separate gate is unnecessary. Branch protection should be updated to require the individual matrix checks (or the 'E2E Summary' job) directly. * ci(e2e): split e2e back into per-OS pipelines for true parallelism Folding OS into the e2e matrix forced every cell to wait for the entire build matrix to finish (Linux + Windows), so a slow Windows build delayed Linux e2e jobs by ~2.5 min unnecessarily (Copilot review feedback). Split back into independent per-OS pipelines: build-linux -> e2e-linux (matrix on plan) build-windows -> e2e-windows (matrix on plan) Both pipelines feed into the shared discover-plans + analyze jobs, so the unified entry point and combined summary are preserved.
1 parent 7be414f commit 11a0645

5 files changed

Lines changed: 442 additions & 171 deletions

File tree

‎.github/workflows/e2eUI.yml‎

Lines changed: 331 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,331 @@
1+
name: E2E UI Tests
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
# Split-pipeline E2E UI workflow.
10+
#
11+
# lint → tslint + checkstyle (ubuntu, OS-agnostic)
12+
# discover-plans → emits a matrix of test-plan basenames
13+
#
14+
# build-linux ─┐
15+
# e2e-linux (×plan) ┤
16+
# ├──→ analyze → unified summary covering both OSes
17+
# build-windows ─┤
18+
# e2e-windows (×plan)┘
19+
#
20+
# Per-OS pipelines run completely independently: Linux e2e jobs do NOT
21+
# wait for the Windows VSIX build (and vice versa), so a slow Windows
22+
# build cannot delay the start of Linux e2e plans. Each matrix cell
23+
# surfaces as its own PR check, so failures are visible without an
24+
# extra gate job.
25+
#
26+
# Inspired by vscode-java-pack/.github/workflows/e2e-autotest.yml.
27+
28+
jobs:
29+
# ── Lint + Checkstyle (OS-agnostic) ─────────────────────
30+
lint:
31+
name: Lint & Checkstyle
32+
runs-on: ubuntu-latest
33+
timeout-minutes: 10
34+
steps:
35+
- uses: actions/checkout@v4
36+
37+
- name: Set up JDK 21
38+
uses: actions/setup-java@v4
39+
with:
40+
java-version: '21'
41+
distribution: 'temurin'
42+
43+
- name: Setup Node.js environment
44+
uses: actions/setup-node@v4
45+
with:
46+
node-version: 20
47+
48+
- name: Install Node.js modules
49+
run: npm install
50+
51+
- name: Lint
52+
run: npm run tslint
53+
54+
- name: Checkstyle
55+
working-directory: ./jdtls.ext
56+
run: ./mvnw checkstyle:check
57+
58+
# ── Discover test plans ─────────────────────────────────
59+
discover-plans:
60+
name: Discover E2E Plans
61+
runs-on: ubuntu-latest
62+
outputs:
63+
matrix: ${{ steps.scan.outputs.matrix }}
64+
steps:
65+
- uses: actions/checkout@v4
66+
67+
- name: Scan test plans
68+
id: scan
69+
shell: bash
70+
run: |
71+
plans=$(ls test/e2e-plans/*.yaml | xargs -n1 basename | sed 's/\.yaml$//' | jq -R . | jq -sc .)
72+
echo "matrix=$plans" >> "$GITHUB_OUTPUT"
73+
echo "Found plans: $plans"
74+
75+
# ── Build VSIX (Linux) ──────────────────────────────────
76+
build-linux:
77+
name: Build VSIX (Linux)
78+
runs-on: ubuntu-latest
79+
timeout-minutes: 20
80+
steps:
81+
- uses: actions/checkout@v4
82+
83+
- name: Set up JDK 21
84+
uses: actions/setup-java@v4
85+
with:
86+
java-version: '21'
87+
distribution: 'temurin'
88+
89+
- name: Setup Node.js environment
90+
uses: actions/setup-node@v4
91+
with:
92+
node-version: 20
93+
94+
- name: Install Node.js modules
95+
run: npm install
96+
97+
- name: Install VSCE
98+
run: npm install -g @vscode/vsce
99+
100+
- name: Build OSGi bundle
101+
run: npm run build-server
102+
103+
- name: Build VSIX file
104+
run: vsce package -o vscode-java-dependency.vsix
105+
106+
- name: Upload VSIX artifact
107+
uses: actions/upload-artifact@v4
108+
with:
109+
name: vsix-linux
110+
path: vscode-java-dependency.vsix
111+
retention-days: 1
112+
113+
# ── Build VSIX (Windows) ────────────────────────────────
114+
build-windows:
115+
name: Build VSIX (Windows)
116+
runs-on: windows-latest
117+
timeout-minutes: 20
118+
steps:
119+
- uses: actions/checkout@v4
120+
121+
- name: Set up JDK 21
122+
uses: actions/setup-java@v4
123+
with:
124+
java-version: '21'
125+
distribution: 'temurin'
126+
127+
- name: Setup Node.js environment
128+
uses: actions/setup-node@v4
129+
with:
130+
node-version: 20
131+
132+
- name: Install Node.js modules
133+
run: npm install
134+
135+
- name: Install VSCE
136+
run: npm install -g @vscode/vsce
137+
138+
- name: Build OSGi bundle
139+
run: npm run build-server
140+
141+
- name: Build VSIX file
142+
run: vsce package -o vscode-java-dependency.vsix
143+
144+
- name: Upload VSIX artifact
145+
uses: actions/upload-artifact@v4
146+
with:
147+
name: vsix-windows
148+
path: vscode-java-dependency.vsix
149+
retention-days: 1
150+
151+
# ── E2E plans (Linux) — depends only on Linux build ─────
152+
e2e-linux:
153+
name: E2E Linux (${{ matrix.plan }})
154+
needs: [ build-linux, discover-plans ]
155+
runs-on: ubuntu-latest
156+
timeout-minutes: 25
157+
strategy:
158+
fail-fast: false
159+
matrix:
160+
plan: ${{ fromJson(needs.discover-plans.outputs.matrix) }}
161+
162+
steps:
163+
- uses: actions/checkout@v4
164+
165+
- name: Setup Build Environment (Xvfb)
166+
run: |
167+
sudo apt-get update
168+
sudo apt-get install -y libxkbfile-dev pkg-config libsecret-1-dev libxss1 dbus xvfb libgtk-3-0 libgbm1
169+
# Use 1920x1080 so the Java Projects view (rendered inside the Explorer
170+
# sidebar) gets enough vertical space. With 1024x768 the sticky
171+
# pane-header overlapped tree rows and intercepted click events.
172+
sudo /usr/bin/Xvfb :99 -screen 0 1920x1080x24 > /dev/null 2>&1 &
173+
sleep 3
174+
175+
- name: Set up JDK 21
176+
uses: actions/setup-java@v4
177+
with:
178+
java-version: '21'
179+
distribution: 'temurin'
180+
181+
- name: Setup Node.js environment
182+
uses: actions/setup-node@v4
183+
with:
184+
node-version: 20
185+
186+
- name: Setup autotest
187+
run: npm install -g @vscjava/vscode-autotest
188+
189+
- name: Download VSIX artifact
190+
uses: actions/download-artifact@v4
191+
with:
192+
name: vsix-linux
193+
path: .
194+
195+
- name: E2E Test — ${{ matrix.plan }}
196+
shell: bash
197+
env:
198+
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
199+
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
200+
AZURE_OPENAI_DEPLOYMENT: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
201+
run: |
202+
DISPLAY=:99 autotest run "test/e2e-plans/${{ matrix.plan }}.yaml" \
203+
--vsix "$(pwd)/vscode-java-dependency.vsix" \
204+
--output "test-results/${{ matrix.plan }}"
205+
206+
- name: Upload test results
207+
if: ${{ always() }}
208+
uses: actions/upload-artifact@v4
209+
with:
210+
name: e2e-results-linux-${{ matrix.plan }}
211+
path: test-results/
212+
retention-days: 7
213+
214+
# ── E2E plans (Windows) — depends only on Windows build ─
215+
e2e-windows:
216+
name: E2E Windows (${{ matrix.plan }})
217+
needs: [ build-windows, discover-plans ]
218+
runs-on: windows-latest
219+
timeout-minutes: 25
220+
strategy:
221+
fail-fast: false
222+
matrix:
223+
plan: ${{ fromJson(needs.discover-plans.outputs.matrix) }}
224+
225+
steps:
226+
- uses: actions/checkout@v4
227+
228+
- name: Set up JDK 21
229+
uses: actions/setup-java@v4
230+
with:
231+
java-version: '21'
232+
distribution: 'temurin'
233+
234+
- name: Setup Node.js environment
235+
uses: actions/setup-node@v4
236+
with:
237+
node-version: 20
238+
239+
- name: Setup autotest
240+
run: npm install -g @vscjava/vscode-autotest
241+
242+
- name: Download VSIX artifact
243+
uses: actions/download-artifact@v4
244+
with:
245+
name: vsix-windows
246+
path: .
247+
248+
- name: E2E Test — ${{ matrix.plan }}
249+
shell: pwsh
250+
env:
251+
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
252+
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
253+
AZURE_OPENAI_DEPLOYMENT: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
254+
run: |
255+
autotest run "test/e2e-plans/${{ matrix.plan }}.yaml" --vsix "$((Get-Location).Path)\vscode-java-dependency.vsix" --output "test-results\${{ matrix.plan }}"
256+
257+
- name: Upload test results
258+
if: ${{ always() }}
259+
uses: actions/upload-artifact@v4
260+
with:
261+
name: e2e-results-windows-${{ matrix.plan }}
262+
path: test-results/
263+
retention-days: 7
264+
265+
# ── Unified analysis across both OSes ───────────────────
266+
analyze:
267+
name: E2E Summary
268+
needs: [ e2e-linux, e2e-windows ]
269+
if: ${{ always() }}
270+
runs-on: ubuntu-latest
271+
steps:
272+
- name: Setup Node.js environment
273+
uses: actions/setup-node@v4
274+
with:
275+
node-version: 20
276+
277+
- name: Setup autotest
278+
run: npm install -g @vscjava/vscode-autotest
279+
280+
- name: Download all plan results
281+
uses: actions/download-artifact@v4
282+
with:
283+
pattern: e2e-results-*
284+
path: all-results
285+
merge-multiple: false
286+
287+
- name: Organize results (prefix each plan by OS)
288+
shell: bash
289+
run: |
290+
mkdir -p test-results
291+
for dir in all-results/e2e-results-linux-*/; do
292+
[ -d "$dir" ] || continue
293+
find "$dir" -name "results.json" -exec dirname {} \; | while read d; do
294+
name=$(basename "$d")
295+
mkdir -p "test-results/linux-$name"
296+
cp -r "$d"/. "test-results/linux-$name"/
297+
done
298+
done
299+
for dir in all-results/e2e-results-windows-*/; do
300+
[ -d "$dir" ] || continue
301+
find "$dir" -name "results.json" -exec dirname {} \; | while read d; do
302+
name=$(basename "$d")
303+
mkdir -p "test-results/windows-$name"
304+
cp -r "$d"/. "test-results/windows-$name"/
305+
done
306+
done
307+
echo "Organized plan result directories:"
308+
ls test-results/ || true
309+
310+
- name: Analyze results
311+
env:
312+
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
313+
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
314+
AZURE_OPENAI_DEPLOYMENT: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
315+
run: autotest analyze test-results --output test-results
316+
317+
- name: Write Job Summary
318+
if: always()
319+
shell: bash
320+
run: |
321+
if [ -f test-results/summary.md ]; then
322+
cat test-results/summary.md >> "$GITHUB_STEP_SUMMARY"
323+
fi
324+
325+
- name: Upload aggregate summary
326+
if: always()
327+
uses: actions/upload-artifact@v4
328+
with:
329+
name: e2e-aggregate-summary
330+
path: test-results/summary.md
331+
retention-days: 30

0 commit comments

Comments
 (0)