Skip to content

Commit bfd230e

Browse files
wenytang-msCopilot
wenytang-ms
and
Copilot
authored
ci: add auto test plan and cases in github actions (#1607)
* feat: add E2E autotest plans and GitHub Action workflow Add 16 YAML test plans covering all wiki Test-Plan.md scenarios: - Basic editing (#1-9): snippets, code actions, imports, rename, new file - Maven / Maven Multimodule / Gradle projects - JDK 25 compatibility (Maven + Gradle) - Single file / Single file without workspace - Debugger, Test Runner, Dependency Viewer - Maven for Java (Resolve Unknown Type) - Java Extension Pack (Configure Classpath) - Fresh Import (Spring Petclinic, auto-clone) GitHub Action (manual trigger): - Runs all test plans on windows-latest - Uploads test-results/ (screenshots + JSON) as artifact - Supports running a single plan via workflow_dispatch input Uses @vscjava/vscode-autotest CLI for execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: adjust GitHub Action checkout paths for CI - Checkout all repos into GITHUB_WORKSPACE (no ../ paths) - Set working-directory to vscode-java-pack for autotest - Upload results from vscode-java-pack/test-results/ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: translate all test plans from Chinese to English Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add JDK 25 setup for java25 e2e test plans The java-gradle-java25 and java-maven-java25 test plans require JDK 25 at 'C:\Program Files\Java\jdk-25', but the pipeline only installed JDK 21. This caused the Language Server to fail to start, timing out on the ls-ready step. - Install JDK 25 via setup-java before JDK 21 - Create a directory junction at the expected path - JDK 21 remains the default JAVA_HOME Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: simplify E2E workflow to clone javaext-autotest directly Instead of installing @vscjava/vscode-autotest globally and maintaining duplicate test plans in this repo, the workflow now: - Clones wenytang-ms/javaext-autotest as a sibling directory - Runs npm ci && npm run build to set up the framework - Executes test plans directly from javaext-autotest/test-plans/ This keeps test plans in a single source of truth (javaext-autotest) and avoids version drift between the npm package and latest changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: update * ci: update * ci: update * refactor: use npm package for autotest CLI and test plans - Install @vscjava/vscode-autotest from npm instead of cloning repo - Test plans now come from npm package (single source of truth) - Support local overlay via test-plans-local/ directory - Matrix CI runs each plan in parallel with rerun support - Analyze job aggregates results with LLM analysis and Job Summary - Delete stale test-plans/ from repo (now in npm package) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: test plans in repo, CLI from npm package - Test plans live in vscode-java-pack repo (source of truth for this project) - CLI tool installed from npm: @vscjava/vscode-autotest - Updated test plans with all latest fixes (debug, resolve-type, etc.) - Removed npm package sync steps from workflow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: update * chore: remove AGENTS.md from PR Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address PR review comments - Replace Invoke-Expression with safe '& autotest @Args' invocation - Fix debugger verify text: stepOver keeps paused, not 'continued' - Fix typo: 'auto-generate' → 'auto-generated' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent a848ef1 commit bfd230e

17 files changed

+513
-313
lines changed

‎.github/workflows/e2e-autotest.yml‎

Lines changed: 212 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,44 @@ on:
88
required: false
99
default: ""
1010
type: string
11+
vsix_urls:
12+
description: "VSIX URLs to install, comma-separated (e.g. https://host/redhat.java-1.42.0.vsix,https://host/vscode-java-debug-0.58.0.vsix)"
13+
required: false
14+
default: ""
15+
type: string
1116

1217
jobs:
18+
# ── Job 1: Discover test plans ──────────────────────────
19+
discover:
20+
if: ${{ inputs.test_plan == '' }}
21+
runs-on: ubuntu-latest
22+
outputs:
23+
matrix: ${{ steps.scan.outputs.matrix }}
24+
steps:
25+
- name: Checkout
26+
uses: actions/checkout@v4
27+
28+
- name: Scan test plans
29+
id: scan
30+
run: |
31+
plans=$(ls test-plans/*.yaml | xargs -I{} basename {} .yaml | grep -v java-fresh-import | jq -R . | jq -sc .)
32+
echo "matrix=$plans" >> "$GITHUB_OUTPUT"
33+
echo "Found plans: $plans"
34+
35+
# ── Job 2: Run each test plan in parallel ───────────────
1336
e2e-test:
37+
if: ${{ inputs.test_plan == '' }}
38+
needs: discover
1439
runs-on: windows-latest
15-
timeout-minutes: 60
40+
timeout-minutes: 30
41+
strategy:
42+
fail-fast: false
43+
matrix:
44+
plan: ${{ fromJson(needs.discover.outputs.matrix) }}
1645

1746
steps:
18-
- name: Checkout vscode-java-pack
47+
- name: Checkout
1948
uses: actions/checkout@v4
20-
with:
21-
path: vscode-java-pack
2249

2350
- name: Checkout vscode-java (test projects)
2451
uses: actions/checkout@v4
@@ -37,7 +64,20 @@ jobs:
3764
with:
3865
node-version: 20
3966

40-
- name: Setup Java
67+
- name: Setup Java 25 (for java25 test plans)
68+
if: contains(matrix.plan, 'java25')
69+
uses: actions/setup-java@v4
70+
with:
71+
distribution: temurin
72+
java-version: 25-ea
73+
74+
- name: Create JDK 25 path
75+
if: contains(matrix.plan, 'java25')
76+
shell: pwsh
77+
run: |
78+
New-Item -ItemType Junction -Path "C:\Program Files\Java\jdk-25" -Target $env:JAVA_HOME
79+
80+
- name: Setup Java 21
4181
uses: actions/setup-java@v4
4282
with:
4383
distribution: temurin
@@ -46,37 +86,179 @@ jobs:
4686
- name: Install autotest CLI
4787
run: npm install -g @vscjava/vscode-autotest
4888

49-
- name: Run test plan(s)
89+
- name: Download VSIX files
90+
if: ${{ inputs.vsix_urls != '' }}
5091
shell: pwsh
51-
working-directory: vscode-java-pack
5292
run: |
53-
$plan = "${{ inputs.test_plan }}"
54-
if ($plan -and $plan -ne "") {
55-
Write-Host "Running: $plan"
56-
autotest run "test-plans/$plan"
57-
} else {
58-
Write-Host "Running all test plans..."
59-
$plans = Get-ChildItem test-plans -Filter "*.yaml" |
60-
Where-Object { $_.Name -ne "java-fresh-import.yaml" } |
61-
Sort-Object Name
62-
$failed = @()
63-
foreach ($p in $plans) {
64-
Write-Host "`n========== $($p.Name) =========="
65-
autotest run "test-plans/$($p.Name)"
66-
if ($LASTEXITCODE -ne 0) { $failed += $p.Name }
67-
}
68-
Write-Host "`n========== Summary =========="
69-
Write-Host "Total: $($plans.Count) Failed: $($failed.Count)"
70-
if ($failed.Count -gt 0) {
71-
Write-Host "Failed: $($failed -join ', ')"
72-
exit 1
73-
}
93+
New-Item -ItemType Directory -Path vsix -Force | Out-Null
94+
$urls = "${{ inputs.vsix_urls }}" -split "," | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne "" }
95+
foreach ($url in $urls) {
96+
$fileName = [System.IO.Path]::GetFileName(($url -split '\?')[0])
97+
Write-Host "Downloading: $url → vsix/$fileName"
98+
Invoke-WebRequest -Uri $url -OutFile "vsix/$fileName" -UseBasicParsing
7499
}
100+
Write-Host "Downloaded VSIX files:"
101+
Get-ChildItem vsix -Filter "*.vsix" | ForEach-Object { Write-Host " $($_.Name) ($([math]::Round($_.Length/1MB, 1)) MB)" }
75102
76-
- name: Upload test results
103+
- name: Run ${{ matrix.plan }}
104+
shell: pwsh
105+
env:
106+
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
107+
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
108+
AZURE_OPENAI_DEPLOYMENT: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
109+
run: |
110+
$autotestArgs = @("run", "test-plans/${{ matrix.plan }}.yaml")
111+
if (Test-Path vsix) {
112+
$vsixFiles = (Get-ChildItem vsix -Filter "*.vsix" | ForEach-Object { $_.FullName }) -join ","
113+
if ($vsixFiles) { $autotestArgs += @("--vsix", $vsixFiles) }
114+
}
115+
Write-Host "Running: autotest $($autotestArgs -join ' ')"
116+
& autotest @autotestArgs
117+
118+
- name: Upload results
119+
if: always()
120+
uses: actions/upload-artifact@v4
121+
with:
122+
name: results-${{ matrix.plan }}
123+
path: test-results/
124+
retention-days: 30
125+
126+
# ── Job 2b: Run a single test plan (when specified) ─────
127+
e2e-single:
128+
if: ${{ inputs.test_plan != '' }}
129+
runs-on: windows-latest
130+
timeout-minutes: 30
131+
132+
steps:
133+
- name: Checkout
134+
uses: actions/checkout@v4
135+
136+
- name: Checkout vscode-java
137+
uses: actions/checkout@v4
138+
with:
139+
repository: redhat-developer/vscode-java
140+
path: vscode-java
141+
142+
- name: Checkout eclipse.jdt.ls
143+
uses: actions/checkout@v4
144+
with:
145+
repository: eclipse-jdtls/eclipse.jdt.ls
146+
path: eclipse.jdt.ls
147+
148+
- name: Setup Node.js
149+
uses: actions/setup-node@v4
150+
with:
151+
node-version: 20
152+
153+
- name: Setup Java 25
154+
if: contains(inputs.test_plan, 'java25')
155+
uses: actions/setup-java@v4
156+
with:
157+
distribution: temurin
158+
java-version: 25-ea
159+
160+
- name: Create JDK 25 path
161+
if: contains(inputs.test_plan, 'java25')
162+
shell: pwsh
163+
run: |
164+
New-Item -ItemType Junction -Path "C:\Program Files\Java\jdk-25" -Target $env:JAVA_HOME
165+
166+
- name: Setup Java 21
167+
uses: actions/setup-java@v4
168+
with:
169+
distribution: temurin
170+
java-version: 21
171+
172+
- name: Install autotest CLI
173+
run: npm install -g @vscjava/vscode-autotest
174+
175+
- name: Download VSIX files
176+
if: ${{ inputs.vsix_urls != '' }}
177+
shell: pwsh
178+
run: |
179+
New-Item -ItemType Directory -Path vsix -Force | Out-Null
180+
$urls = "${{ inputs.vsix_urls }}" -split "," | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne "" }
181+
foreach ($url in $urls) {
182+
$fileName = [System.IO.Path]::GetFileName(($url -split '\?')[0])
183+
Write-Host "Downloading: $url → vsix/$fileName"
184+
Invoke-WebRequest -Uri $url -OutFile "vsix/$fileName" -UseBasicParsing
185+
}
186+
187+
- name: Run ${{ inputs.test_plan }}
188+
shell: pwsh
189+
env:
190+
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
191+
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
192+
AZURE_OPENAI_DEPLOYMENT: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
193+
run: |
194+
$autotestArgs = @("run", "test-plans/${{ inputs.test_plan }}")
195+
if (Test-Path vsix) {
196+
$vsixFiles = (Get-ChildItem vsix -Filter "*.vsix" | ForEach-Object { $_.FullName }) -join ","
197+
if ($vsixFiles) { $autotestArgs += @("--vsix", $vsixFiles) }
198+
}
199+
Write-Host "Running: autotest $($autotestArgs -join ' ')"
200+
& autotest @autotestArgs
201+
202+
- name: Upload results
77203
if: always()
78204
uses: actions/upload-artifact@v4
79205
with:
80206
name: e2e-test-results
81-
path: vscode-java-pack/test-results/
207+
path: test-results/
208+
retention-days: 30
209+
210+
# ── Job 3: Aggregate analysis ───────────────────────────
211+
analyze:
212+
if: ${{ always() && inputs.test_plan == '' }}
213+
needs: e2e-test
214+
runs-on: ubuntu-latest
215+
216+
steps:
217+
- name: Setup Node.js
218+
uses: actions/setup-node@v4
219+
with:
220+
node-version: 20
221+
222+
- name: Install autotest CLI
223+
run: npm install -g @vscjava/vscode-autotest
224+
225+
- name: Download all results
226+
uses: actions/download-artifact@v4
227+
with:
228+
pattern: results-*
229+
path: all-results
230+
merge-multiple: false
231+
232+
- name: Organize results
233+
run: |
234+
mkdir -p test-results
235+
for dir in all-results/results-*/; do
236+
find "$dir" -name "results.json" -exec dirname {} \; | while read d; do
237+
name=$(basename "$d")
238+
cp -r "$d" "test-results/$name"
239+
done
240+
done
241+
echo "Organized results:"
242+
ls test-results/
243+
244+
- name: Analyze results
245+
env:
246+
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
247+
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
248+
AZURE_OPENAI_DEPLOYMENT: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
249+
run: autotest analyze test-results --output test-results
250+
251+
- name: Write Job Summary
252+
if: always()
253+
run: |
254+
if [ -f test-results/summary.md ]; then
255+
cat test-results/summary.md >> "$GITHUB_STEP_SUMMARY"
256+
fi
257+
258+
- name: Upload aggregate results
259+
if: always()
260+
uses: actions/upload-artifact@v4
261+
with:
262+
name: e2e-aggregate-summary
263+
path: test-results/summary.md
82264
retention-days: 30

0 commit comments

Comments
 (0)