Skip to content

Commit 37a2acd

Browse files
nmetulevCopilotJaylyn-BarbeeCopilot
authored
Adding end to end tests for the guides and samples (#351)
## Description Adds end-to-end Pester tests for all 8 guides and samples, running automatically on every PR. These tests exercise the core CLI workflows (`winapp init`, `winapp run`, `winapp pack`, `winapp cert`, `winapp node create-addon`, etc.) across all supported frameworks. This gives us regression coverage so that changes to the CLI — like modifying `winapp init`, package resolution, or the addon template — are validated against real-world workflows before merging. The three bug fixes in this PR were discovered directly by this test coverage. Each sample has a self-contained `test.Tests.ps1` that validates the guide workflow from scratch (Phase 1) and verifies the existing sample code still builds (Phase 2). ### Test Coverage Matrix | Command | cpp | dotnet | electron | flutter | rust | tauri | wpf | packaging | |---------|:---:|:------:|:--------:|:-------:|:----:|:-----:|:---:|:---------:| | `winapp init` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | `winapp restore` | ✅ | | | | | | | | | `winapp run` | ✅ | ✅ | ⚠️| ✅ | ✅ | ✅ | ✅ | | | `winapp pack` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | `winapp sign` | | | | | | | | ✅ | | `winapp cert generate` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | `winapp cert info` | | ✅ | | | ✅ | | ✅ | ✅ | | `winapp manifest generate` | | | | | | | | ✅ | | `manifest add-alias` | ✅ | ✅ | | | ✅ | | | | | `create-debug-identity` | | ✅ | | | | | ✅ | | | `add-electron-debug-identity` | | | ✅ | | | | | | | `node create-addon` (C++) | | | ✅ | | | | | | | `node create-addon` (C#) | | | ✅ | | | | | | | Framework build | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | Sample freshness check | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ⚠️ Tests using --no-launch to ensure identity is properly set up even though the app will not launch using winapp run. ### Infrastructure - New CI workflow `.github/workflows/test-samples.yml` with matrix strategy (8 parallel jobs) - Shared test helpers in `samples/SampleTestHelpers.psm1` - Runner script `scripts/test-samples.ps1` for local execution - Minor guide doc fixes found during test authoring ## Type of Change - 🧪 Tests - 🐛 Bug fix - 🔧 Config/build ## Checklist - [x] New tests added for new functionality (if applicable) - [x] Tested locally on Windows - [x] All CI checks passing ## Notes - Windows App SDK Notification didn't appear for me when going through the notification add on guide. - Windows runners are slow to get picked up and since this PR introduces a lot of tests that need those runners which could be blocking and slow down the development loop. --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jaylyn Barbee <51131738+Jaylyn-Barbee@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent d54b4ff commit 37a2acd

29 files changed

Lines changed: 2326 additions & 761 deletions

File tree

‎.github/workflows/build-package.yml‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -173,34 +173,6 @@ jobs:
173173
});
174174
core.info(`Marked comment ${existing.id} as stale.`);
175175
176-
# E2E test for Electron workflow - runs after build completes
177-
e2e-test:
178-
runs-on: windows-latest
179-
needs: build-and-package
180-
steps:
181-
- name: Checkout
182-
uses: actions/checkout@v5
183-
184-
- name: Enable Windows Developer Mode
185-
run: |
186-
# Registry key to enable Developer Mode
187-
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock" /t REG_DWORD /v "AllowDevelopmentWithoutDevLicense" /d 1 /f
188-
189-
- name: Setup Node.js
190-
uses: actions/setup-node@v5
191-
with:
192-
node-version: '24'
193-
194-
- name: Download npm package artifact
195-
uses: actions/download-artifact@v4
196-
with:
197-
name: npm-package
198-
path: artifacts/npm
199-
200-
- name: Run E2E Electron test
201-
run: |
202-
.\scripts\test-e2e-electron.ps1 -ArtifactsPath "artifacts/npm" -Verbose
203-
204176
# E2E test for winapp ui commands against WinUI 3 sample app
205177
e2e-test-ui:
206178
runs-on: windows-latest

‎.github/workflows/test-samples.yml‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
name: Test Samples & Guides
2+
3+
on:
4+
pull_request:
5+
branches: [ "main" ]
6+
workflow_dispatch:
7+
inputs:
8+
sample:
9+
description: 'Specific sample to test (or "all")'
10+
required: false
11+
default: 'all'
12+
type: choice
13+
options:
14+
- all
15+
- cpp-app
16+
- dotnet-app
17+
- electron
18+
- flutter-app
19+
- packaging-cli
20+
- rust-app
21+
- tauri-app
22+
- wpf-app
23+
24+
permissions:
25+
contents: read
26+
27+
jobs:
28+
# Build npm (and NuGet) package for pull_request and workflow_dispatch events.
29+
# This makes the workflow self-contained — no cross-workflow artifact chaining.
30+
build:
31+
if: github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'
32+
runs-on: windows-latest
33+
steps:
34+
- uses: actions/checkout@v5
35+
- uses: actions/setup-dotnet@v5
36+
with:
37+
dotnet-version: '10.0.x'
38+
- uses: actions/setup-node@v5
39+
with:
40+
node-version: '24'
41+
- name: Build npm package
42+
shell: pwsh
43+
run: .\scripts\build-cli.ps1 -SkipTests -SkipMsix
44+
- uses: actions/upload-artifact@v4
45+
with:
46+
name: npm-package
47+
path: artifacts/*.tgz
48+
- uses: actions/upload-artifact@v4
49+
with:
50+
name: nuget-package
51+
path: artifacts/nuget/*.nupkg
52+
if-no-files-found: ignore
53+
54+
test-sample:
55+
needs: [build]
56+
if: needs.build.result == 'success'
57+
strategy:
58+
fail-fast: false
59+
matrix:
60+
sample: [cpp-app, dotnet-app, electron, flutter-app, packaging-cli, rust-app, tauri-app, wpf-app]
61+
runs-on: windows-latest
62+
name: ${{ matrix.sample }}
63+
64+
steps:
65+
- name: Check if sample should run
66+
id: check
67+
shell: pwsh
68+
run: |
69+
$requested = '${{ github.event.inputs.sample || 'all' }}'
70+
if ($requested -ne 'all' -and $requested -ne '${{ matrix.sample }}') {
71+
echo "skip=true" >> $env:GITHUB_OUTPUT
72+
} else {
73+
echo "skip=false" >> $env:GITHUB_OUTPUT
74+
}
75+
76+
- name: Checkout
77+
if: steps.check.outputs.skip != 'true'
78+
uses: actions/checkout@v5
79+
80+
# Download the npm package artifact built by the `build` job above
81+
- name: Download npm package
82+
if: steps.check.outputs.skip != 'true'
83+
uses: actions/download-artifact@v4
84+
with:
85+
name: npm-package
86+
path: artifacts/npm
87+
88+
# Download NuGet package for .NET samples
89+
- name: Download NuGet package
90+
if: >-
91+
steps.check.outputs.skip != 'true' &&
92+
contains(fromJson('["dotnet-app", "wpf-app"]'), matrix.sample)
93+
uses: actions/download-artifact@v4
94+
with:
95+
name: nuget-package
96+
path: artifacts/nuget
97+
continue-on-error: true
98+
99+
- name: Add local NuGet source
100+
if: >-
101+
steps.check.outputs.skip != 'true' &&
102+
contains(fromJson('["dotnet-app", "wpf-app"]'), matrix.sample)
103+
shell: pwsh
104+
run: |
105+
$nugetPath = "artifacts/nuget"
106+
if (Test-Path $nugetPath) {
107+
$resolvedPath = (Resolve-Path $nugetPath).Path
108+
dotnet nuget add source $resolvedPath --name WinAppLocal
109+
Write-Host "Added local NuGet source: $resolvedPath"
110+
} else {
111+
Write-Warning "No NuGet artifacts found — .NET samples may fail to restore"
112+
}
113+
114+
# --- Toolchain setup (conditional per sample) ---
115+
116+
- name: Setup .NET
117+
if: >-
118+
steps.check.outputs.skip != 'true' &&
119+
contains(fromJson('["dotnet-app", "wpf-app", "packaging-cli", "electron"]'), matrix.sample)
120+
uses: actions/setup-dotnet@v5
121+
with:
122+
dotnet-version: '10.0.x'
123+
124+
- name: Setup Node.js
125+
if: >-
126+
steps.check.outputs.skip != 'true' &&
127+
contains(fromJson('["electron", "tauri-app", "cpp-app", "dotnet-app", "wpf-app", "rust-app", "flutter-app", "packaging-cli"]'), matrix.sample)
128+
uses: actions/setup-node@v5
129+
with:
130+
node-version: '24'
131+
132+
- name: Setup Flutter
133+
if: >-
134+
steps.check.outputs.skip != 'true' &&
135+
matrix.sample == 'flutter-app'
136+
uses: subosito/flutter-action@v2
137+
with:
138+
channel: stable
139+
140+
- name: Setup Rust
141+
if: >-
142+
steps.check.outputs.skip != 'true' &&
143+
contains(fromJson('["rust-app", "tauri-app"]'), matrix.sample)
144+
uses: dtolnay/rust-toolchain@stable
145+
146+
# --- Run the sample's self-contained Pester test ---
147+
148+
- name: Install Pester
149+
if: steps.check.outputs.skip != 'true'
150+
shell: pwsh
151+
run: |
152+
Install-Module -Name Pester -Force -SkipPublisherCheck -Scope CurrentUser -MinimumVersion 5.0
153+
154+
- name: Run ${{ matrix.sample }} test
155+
if: steps.check.outputs.skip != 'true'
156+
shell: pwsh
157+
run: |
158+
$container = New-PesterContainer -Path "samples/${{ matrix.sample }}/test.Tests.ps1" -Data @{
159+
WinappPath = "artifacts/npm"
160+
}
161+
$config = New-PesterConfiguration
162+
$config.Run.Container = $container
163+
$config.Run.Exit = $true
164+
$config.Output.Verbosity = 'Detailed'
165+
$config.TestResult.Enabled = $true
166+
$config.TestResult.OutputPath = "test-results-${{ matrix.sample }}.xml"
167+
$config.TestResult.OutputFormat = 'JUnitXml'
168+
Invoke-Pester -Configuration $config
169+
170+
- name: Upload test results
171+
if: always() && steps.check.outputs.skip != 'true'
172+
uses: actions/upload-artifact@v4
173+
with:
174+
name: test-results-${{ matrix.sample }}
175+
path: test-results-${{ matrix.sample }}.xml
176+
if-no-files-found: ignore
177+
178+
# Summary job to provide a single check status for branch protection
179+
test-samples-result:
180+
if: always()
181+
needs: [build, test-sample]
182+
runs-on: ubuntu-latest
183+
steps:
184+
- name: Check results
185+
run: |
186+
if [ "${{ needs.test-sample.result }}" = "failure" ]; then
187+
echo "::error::One or more sample tests failed"
188+
exit 1
189+
fi
190+
echo "All sample tests passed (or were skipped)"

‎AGENTS.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,47 @@ When adding or changing public facing features, ensure all documentation is also
4747

4848
If a feature is big enough and requires its own docs page, add it under docs\
4949

50+
## Sample & guide testing
51+
52+
Each sample under `samples/` has a self-contained **Pester 5.x** test file (`test.Tests.ps1`) that validates the corresponding guide workflow from scratch (Phase 1) and verifies the existing sample code still builds (Phase 2). Tests share infrastructure via `samples/SampleTestHelpers.psm1`.
53+
54+
### Running sample & guide tests locally
55+
56+
```powershell
57+
# Run all sample tests
58+
.\scripts\test-samples.ps1
59+
60+
# Run a specific sample
61+
.\scripts\test-samples.ps1 -Samples dotnet-app
62+
63+
# Run with a locally built winapp npm tarball (package-npm.ps1 outputs to .\artifacts\)
64+
.\scripts\test-samples.ps1 -WinappPath .\artifacts -Verbose
65+
66+
# Or pass a specific .tgz / a directory containing one (e.g., a CI artifact download)
67+
.\scripts\test-samples.ps1 -WinappPath .\artifacts\npm -Verbose
68+
```
69+
70+
### Writing a new sample & guide test
71+
72+
1. Create `test.Tests.ps1` in the sample directory (Pester naming convention)
73+
2. Use `BeforeDiscovery` for skip logic (prerequisite checks run at discovery time)
74+
3. Import shared helpers in `BeforeAll`: `Import-Module "$PSScriptRoot\..\SampleTestHelpers.psm1" -Force`
75+
4. Accept `$WinappPath` and `$SkipCleanup` parameters via `param()` block
76+
5. Phase 1 (`Context`): from-scratch guide workflow in a temp directory (scaffold, winapp init, build, cert, pack)
77+
6. Phase 2 (`Context`): quick build of existing sample code to verify freshness
78+
7. Add the sample name to the matrix in `.github/workflows/test-samples.yml`
79+
80+
#### Pester conventions for sample tests
81+
82+
- **`BeforeDiscovery`**: Set `$script:skip` using inline `Get-Command` checks (no module import). Pester evaluates `-Skip:$variable` during discovery, before `BeforeAll` runs.
83+
- **`BeforeAll`**: Import `SampleTestHelpers.psm1`, install winapp, create temp directories. Guard with `if ($script:skip) { return }`.
84+
- **`AfterAll`**: Clean up temp directories using `Remove-TempTestDirectory`.
85+
- **`It` blocks**: Use `-Skip:$script:skip` for prerequisite gating. Use Pester `Should` assertions (`Should -Be 0`, `Should -Exist`, `Should -Not -BeNullOrEmpty`). When all setup happens in `BeforeAll` and depends on the prerequisite, you may apply `-Skip:$script:skip` to the enclosing `Context` instead.
86+
- **Shared helpers**: `Invoke-WinappCommand` (throws on failure), `Test-Prerequisite` (returns bool), `New-TempTestDirectory`, `Remove-TempTestDirectory`, `Install-WinappGlobal`.
87+
88+
### CI integration
89+
90+
Sample & guide tests run via `.github/workflows/test-samples.yml` using a GitHub Actions matrix strategy. Each sample runs in its own parallel job after the main build completes. The workflow downloads the npm package artifact from the `Build and Package` workflow. Test results are uploaded as JUnit XML via `Invoke-Pester` with `TestResult` configuration.
5091

5192
## Where to look first
5293

0 commit comments

Comments
 (0)