Skip to content

Commit 49f54a5

Browse files
sstidlstefanstidlffg
sstidl
and
stefanstidlffg
authored
add e2e testing (#777)
* docs(test): add playwright modes test plan * test(e2e): add playwright phase-1 mode coverage * test(e2e): add classic standalone no-server regression * ci(docker): gate image build on playwright e2e * ci(e2e): run on master push and allow manual dispatch --------- Co-authored-by: Stefan Stidl <stefan.stidl@ffg.at>
1 parent 1a45664 commit 49f54a5

16 files changed

Lines changed: 476 additions & 0 deletions

‎.github/workflows/docker-publish.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,30 @@ env:
2424
IMAGE_NAME: ${{ github.repository }}
2525

2626
jobs:
27+
e2e:
28+
runs-on: ubuntu-latest
29+
timeout-minutes: 45
30+
31+
steps:
32+
- name: Checkout
33+
uses: actions/checkout@v6
34+
35+
- name: Setup Node
36+
uses: actions/setup-node@v4
37+
with:
38+
node-version: 20
39+
40+
- name: Install dependencies
41+
run: npm install
42+
43+
- name: Install Playwright Chromium
44+
run: npx playwright install --with-deps chromium
45+
46+
- name: Run Playwright tests
47+
run: npm run test:e2e
48+
2749
build:
50+
needs: e2e
2851
runs-on: ubuntu-latest
2952
strategy:
3053
fail-fast: false

‎.github/workflows/playwright.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Playwright E2E
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
workflow_dispatch:
8+
9+
jobs:
10+
e2e:
11+
runs-on: ubuntu-latest
12+
timeout-minutes: 45
13+
14+
steps:
15+
- name: Checkout
16+
uses: actions/checkout@v4
17+
18+
- name: Setup Node
19+
uses: actions/setup-node@v4
20+
with:
21+
node-version: 20
22+
23+
- name: Install dependencies
24+
run: npm install
25+
26+
- name: Install Playwright Chromium
27+
run: npx playwright install --with-deps chromium
28+
29+
- name: Run Playwright tests
30+
run: npm run test:e2e

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ db-dir/
44
.vscode/
55
node_modules/
66
package-lock.json
7+
playwright-report/
8+
test-results/
79
results/speedtest_telemetry.db
810
results/speedtest_telemetry.db-shm
911
results/speedtest_telemetry.db-wal

‎package.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
"main": "speedtest.js",
66
"scripts": {
77
"test": "echo \"No automated tests configured yet\" && exit 0",
8+
"test:e2e": "playwright test",
9+
"test:e2e:headed": "playwright test --headed",
810
"lint": "eslint speedtest.js speedtest_worker.js",
911
"lint:fix": "eslint --fix speedtest.js speedtest_worker.js",
1012
"format": "prettier --write \"*.js\"",
@@ -37,6 +39,7 @@
3739
},
3840
"homepage": "https://github.com/librespeed/speedtest#readme",
3941
"devDependencies": {
42+
"@playwright/test": "^1.55.0",
4043
"eslint": "^8.57.0",
4144
"prettier": "^3.1.1"
4245
},

‎playwright.config.js‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
const { defineConfig, devices } = require('@playwright/test');
2+
3+
module.exports = defineConfig({
4+
testDir: './tests/e2e',
5+
fullyParallel: true,
6+
forbidOnly: !!process.env.CI,
7+
retries: process.env.CI ? 1 : 0,
8+
workers: process.env.CI ? 2 : undefined,
9+
timeout: 30_000,
10+
expect: {
11+
timeout: 7_500,
12+
},
13+
reporter: process.env.CI ? [['github'], ['html', { open: 'never' }]] : [['list']],
14+
use: {
15+
trace: 'on-first-retry',
16+
screenshot: 'only-on-failure',
17+
video: 'off',
18+
},
19+
globalSetup: require.resolve('./tests/e2e/global-setup.js'),
20+
globalTeardown: require.resolve('./tests/e2e/global-teardown.js'),
21+
projects: [
22+
{
23+
name: 'chromium',
24+
use: { ...devices['Desktop Chrome'] },
25+
},
26+
],
27+
});

‎tests/PLAYWRIGHT_MODES_PLAN.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# Playwright Plan: Test All Runtime and UI Modes
2+
3+
## Objective
4+
Build a deterministic Playwright test suite that validates LibreSpeed behavior across all supported deployment modes and UI design modes, without asserting real network throughput values.
5+
6+
## Modes to Cover
7+
8+
### Docker runtime modes
9+
- `standalone`
10+
- `backend`
11+
- `frontend`
12+
- `dual`
13+
14+
### UI design modes
15+
- Classic (`index-classic.html`)
16+
- Modern (`index-modern.html`)
17+
- Switcher behavior from `index.html`:
18+
- default from `config.json` (`useNewDesign`)
19+
- `?design=new` override
20+
- `?design=old` override
21+
22+
## Test Strategy
23+
24+
### 1. Keep assertions deterministic
25+
Do not assert real bandwidth numbers. Focus on:
26+
- HTTP availability of expected files/endpoints
27+
- correct redirects/switching behavior
28+
- expected UI controls rendered
29+
- expected server-list loading behavior
30+
- ability to initiate/abort flow at UI level
31+
32+
### 2. Separate test types
33+
- **Mode smoke tests** (fast, always-run): verify each runtime mode serves the right surfaces.
34+
- **UI mode tests**: verify classic/modern pages and switcher rules.
35+
- **Optional flow tests** (later): mock `Speedtest` in browser to simulate state changes and verify UI updates.
36+
37+
### 3. Use Docker Compose as the environment contract
38+
Run Playwright against containers started with explicit `MODE` values to mirror production entrypoint behavior.
39+
40+
## Proposed Test Matrix
41+
42+
### A) `standalone`
43+
Expectations:
44+
- `GET /` responds and serves UI (classic by default unless overridden)
45+
- `GET /backend/empty.php`, `GET /backend/garbage.php`, `GET /backend/getIP.php` available
46+
- `GET /results/telemetry.php` reachable (even if telemetry disabled behavior differs)
47+
- `GET /index.html?design=new` resolves to modern page
48+
- `GET /index.html?design=old` resolves to classic page
49+
50+
### B) `backend`
51+
Expectations:
52+
- backend endpoints return success (`/backend/empty.php`, `/backend/garbage.php`, `/backend/getIP.php`)
53+
- tests only assert local backend endpoint contracts in this mode
54+
55+
### C) `frontend`
56+
Expectations:
57+
- UI entrypoint available
58+
- server list loads from `/servers.json` (or `SERVER_LIST_URL` if set)
59+
- backend test endpoints should not be treated as local testpoint contract in this mode
60+
- selecting server and pressing start does not crash UI shell
61+
62+
### D) `dual`
63+
Expectations:
64+
- combines frontend + local backend availability
65+
- UI can load multi-server list
66+
- backend endpoints available locally
67+
68+
## Playwright Architecture
69+
70+
### Files
71+
- `playwright.config.ts`
72+
- `tests/e2e/modes.spec.ts` (runtime-mode smoke)
73+
- `tests/e2e/design-switch.spec.ts` (classic/modern/switch overrides)
74+
- `tests/e2e/helpers/env.ts` (base URLs + mode metadata)
75+
- `tests/e2e/helpers/ui.ts` (shared selectors, start/abort helpers)
76+
77+
### Environment boot
78+
- `docker compose -f tests/docker-compose-playwright.yml up -d --build`
79+
- dedicate one service per runtime mode on separate ports
80+
- for `frontend` and `dual`, mount a stable `servers.json`
81+
82+
### Selector policy
83+
Use role/text selectors anchored on stable labels and IDs already in pages; avoid brittle CSS-path selectors.
84+
85+
## Phased Rollout
86+
87+
### Phase 1 (recommended first PR)
88+
- Add Playwright scaffolding and CI job
89+
- Add smoke coverage for 4 Docker runtime modes
90+
- Add design switch tests (`index.html`, `?design=new`, `?design=old`)
91+
- No full speed measurement assertions
92+
93+
### Phase 2
94+
- Add deterministic UI flow tests with mocked Speedtest state updates
95+
- Validate button states (`Start` -> running -> abort/end)
96+
- Validate result widgets receive simulated values
97+
98+
### Phase 3
99+
- Add telemetry-enabled scenario tests (`TELEMETRY=true`) for link visibility and stats exposure
100+
- Add negative tests (missing/invalid `servers.json` in frontend/dual)
101+
102+
## Risks and Mitigations
103+
- Flaky speed measurements due to host/network variance
104+
- Mitigation: avoid throughput assertions; use mocked state for UI behavior.
105+
- Divergence between local static run and Docker entrypoint behavior
106+
- Mitigation: run all mode tests against Docker services.
107+
- Selector drift between classic and modern UIs
108+
- Mitigation: maintain per-design helper selectors with minimal coupling.
109+
110+
## CI Proposal
111+
- Trigger on PR + main branch
112+
- Build test image once, run mode services in parallel ports
113+
- Playwright retries: `1` in CI, `0` locally
114+
- Upload traces/screenshots on failure only
115+
- Browser scope for v1: Chromium only
116+
117+
## Confirmed Decisions
118+
1. Browser scope for v1: Chromium only.
119+
2. Telemetry checks are deferred to Phase 3.
120+
3. `backend` mode tests assert backend endpoint contracts only.

‎tests/docker-compose-playwright.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
services:
2+
backend-testpoint:
3+
build:
4+
context: ..
5+
dockerfile: Dockerfile
6+
environment:
7+
- MODE=backend
8+
- WEBPORT=8080
9+
10+
standalone:
11+
build:
12+
context: ..
13+
dockerfile: Dockerfile
14+
environment:
15+
- MODE=standalone
16+
- WEBPORT=8080
17+
- USE_NEW_DESIGN=false
18+
ports:
19+
- "18180:8080"
20+
21+
standalone-new:
22+
build:
23+
context: ..
24+
dockerfile: Dockerfile
25+
environment:
26+
- MODE=standalone
27+
- WEBPORT=8080
28+
- USE_NEW_DESIGN=true
29+
ports:
30+
- "18185:8080"
31+
32+
backend:
33+
build:
34+
context: ..
35+
dockerfile: Dockerfile
36+
environment:
37+
- MODE=backend
38+
- WEBPORT=8080
39+
ports:
40+
- "18181:8080"
41+
42+
frontend:
43+
build:
44+
context: ..
45+
dockerfile: Dockerfile
46+
depends_on:
47+
- backend-testpoint
48+
environment:
49+
- MODE=frontend
50+
- WEBPORT=8080
51+
volumes:
52+
- ./e2e/fixtures/servers-frontend.json:/servers.json:ro
53+
ports:
54+
- "18182:8080"
55+
56+
dual:
57+
build:
58+
context: ..
59+
dockerfile: Dockerfile
60+
depends_on:
61+
- backend-testpoint
62+
environment:
63+
- MODE=dual
64+
- WEBPORT=8080
65+
volumes:
66+
- ./e2e/fixtures/servers-dual.json:/servers.json:ro
67+
ports:
68+
- "18183:8080"

‎tests/e2e/classic-standalone-regression.spec.js‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
const { test, expect } = require('@playwright/test');
2+
const { baseUrls } = require('./helpers/env');
3+
const { classicStartButton } = require('./helpers/ui');
4+
5+
test.describe('Classic standalone regression coverage', () => {
6+
test('standalone classic does not get stuck on "No servers available"', async ({ page }) => {
7+
await page.goto(`${baseUrls.standalone}/index-classic.html`);
8+
9+
// In standalone mode, classic UI should show the test wrapper directly.
10+
await expect(page.locator('#testWrapper')).toHaveClass(/visible/, { timeout: 10_000 });
11+
await expect(page.locator('#loading')).toHaveClass(/hidden/, { timeout: 10_000 });
12+
13+
await expect(page.locator('#message')).not.toContainText(/No servers available/i);
14+
await expect(classicStartButton(page)).toBeVisible();
15+
});
16+
});

‎tests/e2e/design-switch.spec.js‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
const { test, expect } = require('@playwright/test');
2+
const { baseUrls } = require('./helpers/env');
3+
const { classicStartButton, modernStartButton } = require('./helpers/ui');
4+
5+
test.describe('Design switch behavior', () => {
6+
test('index.html defaults to classic when useNewDesign=false', async ({ page }) => {
7+
await page.goto(`${baseUrls.standalone}/index.html`);
8+
await expect(page).toHaveURL(/index-classic\.html/);
9+
await expect(classicStartButton(page)).toBeVisible();
10+
});
11+
12+
test('index.html defaults to modern when useNewDesign=true', async ({ page }) => {
13+
await page.goto(`${baseUrls.standaloneNew}/index.html`);
14+
await expect(page).toHaveURL(/index-modern\.html/);
15+
await expect(modernStartButton(page)).toBeVisible();
16+
});
17+
18+
test('query override design=new forces modern', async ({ page }) => {
19+
await page.goto(`${baseUrls.standalone}/index.html?design=new`);
20+
await expect(page).toHaveURL(/index-modern\.html\?design=new/);
21+
await expect(modernStartButton(page)).toBeVisible();
22+
});
23+
24+
test('query override design=old forces classic', async ({ page }) => {
25+
await page.goto(`${baseUrls.standaloneNew}/index.html?design=old`);
26+
await expect(page).toHaveURL(/index-classic\.html\?design=old/);
27+
await expect(classicStartButton(page)).toBeVisible();
28+
});
29+
});

‎tests/e2e/fixtures/servers-dual.json‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
[
2+
{
3+
"name": "Local dual backend",
4+
"server": "/",
5+
"dlURL": "backend/garbage.php",
6+
"ulURL": "backend/empty.php",
7+
"pingURL": "backend/empty.php",
8+
"getIpURL": "backend/getIP.php",
9+
"id": 1
10+
},
11+
{
12+
"name": "External backend testpoint",
13+
"server": "http://backend-testpoint/",
14+
"dlURL": "garbage.php",
15+
"ulURL": "empty.php",
16+
"pingURL": "empty.php",
17+
"getIpURL": "getIP.php",
18+
"id": 2
19+
}
20+
]

0 commit comments

Comments
 (0)