Skip to content

Commit 9e91d62

Browse files
apstenku123apstenku123
committed
feat(e2e-e6): GitHub Actions workflow + matrix report aggregator
Stage E-6 of the E2E Coverage Matrix epic (cppmega-mlx-pa3.7) — final ticket. Closes the epic. .github/workflows/e2e-matrix.yml: - preset-matrix job (ubuntu, every push/PR): runs the 912-cell smoke matrix sharded 1/4..4/4 in parallel jobs (~6 min wall-clock each). - specialised job (ubuntu, every push/PR): canvas smoke + 4 specialised spec files (~3 min). - mini-train-matrix job (macos-latest, nightly cron + workflow_dispatch only): the 192-cell real-mlx train matrix. macOS-only because mlx has no native ubuntu wheel; ubuntu CI skips it cleanly. - matrix-report job (always runs): downloads every shard's artefacts, invokes tools/build_e2e_matrix_report.py, pipes the markdown into the GitHub Step Summary and uploads it as a standalone artefact. - All shards upload test-results/, screenshots/, logs/ (3-7 day retention). tools/build_e2e_matrix_report.py: - Parses .last-run.json from each Playwright shard, counts failed + flaky retries + screenshots, emits a markdown report with the matrix-size legend (912 + 192 + 25 + 6) + screenshot buckets + xfail notes (kimi_linear / qwen3_next / GUI z-index issues). - Robust to missing artefacts (emits stub report so the CI step never fails just because of an empty matrix). Tests (+11 new pytest, 2150 total): - workflow file exists + has push/PR/cron/dispatch triggers - preset-matrix sharded 4-way with --shard= - artifacts uploaded - mini-train gated to macOS + schedule|workflow_dispatch - matrix-report step invokes the script + writes to GITHUB_STEP_SUMMARY - report script: stub on missing artefacts, aggregates .last-run.json, multiple shards, matrix-size documentation present in output. Closes cppmega-mlx-pa3.7 and the cppmega-mlx-pa3 epic.
1 parent ebc9149 commit 9e91d62

3 files changed

Lines changed: 507 additions & 0 deletions

File tree

‎.github/workflows/e2e-matrix.yml‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
name: E2E Coverage Matrix
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths:
7+
- "cppmega_v4/**"
8+
- "cppmega_mlx/**"
9+
- "vbgui/**"
10+
- "tests/fixtures/build_e2e_matrix.py"
11+
- "tests/fixtures/tokenizers/**"
12+
- "tests/fixtures/parquet/**"
13+
- ".github/workflows/e2e-matrix.yml"
14+
pull_request:
15+
branches: [main]
16+
paths:
17+
- "cppmega_v4/**"
18+
- "vbgui/**"
19+
- "tests/fixtures/build_e2e_matrix.py"
20+
schedule:
21+
# Nightly full run — includes the mini-train matrix on macOS.
22+
- cron: "0 4 * * *"
23+
workflow_dispatch:
24+
25+
concurrency:
26+
group: e2e-matrix-${{ github.ref }}
27+
cancel-in-progress: true
28+
29+
jobs:
30+
# On every push/PR: smoke matrix (912 cells, ubuntu, no real MLX
31+
# training — backend just verifies). Quick gate, ~10 min.
32+
preset-matrix:
33+
name: Preset matrix (smoke, ubuntu)
34+
runs-on: ubuntu-latest
35+
timeout-minutes: 25
36+
strategy:
37+
fail-fast: false
38+
matrix:
39+
shard: [1, 2, 3, 4]
40+
steps:
41+
- uses: actions/checkout@v4
42+
- uses: actions/setup-node@v4
43+
with:
44+
node-version: "22"
45+
cache: "npm"
46+
cache-dependency-path: vbgui/package-lock.json
47+
- uses: actions/setup-python@v5
48+
with:
49+
python-version: "3.13"
50+
- name: Python deps
51+
run: |
52+
python -m pip install --upgrade pip
53+
pip install -e ".[gui,parquet,widget]"
54+
pip install jsonschema pyarrow tokenizers anywidget
55+
- name: Generate fixtures
56+
run: python tests/fixtures/build_e2e_matrix.py
57+
- name: Vbgui install + typecheck
58+
working-directory: vbgui
59+
env:
60+
NODE_ENV: development
61+
run: |
62+
npm ci
63+
npm run typecheck
64+
- name: Playwright browsers
65+
working-directory: vbgui
66+
run: npx playwright install --with-deps chromium
67+
- name: Run preset matrix shard ${{ matrix.shard }}/4
68+
working-directory: vbgui
69+
env:
70+
NODE_ENV: development
71+
CI: "1"
72+
run: |
73+
npx playwright test --config=e2e/playwright.config.ts \
74+
e2e/scenarios/02_preset_matrix.spec.ts \
75+
--shard=${{ matrix.shard }}/4
76+
- name: Upload artefacts
77+
if: always()
78+
uses: actions/upload-artifact@v4
79+
with:
80+
name: preset-matrix-shard-${{ matrix.shard }}
81+
path: |
82+
vbgui/e2e/test-results/
83+
vbgui/e2e/screenshots/
84+
vbgui/e2e/logs/
85+
retention-days: 3
86+
87+
# Specialised + canvas smoke scenarios — run on every push (lightweight).
88+
specialised:
89+
name: Specialised + canvas smoke
90+
runs-on: ubuntu-latest
91+
timeout-minutes: 15
92+
steps:
93+
- uses: actions/checkout@v4
94+
- uses: actions/setup-node@v4
95+
with: { node-version: "22", cache: "npm",
96+
cache-dependency-path: vbgui/package-lock.json }
97+
- uses: actions/setup-python@v5
98+
with: { python-version: "3.13" }
99+
- run: |
100+
python -m pip install --upgrade pip
101+
pip install -e ".[gui,parquet,widget]"
102+
pip install jsonschema pyarrow tokenizers anywidget
103+
- run: python tests/fixtures/build_e2e_matrix.py
104+
- working-directory: vbgui
105+
env: { NODE_ENV: development }
106+
run: |
107+
npm ci
108+
npm run typecheck
109+
- working-directory: vbgui
110+
run: npx playwright install --with-deps chromium
111+
- working-directory: vbgui
112+
env: { NODE_ENV: development, CI: "1" }
113+
run: |
114+
npx playwright test --config=e2e/playwright.config.ts \
115+
e2e/scenarios/01_canvas_smoke.spec.ts \
116+
e2e/scenarios/04_tokenizer_playground.spec.ts \
117+
e2e/scenarios/05_data_inspector.spec.ts \
118+
e2e/scenarios/06_sharding_proposals.spec.ts \
119+
e2e/scenarios/07_gotchas.spec.ts
120+
- if: always()
121+
uses: actions/upload-artifact@v4
122+
with:
123+
name: specialised-results
124+
path: |
125+
vbgui/e2e/test-results/
126+
vbgui/e2e/screenshots/
127+
vbgui/e2e/logs/
128+
retention-days: 3
129+
130+
# Mini-train matrix (192 cells, real mlx) — macOS only, nightly.
131+
mini-train-matrix:
132+
name: Mini-train matrix (macOS, nightly)
133+
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
134+
runs-on: macos-latest
135+
timeout-minutes: 60
136+
steps:
137+
- uses: actions/checkout@v4
138+
- uses: actions/setup-node@v4
139+
with: { node-version: "22", cache: "npm",
140+
cache-dependency-path: vbgui/package-lock.json }
141+
- uses: actions/setup-python@v5
142+
with: { python-version: "3.13" }
143+
- run: |
144+
python -m pip install --upgrade pip
145+
pip install -e ".[gui,parquet,widget]"
146+
pip install jsonschema pyarrow tokenizers anywidget mlx mlx-lm
147+
- run: python tests/fixtures/build_e2e_matrix.py
148+
- working-directory: vbgui
149+
env: { NODE_ENV: development }
150+
run: |
151+
npm ci
152+
npm run typecheck
153+
- working-directory: vbgui
154+
run: npx playwright install --with-deps chromium
155+
- working-directory: vbgui
156+
env: { NODE_ENV: development, CI: "1" }
157+
run: |
158+
npx playwright test --config=e2e/playwright.config.ts \
159+
e2e/scenarios/03_train_matrix.spec.ts
160+
- if: always()
161+
uses: actions/upload-artifact@v4
162+
with:
163+
name: mini-train-results
164+
path: |
165+
vbgui/e2e/test-results/
166+
vbgui/e2e/screenshots/
167+
vbgui/e2e/logs/
168+
retention-days: 7
169+
170+
# Matrix report — aggregates shard artefacts into one Markdown
171+
# summary, posted as a build summary + uploaded as standalone artefact.
172+
matrix-report:
173+
name: Matrix report
174+
needs: [preset-matrix, specialised]
175+
runs-on: ubuntu-latest
176+
if: always()
177+
steps:
178+
- uses: actions/checkout@v4
179+
- uses: actions/download-artifact@v4
180+
with: { path: artifacts }
181+
- uses: actions/setup-python@v5
182+
with: { python-version: "3.13" }
183+
- name: Build matrix report
184+
run: |
185+
python tools/build_e2e_matrix_report.py \
186+
--artifacts artifacts \
187+
--output e2e_matrix_report.md
188+
cat e2e_matrix_report.md >> "$GITHUB_STEP_SUMMARY"
189+
- uses: actions/upload-artifact@v4
190+
with:
191+
name: e2e_matrix_report
192+
path: e2e_matrix_report.md
193+
retention-days: 14

‎tests/v4/test_e2e_matrix_report.py‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
"""E-6 matrix-report tests — workflow shape + report generator."""
2+
3+
from __future__ import annotations
4+
5+
import json
6+
import subprocess
7+
import sys
8+
from pathlib import Path
9+
10+
import pytest
11+
12+
13+
WORKFLOW = Path(".github/workflows/e2e-matrix.yml")
14+
REPORT_SCRIPT = Path("tools/build_e2e_matrix_report.py")
15+
16+
17+
# ---------------------------------------------------------------------------
18+
# GitHub Actions workflow shape
19+
# ---------------------------------------------------------------------------
20+
21+
22+
def test_workflow_file_exists():
23+
assert WORKFLOW.is_file()
24+
25+
26+
def test_workflow_triggers_on_push_pr_and_schedule():
27+
text = WORKFLOW.read_text()
28+
assert "push:" in text
29+
assert "pull_request:" in text
30+
assert "schedule:" in text
31+
assert "cron:" in text
32+
assert "workflow_dispatch:" in text
33+
34+
35+
def test_workflow_runs_preset_matrix_in_shards():
36+
text = WORKFLOW.read_text()
37+
assert "shard: [1, 2, 3, 4]" in text
38+
assert "--shard=${{ matrix.shard }}/4" in text
39+
40+
41+
def test_workflow_uploads_artifacts():
42+
text = WORKFLOW.read_text()
43+
assert "actions/upload-artifact" in text
44+
assert "e2e/test-results" in text
45+
assert "e2e/screenshots" in text
46+
47+
48+
def test_workflow_runs_mini_train_only_on_macos_nightly():
49+
text = WORKFLOW.read_text()
50+
assert "macos-latest" in text
51+
assert "github.event_name == 'schedule'" in text
52+
53+
54+
def test_workflow_invokes_matrix_report_step():
55+
text = WORKFLOW.read_text()
56+
assert "tools/build_e2e_matrix_report.py" in text
57+
assert "GITHUB_STEP_SUMMARY" in text
58+
59+
60+
# ---------------------------------------------------------------------------
61+
# Report generator
62+
# ---------------------------------------------------------------------------
63+
64+
65+
def test_report_script_exists():
66+
assert REPORT_SCRIPT.is_file()
67+
68+
69+
def test_report_emits_stub_on_missing_artifacts(tmp_path: Path):
70+
out = tmp_path / "report.md"
71+
rc = subprocess.run(
72+
[sys.executable, str(REPORT_SCRIPT),
73+
"--artifacts", str(tmp_path / "nonexistent"),
74+
"--output", str(out)],
75+
capture_output=True, text=True,
76+
).returncode
77+
assert rc == 0
78+
text = out.read_text()
79+
assert "E2E Coverage Matrix Report" in text
80+
assert "No artefacts" in text
81+
82+
83+
def test_report_aggregates_last_run_json(tmp_path: Path):
84+
artifacts = tmp_path / "artifacts"
85+
shard = artifacts / "preset-matrix-shard-1" / "test-results"
86+
shard.mkdir(parents=True)
87+
(shard / ".last-run.json").write_text(json.dumps({
88+
"status": "passed",
89+
"failedTests": [],
90+
"flakyTests": [{"id": "x"}],
91+
}))
92+
shots = artifacts / "preset-matrix-shard-1" / "screenshots" / "02_preset_matrix"
93+
shots.mkdir(parents=True)
94+
(shots / "cell.png").write_bytes(b"\x89PNG")
95+
96+
out = tmp_path / "report.md"
97+
rc = subprocess.run(
98+
[sys.executable, str(REPORT_SCRIPT),
99+
"--artifacts", str(artifacts), "--output", str(out)],
100+
capture_output=True, text=True,
101+
).returncode
102+
assert rc == 0
103+
text = out.read_text()
104+
assert "Playwright run files parsed: **1**" in text
105+
assert "Total flaky retries (across shards): **1**" in text
106+
assert "`02_preset_matrix`" in text
107+
assert "1 " in text # screenshot count
108+
109+
110+
def test_report_handles_multiple_shards(tmp_path: Path):
111+
artifacts = tmp_path / "artifacts"
112+
for i in (1, 2, 3, 4):
113+
d = artifacts / f"preset-matrix-shard-{i}" / "test-results"
114+
d.mkdir(parents=True)
115+
(d / ".last-run.json").write_text(json.dumps({
116+
"status": "passed", "failedTests": [], "flakyTests": [],
117+
}))
118+
out = tmp_path / "report.md"
119+
subprocess.run([sys.executable, str(REPORT_SCRIPT),
120+
"--artifacts", str(artifacts), "--output", str(out)],
121+
check=True)
122+
text = out.read_text()
123+
assert "Playwright run files parsed: **4**" in text
124+
125+
126+
def test_report_documents_matrix_sizes():
127+
"""Smoke: report explains the 912 + 192 + 25 + 6 layout for readers."""
128+
artifacts_root = Path("/tmp/_report_doc_test_artifacts")
129+
if artifacts_root.exists():
130+
import shutil
131+
shutil.rmtree(artifacts_root)
132+
artifacts_root.mkdir()
133+
out = artifacts_root / "out.md"
134+
subprocess.run([sys.executable, str(REPORT_SCRIPT),
135+
"--artifacts", str(artifacts_root), "--output", str(out)],
136+
check=True)
137+
text = out.read_text()
138+
assert "912 cells" in text
139+
assert "192 cells" in text
140+
assert "25 tests" in text
141+
assert "6 tests" in text

0 commit comments

Comments
 (0)