Skip to content

Commit d81124a

Browse files
CreatmanCEOclaude
CreatmanCEO
and
claude
committed
M2: tests, CI, demo fixture (94 tests, ruff+mypy clean)
- pyproject.toml — pytest/coverage/ruff/mypy config - requirements-dev.txt — pytest, ruff, mypy, pytest-cov - tests/conftest.py — shared fixtures (sample Playwright JSON, raw bugs, console messages) - tests/python/test_*.py — 9 test files, 94 tests covering all 9 scripts - .github/workflows/ci.yml — lint + matrix pytest (Linux/macOS/Windows × py3.10/3.12) + E2E smoke - tests/fixtures/demo-app/ — minimal HTML site with planted bugs for E2E smoke - README.md — CI/license/version/python badges + beta disclaimer Bug fixes surfaced by tests: - fingerprint_bugs: regex now matches both ASCII 'x' and Unicode '×' for node counts - triage_console: triage() type annotations clean - generate_report: variable shadowing fix (sev dict vs bug_sev str) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 4c75e09 commit d81124a

29 files changed

Lines changed: 1396 additions & 37 deletions

‎.coverage‎

84 KB
Binary file not shown.

‎.github/workflows/ci.yml‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
workflow_dispatch:
9+
10+
# Cancel previous runs on the same branch when a new commit is pushed
11+
concurrency:
12+
group: ${{ github.workflow }}-${{ github.ref }}
13+
cancel-in-progress: true
14+
15+
jobs:
16+
lint:
17+
name: Lint (ruff + mypy)
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@v4
21+
- uses: actions/setup-python@v5
22+
with:
23+
python-version: "3.10"
24+
cache: pip
25+
- name: Install dev deps
26+
run: pip install -r requirements-dev.txt
27+
- name: Ruff
28+
run: ruff check scripts/ tests/
29+
- name: Mypy
30+
run: mypy scripts/
31+
32+
test:
33+
name: Pytest (${{ matrix.os }} / py${{ matrix.python-version }})
34+
needs: lint
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
os: [ubuntu-latest, macos-latest, windows-latest]
39+
python-version: ["3.10", "3.12"]
40+
# Trim matrix — keep ubuntu fully tested, others only on 3.12
41+
exclude:
42+
- os: macos-latest
43+
python-version: "3.10"
44+
- os: windows-latest
45+
python-version: "3.10"
46+
runs-on: ${{ matrix.os }}
47+
steps:
48+
- uses: actions/checkout@v4
49+
- uses: actions/setup-python@v5
50+
with:
51+
python-version: ${{ matrix.python-version }}
52+
cache: pip
53+
- name: Install dev deps
54+
run: pip install -r requirements-dev.txt
55+
- name: Pytest with coverage
56+
run: pytest --cov=scripts --cov-report=term --cov-report=xml -q
57+
- name: Upload coverage (Linux py3.12 only)
58+
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
59+
uses: actions/upload-artifact@v4
60+
with:
61+
name: coverage-xml
62+
path: coverage.xml
63+
if-no-files-found: ignore
64+
65+
smoke-e2e:
66+
name: E2E smoke (Linux)
67+
needs: test
68+
runs-on: ubuntu-latest
69+
steps:
70+
- uses: actions/checkout@v4
71+
- uses: actions/setup-python@v5
72+
with:
73+
python-version: "3.12"
74+
cache: pip
75+
- uses: actions/setup-node@v4
76+
with:
77+
node-version: "20"
78+
- name: Install dev deps
79+
run: pip install -r requirements-dev.txt
80+
- name: Bootstrap demo project
81+
working-directory: tests/fixtures/demo-app
82+
run: |
83+
npm init -y >/dev/null
84+
npm install --silent --save-dev @playwright/test @axe-core/playwright dotenv
85+
npx playwright install --with-deps chromium
86+
- name: Serve demo HTML
87+
run: |
88+
cd tests/fixtures/demo-app
89+
python3 -m http.server 8765 &
90+
echo $! > /tmp/server.pid
91+
sleep 1
92+
curl -fsS http://127.0.0.1:8765/ >/dev/null
93+
- name: Run skill pipeline against demo
94+
working-directory: tests/fixtures/demo-app
95+
run: |
96+
mkdir -p reports/run-ci
97+
npx playwright test --project chromium-mobile --reporter=json,list \
98+
> reports/run-ci/playwright-stdout.txt 2>&1 || true
99+
# Move Playwright's results.json into the run dir
100+
cp test-results/results.json reports/run-ci/results.json || \
101+
cp playwright-report/results.json reports/run-ci/results.json
102+
python ../../../scripts/run_suite.py --cwd . --out reports/run-ci --skip-run
103+
python ../../../scripts/fingerprint_bugs.py \
104+
--current reports/run-ci/raw_bugs.json \
105+
--out reports/run-ci/bugs.json \
106+
--diff reports/run-ci/diff.json
107+
python ../../../scripts/generate_report.py \
108+
--run-dir reports/run-ci --app-name "Demo App"
109+
- name: Verify report artefacts exist
110+
working-directory: tests/fixtures/demo-app
111+
run: |
112+
test -f reports/run-ci/bugs.json
113+
test -f reports/run-ci/diff.json
114+
test -f reports/run-ci/report.md
115+
test -f reports/run-ci/index.html
116+
echo "--- report.md preview ---"
117+
head -40 reports/run-ci/report.md
118+
- name: Upload run artefacts
119+
if: always()
120+
uses: actions/upload-artifact@v4
121+
with:
122+
name: smoke-e2e-run
123+
path: tests/fixtures/demo-app/reports/run-ci/
124+
if-no-files-found: warn
125+
- name: Stop demo server
126+
if: always()
127+
run: kill $(cat /tmp/server.pid) || true

‎README.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1-
# webapp-test-orchestrator
1+
# webtest-orch — webapp-test-orchestrator
2+
3+
[![CI](https://github.com/CreatmanCEO/webtest-orch/actions/workflows/ci.yml/badge.svg)](https://github.com/CreatmanCEO/webtest-orch/actions/workflows/ci.yml)
4+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
5+
[![Version](https://img.shields.io/badge/version-0.1.0--beta-orange.svg)](./CHANGELOG.md)
6+
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
27

38
**Universal e2e testing skill for Claude Code.** Заменяет ad-hoc промпты с Playwright MCP на одну переиспользуемую сущность для тестирования любого web-приложения (Next.js, FastAPI, статика, Telegram WebApp, и т.д.).
49

10+
> ⚠️ **Public beta (`0.1.0-beta`)** — looking for early feedback, especially OS-compatibility reports. See [issue templates](.github/ISSUE_TEMPLATE/).
11+
512
---
613

714
## Зачем

‎pyproject.toml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
[tool.pytest.ini_options]
2+
testpaths = ["tests"]
3+
python_files = ["test_*.py"]
4+
python_classes = ["Test*"]
5+
python_functions = ["test_*"]
6+
addopts = [
7+
"-ra",
8+
"--strict-markers",
9+
"--tb=short",
10+
]
11+
markers = [
12+
"slow: tests that take more than a few seconds",
13+
]
14+
15+
[tool.coverage.run]
16+
source = ["scripts"]
17+
branch = true
18+
19+
[tool.coverage.report]
20+
skip_empty = true
21+
show_missing = true
22+
exclude_lines = [
23+
"pragma: no cover",
24+
"if __name__ == .__main__.:",
25+
"raise NotImplementedError",
26+
]
27+
28+
[tool.ruff]
29+
line-length = 110
30+
target-version = "py310"
31+
extend-exclude = ["tests/fixtures/demo-app"]
32+
33+
[tool.ruff.lint]
34+
select = ["E", "F", "W", "I", "B", "UP", "C4", "SIM", "RUF"]
35+
ignore = [
36+
"E501", # line too long — handled by formatter
37+
"RUF001", # ambiguous unicode chars — we use em-dashes intentionally
38+
"RUF002",
39+
"RUF003",
40+
"SIM105", # try/except/pass is clearer than contextlib.suppress for stdout reconfigure
41+
"SIM108", # ternary not always more readable
42+
]
43+
44+
[tool.mypy]
45+
python_version = "3.10"
46+
warn_unused_ignores = true
47+
warn_redundant_casts = true
48+
ignore_missing_imports = true
49+
files = ["scripts"]

‎requirements-dev.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pytest>=8.0
2+
pytest-cov>=5.0
3+
ruff>=0.6
4+
mypy>=1.10
5+
PyYAML>=6.0

‎scripts/_image_isolation_check.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
# em-dashes don't crash with UnicodeEncodeError. No-op on Linux/macOS.
2828
for _stream in (sys.stdout, sys.stderr):
2929
try:
30-
_stream.reconfigure(encoding="utf-8")
30+
_stream.reconfigure(encoding="utf-8") # type: ignore[union-attr]
3131
except (AttributeError, ValueError):
3232
pass
3333

@@ -88,7 +88,7 @@ def gen_fixtures() -> int:
8888
print(f"[image_isolation_check] fixtures written to {fdir}")
8989
print("Next step: dispatch a Task subagent (general-purpose) with this prompt:")
9090
print("---")
91-
print(f"Read these 3 files with Read and return one short text description per file:")
91+
print("Read these 3 files with Read and return one short text description per file:")
9292
for name in FIXTURE_NAMES:
9393
print(f" {fdir / name}")
9494
print("Output 3 lines, no preamble, no inline images.")

‎scripts/detect_state.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
# em-dashes don't crash with UnicodeEncodeError. No-op on Linux/macOS.
2626
for _stream in (sys.stdout, sys.stderr):
2727
try:
28-
_stream.reconfigure(encoding="utf-8")
28+
_stream.reconfigure(encoding="utf-8") # type: ignore[union-attr]
2929
except (AttributeError, ValueError):
3030
pass
3131

‎scripts/fingerprint_bugs.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525

2626
for _stream in (sys.stdout, sys.stderr):
2727
try:
28-
_stream.reconfigure(encoding="utf-8")
28+
_stream.reconfigure(encoding="utf-8") # type: ignore[union-attr]
2929
except (AttributeError, ValueError):
3030
pass
3131

@@ -163,9 +163,11 @@ def compute_fingerprint(bug: dict) -> str:
163163

164164
# Issue line is most discriminating when present (one bug per issue)
165165
if issue_line:
166-
# Normalize: strip variable counts like "(3× nodes)" → ""
167-
normalized_issue = re.sub(r"\(\d+× nodes\)", "", issue_line)
168-
normalized_issue = re.sub(r"\d+x\d+", "WxH", normalized_issue) # strip viewport sizes
166+
# Strip variable node counts: "(3x nodes)" or "(10× nodes)" → ""
167+
normalized_issue = re.sub(r"\s*\(\d+\s*[x×]\s*nodes\)", "", issue_line)
168+
# Strip viewport sizes "390x844" → "WxH" so the same bug across
169+
# viewports collapses to one fingerprint
170+
normalized_issue = re.sub(r"\d+x\d+", "WxH", normalized_issue)
169171
composite = f"{spec_file}|{normalized_issue}"
170172
else:
171173
composite = f"{selector}|{assertion}|{error_class}|{url_path}|{msg[:80]}|{spec_file}"

‎scripts/generate_report.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222

2323
for _stream in (sys.stdout, sys.stderr):
2424
try:
25-
_stream.reconfigure(encoding="utf-8")
25+
_stream.reconfigure(encoding="utf-8") # type: ignore[union-attr]
2626
except (AttributeError, ValueError):
2727
pass
2828

@@ -104,22 +104,22 @@ def render_markdown(bugs: list, summary: dict, run_id: str, app_name: str) -> st
104104
if sorted_open:
105105
lines += ["", "## 🚨 Open issues", ""]
106106
for i, b in enumerate(sorted_open, 1):
107-
sev = b.get("severity") or "S2"
107+
bug_sev = b.get("severity") or "S2"
108108
pri = b.get("priority") or "P2"
109109
state = (b.get("diff") or {}).get("state") or "new"
110-
emoji = SEV_EMOJI.get(sev, "•") + " " + VERDICT_EMOJI.get(state, "")
110+
emoji = SEV_EMOJI.get(bug_sev, "•") + " " + VERDICT_EMOJI.get(state, "")
111111
title = b.get("title") or "untitled"
112112
bug_id = b.get("id") or "BUG-?"
113113
occ = b.get("occurrenceCount", 1)
114114
lines += [
115-
f"### {i}. [{bug_id}] {sev}/{pri} — {title} {emoji}",
115+
f"### {i}. [{bug_id}] {bug_sev}/{pri} — {title} {emoji}",
116116
"",
117117
f"- **State:** `{state}` · **Occurrences:** {occ}",
118118
f"- **Spec:** `{b.get('specFile', '?')}` :: {b.get('specTitle', '?')}",
119119
f"- **Project:** `{b.get('project', '?')}`",
120120
]
121121
err = b.get("error") or {}
122-
msg = strip_ansi((err.get("message") or "")).strip()
122+
msg = strip_ansi(err.get("message") or "").strip()
123123
if msg:
124124
lines += ["", "```", msg[:600], "```"]
125125
screenshots = b.get("screenshots") or []
@@ -138,7 +138,7 @@ def render_markdown(bugs: list, summary: dict, run_id: str, app_name: str) -> st
138138
for b in fixed_bugs:
139139
lines += [f"- [{b.get('id')}] {b.get('title', '?')}"]
140140

141-
lines += ["", "---", "", f"_Report generated by webapp-test-orchestrator. bugs.json + diff.json sit alongside this file._"]
141+
lines += ["", "---", "", "_Report generated by webapp-test-orchestrator. bugs.json + diff.json sit alongside this file._"]
142142
return "\n".join(lines) + "\n"
143143

144144

‎scripts/run_suite.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323

2424
for _stream in (sys.stdout, sys.stderr):
2525
try:
26-
_stream.reconfigure(encoding="utf-8")
26+
_stream.reconfigure(encoding="utf-8") # type: ignore[union-attr]
2727
except (AttributeError, ValueError):
2828
pass
2929

@@ -165,7 +165,7 @@ def normalize_results(results_path: Path, out_path: Path, run_id: str) -> tuple[
165165
print(f"[run_suite] missing {results_path}", file=sys.stderr)
166166
return 1, 0, 0
167167

168-
with open(results_path, "r", encoding="utf-8") as f:
168+
with open(results_path, encoding="utf-8") as f:
169169
results = json.load(f)
170170

171171
bugs: list = []
@@ -228,7 +228,7 @@ def main(argv: list[str] | None = None) -> int:
228228
shutil.copy2(cand, results_path)
229229
break
230230

231-
rc_norm, total, failed = normalize_results(results_path, out_dir / "raw_bugs.json", run_id)
231+
rc_norm, _total, _failed = normalize_results(results_path, out_dir / "raw_bugs.json", run_id)
232232

233233
return rc_pw if not args.skip_run else rc_norm
234234

0 commit comments

Comments
 (0)