Skip to content

Commit b6d5c2d

Browse files
Stevengredkcumming
andauthored
test(stable-mir-ui): add external UI test harness with skip management (#968)
## Summary - Add parametrized pytest harness running `kmir prove-rs` against every entry in `stable-mir-json`'s `passing.tsv` - Ship `skip.txt` (2859 entries) for known-failing cases, with `--update-skip` mode to shrink it by re-proving skipped cases - 300s per-test timeout via `pytest-timeout`; proof show output saved to `tmp_path` on failure ## Usage ```bash # Normal run (skipped cases are skipped) RUST_DIR_ROOT=/path/to/rust make test-stable-mir-ui # Shrink skip.txt by re-proving skipped cases RUST_DIR_ROOT=/path/to/rust make test-stable-mir-ui TEST_ARGS="--update-skip" ``` ## Test plan - [x] `make check` passes (B011 lint fixed) - [x] Single squashed commit, rebased on `origin/master` --------- Co-authored-by: Daniel Cumming <124537596+dkcumming@users.noreply.github.com> Co-authored-by: dkcumming <daniel.cumming@runtimeverification.com>
1 parent 53e9dab commit b6d5c2d

6 files changed

Lines changed: 2987 additions & 2 deletions

File tree

Makefile

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,11 @@ test-integration: stable-mir-json build
5656
$(UV_RUN) pytest $(TOP_DIR)/kmir/src/tests/integration --maxfail=1 --verbose \
5757
--durations=0 --numprocesses=$(PARALLEL) --dist=worksteal $(TEST_ARGS)
5858

59+
.PHONY: test-stable-mir-ui
60+
test-stable-mir-ui: stable-mir-json build
61+
@test -n "$(RUST_DIR_ROOT)" || (echo "RUST_DIR_ROOT is required. Example: RUST_DIR_ROOT=/path/to/rust make test-stable-mir-ui"; exit 2)
62+
$(UV_RUN) pytest $(TOP_DIR)/kmir/src/tests/external/test_stable_mir_ui_pass.py --maxfail=1 --verbose $(TEST_ARGS)
63+
5964
# Checks and formatting
6065

6166
format: autoflake isort black nix-fmt

kmir/pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ dev = [
4242
"pytest",
4343
"pytest-cov",
4444
"pytest-mock",
45+
"pytest-timeout",
4546
"pytest-xdist",
4647
"pyupgrade",
4748
]
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
from __future__ import annotations
2+
3+
from typing import TYPE_CHECKING
4+
5+
import pytest
6+
7+
if TYPE_CHECKING:
8+
from pytest import FixtureRequest, Parser
9+
10+
11+
def pytest_addoption(parser: Parser) -> None:
12+
parser.addoption(
13+
'--update-skip',
14+
action='store_true',
15+
default=False,
16+
help='Shrink stable-mir-ui skip entries by rerunning only current skip.txt cases.',
17+
)
18+
19+
20+
@pytest.fixture(scope='session')
21+
def update_skip_mode(request: FixtureRequest) -> bool:
22+
return request.config.getoption('--update-skip')

0 commit comments

Comments
 (0)