Skip to content

Commit f0257bc

Browse files
Stevengreehildenbdkcumming
authored
perf(test): remove redundant integration test executions (#972)
## Summary - Skip LLVM backend `test_exec_smir` tests in CI — keep Haskell only, since it's the backend used for proving and bugs there have higher impact - Skip `test_prove_termination` in CI — the same 19 programs are already executed via `test_exec_smir[*-haskell]` This is done via a pytest `-k` filter in the CI workflow only — **no test code is modified**, so all tests remain available for local development. **Deselected: 58 of 247 tests** (39 LLVM exec_smir + 19 prove_termination) ## Risk Analysis - LLVM backend regression will be missed in the previous test, which should be handled by future test framework refactoring. But if we don't add new `exec_smir` test or add new `exec_smir` test with llvm to update expected files, the result is the same as we run CI before. - `test_prove_termination` just uses `prove-rs` to validate the termination, which is the same as the comparison with the current expected files. I believe that there is no risk to remove them in this way. ## Expected CI time reduction ~2h37m → ~1h20m (based on [this run](https://github.com/runtimeverification/mir-semantics/actions/runs/22658250856/job/65672639933?pr=957)) ## Test plan - [x] Integration tests pass with the `-k` filter applied - [x] Verify CI time improvement Resolves #971 (Phase 1) --------- Co-authored-by: Everett Hildenbrandt <everett.hildenbrandt@gmail.com> Co-authored-by: dkcumming <daniel.cumming@runtimeverification.com>
1 parent f70f025 commit f0257bc

1 file changed

Lines changed: 30 additions & 3 deletions

File tree

.github/workflows/test.yml

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,37 @@ jobs:
4747
with:
4848
version: ${{ steps.uv_release.outputs.uv_version }}
4949
- name: 'Run unit tests'
50-
run: make test-unit
50+
run: make test-unit PARALLEL=12
5151

5252
integration-tests:
5353
needs: code-quality-checks
54-
name: 'Integration Tests'
54+
name: 'Integration Tests ${{ matrix.name }}'
5555
runs-on: [self-hosted, linux, normal]
56+
timeout-minutes: ${{ matrix.timeout }}
57+
strategy:
58+
fail-fast: true
59+
matrix:
60+
include:
61+
- name: 'LLVM Concrete Tests'
62+
test-args: '-k "llvm or test_run_smir_random"'
63+
parallel: 12
64+
timeout: 20
65+
- name: 'Haskell Exec SMIR'
66+
test-args: '-k "test_exec_smir and haskell"'
67+
parallel: 6
68+
timeout: 20
69+
- name: 'Haskell Termination'
70+
test-args: '-k test_prove_termination'
71+
parallel: 6
72+
timeout: 20
73+
- name: 'Haskell Proofs'
74+
test-args: '-k "test_prove and not test_prove_termination"'
75+
parallel: 6
76+
timeout: 60
77+
- name: 'Remainder'
78+
test-args: '-k "not llvm and not test_run_smir_random and not test_exec_smir and not test_prove_termination and not test_prove"'
79+
parallel: 6
80+
timeout: 20
5681
steps:
5782
- name: 'Check out code'
5883
uses: actions/checkout@v4
@@ -66,7 +91,9 @@ jobs:
6691
- name: 'Build stable-mir-json and kmir'
6792
run: docker exec --user github-user mir-semantics-ci-${GITHUB_SHA} make build
6893
- name: 'Run integration tests'
69-
run: docker exec --user github-user mir-semantics-ci-${GITHUB_SHA} make test-integration
94+
run: |
95+
docker exec --user github-user mir-semantics-ci-${GITHUB_SHA} make test-integration \
96+
TEST_ARGS='${{ matrix.test-args }}' PARALLEL=${{ matrix.parallel }}
7097
- name: 'Tear down Docker'
7198
if: always()
7299
run: docker stop --time 0 mir-semantics-ci-${GITHUB_SHA}

0 commit comments

Comments
 (0)