You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+27-13Lines changed: 27 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,18 +16,24 @@ This repository contains a testing script for [`partdiff`](https://github.com/pa
16
16
17
17
## How it works
18
18
19
-
`partdiff_tester` is based on `pytest` which is a testing framework for Python.
20
-
`pytest` automatically picks up the tests contained in `test_partdiff.py`[^1] and uses `conftest.py` for configuration.
21
-
The test cases are loaded from `test_cases.txt`; `pytest` generates the test cases automatically. The selection of test cases can be modified via the arguments `--num-threads`, `--filter`, `--shuffle`, and `--max-num-tests`.
19
+
`partdiff_tester` is based on [`pytest`](https://docs.pytest.org/en/stable/) which is a testing framework for Python.
20
+
`pytest` automatically picks up all `test*.py` files (here, that's only [`test_partdiff.py`](test_partdiff.py))[^1] and uses [`conftest.py`](conftest.py) for configuration[^2].
21
+
The test cases are auto-generated from the content [`test_cases.txt`](test_cases.txt) via `pytest_generate_tests`[^3]. The selection of test cases can be modified via arguments
22
+
[`--num-threads`](#num-threads),
23
+
[`--filter`](#filter),
24
+
[`--shuffle`](#shuffle), and
25
+
[`--max-num-tests`](#max-num-tests) (see below).
22
26
23
27
[^1]: See https://docs.python.org/3/library/unittest.html#unittest-test-discovery
28
+
[^2]: See https://docs.pytest.org/en/stable/reference/fixtures.html#conftest-py-sharing-fixtures-across-multiple-files
29
+
[^3]: See https://docs.pytest.org/en/stable/how-to/parametrize.html#pytest-generate-tests
24
30
25
31
The output of a partdiff executable is simply compared to the output of the known good reference implementation.
26
-
The strictness of this check is configurable via the `--strictness` argument, e.g. with `--strictness=0`, only the matrix is compared, and with `--strictness=4`, the output has to match completely (except for the actual values of runtime and memory consumption).
27
-
If desired, a `valgrind` check can also be performed (see `--valgrind`).
32
+
The strictness of this check is configurable via the [`--strictness`](#strictness) argument, e.g. with `--strictness=0`, only the matrix is compared, and with `--strictness=4`, the output has to match completely (except for the actual values of runtime and memory consumption).
33
+
If desired, a `valgrind` check can also be performed (see [`--valgrind`](#valgrind)).
28
34
29
35
The directory `reference_output` contains a collection of cached reference outputs.
30
-
If a test case is supposed to test a parameter configuration of which the output isn't cached, the reference implementation can also be used instead (see `--reference-source`).
36
+
If a test case is supposed to test a parameter configuration of which the output isn't cached, the reference implementation can also be used instead (see [`--reference-source`](#reference-source)).
31
37
The content of `reference_output` and `test_cases.txt` is generated with the `make_reference_output.sh` script.
32
38
33
39
## Usage
@@ -38,9 +44,15 @@ Example usage:
38
44
$ uv run pytest --executable='/path/to/partdiff'
39
45
```
40
46
47
+
> [!TIP]
48
+
> If you want to use `partdiff_tester` in a GitHub workflow, you can easily do that with the GitHub action contained in [`action.yaml`](action.yaml).
49
+
> The input parameters closely reflect the CLI parameters described below.
50
+
> You can find a usage example in the [partdiff repository](https://github.com/parcio/partdiff/blob/main/.github/workflows/correctness_check.yaml).
51
+
> ```
52
+
41
53
Of course, you can also install `pytest` and execute it directly.
42
54
43
-
We can use the [`pytest-xdist` plugin](https://pypi.org/project/pytest-xdist/) to execute the test cases in parallel. It is recommended to do so for all serial implementations, because the parallelism greatly accelerates the tests. To do this, we simply pass the `-n auto` arguments to `pytest` (special care must be taken if you want to use `--shuffle`!)
55
+
We can use the [`pytest-xdist` plugin](https://pypi.org/project/pytest-xdist/) to execute the test cases in parallel. It is recommended to do so for all serial implementations, because the parallelism greatly accelerates the tests. To do this, we simply pass the `-n auto` arguments to `pytest` (special care must be taken if you want to use [`--shuffle`](#shuffle)!)
44
56
45
57
The following performs a soundness check by essentially testing the reference implementation against itself.
46
58
@@ -92,16 +104,18 @@ The custom options are explained below.
92
104
93
105
> [!NOTE]
94
106
> Some parameters modify the set of test cases. They are applied in this order:
95
-
> 1.`--num-threads`
96
-
> 2.`--filter`
97
-
> 3.`--shuffle`
98
-
> 4.`--max-num-tests`
107
+
> 1.`--num-threads` (may increase number of tests)
108
+
> 2.`--filter` (may decrease number of tests)
109
+
> 3.`--shuffle` (re-orders tests)
110
+
> 4.`--max-num-tests` (may decrease number of tests)
99
111
100
112
### `executable`
101
113
102
114
Path to the partdiff executable.
103
115
104
-
Use `--executable=/path/to/partdiff` (not `--executable /path/to/partdiff` because pytest's argparser is a bit dumb).
116
+
Use `--executable=/path/to/partdiff` (not `--executable /path/to/partdiff` because pytest's argparser is a bit dumb[^4]).
117
+
118
+
[^4]: The issue here is that pytest tries to automatically interpret positional arguments as test paths. Here, the partdiff path would be consumed before the `executable` argument is parsed. The `=` sign prevents this.
105
119
106
120
You can pass a space-separated list to do something like this:
107
121
@@ -254,7 +268,7 @@ See the code blocks below to get a graphical overview over what is checked exact
254
268
```
255
269
</details>
256
270
257
-
## `valgrind`
271
+
###`valgrind`
258
272
259
273
Start the executable with `valgrind --leak-check=full`.
0 commit comments