Skip to content

Commit 98e15ee

Browse files
committed
Update README
1 parent c2cc58a commit 98e15ee

1 file changed

Lines changed: 27 additions & 13 deletions

File tree

README.md

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -16,18 +16,24 @@ This repository contains a testing script for [`partdiff`](https://github.com/pa
1616

1717
## How it works
1818

19-
`partdiff_tester` is based on `pytest` which is a testing framework for Python.
20-
`pytest` automatically picks up the tests contained in `test_partdiff.py`[^1] and uses `conftest.py` for configuration.
21-
The test cases are loaded from `test_cases.txt`; `pytest` generates the test cases automatically. The selection of test cases can be modified via the arguments `--num-threads`, `--filter`, `--shuffle`, and `--max-num-tests`.
19+
`partdiff_tester` is based on [`pytest`](https://docs.pytest.org/en/stable/) which is a testing framework for Python.
20+
`pytest` automatically picks up all `test*.py` files (here, that's only [`test_partdiff.py`](test_partdiff.py))[^1] and uses [`conftest.py`](conftest.py) for configuration[^2].
21+
The test cases are auto-generated from the content [`test_cases.txt`](test_cases.txt) via `pytest_generate_tests`[^3]. The selection of test cases can be modified via arguments
22+
[`--num-threads`](#num-threads),
23+
[`--filter`](#filter),
24+
[`--shuffle`](#shuffle), and
25+
[`--max-num-tests`](#max-num-tests) (see below).
2226

2327
[^1]: See https://docs.python.org/3/library/unittest.html#unittest-test-discovery
28+
[^2]: See https://docs.pytest.org/en/stable/reference/fixtures.html#conftest-py-sharing-fixtures-across-multiple-files
29+
[^3]: See https://docs.pytest.org/en/stable/how-to/parametrize.html#pytest-generate-tests
2430

2531
The output of a partdiff executable is simply compared to the output of the known good reference implementation.
26-
The strictness of this check is configurable via the `--strictness` argument, e.g. with `--strictness=0`, only the matrix is compared, and with `--strictness=4`, the output has to match completely (except for the actual values of runtime and memory consumption).
27-
If desired, a `valgrind` check can also be performed (see `--valgrind`).
32+
The strictness of this check is configurable via the [`--strictness`](#strictness) argument, e.g. with `--strictness=0`, only the matrix is compared, and with `--strictness=4`, the output has to match completely (except for the actual values of runtime and memory consumption).
33+
If desired, a `valgrind` check can also be performed (see [`--valgrind`](#valgrind)).
2834

2935
The directory `reference_output` contains a collection of cached reference outputs.
30-
If a test case is supposed to test a parameter configuration of which the output isn't cached, the reference implementation can also be used instead (see `--reference-source`).
36+
If a test case is supposed to test a parameter configuration of which the output isn't cached, the reference implementation can also be used instead (see [`--reference-source`](#reference-source)).
3137
The content of `reference_output` and `test_cases.txt` is generated with the `make_reference_output.sh` script.
3238

3339
## Usage
@@ -38,9 +44,15 @@ Example usage:
3844
$ uv run pytest --executable='/path/to/partdiff'
3945
```
4046

47+
> [!TIP]
48+
> If you want to use `partdiff_tester` in a GitHub workflow, you can easily do that with the GitHub action contained in [`action.yaml`](action.yaml).
49+
> The input parameters closely reflect the CLI parameters described below.
50+
> You can find a usage example in the [partdiff repository](https://github.com/parcio/partdiff/blob/main/.github/workflows/correctness_check.yaml).
51+
> ```
52+
4153
Of course, you can also install `pytest` and execute it directly.
4254
43-
We can use the [`pytest-xdist` plugin](https://pypi.org/project/pytest-xdist/) to execute the test cases in parallel. It is recommended to do so for all serial implementations, because the parallelism greatly accelerates the tests. To do this, we simply pass the `-n auto` arguments to `pytest` (special care must be taken if you want to use `--shuffle`!)
55+
We can use the [`pytest-xdist` plugin](https://pypi.org/project/pytest-xdist/) to execute the test cases in parallel. It is recommended to do so for all serial implementations, because the parallelism greatly accelerates the tests. To do this, we simply pass the `-n auto` arguments to `pytest` (special care must be taken if you want to use [`--shuffle`](#shuffle)!)
4456
4557
The following performs a soundness check by essentially testing the reference implementation against itself.
4658
@@ -92,16 +104,18 @@ The custom options are explained below.
92104

93105
> [!NOTE]
94106
> Some parameters modify the set of test cases. They are applied in this order:
95-
> 1. `--num-threads`
96-
> 2. `--filter`
97-
> 3. `--shuffle`
98-
> 4. `--max-num-tests`
107+
> 1. `--num-threads` (may increase number of tests)
108+
> 2. `--filter` (may decrease number of tests)
109+
> 3. `--shuffle` (re-orders tests)
110+
> 4. `--max-num-tests` (may decrease number of tests)
99111
100112
### `executable`
101113

102114
Path to the partdiff executable.
103115

104-
Use `--executable=/path/to/partdiff` (not `--executable /path/to/partdiff` because pytest's argparser is a bit dumb).
116+
Use `--executable=/path/to/partdiff` (not `--executable /path/to/partdiff` because pytest's argparser is a bit dumb[^4]).
117+
118+
[^4]: The issue here is that pytest tries to automatically interpret positional arguments as test paths. Here, the partdiff path would be consumed before the `executable` argument is parsed. The `=` sign prevents this.
105119

106120
You can pass a space-separated list to do something like this:
107121

@@ -254,7 +268,7 @@ See the code blocks below to get a graphical overview over what is checked exact
254268
```
255269
</details>
256270

257-
## `valgrind`
271+
### `valgrind`
258272

259273
Start the executable with `valgrind --leak-check=full`.
260274

0 commit comments

Comments
 (0)