Skip to content

Commit 9cab944

Browse files
committed
docs: add code coverage section to test README
Document how contributors can measure code coverage locally with pytest-cov, link to the project's Codecov dashboard, and note the ~80% coverage goal. Mentions the pyproject.toml configuration and the LONG_TESTS / EXTERNAL_SYSTEM gating that affects local figures versus what CI measures. fixes #4952 Signed-off-by: ABD <abdullahfazili3614@gmail.com>
1 parent 307027e commit 9cab944

1 file changed

Lines changed: 31 additions & 0 deletions

File tree

doc/test/README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ You can see all existing tests in [`test/`](https://github.com/ossf/cve-bin-tool
66
- [Running all tests](#running-all-tests)
77
- [Running a single test](#running-a-single-test)
88
- [Running tests on different versions of Python](#running-tests-on-different-versions-of-python)
9+
- [Measuring code coverage](#measuring-code-coverage)
910
- [Adding new tests: CVE mapping tests](#adding-new-tests-cve-mapping-tests)
1011
- [Adding new tests: Signature tests against real files](#adding-new-tests-signature-tests-against-real-files)
1112
- [Adding new tests: Checker filename mappings](#adding-new-tests-checker-filename-mappings)
@@ -87,6 +88,36 @@ deactivate
8788

8889
```
8990

91+
## Measuring code coverage
92+
93+
CVE Binary Tool uses [`pytest-cov`](https://pytest-cov.readthedocs.io/) to measure code coverage during test runs, and CI uploads the results to [Codecov](https://codecov.io/gh/ossf/cve-bin-tool). The current goal is around 80% coverage as reported by Codecov.
94+
95+
`pytest-cov` is included in the `[dev]` extras, so if you installed the development dependencies (`pip install .[dev]`) it is already available.
96+
97+
To measure coverage locally, run pytest with the `--cov` flag:
98+
99+
```console
100+
pytest --cov
101+
```
102+
103+
For a human-readable HTML report, add `--cov-report=html`:
104+
105+
```console
106+
pytest --cov --cov-report=html
107+
```
108+
109+
This writes an `htmlcov/` directory in the repository root; open `htmlcov/index.html` in a browser to browse the report.
110+
111+
Coverage settings live in the `[tool.coverage.run]` and `[tool.coverage.report]` sections of `pyproject.toml`. By default, coverage is collected for the `cve_bin_tool` and `test` packages with branch coverage enabled.
112+
113+
Note that, as with the regular test suite, longer tests are gated behind the `LONG_TESTS` and `EXTERNAL_SYSTEM` environment variables described above. Coverage figures from a default `pytest --cov` run will only reflect the short test suite. To approximate what CI measures, run with the same gating variables and report flags CI uses:
114+
115+
```console
116+
LONG_TESTS=1 EXTERNAL_SYSTEM=1 pytest --cov --cov-report=xml --cov-append -n 8
117+
```
118+
119+
Be aware that the long test suite can take a substantial amount of time to complete.
120+
90121
## Adding new tests: CVE mapping tests
91122

92123
- You can see the code for scanner tests in ['test/test_scanner.py'](https://github.com/ossf/cve-bin-tool/blob/main/test/test_scanner.py)

0 commit comments

Comments
 (0)