Skip to content

Commit a89183e

Browse files
0saurabh0jacobtylerwalls
authored andcommitted
Fixed #36620 -- Added coverage workflow to summarize coverage in pull requests.
Part of GSoC 2025. Thanks Lily for mentorship, and Sarah Boyce and Jacob Walls for reviews.
1 parent 846613e commit a89183e

4 files changed

Lines changed: 151 additions & 0 deletions

File tree

.github/workflows/coverage.yml

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
name: Coverage
2+
3+
on:
4+
pull_request_target:
5+
paths:
6+
- 'django/**/*.py'
7+
- 'tests/**/*.py'
8+
branches:
9+
- main
10+
11+
concurrency:
12+
group: ${{ github.workflow }}-${{ github.ref }}
13+
cancel-in-progress: true
14+
15+
permissions:
16+
contents: read
17+
pull-requests: write
18+
19+
jobs:
20+
diff-coverage:
21+
if: github.repository == 'django/django'
22+
name: Diff Coverage (Windows)
23+
runs-on: windows-latest
24+
steps:
25+
- name: Checkout
26+
uses: actions/checkout@v5
27+
with:
28+
fetch-depth: 0
29+
persist-credentials: false
30+
31+
- name: Set up Python
32+
uses: actions/setup-python@v6
33+
with:
34+
python-version: '3.14'
35+
cache: 'pip'
36+
cache-dependency-path: 'tests/requirements/py3.txt'
37+
38+
- name: Install dependencies
39+
run: |
40+
python -m pip install --upgrade pip wheel
41+
python -m pip install -r tests/requirements/py3.txt -e .
42+
python -m pip install 'coverage[toml]' diff-cover
43+
44+
- name: Run tests with coverage
45+
env:
46+
PYTHONPATH: ${{ github.workspace }}/tests
47+
COVERAGE_PROCESS_START: ${{ github.workspace }}/tests/.coveragerc
48+
RUNTESTS_DIR: ${{ github.workspace }}/tests
49+
run: |
50+
python -Wall tests/runtests.py -v2
51+
52+
- name: Generate coverage report
53+
if: success()
54+
env:
55+
COVERAGE_RCFILE: ${{ github.workspace }}/tests/.coveragerc
56+
RUNTESTS_DIR: ${{ github.workspace }}/tests
57+
run: |
58+
python -m coverage combine
59+
python -m coverage report --show-missing
60+
python -m coverage xml -o tests/coverage.xml
61+
62+
- name: Run diff-cover
63+
if: success()
64+
run: |
65+
if (Test-Path 'tests/coverage.xml') {
66+
diff-cover tests/coverage.xml --compare-branch=origin/main --fail-under=0 > tests/diff-cover-report.md
67+
} else {
68+
Set-Content -Path tests/diff-cover-report.md -Value 'No coverage.xml found; skipping diff-cover.'
69+
}
70+
71+
- name: Post/update PR comment
72+
if: success()
73+
uses: actions/github-script@v8
74+
with:
75+
script: |
76+
const fs = require('fs');
77+
const reportPath = 'tests/diff-cover-report.md';
78+
let body = 'No coverage data available.';
79+
if (fs.existsSync(reportPath)) {
80+
body = fs.readFileSync(reportPath, 'utf8');
81+
}
82+
const commentBody = '### 📊 Coverage Report for Changed Files\n\n```\n' + body + '\n```\n\n**Note:** Missing lines are warnings only. Some lines may not be covered by SQLite tests as they are database-specific.\n\nFor more information about code coverage on pull requests, see the [contributing documentation](https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#code-coverage-on-pull-requests).';
83+
84+
const { data: comments } = await github.rest.issues.listComments({
85+
owner: context.repo.owner,
86+
repo: context.repo.repo,
87+
issue_number: context.issue.number,
88+
});
89+
for (const c of comments) {
90+
if ((c.body || '').includes('📊 Coverage Report for Changed Files')) {
91+
await github.rest.issues.deleteComment({
92+
owner: context.repo.owner,
93+
repo: context.repo.repo,
94+
comment_id: c.id,
95+
});
96+
}
97+
}
98+
99+
await github.rest.issues.createComment({
100+
owner: context.repo.owner,
101+
repo: context.repo.repo,
102+
issue_number: context.issue.number,
103+
body: commentBody,
104+
});

docs/internals/contributing/writing-code/submitting-patches.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,9 @@ All code changes
438438
* If the change is backwards incompatible in any way, is there a note
439439
in the release notes (``docs/releases/A.B.txt``)?
440440
* Is Django's test suite passing?
441+
* If there is a :ref:`code coverage report <code-coverage-on-pull-requests>`
442+
comment on the pull request, have you reviewed the missing coverage in
443+
context (considering database/platform-specific limitations)?
441444
* If the change affects the Django admin or rendered HTML output, has
442445
:ref:`accessibility testing <accessibility-testing-baseline>` been done?
443446

docs/internals/contributing/writing-code/unit-tests.txt

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,6 +394,49 @@ settings file defines ``coverage_html`` as the output directory for the report
394394
and also excludes several directories not relevant to the results
395395
(test code or external code included in Django).
396396

397+
.. _code-coverage-on-pull-requests:
398+
399+
Code coverage on pull requests
400+
------------------------------
401+
402+
Django's continuous integration (CI) system automatically runs code coverage
403+
analysis on pull requests and posts a comment with a diff coverage report. This
404+
helps reviewers see which lines in the changed code are covered by tests.
405+
406+
**What the coverage report shows:**
407+
408+
The coverage report posted on pull requests uses `diff-cover`_ to analyze only
409+
the lines that were changed or added in the PR. It shows:
410+
411+
* Lines that are covered by tests (✓)
412+
* Lines that are not covered by tests (✗)
413+
* Lines that cannot be covered (e.g., comments, blank lines)
414+
415+
.. _diff-cover: https://github.com/Bachmann1234/diff_cover
416+
417+
**Important limitations:**
418+
419+
When reviewing coverage reports on pull requests, keep these limitations in
420+
mind:
421+
422+
* **Database-specific code:** The CI coverage job runs tests using SQLite on
423+
Windows. Code paths specific to other databases (PostgreSQL, MySQL, Oracle)
424+
will appear as "not covered" even if database-specific tests exist. This is
425+
expected and acceptable.
426+
427+
* **Platform-specific code:** Similarly, code that only runs on certain
428+
operating systems (Linux, macOS) will appear as not covered when run on
429+
Windows.
430+
431+
* **Coverage doesn't equal quality:** A line being "covered" only means it was
432+
executed during tests. It doesn't guarantee the line is well-tested or that
433+
all edge cases are handled. During review, assess test quality beyond just
434+
coverage numbers.
435+
436+
437+
Missing coverage should be considered a warning rather than a blocker and
438+
should be evaluated in context.
439+
397440
.. _contrib-apps:
398441

399442
Contrib apps

zizmor.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
rules:
22
dangerous-triggers:
33
ignore:
4+
- coverage.yml
45
- labels.yml
56
- new_contributor_pr.yml
67
unpinned-uses:

0 commit comments

Comments
 (0)