Skip to content

Commit 868a49d

Browse files
sayakpauldanieldk
sayakpaul
and
danieldk
authored
feat: implement coverage reporting (#582)
* up * limit marker to github bot. * COVERAGE_CELL -> CHECK_COVERAGE * Apply suggestions from code review Co-authored-by: Daniël de Kok <me@danieldk.eu> --------- Co-authored-by: Daniël de Kok <me@danieldk.eu>
1 parent dd6b5be commit 868a49d

4 files changed

Lines changed: 165 additions & 1 deletion

File tree

‎.github/workflows/coverage_comment.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
name: Post coverage comment
2+
3+
on:
4+
workflow_run:
5+
workflows: ["Test kernels"]
6+
types: [completed]
7+
8+
concurrency:
9+
group: coverage-comment-${{ github.event.workflow_run.head_sha }}
10+
cancel-in-progress: true
11+
12+
jobs:
13+
post:
14+
name: Post coverage comment
15+
runs-on: ubuntu-latest
16+
if: >-
17+
github.event.workflow_run.conclusion == 'success' &&
18+
github.event.workflow_run.event == 'pull_request'
19+
permissions:
20+
contents: read
21+
pull-requests: write
22+
23+
steps:
24+
- name: Download coverage artifact
25+
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
26+
with:
27+
name: coverage-report
28+
path: coverage-artifact
29+
github-token: ${{ secrets.GITHUB_TOKEN }}
30+
run-id: ${{ github.event.workflow_run.id }}
31+
32+
- name: Post or update sticky comment
33+
env:
34+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
35+
REPO: ${{ github.repository }}
36+
run: |
37+
set -euo pipefail
38+
39+
# PR number from artifact (not workflow_run.pull_requests, which is
40+
# empty for fork PRs).
41+
PR_NUMBER=$(cat coverage-artifact/pr_number.txt)
42+
43+
# Validate: must be a positive integer. Defends against a malicious
44+
# PR replacing pr_number.txt with arbitrary content that we'd then
45+
# interpolate into a URL.
46+
if ! [[ "${PR_NUMBER}" =~ ^[1-9][0-9]*$ ]]; then
47+
echo "::error::pr_number.txt does not contain a valid PR number: ${PR_NUMBER}"
48+
exit 1
49+
fi
50+
51+
# Only match comments authored by the GITHUB_TOKEN bot, so a human
52+
# comment that happens to start with the marker doesn't get
53+
# silently overwritten.
54+
EXISTING=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
55+
--jq 'first(
56+
.[]
57+
| select(.user.login == "github-actions[bot]"
58+
and (.body | startswith("<!-- coverage-report -->")))
59+
| .id
60+
) // empty')
61+
62+
# Body comes from the untrusted upstream run; pass via --rawfile +
63+
# --input - so it's never shell-interpolated.
64+
if [ -n "$EXISTING" ]; then
65+
jq -n --rawfile body coverage-artifact/body.md '{body: $body}' \
66+
| gh api --method PATCH "repos/${REPO}/issues/comments/${EXISTING}" --input -
67+
else
68+
jq -n --rawfile body coverage-artifact/body.md '{body: $body}' \
69+
| gh api --method POST "repos/${REPO}/issues/${PR_NUMBER}/comments" --input -
70+
fi

‎.github/workflows/test_kernels.yaml‎

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ jobs:
3131

3232
env:
3333
UV_PYTHON_PREFERENCE: only-managed
34+
CHECK_COVERAGE: ${{ matrix.python-version == '3.10' && matrix.torch-version == '2.12.0' }}
3435

3536
steps:
3637
- name: Checkout code
@@ -66,7 +67,14 @@ jobs:
6667
env:
6768
HF_HUB_DOWNLOAD_TIMEOUT: 60
6869
run: |
69-
uv run pytest tests
70+
if [ "${CHECK_COVERAGE}" = "true" ]; then
71+
uv run pytest tests \
72+
--cov=kernels \
73+
--cov-report=term-missing \
74+
--cov-report=json:coverage.json
75+
else
76+
uv run pytest tests
77+
fi
7078
7179
- name: Re-run dependency test with dependencies installed
7280
working-directory: ./kernels
@@ -98,3 +106,63 @@ jobs:
98106
run: |
99107
uv run kernels check kernels-community/activation
100108
uv run kernels versions kernels-community/activation
109+
110+
# This is done to securely run the coverage test and to post comments even
111+
# on fork PRs.
112+
- name: Render coverage report for PR comment
113+
if: env.CHECK_COVERAGE == 'true' && github.event_name == 'pull_request'
114+
working-directory: ./kernels
115+
env:
116+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
117+
PR_NUMBER: ${{ github.event.pull_request.number }}
118+
PY_VERSION: ${{ matrix.python-version }}
119+
TORCH_VERSION: ${{ matrix.torch-version }}
120+
run: |
121+
set -euo pipefail
122+
mkdir -p coverage-artifact
123+
124+
PCT=$(jq -r '.totals.percent_covered' coverage.json)
125+
PCT_INT=$(printf '%.0f' "$PCT")
126+
PCT_DISPLAY=$(printf '%.1f' "$PCT")
127+
if [ "$PCT_INT" -ge 80 ]; then
128+
EMOJI=":white_check_mark:"
129+
elif [ "$PCT_INT" -ge 70 ]; then
130+
EMOJI=":warning:"
131+
else
132+
EMOJI=":red_circle:"
133+
fi
134+
135+
uv run coverage report --format=markdown > coverage-artifact/cov-table.md
136+
137+
{
138+
echo "<!-- coverage-report -->"
139+
echo "## Coverage report — \`kernels/\`"
140+
echo
141+
echo "> Measured on: **Python ${PY_VERSION} / Torch ${TORCH_VERSION}**."
142+
echo "> Other CI configurations are not included in this number."
143+
echo "> Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner."
144+
echo
145+
echo "**Total coverage: \`${PCT_DISPLAY}%\`** — threshold: 80% — ${EMOJI}"
146+
echo
147+
echo "<details>"
148+
echo "<summary>Per-file breakdown</summary>"
149+
echo
150+
cat coverage-artifact/cov-table.md
151+
echo
152+
echo "</details>"
153+
echo
154+
echo "_Updated by the \`Test kernels\` workflow on commit \`${HEAD_SHA}\`._"
155+
} > coverage-artifact/body.md
156+
157+
# `workflow_run.pull_requests` is empty for fork PRs, so the
158+
# downstream comment workflow reads the PR number from here.
159+
echo "${PR_NUMBER}" > coverage-artifact/pr_number.txt
160+
161+
- name: Upload coverage artifact
162+
if: env.CHECK_COVERAGE == 'true' && github.event_name == 'pull_request'
163+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
164+
with:
165+
name: coverage-report
166+
path: kernels/coverage-artifact/
167+
retention-days: 1
168+
if-no-files-found: error

‎kernels/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,13 @@ the Hub.
4848
## 📚 Documentation
4949

5050
Read the [documentation of kernels](https://huggingface.co/docs/kernels/).
51+
52+
## Test coverage
53+
54+
To reproduce the coverage number reported on PRs locally:
55+
56+
```bash
57+
uv run pytest --cov=kernels --cov-report=term-missing tests
58+
```
59+
60+
CI measures coverage on a single canonical matrix cell (Python 3.10 / Torch 2.12.0) and posts a sticky comment on the PR; the threshold is 80% (warn-only — the check stays green either way).

‎kernels/pyproject.toml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ dev = [
3030
"pytest>=8",
3131
# Whatever version is compatible with pytest.
3232
"pytest-benchmark",
33+
"pytest-cov>=5",
3334
"torch>=2.5",
3435
"apache-tvm-ffi>=0.1.9,<0.2.0",
3536
"types-pyyaml",
@@ -89,3 +90,18 @@ lint.ignore = ["E501"]
8990
lint.select = ["E", "F", "I", "W"]
9091

9192
[tool.ruff.format]
93+
94+
[tool.coverage.run]
95+
source = ["kernels"]
96+
branch = false
97+
relative_files = true
98+
omit = [
99+
"*/benchmarks/*",
100+
"*/benchmark.py",
101+
"*/cli/*",
102+
"*/_windows.py",
103+
]
104+
105+
[tool.coverage.report]
106+
show_missing = true
107+
skip_empty = true

0 commit comments

Comments
 (0)