feat(ci): add per-integration unit test coverage reporting on PRs#365
feat(ci): add per-integration unit test coverage reporting on PRs#365Shubhank-Jonnada wants to merge 3 commits into
Conversation
Adds a coverage.yml workflow that: - Detects which integration directories changed on a PR - Runs pytest --cov scoped to each changed integration - Posts a PR comment with coverage % and trend vs master baseline - Updates the master baseline on merge via .coverage-baselines/
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a81e6b3d5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| python -m pytest ${{ matrix.integration }}/tests/test_*_unit.py \ | ||
| --cov=${{ matrix.integration }} \ | ||
| --cov-report=json:coverage.json \ | ||
| --cov-report=term-missing \ | ||
| -m unit \ | ||
| --tb=short \ | ||
| -q 2>&1 | tee test_output.txt |
There was a problem hiding this comment.
When any unit test fails, this step can still succeed because the workflow uses the unspecified GitHub Actions shell, which runs as bash -e {0} without pipefail; the pipeline therefore returns tee's exit status instead of pytest's. In that scenario the job continues, extracts whatever coverage file exists, and posts a successful coverage comment even though the unit-test command failed.
Useful? React with 👍 / 👎.
| pull_request: | ||
| branches: [master, main] |
There was a problem hiding this comment.
With this workflow triggered only by pull_request, the update-baseline job's github.ref_name == 'master' condition is never true for the merge-to-master event; pull_request runs use the PR ref rather than executing after the target branch is updated. As a result .coverage-baselines/ never refreshes on merge, so future PR comments will keep showing no baseline or stale trends.
Useful? React with 👍 / 👎.
| - name: Post coverage comment | ||
| uses: actions/github-script@v7 |
There was a problem hiding this comment.
For PRs opened from forks that touch an integration, this direct pull_request job attempts to create or update the coverage comment with the downgraded read-only token and will fail the otherwise successful coverage run. The existing validation workflow explicitly avoids direct PR-run commenting for this reason (validate-integration.yml lines 24-28), so coverage comments need the same base-context posting path or a fork-safe guard.
Useful? React with 👍 / 👎.
🔍 Integration Validation ResultsCommit: Changed directories:
|
Unit test coverage —
|
1 participant
Summary
.github/workflows/coverage.ymlthat runs on every PR targetingmaster.coverage-baselines/on merge to master so future PRs have a comparison pointrequirements-test.txtwhich already haspytest-cov— no per-integration changes neededHow it works
config.json+ unit tests)pytest --cov.coverage-baselines/<integration>.jsonTest plan
.coverage-baselines/is updated