ci: enforce coverage quality gates

[Pouyanpi]

Description

This change turns the existing package coverage signal into a staged quality
gate now that the full test suite consistently reports approximately 86%
coverage.

It makes the following changes:

• Enforces an 85% package-wide floor in the canonical coverage job with
  pytest-cov's --cov-fail-under option.
• Restores a terminal coverage report alongside the existing XML report so the
  measured total and failure reason are visible directly in CI logs.
• Makes the Codecov project status non-informational, compares it with the base
  commit using target: auto, and permits a 0.25 percentage-point tolerance.
• Sets a 90% patch coverage target while keeping the patch status informational
  during the initial observation period.
• Limits Codecov PR comments to project coverage drops or uncovered patch lines
  to reduce comments that do not require contributor attention.

The absolute 85% CI floor protects against a large package-wide regression even
when no usable Codecov base report is available. The relative Codecov project
status prevents gradual erosion from the current baseline. The patch target
encourages changed code to exceed the historical package average without making
that new policy a merge gate before its behavior has been observed across
representative pull requests.

The coverage run remains limited to the existing Ubuntu/Python 3.11 matrix entry
in .github/workflows/pr-tests.yml; this avoids repeating an equivalent coverage
measurement across the full Python compatibility matrix.

References

• Codecov project and patch status semantics:
  Status Checks
• Codecov blocking, relative, and patch coverage examples:
  Common Configurations
• Codecov PR comment filters, including coverage_drop and uncovered_patch:
  Pull Request Comments
• Codecov repository YAML locations and validation:
  Codecov YAML
• pytest-cov definitions for --cov-fail-under and multiple report formats:
  Configuration
• pytest-cov explanation that explicit non-terminal reports suppress the default
  terminal report:
  Reporting

Verification

• uv run --locked pre-commit run --files .github/codecov.yml .github/workflows/_test.yml
  • Passed YAML, whitespace, zizmor, and ty checks.
• curl --fail-with-body --silent --show-error --data-binary @.github/codecov.yml https://api.codecov.io/validate
  • Codecov returned Valid! and normalized the configured project and patch
    statuses as expected.
• make test-coverage ARGS="--cov-fail-under=85 --cov-report=term"
  • 5,254 tests passed and 178 were skipped.
  • Total package coverage was 86.05%.
  • The 85% coverage requirement passed.
  • The run used macOS, Python 3.12.7, and no live provider credentials.
• git diff --check
  • Passed.

Residual CI-specific behavior will be exercised by the PR's canonical Ubuntu,
Python 3.11 coverage job. The 90% patch target is intentionally informational;
promoting codecov/patch to a required check should be a separate maintainer
decision after observing representative PRs. Repository branch protection was
not modified by this change.

Checklist

• I've read the CONTRIBUTING guidelines.
• This PR links to a triaged issue assigned to me.
• My PR title follows the project commit convention.
• I've updated the documentation if applicable. No user documentation change is required for this CI-only policy.
• I've added tests if applicable. No new test behavior is introduced; the full package coverage suite passed.
• I've noted any verification beyond CI and any checks I couldn't run.
• I did not update generated changelog files manually.
• I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed. Complete after automated review runs.
• @ responsible for reviewing the proposed CI and Codecov policy.

Summary by CodeRabbit

• Tests

  • Increased automated test coverage requirements to 85%.
  • Added terminal coverage reporting to test runs.

• Chores

  • Updated coverage reporting rules to flag coverage drops and uncovered changes.
  • Applied stricter project coverage thresholds.

[coderabbitai]

📝 Walkthrough

Walkthrough

Coverage reporting and enforcement are tightened across Codecov configuration and the reusable test workflow. Patch and project thresholds, change requirements, and pytest coverage arguments are updated.

Changes

Coverage enforcement

Layer / File(s)	Summary
Coverage thresholds and CI invocation .github/codecov.yml, .github/workflows/_test.yml	Codecov project and patch settings update targets, thresholds, and change requirements; the coverage test command now enforces 85% coverage and prints a terminal report.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Test Results For Major Changes	✅ Passed	PASS: This is CI/config-only work, and the PR description includes a Verification section with concrete test results and coverage numbers.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: tightening CI coverage gates and Codecov thresholds.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch pouyanpi/coverage-threshold

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR converts the existing package-wide coverage measurement into a staged quality gate. The absolute CI floor (--cov-fail-under=85) ensures no base-report-free regression slips through, while the relative Codecov project status (target: auto, informational: false) guards against gradual erosion from the current ~86% baseline.

• .github/workflows/_test.yml: Adds --cov-fail-under=85 and --cov-report=term to the make test-coverage invocation; both flags accumulate correctly with the existing --cov-report=xml in the Makefile, so Codecov still receives its XML report.
• .github/codecov.yml: Promotes project status to a blocking gate (informational: false, target: auto, threshold: 0.25%), adds a non-blocking 90% patch target, and narrows PR comment noise to coverage drops or uncovered patch lines only.

Confidence Score: 5/5

Safe to merge — the change is additive CI policy with no production code impact and a verified 1 pp margin above the new floor.

Both changed files are CI/Codecov configuration only. The pytest-cov flags compose correctly with the existing Makefile target (multiple --cov-report options accumulate, XML output for Codecov is preserved). The ~86% current coverage gives roughly a 1 pp buffer above the enforced 85% floor, and the patch gate remains informational, so no PR is hard-blocked by an unproven policy. The Codecov YAML was validated against the Codecov API. No logic, security, or correctness concerns were identified.

No files require special attention.

Important Files Changed

Filename	Overview
.github/workflows/_test.yml	Appends --cov-fail-under=85 and --cov-report=term to the coverage make target; both flags accumulate correctly alongside the existing --cov-report=xml in the Makefile.
.github/codecov.yml	Promotes project status to a blocking gate (informational: false, target: auto, threshold: 0.25%), adds an informational 90% patch target, and restricts PR comments to coverage drops or uncovered patch lines.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PR opened / push] --> B{with-coverage?}
    B -- false --> C[make test]
    B -- true --> D["make test-coverage ARGS=--cov-fail-under=85 --cov-report=term"]
    D --> E[pytest-cov runs\nXML + terminal reports generated]
    E --> F{coverage >= 85%?}
    F -- No --> G[❌ CI fails — job exits non-zero]
    F -- Yes --> H[Upload coverage.xml to Codecov]
    H --> I{Codecov project status\ntarget: auto / threshold 0.25%}
    I -- drop > 0.25pp --> J[❌ Codecov project check fails\ninformational: false]
    I -- within threshold --> K[✅ Codecov project check passes]
    H --> L{Patch coverage >= 90%?}
    L -- No --> M[⚠️ Codecov patch check fails\ninformational: true — not blocking]
    L -- Yes --> N[✅ Patch check passes]
    H --> O{coverage_drop OR uncovered_patch?}
    O -- Yes --> P[Codecov PR comment posted]
    O -- No --> Q[No PR comment]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[PR opened / push] --> B{with-coverage?}
    B -- false --> C[make test]
    B -- true --> D["make test-coverage ARGS=--cov-fail-under=85 --cov-report=term"]
    D --> E[pytest-cov runs\nXML + terminal reports generated]
    E --> F{coverage >= 85%?}
    F -- No --> G[❌ CI fails — job exits non-zero]
    F -- Yes --> H[Upload coverage.xml to Codecov]
    H --> I{Codecov project status\ntarget: auto / threshold 0.25%}
    I -- drop > 0.25pp --> J[❌ Codecov project check fails\ninformational: false]
    I -- within threshold --> K[✅ Codecov project check passes]
    H --> L{Patch coverage >= 90%?}
    L -- No --> M[⚠️ Codecov patch check fails\ninformational: true — not blocking]
    L -- Yes --> N[✅ Patch check passes]
    H --> O{coverage_drop OR uncovered_patch?}
    O -- Yes --> P[Codecov PR comment posted]
    O -- No --> Q[No PR comment]

Loading

_{Reviews (4): Last reviewed commit: "ci: enforce coverage quality gates" | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
nemoguardrails/utils.py	75.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!