You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ci: gate docstring quality and coverage in CI (#616) (#689)
* ci: gate docstring quality and coverage in CI (#616)
Add a hard-fail docstring quality gate to the docs-publish workflow:
- New 'Docstring quality gate' step runs --quality --fail-on-quality
--threshold 100; fails if any quality issue is found or coverage
drops below 100% (both currently pass in CI)
- Existing audit_coverage step (soft-fail, threshold 80) retained for
the summary coverage metric
Add typeddict_mismatch checks to audit_coverage.py:
- typeddict_phantom: Attributes: documents a field not declared in the TypedDict
- typeddict_undocumented: declared field absent from Attributes: section
- Mirrors the existing param_mismatch logic for functions
Pre-commit: enable --fail-on-quality on the manual-stage hook (CI is
the hard gate; hook remains stages: [manual] as docs must be pre-built).
Update CONTRIBUTING.md and docs/docs/guide/CONTRIBUTING.md with TypedDict
docstring requirements and the two new audit check kinds.
* style: fix ruff formatting in audit_coverage.py
* ci: remove soft-fail mode labels from docs workflow summary
* docs: improve doc check reporting with line numbers, GHA annotations, and fix hints
audit_coverage.py:
- Add file/line fields to every issue dict (repo-relative path + def line)
- _print_quality_report now shows [file:line] per issue, per-kind Fix:/Ref:
hints linking to CONTRIBUTING.md anchors, and emits ::error file=...,line=...
GHA annotations so issues appear inline in PR diffs
- Cap GHA annotations at 10 per check kind with "N more in job log" notice
- Add _KIND_FIX_HINTS and _gha_file_annotation helpers; _CONTRIB_DOCS_URL constant
validate.py:
- Convert all check functions from list[str] to list[dict] errors (file/line/message)
- Add line-number tracking to validate_source_links, validate_internal_links,
validate_anchor_collisions, and validate_doc_imports
- Emit per-error GHA annotations with file/line; shared 20-annotation budget
across all checks so every category gets representation in PR diff
- Fix icon bug: summary rows now use correct pass/fail icon
- Group detailed errors by check type with section headers
docs-publish.yml:
- Add --orphans and --output /tmp/quality_report.json to quality gate step
- Upload quality_report.json as docstring-quality-report artifact (30-day retention)
pyproject.toml:
- cli/**/*.py: suppress only D2/D3/D4xx style rules; enable D1xx (missing
docstrings) as a ruff-level complement to the audit_coverage quality gate
docs/docs/guide/CONTRIBUTING.md:
- Add CI docstring checks reference section with per-kind tables (fix instructions
+ anchors) for all 11 check kinds across 4 categories
- Add callout explaining GHA annotation cap (10 per kind) and where to find
the full list (job log + JSON artifact)
* fix: repair orphan check for Mintlify v2 docs.json and improve summary links
audit_coverage.py (audit_nav_orphans):
- Probe docs/docs/docs.json before docs/mint.json so both Mintlify v1
and v2 nav configs are supported
- Extend _extract to handle plain string page entries used by docs.json
(v2 uses "pages": ["api/..."] strings; v1 used {"page": "api/..."} dicts)
- Previously mint.json was never found, nav_refs stayed empty, and every
MDX file was reported as an orphan
docs-publish.yml (Write job summary):
- When the quality gate fails, render a prominent markdown callout with a
direct link to the CI docstring checks reference section in CONTRIBUTING.md
- Add a per-kind fix reference table with clickable anchor links to each
category section (missing/short, args/returns, class Option C, TypedDict)
- Per-kind Ref: URLs in the raw log are inside a text block and do
not render as links in the step summary; this table surfaces them rendered
* ci: per-kind quality breakdown in summary table, drop misleading skipped notice
docs-publish.yml:
- Parse per-kind counts from _print_quality_report section headers in the
quality gate log (e.g. "Missing docstrings (12)") and show them as a
comma-separated breakdown in the Docstring Quality table cell instead of
just the total — gives developers an immediate view of which categories
are failing without expanding the log
audit_coverage.py:
- Remove the "skipped (pass --quality to enable)" GHA notice emitted by
the coverage-only step; there is always a dedicated quality gate step
immediately after so the notice was misleading and redundant
* docs: add fix hints and doc refs to coverage and MDX validation output
audit_coverage.py:
- Coverage miss section now shows a structured Fix:/Ref: block with the
exact generate-ast.py command and a link to CONTRIBUTING.md#validating-docstrings
- Missing symbols listed one per line (symbol indented under module) for
scannability instead of comma-joined on one long line
- Emit a ::error or ::warning GHA annotation with symbol/module counts
when coverage symbols are undocumented
validate.py:
- Add _CHECK_FIX_HINTS dict mapping each check label to a (fix text, ref URL)
pair, covering all 8 check types with specific fix instructions and links
into CONTRIBUTING.md (root or guide as appropriate)
- _print_check_errors now prints Fix:/Ref: under each section header,
matching the pattern established by _print_quality_report
* docs: add annotation gap checks and public-API-only doc filter
audit_coverage.py:
- Add missing_param_type check: fires when Args: section exists but one
or more concrete params lack Python type annotations; naturally
non-overlapping with no_args (which fires when section is absent)
- Add missing_return_type check: fires when Returns: section is
documented but the function has no return annotation; naturally
non-overlapping with no_returns (annotation exists but section absent)
- Add fix hints and CONTRIBUTING.md anchors for both new check kinds
- Update kind_labels and iteration order in _print_quality_report
generate-ast.py:
- Add remove_internal_modules() post-generation filter step
- Uses AST-based import analysis: a submodule is internal when the
parent __init__.py imports from at least one sibling submodule but
not from this one (import-based visibility, not __all__ name-matching)
- Conservative: keeps module when parent imports nothing (indeterminate)
or __init__.py cannot be parsed
- _CONFIRMED_INTERNAL_MODULES hardcoded set for known internals where
parent imports nothing (json_util, backend_instrumentation); these
should eventually be renamed with _ prefix per Python convention
- Package index files (stem == parent dir) are never filtered
docs.json: nav regenerated by build; internal modules removed from nav
CONTRIBUTING.md: add missing_param_type / missing_return_type to CI
docstring checks reference table
docs-publish.yml: add both new kinds to summary kind_short and
kind_anchors fix-reference table
* fix: align coverage scope with doc generator's public-API filter
audit_coverage.py was walking all non-_-prefixed source modules via
Griffe, including internal modules (json_util, backend_instrumentation,
etc.) whose MDX files were removed by remove_internal_modules() in
generate-ast.py. This caused coverage to drop because those symbols
were no longer 'documented' but were still counted in the denominator.
Apply the same import-based public-API filter in discover_public_symbols():
skip submodules that the parent __init__.py does not import from, mirroring
the generate-ast.py logic. _CONFIRMED_INTERNAL_MODULES kept in sync.
Also drop the per-kind anchor table from the GHA job summary. Anchors
in GitHub Actions summaries only navigate to the top of the referenced
document, so the table added noise without working links.
* feat: add param_type_mismatch and return_type_mismatch docstring checks
Adds two new quality check kinds that fire when the type explicitly
stated in an Args:/Returns: docstring entry disagrees with the Python
type annotation in the function signature:
param_type_mismatch — 'param (OldType): ...' vs annotation 'NewType'
return_type_mismatch — 'Returns: OldType: ...' vs annotation '-> NewType'
Both checks fire only when BOTH sides have an explicit type; one-sided
absence is already handled by missing_param_type / missing_return_type.
Type comparison uses _types_match() / _normalize_type() which handles:
- typing aliases: List→list, Dict→dict, Optional→X|None, Union→A|B
- typing. prefix stripping
- pipe-union component ordering (str|None == None|str)
- incidental whitespace
Known conservative suppressions (prefer false negatives over false positives,
since there is no per-site suppression mechanism):
- Nested generics not fully expandable by regex (e.g. Optional[list[str]])
are silently skipped — both sides must fully normalise to be compared
- Union with bracket-containing members
- Callable argument ordering
* docs: document param_type_mismatch and return_type_mismatch check kinds
* fix: correct parent __init__.py path for module files in _is_public_submodule
For a module file mellea/pkg/submodule.py, Griffe gives filepath ending in
.py (not __init__.py). The parent __init__.py is fp.parent/__init__.py.
The previous code used fp.parent.parent which is correct for packages
(whose filepath IS the __init__.py) but goes one level too far for plain
module files — it was checking the grandparent init instead of the parent.
Effect: genslot, react, unit_test_eval and similar non-exported modules
in stdlib/components were incorrectly counted as public symbols, inflating
the denominator and lowering the reported coverage percentage.
* style: fix ruff formatting and EN dash in validate.py
* revert: restore full D suppression for cli/ (see #705)
* docs: add artifact download link to quality gate failure summary
* feat: add --warn-only to validate.py for pre-commit informational mode
* docs: fix 36 docstring quality gate failures across 17 files
- Fix missing_param_type, missing_return_type, param_type_mismatch,
return_type_mismatch, no_args, no_returns, and missing docstring issues
- Add TYPE_CHECKING imports for HuggingFace types in util.py with
type: ignore[union-attr] for pre-existing None-safety gaps
- Add Granite3ChatCompletion import to granite32/33 input.py for
correct sanitize() parent signature match
- Convert reST-style docstrings to Google style in intrinsics/input.py
- Document AST single-quote normalization for Literal types in
CONTRIBUTING.md
* fix: suppress pre-existing mypy errors exposed by new type annotations
Adding TYPE_CHECKING annotations to util.py made mypy check function
bodies it previously skipped (untyped params = implicit Any = no body
checking). This exposed a pre-existing Tensor-not-callable issue and
a dict-variance issue in mobject.py. Suppress with targeted type:
ignore comments — these are not new bugs, just newly visible ones.
0 commit comments