Skip to content

ENH: Add fix-nightly-warnings skill and CDash query scripts#5923

Open
blowekamp wants to merge 12 commits intoInsightSoftwareConsortium:mainfrom
blowekamp:skill_fix_nightly_warnings
Open

ENH: Add fix-nightly-warnings skill and CDash query scripts#5923
blowekamp wants to merge 12 commits intoInsightSoftwareConsortium:mainfrom
blowekamp:skill_fix_nightly_warnings

Conversation

@blowekamp
Copy link
Copy Markdown
Member

@blowekamp blowekamp commented Mar 11, 2026

Summary

Adds a GitHub Copilot agent skill for the workflow of fixing ITK nightly build warnings reported on CDash, along with two helper Python scripts for querying the CDash GraphQL API.

Changes

  • SKILL.md — Skill definition describing the step-by-step procedure: fetch nightly build warnings from CDash, identify warning patterns, apply minimal source fixes, build affected modules to verify, and open a draft PR.
  • list_nightly_warnings.py — CLI script to list recent CDash nightly builds and their warning counts via the GraphQL API.
  • get_build_warnings.py — CLI script to fetch and display detailed warning messages for a specific CDash build ID via the GraphQL API.

Notes

No production source changes; purely development tooling.

@github-actions github-actions bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Enhancement Improvement of existing methods or implementation area:Python wrapping Python bindings for a class labels Mar 11, 2026
@blowekamp
Copy link
Copy Markdown
Member Author

I have had some success with this skill. Adding the scripts is a way to reduce the number of requests needed for the AI, and improve efficiency. I have am running low on available requests with the initial development of this skill.

Someone else is welcome to further refine this workflow.

Copy link
Copy Markdown
Member

@dzenanz dzenanz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good on a glance.

Copy link
Copy Markdown
Member

@thewtex thewtex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool!!

We may want to set up a GitHub Agentic Workflow to run this on a regular basis.

@hjmjohnson
Copy link
Copy Markdown
Member

hjmjohnson commented Mar 18, 2026

@blowekamp I have some improvements ready for this PR (typo fixes, ThirdParty filtering, MSVC warning support, a triage_nightly.py entry point, etc.).

To let me push directly to your PR branch, could you enable "Allow edits from maintainers"? You can do this by:

  1. Scrolling to the bottom of this PR page
  2. Checking the box "Allow edits and access to secrets by maintainers"

(This option is in the right sidebar under the PR description when you're editing the PR, or at the bottom of the PR creation/edit page.)

Once enabled, I'll be able to git push to your fork's skill_fix_nightly_warnings branch directly. Alternatively, I can close this PR and open a new one from upstream/skill_fix_nightly_warnings where I've already pushed the changes — let me know your preference.

======
FYI: I asked claude to make separate commits with recommendations to make this skill more automated and efficient:

https://github.com/InsightSoftwareConsortium/ITK/tree/skill_fix_nightly_warnings

The 10 commits should probably be squished, but they seem reasonable for draft consideration.

@blowekamp
Copy link
Copy Markdown
Member Author

That option is already enabled.

blowekamp and others added 11 commits April 9, 2026 13:25
Address PR review comment (opens -> open), fix --limit flag typo
(-limit -> --limit), correct jq pipeline to operate on .entries
for get_build_warnings.py JSON output, and add explicit script
directory path for agentic automation clarity.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --exclude-thirdparty flag to get_build_warnings.py to filter
out warnings from Modules/ThirdParty/ paths. Update jq pipeline
example in SKILL.md to filter ThirdParty in the JSON output.
Warnings from vendored third-party code should not be fixed in ITK.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Provide a specific build command for local verification of warning
fixes. Reference CMake/CTestCustom.cmake.in as the authoritative
list of already-suppressed warning patterns to avoid duplicate work.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instruct agents to deduplicate warnings by (sourceFile, flag)
across multiple CDash builds before analyzing. Make user-review
step conditional on interactive vs. agentic execution mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The display limit was computed into displayed_nodes but never used;
JSON output iterated the untruncated nodes list instead. Replace
displayed_nodes with in-place truncation of nodes so --limit is
applied consistently to both human-readable and JSON output modes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add MSVC_FLAG_RE pattern to extract C4805, C4267, etc. warning
codes from MSVC compiler output. Falls back to MSVC pattern when
no GCC/Clang -W flag is found, improving grouping accuracy for
Windows CDash builds.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a quick-reference table mapping common compiler warning flags
to their typical mechanical fixes. Reduces agent research time for
routine warnings like -Wshadow, -Wunused-parameter, and MSVC C4805.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add repeatable --exclude PATTERN flag to get_build_warnings.py
that filters entries whose sourceFile contains the given pattern.
Consolidates --exclude-thirdparty into the same filtering logic.
Useful for excluding specific directories beyond ThirdParty.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a script that chains list -> fetch -> deduplicate into one
invocation, producing an actionable summary of warnings grouped
by flag. Automatically excludes ThirdParty paths and deduplicates
across builds. Reduces tool calls needed for agentic automation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a flag that filters builds to only those with more warnings
than the minimum across recent builds. This identifies regressions
from a clean baseline rather than showing all builds with any
warnings, which is more useful for automated triage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@hjmjohnson hjmjohnson force-pushed the skill_fix_nightly_warnings branch from e0bcf58 to 7997b88 Compare April 9, 2026 18:26
- Fix typo: "opens a PR" -> "open a PR" in SKILL.md (thewtex review)
- Rename nodes -> displayed_nodes after limit is applied in
  list_nightly_warnings.py to distinguish the display-limited subset
  from the full filtered list (hjmjohnson review)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@hjmjohnson hjmjohnson marked this pull request as ready for review April 9, 2026 18:37
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 9, 2026

Greptile Summary

This PR adds a GitHub Copilot agent skill (fix-nightly-warnings) with three Python helper scripts for querying the CDash GraphQL API, triaging ITK nightly build warnings, and automating the fix-and-PR workflow. There are no production source changes.

  • triage_nightly.py line 116: the empty-builds JSON branch emits \"warnings\": [] while the normal path emits \"warnings_by_flag\": [...] — any agent or jq pipeline reading .warnings_by_flag will silently get null instead of an empty list.

Confidence Score: 4/5

Safe to merge after fixing the JSON key inconsistency in triage_nightly.py.

One P1 defect: the empty-builds branch in triage_nightly.py emits "warnings" instead of "warnings_by_flag", which silently breaks any consumer (agent or jq pipeline) that reads the normal key. The remaining findings are P2 style/quality issues. Per the confidence guidance, a P1 finding keeps the score at 4.

triage_nightly.py — fix the JSON key mismatch in the early-return branch (line 116).

Vulnerabilities

No security concerns identified. The scripts only perform read-only queries against a public CDash GraphQL API endpoint; no credentials are handled, no user data is processed, and no code is executed from remote sources.

Important Files Changed

Filename Overview
.github/skills/fix-nightly-warnings/SKILL.md Agent skill definition for the nightly-warning triage workflow; minor: Quality Checks section uses bare [] instead of - [ ], so checkboxes won't render correctly on GitHub.
.github/skills/fix-nightly-warnings/scripts/get_build_warnings.py Paginated CDash GraphQL fetch for per-build warnings; pagination, exclusion filters, and output modes are all correct.
.github/skills/fix-nightly-warnings/scripts/list_nightly_warnings.py Lists CDash builds with warnings/errors; logic for nightly-start window, sorting, and --new-warnings-only baseline heuristic is correct.
.github/skills/fix-nightly-warnings/scripts/triage_nightly.py Orchestrates list + fetch into a deduplicated triage report; has a P1 JSON key inconsistency ("warnings" vs "warnings_by_flag" in the empty-builds branch) and silently swallows per-build fetch errors with no diagnostic output.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant triage_nightly.py
    participant get_build_warnings.py
    participant CDash GraphQL

    Agent->>triage_nightly.py: run (--type, --since, --json)
    triage_nightly.py->>CDash GraphQL: project builds query (QUERY_TEMPLATE)
    CDash GraphQL-->>triage_nightly.py: edges [{node: {id, name, warnings, errors}}]
    loop for each build
        triage_nightly.py->>get_build_warnings.py: fetch_entries(build_id, WARNING, limit)
        get_build_warnings.py->>CDash GraphQL: build warnings query (paginated)
        CDash GraphQL-->>get_build_warnings.py: warning entries
        get_build_warnings.py-->>triage_nightly.py: (build_meta, entries)
        triage_nightly.py->>triage_nightly.py: deduplicate by (sourceFile, flag)
    end
    triage_nightly.py-->>Agent: JSON {builds_inspected, builds, warnings_by_flag}
Loading

Reviews (1): Last reviewed commit: "STYLE: Address review comments on fix-ni..." | Re-trigger Greptile

Comment on lines +114 to +119
if not builds_with_warnings:
if args.json_output:
print(json.dumps({"builds_inspected": 0, "warnings": []}, indent=2))
else:
print("No builds with warnings found.", file=sys.stderr)
return
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Inconsistent JSON key in empty-builds case

When no builds are found, the JSON output uses the key "warnings", but the normal execution path uses "warnings_by_flag". Any downstream consumer (e.g., an agent or jq pipeline) that reads .warnings_by_flag will get null for the empty case instead of [], silently producing incorrect behaviour.

Suggested change
if not builds_with_warnings:
if args.json_output:
print(json.dumps({"builds_inspected": 0, "warnings": []}, indent=2))
else:
print("No builds with warnings found.", file=sys.stderr)
return
if not builds_with_warnings:
if args.json_output:
print(json.dumps({"builds_inspected": 0, "warnings_by_flag": []}, indent=2))
else:
print("No builds with warnings found.", file=sys.stderr)
return

Comment on lines +132 to +137
try:
_meta, entries = fetch_entries(
str(build_id), "WARNING", args.limit_warnings
)
except SystemExit:
continue
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Silent failure when all warning-fetches fail

fetch_entries calls sys.exit(2) on any network error. Catching SystemExit swallows those errors, so if every per-build fetch fails the script exits with code 0 and produces an empty result — the agent will interpret this as "no warnings" and skip the triage step entirely. At minimum, a warning should be printed for each skipped build so the caller can tell whether the result is complete.

Suggested change
try:
_meta, entries = fetch_entries(
str(build_id), "WARNING", args.limit_warnings
)
except SystemExit:
continue
try:
_meta, entries = fetch_entries(
str(build_id), "WARNING", args.limit_warnings
)
except SystemExit:
print(f"Warning: failed to fetch warnings for build {build_id} ({build_name}), skipping.", file=sys.stderr)
continue

Comment on lines +118 to +121
[] All targeted warnings are fixed
[] No new warnings or errors introduced
[] Changes are limited to the files affected by the warnings
[] Commit message clearly describes the fix and references the CDash issue if applicable
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Checklist items won't render as checkboxes

GitHub Flavored Markdown requires the - [ ] syntax for interactive checkboxes. The bare [] used here will render as plain text, not tick-boxes.

Suggested change
[] All targeted warnings are fixed
[] No new warnings or errors introduced
[] Changes are limited to the files affected by the warnings
[] Commit message clearly describes the fix and references the CDash issue if applicable
- [ ] All targeted warnings are fixed
- [ ] No new warnings or errors introduced
- [ ] Changes are limited to the files affected by the warnings
- [ ] Commit message clearly describes the fix and references the CDash issue if applicable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Python wrapping Python bindings for a class type:Enhancement Improvement of existing methods or implementation type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants