Skip to content

CI: auto-file ai-fix issues from coverage manifest diffs after data updates #69

Description

@jay9297

Goal

Close the loop from audit to fix: when a data update lands on dev and the coverage manifests change, automatically file well-scoped issues for the newly-unsupported items so the existing ai-fix pipeline can work through them. This turns "new patch merged" into "triaged work queue" with zero manual effort.

Depends on: #67 (mod-coverage manifest) and ideally #68 (skill-stat manifest). Do not start until #67 is merged.

Background (read first)

What to build

A workflow .github/workflows/audit-triage.yml:

Trigger: push to dev, with a paths filter on audit/*.txt.

Steps:

  1. Check out with enough history to diff (fetch-depth: 2 is sufficient for the pushed range; use git diff HEAD^ HEAD -- audit/ — handle the first-commit edge case by skipping).
  2. Parse the diff with a small script (Python is fine for workflow tooling; keep it in tools/triage_audit_diff.py):
    • Added unsupported/partial lines in audit/mod-coverage.txt → new unsupported mod forms.
    • Added lines in audit/skill-stat-coverage.txt → new unmapped stat IDs.
    • Removed lines are good news — ignore them (the manifest commit itself is the record).
  3. Group added mod forms into batches of at most 15 per issue (one giant issue is not actionable; per-line issues are spam). Group by shared leading keyword where easy, otherwise just chunk.
  4. For each batch, create an issue via gh issue create (the default GITHUB_TOKEN with issues: write permission) with:
    • Title: Coverage: support N new unsupported mod lines (from <short-sha>) or Coverage: map N new skill stat IDs (from <short-sha>)
    • Body: the exact manifest lines in a table, a pointer to CLAUDE.md rules (ModParser changes need spec/System/TestModParser_spec.lua cases; SkillStatMap changes need --tags data to pass), and a note that src/Data/ModCache.lua must be regenerated and committed.
    • Labels: enhancement and ai-fix — applying ai-fix is what queues the fix agent.
  5. De-duplication guard: before creating, search existing open issues (gh issue list --search) for the same manifest lines / short-sha and skip duplicates. The workflow must be safe to re-run.
  6. Rate guard: cap at 10 issues per run; if the diff would produce more, create the first 10 plus one summary issue listing the remainder (a giant upstream merge should not file 60 issues).

Acceptance criteria

  • Workflow YAML passes actionlint (or at minimum loads without syntax errors on a dummy push).
  • tools/triage_audit_diff.py has a unit-testable core: given two manifest texts, returns the batched issue payloads. Include a few test cases (plain python -m unittest style is fine, or a fixture-driven main; this is workflow tooling, not Lua, so the Busted suite is not required for it).
  • Running the script against a synthetic diff (5 added unsupported lines) produces exactly one issue payload with the right body content.
  • No issue creation happens when the manifests didn't change.
  • Re-running on the same commit creates no duplicates.

Out of scope

  • Changing ai-fix.yml itself.
  • Fixing any mods.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions