ci: stop PR bot over-closing on missing checklist + fix reopen loop#4581
Conversation
Add a pull_request_target workflow that checks PRs from external contributors (org members and bots are exempt via author_association): the PR template must be filled in, and any PR touching functional code (SDK, API, or frontend) must include a demo recording. Non-compliant PRs get a sticky comment, an incomplete-pr label, and are closed, then auto-reopened once fixed. Adds workflow_dispatch dry_run/force_external inputs for testing and registers the new label in .labels.yml.
ci: auto-close incomplete external PRs
Two fixes found while auditing the bot against all open PRs: - Only a non-empty Summary plus a demo (for functional changes) are required. Missing Testing/Checklist sections no longer close a PR. The demo is now detected anywhere in the body, not just the Demo section. This fixes a PR that had a YouTube demo and full testing notes but was closed for lacking the checklist section. - Drop the 'reopened' trigger so a maintainer who manually reopens a flagged PR wins, instead of the bot immediately re-closing it. Auto-reopen on a fixed description still works via 'edited'/'synchronize'.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Disabled knowledge base sources:
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR relaxes PR contribution checks in a GitHub Actions workflow by removing the ChangesPR Validation Workflow Updates
Possibly related PRs
🎯 2 (Simple) | ⏱️ ~8 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary
Follow-up to #4579 / #4580. I ran the contribution bot in live mode against all 18 open non-draft PRs and graded each decision. It got 17 of 18 right: it correctly skipped all internal members (including private-membership ones), passed the two compliant external PRs, and closed 8 genuine low-quality PRs. But it wrongly closed one good PR and revealed a reopen loop. This PR fixes both.
Bug 1 — over-strict template check. PR #4551 included a YouTube demo and detailed testing notes but was auto-closed only because it lacked the
## Checklistsection. The bot required all four template sections and read the demo only from the## Demosection.Fix: require just a non-empty
## Summaryplus a demo for functional changes. Missing Testing/Checklist sections no longer close a PR, and the demo is now detected anywhere in the body.Bug 2 — reopen loop. When a maintainer manually reopened a flagged PR, the
reopenedevent re-ran the bot and it immediately re-closed the PR (observed live on #4551).Fix: drop
reopenedfrom the triggers. A maintainer reopen now wins. Auto-reopen on a fixed description still works viaedited/synchronize.Demo policy stays strict: every functional change (including SDK and config) still needs a demo, so #4519 and #4391 remain correctly closed.
Testing
Verified locally
Re-ran the decision logic against the real bodies and file lists of all 13 PRs of interest:
reopenedis no longer in the trigger list and the YAML parses.Added or updated tests
N/A. CI workflow.
QA follow-up
After merge: re-enable the workflow (it is disabled right now), then dispatch a dry-run against #4551 to confirm it now passes, and against a no-demo external PR to confirm it still flags.
Demo
N/A. Touches only
.github/**.Checklist