[CRCR] Implement on-call bot and allowlist functionality for downstream CI failures by KarhouTam · Pull Request #8183 · pytorch/test-infra

KarhouTam · 2026-06-16T09:18:55Z

Summary

Implement CRCR on-call bot and merge-blocking logic for downstream CI failures, driven by a structured allowlist (L1-L4) that maps downstream repos to severity levels and on-call contacts.

Changes

CRCR Allowlist (`lib/crcrAllowlist.ts`)

Parses .github/allowlist.yml from pytorch/pytorch, mapping downstream repos to four levels:

L1/L2: Tracked only, no on-call behavior.
L3: Non-blocking failure — on-call is notified but merge is not blocked.
L4: Blocking failure — on-call is notified and merge is blocked.

Includes a 15-minute in-memory cache and strict YAML validation. Errors fail open.

On-call Bot (`lib/bot/crcrOncallBot.ts`)

A Probot bot on check_run.completed: when a CRCR check run fails, looks up on-calls from the allowlist and posts a tagged comment on the associated PR. Deduplicates via an HTML comment marker (theoretical TOCTOU race exists but is negligible in practice since checks arrive sequentially).

Merge Blocking (`lib/bot/pytorchBotHandler.ts`)

@pytorchbot merge checks the GitHub Checks API for CRCR failures on the PR head commit. L4 failures block the merge with a comment listing failing repos. -f bypasses this, consistent with existing force-merge semantics. Errors fail open.

Introduced ensureHeadSha() helper to deduplicate the lazy-load pattern for headSha across methods.

Dr.CI Integration

fetchRecentWorkflows.ts: New fetchOotWorkflows() queries oot_workflow_job from ClickHouse. Uses a new downstreamLevel field on RecentWorkflowsData instead of overloading failure_context.
drci.ts: L4 failures appear as blocking; L3 failures appear in a new "OUT OF TREE (non-blocking)" section.

Tests

Unit tests for allowlist parsing and on-call bot behavior, plus updated Dr.CI test helpers.

vercel · 2026-06-16T09:18:59Z

@KarhouTam is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@albanD

…8119) # Summary - The current implementation focuses on L3/L4 levels defined in the RFC: pytorch/rfcs#90 - For detailed design for L3/L4, please refer to pytorch/rfcs#93 # Architecture - `webhook` function: - [x] Create PR label handling function for L3 repo (refer to the 3 scenario cases mentioned in pytorch/rfcs#93) - **Scenario 1**: Label arrives before the downstream workflow job is triggered. So we cached this information in Redis using `mark_check_run_wanted` and let the `callback` lambda create this check run. - **Scenario 2**: Label arrives during the downstream workflow job is running. The `callback` lambda will cache workflow information beforehand so it can immediately create the `in_progress` check run. - **Scenario 3**: Label arrives after the downstream workflow job is done. If the cached workflow information is still alive in Redis (3 days by default, could be set by `CRCR_STATUS_TTL`), it will create a `completed` check run immediately. Otherwise, it will not create a check run. - [x] Dispatch function will check for L3 label or L4, and store this information in Redis for the `callback` to check whether a check run is needed. - [x] Create check run and check suite handling functions for the downstream workflow jobs re-run mechanism within the check run. - `callback` function: - [x] Handle upstream check-run creation/update for L3/L4 - **Scenario 1 & L4**: Check in Redis by calling `is_check_run_wanted` to see if this PR needs a check run. If so, immediately create one. - **All Scenarios**: Store workflow information in Redis for check run creation in the `webhook`. - [x] Set "in_progress" zombie check-runs to "time_out" through a sweeper periodically # Changes ```md .github/actions/cross-repo-ci-relay └── action.yml # Add a step to capture job-name for re-run aws/lambda/cross_repo_ci_relay/ ├── tests/ # Add more unit tests ├── allowlist.py # Update utils function for L3 ├── redis_helper.py # Set more keys for L3 ├── gh_helper.py # Update utils function for L3 ├── misc.py # Update utils function for L3 ├── event_handler.py # Create PR label/check run/check suites handling function ├── cleanup_handler.py # Handle zombie check-run └── callback_handler.py # Handle upstream check-run creation ``` # Verification We performed the following scenario verification on our AWS Lambda instance: - L3: - [x] L3 labels named `ciflow/crcr/{device}` are added immediately after the PR is created, and show up in the corresponding check-run on the PR with the name `crcr/{repo}/{workflow_name}/{job_name}`. - [x] After clicking into the check-run, the corresponding information is correct. - [x] L3 labels are added while the workflow job is running, which should show up the `in_progress` check-run. - [x] L3 labels are added after the workflow job is done, which should show up the `completed` check-run. - [x] Check run is updated when the PR with L3 labels is reopened or synchronized. - L4: - [x] Check run should be created after the PR is opened. - [x] Check run is updated when the PR is reopened or synchronized. - Re-run - [x] Clicking the `Re-run` button in each failed check run will trigger the corresponding downstream workflow failed jobs to re-run and update the check run status to `in_progress`. - [x] Clicking the `Re-run all jobs` or `Re-run all failed jobs` button will trigger the corresponding downstream workflow jobs in the check suite and update the corresponding check run status to `in_progress`. # Unit Tests - [x] Unit Tests (Mock) # TODO - Modify PyTorchBot for L3 non-blocking merge: pytorch/pytorch#185612 - Add a new feature in PyTorchBot that automatically mentions related on-call maintainers in L3/L4 repos when check-run fails: #8183 cc @albanD @fffrog @KarhouTam @atalman @huydhn @zxiiro @subinz1 @jewelkm89

KarhouTam · 2026-07-13T02:25:07Z

Hi, @atalman, @huydhn. Since CRCR L3/L4 PR (#8119) is already merged. I think this mergebot helper PR and pytorch/pytorch#185612 are ready to review. Please take a look and leave your comments if you have a chance!

can-gaa-hou

Hi @KarhouTam, thanks for the PR. Overall looks good. Please take a look at these comments and consider some changes.

can-gaa-hou · 2026-07-14T02:24:09Z

+ *   L4:
+ *     - org5/repo5: oncall1, oncall2
+ */
+export class CrcrAllowlist {


We already have a helper function to parse the allowlist from GitHub. Refer to https://github.com/pytorch/test-infra/blob/main/torchci/pages/api/crcr/allowlist.ts

They serve different purposes: pages/api/crcr/allowlist.ts is for external users, while lib/crcrAllowlist.ts is for internal use. Of course, some code can be reused, and I'll handle that properly. Thanks for the heads-up!

## Summary Update mergebot (`trymerge.py`) to recognize CRCR L3 out-of-tree CI failures as non-blocking, preventing them from halting merges. Depends on the Dr.CI + CRCR infrastructure in [pytorch/test-infra#8183](pytorch/test-infra#8183). ## Changes ### New classification function: `is_crcr_l3` Mirrors the existing `is_flaky` / `is_broken_trunk` / `is_unstable` pattern. Consults the `CRCR_L3` category returned by the Dr.CI API and matches on the standard `name` and `id` fields of the `RecentWorkflowsData` schema. ### Classification in `get_classifications` When a check matches the `CRCR_L3` category from Dr.CI, it is classified as `"CRCR_L3"` and excluded from the unclassified failures pool. ### Non-blocking categorization in `categorize_checks` `CRCR_L3` is added to the list of non-blocking classifications (alongside `BROKEN_TRUNK`, `FLAKY`, `UNSTABLE`). CRCR_L3 failures are intentionally excluded from the `flaky_or_broken_trunk` threshold logic — L3 failures are definitionally non-blocking regardless of count. ## Data Flow ``` CRCR check run completes (oot_workflow_job) ↓ Dr.CI fetches from ClickHouse oot_workflow_job ↓ Dr.CI classifies: L3 → CRCR_L3, L4 → FAILED ↓ Dr.CI API returns {"CRCR_L3": [...], "FAILED": [...]} ↓ trymerge.py reads Dr.CI API response ├── CRCR_L3 → non-blocking (treated like FLAKY/BROKEN_TRUNK) └── FAILED → blocking (L4 failures land here, naturally stopping merge) ``` ## Design Notes - **L4 failures stay blocking**: Dr.CI merges L4 failures into `failedJobs`, not `CRCR_L3`. They flow through the normal failure path in trymerge.py and block the merge. No extra L4-specific logic is needed on this side. - **The classification authority is Dr.CI**: trymerge.py does not query the allowlist or check GitHub check run names itself. It trusts the `CRCR_L3` category from Dr.CI. - **Errors fail closed**: If Dr.CI is unreachable or returns no classifications, checks remain unclassified and block the merge. This is the same behavior as the other classification categories. Pull Request resolved: #185612 Approved by: https://github.com/can-gaa-hou, https://github.com/subinz1, https://github.com/atalman

…am CI failures

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 16, 2026

can-gaa-hou mentioned this pull request Jun 16, 2026

[CRCR] Implement CRCR upstream check run management for L3/L4 jobs #8119

Merged

15 tasks

KarhouTam force-pushed the crcr-mergebot branch from e0b1f38 to f9219b7 Compare June 24, 2026 09:44

KarhouTam force-pushed the crcr-mergebot branch from f9219b7 to fb3d24b Compare July 9, 2026 02:47

KarhouTam marked this pull request as ready for review July 9, 2026 02:48

KarhouTam force-pushed the crcr-mergebot branch from fb3d24b to 277921c Compare July 9, 2026 03:04

KarhouTam mentioned this pull request Jul 9, 2026

[CI][CRCR] Update mergebot rule to adapt CRCR L3 pytorch/pytorch#185612

Closed

can-gaa-hou reviewed Jul 14, 2026

View reviewed changes

Comment thread torchci/lib/bot/crcrOncallBot.ts Outdated

KarhouTam force-pushed the crcr-mergebot branch from 0dd5fcf to c27adc0 Compare July 14, 2026 07:01

subinz1 reviewed Jul 14, 2026

View reviewed changes

Comment thread torchci/lib/fetchRecentWorkflows.ts

[CRCR] Implement on-call bot and allowlist functionality for downstre…

9d7ea10

…am CI failures

KarhouTam force-pushed the crcr-mergebot branch from 34cebfa to 9d7ea10 Compare July 15, 2026 02:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CRCR] Implement on-call bot and allowlist functionality for downstream CI failures#8183

[CRCR] Implement on-call bot and allowlist functionality for downstream CI failures#8183
KarhouTam wants to merge 1 commit into
pytorch:mainfrom
KarhouTam:crcr-mergebot

KarhouTam commented Jun 16, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jun 16, 2026

Uh oh!

KarhouTam commented Jul 13, 2026

Uh oh!

can-gaa-hou left a comment

Uh oh!

Uh oh!

Uh oh!

can-gaa-hou Jul 14, 2026

Uh oh!

KarhouTam Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

KarhouTam commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

CRCR Allowlist (lib/crcrAllowlist.ts)

On-call Bot (lib/bot/crcrOncallBot.ts)

Merge Blocking (lib/bot/pytorchBotHandler.ts)

Dr.CI Integration

Tests

Uh oh!

vercel Bot commented Jun 16, 2026

Uh oh!

KarhouTam commented Jul 13, 2026

Uh oh!

can-gaa-hou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

can-gaa-hou Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

KarhouTam Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

KarhouTam commented Jun 16, 2026 •

edited

Loading

CRCR Allowlist (`lib/crcrAllowlist.ts`)

On-call Bot (`lib/bot/crcrOncallBot.ts`)

Merge Blocking (`lib/bot/pytorchBotHandler.ts`)