diff --git a/.agents/skills/ort-ci/SKILL.md b/.agents/skills/ort-ci/SKILL.md new file mode 100644 index 0000000000000..16435834196f9 --- /dev/null +++ b/.agents/skills/ort-ci/SKILL.md @@ -0,0 +1,295 @@ +--- +name: ort-ci +description: Trigger, re-run, and unblock ONNX Runtime CI checks on a GitHub pull request. Use this skill when a required check is stuck, missing, failed, or needs re-running. Nearly all ORT CI runs as GitHub Actions workflows; only the "Linux Android Emulator QNN CI Pipeline" remains on Azure Pipelines, plus the bot-driven "license/cla" and "Python format" lint checks. Triggers on: rerun CI, retrigger checks, stuck check, missing pipeline, failed CI, license/cla, Python format failure, Doc Gen CI failure, operator docs out of date, /azp run. +--- + +# ONNX Runtime CI Management + +Workflows for triggering, re-running, and unblocking CI checks on an ONNX Runtime PR. +The repository is `microsoft/onnxruntime`. As of 2026-07, nearly all CI runs as **GitHub +Actions** workflows (~80+ checks per PR). Only **one** required check still runs on **Azure +Pipelines** — `Linux Android Emulator QNN CI Pipeline` (host `aiinfra.visualstudio.com`). +There is also the bot-driven `license/cla` status check. + +**Failures are not all the same.** Before touching anything, diagnose each failure (see +[Triage: Diagnose Before Re-running](#triage-diagnose-before-re-running)): most failures need a +**code change** and re-running them just fails again; only genuinely transient (network/disk) +failures should be re-run via §1. §4 (Azure Pipelines) applies only to the single QNN pipeline. + +Before doing anything, inspect current state so you do not queue duplicate runs. + +### Classify a check's provider + +GitHub Actions checks have a non-empty `workflowName` and a `detailsUrl` on `github.com`; the +Azure Pipelines check has an empty `workflowName` and a `detailsUrl` on `aiinfra.visualstudio.com`: + +```bash +gh pr view --repo microsoft/onnxruntime --json statusCheckRollup \ + --jq '[.statusCheckRollup[] | {name, host:(.detailsUrl|split("/")[2])}] + | group_by(.host) | map({host:.[0].host, count:length})' +``` + +## Gather Context + +```bash +# PR metadata + all checks grouped by state +gh pr view --repo microsoft/onnxruntime \ + --json number,title,url,state,isDraft,headRefName,headRefOid,baseRefName,statusCheckRollup + +# Head / merge SHAs for external CI +gh api repos/microsoft/onnxruntime/pulls/ --jq '{head:.head.sha, merge:.merge_commit_sha}' +``` + +Inspect `statusCheckRollup` and note, for each requested check, whether it is missing, queued, +in progress, failed, canceled, skipped, or already successful. **Do not re-trigger a check +that is already `queued`/`in_progress`/`SUCCESS`** unless the user explicitly asks. + +Quickly list just the failed/pending checks: + +```bash +gh pr view --repo microsoft/onnxruntime --json statusCheckRollup \ + --jq '.statusCheckRollup[] + | {name:(.name//.context), status:(.status//.state), conclusion:.conclusion} + | select(.conclusion!="SUCCESS" and .status!="SUCCESS")' +``` + +## Triage: Diagnose Before Re-running + +**Never blindly re-run failed CI.** Most failures need a code change and will fail again +identically on re-run. Only *transient* failures should be re-run. The process is: **download +the failed job's log, read the actual error, classify it, then fix or re-run case by case.** + +### Step 1 — Download the failed log + +For a **GitHub Actions** check (the vast majority), get the run and read only the failed steps: + +```bash +HEAD_SHA=$(gh api repos/microsoft/onnxruntime/pulls/ --jq .head.sha) + +# Map failed checks to their workflow run IDs +gh run list --repo microsoft/onnxruntime --commit "$HEAD_SHA" --limit 100 \ + --json databaseId,workflowName,status,conclusion,url \ + --jq '.[] | select(.conclusion=="failure" or .conclusion=="cancelled")' # cspell:ignore cancelled -- literal GitHub API value + +# Dump just the failed steps of a run (grep for the real error) +gh run view --repo microsoft/onnxruntime --log-failed > /tmp/ci_.log +grep -nE "error:|FAILED|warning:|Traceback|fatal error|No space left|Could not resolve|timed out" \ + /tmp/ci_.log | head -50 +``` + +For the **Azure Pipelines** QNN check, open its `detailsUrl` (a `dev.azure.com` / +`aiinfra.visualstudio.com` build page) and download the job log, or use the Azure DevOps +`.../builds//timeline` + log APIs (see the `ci-failure-retrieval` skill for the exact +requests). + +### Step 2 — Classify the failure and act + +| # | Failure class | How to recognize it in the log | Action — **re-run or fix?** | +|---|---|---|---| +| 1 | **C/C++ warning-as-error** | `error:` on a `-Werror`/`/WX` line — e.g. implicit type-cast/narrowing (`-Werror=conversion`), `unused variable`/`unused parameter` (`-Werror=unused-*`), sign-compare, maybe-uninitialized | **Fix code.** Re-run will not help. Remove/`[[maybe_unused]]` the unused symbol, add an explicit `static_cast()` / `gsl::narrow_cast()` for the cast, or fix the real logic. Rebuild locally to confirm the warning is gone. | +| 2 | **Test failure** | `[ FAILED ] Suite.Case` (gtest) or `FAILED test_*.py::... - AssertionError` (pytest); often only on some EPs | **Fix code/test.** If a newly added op test fails only on EPs that don't support the op, restrict the test to supported EPs (e.g. gtest `OpTester::Run(..., {kCpuExecutionProvider, kCudaExecutionProvider})` / `excluded_provider_types`, or skip via `SetUp`), or fix the kernel. Don't re-run unchanged. See the `ort-test` skill. | +| 3 | **Transient / infra failure** | `Could not resolve host`, `Connection timed out`, `429 Too Many Requests`, `No space left on device`, package/download 5xx, agent lost, submodule clone timeout — with **no** compile/test error | **Re-run** (§1). This is the one class that a plain re-run fixes. If it recurs 2–3×, escalate — it may be a real infra/proxy issue, not noise. | +| 4 | **Lint / Python format** | `Python format` check fails; `lintrunner` reports diffs | **Fix code** with `lintrunner -a`, commit, push (§3). Re-run alone won't fix it. | + +Rules of thumb: +- A compile `error:` or a `[ FAILED ]`/`FAILED` line means **fix the code** — re-running reruns + the same failing commit and fails identically. +- Only re-run when the log shows a network/disk/agent problem and **no** compile or assertion error. +- When unsure, download the log and read it; do not guess from the check name alone. +- After a code fix, push a new commit — CI re-runs automatically on the new head SHA; you do not + need §1. + +## 1. Re-run Failed GitHub Actions (transient failures only) + +The repo ships a helper that re-runs **only** the GitHub Actions workflows whose latest run +for the PR's current head commit failed/canceled — and skips any workflow that already has a +newer run queued or in progress. Use it **only after triage** confirms the failures are +transient (network/disk/agent) — see [Triage](#triage-diagnose-before-re-running). It is the +safest way to retry those without piling on duplicates. + +Script: [tools/scripts/rerun_failed_ci.sh](../../../tools/scripts/rerun_failed_ci.sh) + +```bash +# Dry run first — shows what would be re-run, triggers nothing +./tools/scripts/rerun_failed_ci.sh --dry-run + +# Actually re-run the failed/canceled workflows for the PR's head commit +./tools/scripts/rerun_failed_ci.sh + +# Explicit repo (auto-detected from cwd when omitted) +./tools/scripts/rerun_failed_ci.sh microsoft/onnxruntime +``` + +It prefers `gh run rerun --failed` (retry only failed jobs) and falls back to a full +rerun for fully canceled runs that have no discrete failed jobs. Requires an authenticated +`gh`. Always run `--dry-run` first and confirm the list looks right before the real run. + +To re-run one specific workflow manually: + +```bash +gh run list --repo microsoft/onnxruntime --commit --limit 100 \ + --json databaseId,workflowName,status,conclusion,url +gh run rerun --repo microsoft/onnxruntime --failed # only failed jobs +gh run rerun --repo microsoft/onnxruntime # full rerun +``` + +## 2. Unblock `license/cla` (CLA bot) + +The `license/cla` check is posted by Microsoft's CLA bot, **independent of the CI pipelines**. +When it is stuck as *"Expected — Waiting for status to be reported"*, re-trigger only the bot — +no CI jobs are re-run — by posting this comment on the PR: + +```bash +gh pr comment --repo microsoft/onnxruntime \ + --body "@microsoft-github-policy-service rerun" +``` + +Then verify it flips to success: + +```bash +gh pr view --repo microsoft/onnxruntime --json statusCheckRollup \ + --jq '.statusCheckRollup[] | select((.name//.context)=="license/cla") + | {status, conclusion}' +``` + +Expect `{"status":"COMPLETED","conclusion":"SUCCESS"}`. + +## 3. Fix the "Python format" required check + +The required **Python format** check (job `lint-python-format` in +[.github/workflows/lint.yml](../../../.github/workflows/lint.yml)) runs +`lintrunner --all-files` and fails on any formatting/lint violation. Re-running it will **not** +help — you must fix the code, commit, and push. See +[docs/Coding_Conventions_and_Standards.md](../../../docs/Coding_Conventions_and_Standards.md#linting) +and the `ort-lint` skill. + +```bash +# One-time setup (in an activated Python venv) +pip install -r requirements-lintrunner.txt +lintrunner init + +# Auto-fix. Prefer changed files; use --all-files to match CI exactly. +lintrunner -a # changed files only +lintrunner -a --all-files # everything (what CI checks) + +# Verify clean (no changes reported == pass) +lintrunner --all-files +``` + +Then commit and push the formatting fixes; the check re-runs automatically on the new commit: + +```bash +git add -u && git commit -m "Fix lint" && git push +``` + +Notes: +- CI runs `lintrunner --all-files`, so a local `lintrunner -a` on only changed files can miss a + pre-existing violation the CI reports. If the check still fails, run `--all-files` locally. +- The same job also covers C++ clang-format and other adapters; the fix is the same + (`lintrunner -a`). + +## 4. Trigger the Azure Pipelines check (QNN Android Emulator only) + +As of 2026-07, the **only** ORT check still on Azure Pipelines is +`Linux Android Emulator QNN CI Pipeline`. Everything else is GitHub Actions (use §1). Trigger +it through the PR comment integration: + +```bash +gh pr comment --repo microsoft/onnxruntime \ + --body "/azp run Linux Android Emulator QNN CI Pipeline" +``` + +Then wait briefly and check for a reply from `azure-pipelines[bot]`: + +```bash +gh pr view --repo microsoft/onnxruntime --json comments \ + --jq '.comments[] | select(.author.login=="azure-pipelines") + | {createdAt, body}' | tail +``` + +- If the bot replies *"No pipelines are associated with this pull request"*, the pipeline is not + wired to the comment app — use the direct Azure DevOps API fallback (see the + `Trigger CI Pipelines` section of the private `gh-pr-management` skill for the + `dev.azure.com` project/definition discovery and `POST .../runs` payload using + `refName: refs/pull//merge` and the PR `merge_commit_sha`). +- Keep it to **one batch `/azp run` comment per attempt**; do not spam repeated comments. + +## 5. Fix the "Windows GPU Doc Gen CI" check (operator docs out of date) + +The **ONNX Runtime Windows GPU Doc Gen CI** check (workflow +[.github/workflows/windows_gpu_doc_gen.yml](../../../.github/workflows/windows_gpu_doc_gen.yml)) +builds ORT and runs `build.py --gen_doc validate`. It **fails when the generated operator docs +no longer match what's committed** — typically after you add/modify an operator or its kernel +registrations but forget to regenerate `docs/ContribOperators.md` / `docs/OperatorKernels.md`. +Re-running will **not** help; you must update the docs. + +The easiest fix is to download the regenerated docs the failed job already produced: on failure +the workflow uploads a **single artifact named `updated-docs`** that contains both +`OperatorKernels.md` and `ContribOperators.md` at its top level, so you can replace the committed +copies without building locally. + +```bash +HEAD_SHA=$(gh api repos/microsoft/onnxruntime/pulls/ --jq .head.sha) + +# Find the failed Doc Gen run +run_id=$(gh run list --repo microsoft/onnxruntime --commit "$HEAD_SHA" --limit 100 \ + --json databaseId,workflowName,conclusion \ + --jq '.[] | select(.workflowName|test("Doc Gen")) | select(.conclusion=="failure") | .databaseId' | head -1) + +# Confirm the artifact is present (expect: updated-docs) +gh api repos/microsoft/onnxruntime/actions/runs/$run_id/artifacts --jq '.artifacts[].name' + +# Download the updated docs straight into docs/ (the artifact holds both .md files) +gh run download "$run_id" --repo microsoft/onnxruntime -n updated-docs --dir docs/ +``` + +Then review, commit, and push — the check re-runs on the new commit: + +```bash +git diff --stat docs/ContribOperators.md docs/OperatorKernels.md +git add docs/ContribOperators.md docs/OperatorKernels.md +git commit -m "Update operator docs" && git push +``` + +Notes: +- The `updated-docs` artifact always contains **both** `OperatorKernels.md` and + `ContribOperators.md` at its top level; `--dir docs/` drops them into place. Only the file(s) + that actually changed will show up in `git diff` after extraction. +- Equivalent local alternative: `python tools/ci_build/build.py --config Release --build_dir + build/Linux --gen_doc` after a build, then commit the regenerated files. Downloading the + artifact is faster since it avoids a full build. + +## Special-Case Check Handling + +| Check / job name | Owner | How to unblock | +|---|---|---| +| `license/cla` | CLA bot | Comment `@microsoft-github-policy-service rerun` (§2) | +| `Python format` (`lint-python-format`) | GitHub Actions | Fix with `lintrunner -a`, commit, push (§3) — rerun alone won't fix | +| `ONNX Runtime Windows GPU Doc Gen CI` | GitHub Actions | Download the `updated-docs` artifact into `docs/`, commit, push (§5) — rerun alone won't fix | +| `Optional Lint`, `Optional Lint C++` | GitHub Actions | Non-required reviewdog checks; fix warnings or ignore | +| Most CI (Linux/Windows/Mac/CUDA/TensorRT/WebGPU/Web/Android/iOS, `windows_x64_*`, Builds, PR Checks) | GitHub Actions | `rerun_failed_ci.sh ` (§1) | +| `Linux Android Emulator QNN CI Pipeline` | Azure DevOps | `/azp run Linux Android Emulator QNN CI Pipeline` (§4), then API fallback | + +## Safety Rules + +- **Never** trigger release, publish, official, nightly, signing (`ESRP`), deployment, or + package-upload pipelines unless the user explicitly asks for that class of pipeline. Treat + names containing `release`, `publish`, `official`, `nightly`, `sign`, `ESRP`, `production`, or + `deploy` as high-risk and confirm first. +- Always `--dry-run` the rerun script before the real run. +- Do not re-run green checks or dispatch unrelated workflows. +- Prefer PR merge refs (`refs/pull//merge`) over branch refs for external CI so the run + validates the merge result. +- Do not claim success until the provider returns a queued/in-progress/completed run ID or URL, + or the check flips state in `statusCheckRollup`. + +## Verify After Triggering + +```bash +gh pr view --repo microsoft/onnxruntime --json statusCheckRollup \ + --jq '.statusCheckRollup | group_by(.status//.state) + | map({state:(.[0].status//.[0].state), count:length})' +``` + +Confirm the previously stuck/failed check moved to `queued`/`in_progress` (or `SUCCESS` for the +CLA bot), and that no duplicate runs were created for the same head SHA. diff --git a/AGENTS.md b/AGENTS.md index 44e7257d5d536..6b7c40c5cb816 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,10 @@ See the `/ort-build`, `/ort-test`, and `/ort-lint` skills (in `.agents/skills/`) for detailed instructions. +## CI + +See the `/ort-ci` skill (in `.agents/skills/`) for triggering, re-running, and unblocking CI checks on a pull request (GitHub Actions, Azure Pipelines, `Python format`, and `license/cla`). + ## Architecture Overview ONNX Runtime is a cross-platform inference and training engine for ONNX models. The core pipeline is: **Load model → Build graph → Optimize graph → Partition across Execution Providers → Execute**. diff --git a/tools/scripts/rerun_failed_ci.sh b/tools/scripts/rerun_failed_ci.sh new file mode 100755 index 0000000000000..17f769c1eaf34 --- /dev/null +++ b/tools/scripts/rerun_failed_ci.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +# +# rerun_failed_ci.sh - Re-run failed/canceled GitHub Actions CI for a pull request. +# +# For the PR's current head commit, this finds every GitHub Actions workflow run +# whose latest attempt ended in a non-success state (failure, canceled, timed out, +# etc.) and re-runs it -- but ONLY when no newer run for that same workflow is +# already queued or in progress. This avoids piling duplicate runs on a workflow +# that someone (or a previous invocation of this script) has already re-triggered. +# +# Usage: +# ./rerun_failed_ci.sh [owner/repo] [--dry-run] +# +# Examples: +# ./rerun_failed_ci.sh 29719 +# ./rerun_failed_ci.sh 29719 microsoft/onnxruntime +# ./rerun_failed_ci.sh 29719 --dry-run +# +# Requirements: gh (GitHub CLI), authenticated. (Uses gh's built-in --jq; no +# external jq binary required.) +# +# Note: "cancelled" (double l) below is the literal value GitHub's API returns +# for a canceled run's `conclusion`; it must match the API and is not a typo. +# cspell:ignore cancelled +set -euo pipefail + +usage() { + echo "Usage: $0 [owner/repo] [--dry-run]" >&2 + exit 2 +} + +PR_NUMBER="" +REPO="" +DRY_RUN=0 + +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=1 ;; + -h|--help) usage ;; + */*) REPO="$arg" ;; + *[!0-9]*) echo "Unrecognized argument: $arg" >&2; usage ;; + *) PR_NUMBER="$arg" ;; + esac +done + +[[ -n "$PR_NUMBER" ]] || usage + +command -v gh >/dev/null 2>&1 || { echo "error: gh (GitHub CLI) is not installed" >&2; exit 1; } + +# Auto-detect the repository from the current directory when not supplied. +if [[ -z "$REPO" ]]; then + REPO="$(gh repo view --json nameWithOwner --jq .nameWithOwner)" +fi + +# Resolve the PR's current head commit; all CI decisions are made against this SHA. +HEAD_SHA="$(gh pr view "$PR_NUMBER" --repo "$REPO" --json headRefOid --jq .headRefOid)" +echo "Repo: $REPO" +echo "PR: #$PR_NUMBER" +echo "Head: $HEAD_SHA" +echo + +# Pull every workflow run for this exact commit and decide what to re-run. +# Group runs by workflow. For each workflow: +# - if any run is still active (queued/in_progress/...), skip it (a rerun already exists); +# - otherwise take the most recent run and, if it did not succeed, mark it for rerun. +# +# Output: one "\t\t" line per run to rerun. +# gh embeds jq (--jq), so no external jq binary is required. +# shellcheck disable=SC2016 # the single-quoted jq program is intentional; $s/\(...) are jq, not shell +TO_RERUN="$(gh run list --repo "$REPO" --commit "$HEAD_SHA" --limit 300 \ + --json databaseId,workflowName,name,status,conclusion,createdAt \ + --jq ' + def active: ["queued","in_progress","requested","pending","waiting"]; + def needs_rerun: ["failure","cancelled","timed_out","startup_failure","action_required","stale"]; + group_by(.workflowName)[] + | { latest: (sort_by(.createdAt) | last), + has_active: (any(.[]; .status as $s | active | index($s) != null)) } + | select(.has_active | not) + | .latest + | select(.conclusion as $c | needs_rerun | index($c) != null) + | "\(.databaseId)\t\(.workflowName)\t\(.conclusion)" + ')" + +if [[ -z "$TO_RERUN" ]]; then + echo "No failed workflows to re-run (either all passed, or reruns are already queued/in progress)." + exit 0 +fi + +echo "Workflows to re-run:" +echo "$TO_RERUN" | while IFS=$'\t' read -r id name conclusion; do + printf ' %-45s (%s) run %s\n' "$name" "$conclusion" "$id" +done +echo + +if [[ "$DRY_RUN" -eq 1 ]]; then + echo "Dry run: no reruns triggered." + exit 0 +fi + +rerun_count=0 +fail_count=0 +while IFS=$'\t' read -r id name conclusion; do + [[ -n "$id" ]] || continue + # Prefer re-running only the failed jobs; fall back to a full rerun for runs + # (e.g. fully canceled ones) that have no discrete "failed" jobs to target. + if gh run rerun "$id" --repo "$REPO" --failed >/dev/null 2>&1 \ + || gh run rerun "$id" --repo "$REPO" >/dev/null 2>&1; then + echo " re-ran: $name (run $id)" + rerun_count=$((rerun_count + 1)) + else + echo " FAILED to re-run: $name (run $id)" >&2 + fail_count=$((fail_count + 1)) + fi +done <<<"$TO_RERUN" + +echo +echo "Done: $rerun_count re-run, $fail_count failed." +[[ "$fail_count" -eq 0 ]]