Skip to content

CI: stabilize Aiter wheel prebuilds#3126

Merged
gyohuangxin merged 4 commits into
mainfrom
ci/aiter-prebuild-max-jobs-summary
May 12, 2026
Merged

CI: stabilize Aiter wheel prebuilds#3126
gyohuangxin merged 4 commits into
mainfrom
ci/aiter-prebuild-max-jobs-summary

Conversation

@gyohuangxin
Copy link
Copy Markdown
Member

Summary

  • Lower Aiter wheel prebuild concurrency from 128 to 64 via a workflow env knob.
  • Capture prebuild logs and print a timing summary with runner, wall time, kernel count, and per-module compile costs.

Test plan

  • python3 -m py_compile .github/scripts/aiter_prebuild_summary.py
  • Parsed the workflow YAML successfully with PyYAML.
  • Syntax-checked the generated build shell with bash -n.
  • Ran git diff --check on the changed files.

Lower prebuild concurrency and emit timing summaries so CI can distinguish runner slowdowns from kernel count changes.
@gyohuangxin gyohuangxin requested review from a team and Copilot May 11, 2026 09:42
@github-actions
Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 3126 --add-label <label>

Apply Black formatting to keep the CI style check green.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Aiter CI wheel prebuild step to reduce build parallelism via a workflow-level knob and to emit a timing summary from the setup.py build logs, aiming to improve prebuild stability and observability in CI.

Changes:

  • Introduces a workflow env knob (AITER_PREBUILD_MAX_JOBS) to reduce wheel build parallelism (MAX_JOBS) from 128 to 64.
  • Captures setup.py output via tee and runs a new parser script to print a prebuild timing summary (runner, wall time, kernel count, and per-module compile costs).
  • Adds .github/scripts/aiter_prebuild_summary.py to parse “finish build [...] cost ...” lines and report compilation timings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/aiter-test.yaml Adds env knob for MAX_JOBS and runs a new summary pass over prebuild logs during wheel build.
.github/scripts/aiter_prebuild_summary.py New helper script that parses prebuild logs to print a timing/kernel-count summary.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/aiter-test.yaml Outdated
Comment on lines +118 to +125
prebuild_start=$(date +%s) &&
set +e &&
python setup.py bdist_wheel 2>&1 | tee /tmp/aiter-prebuild.log
build_status=${PIPESTATUS[0]}
set -e
prebuild_end=$(date +%s)
if ! python3 .github/scripts/aiter_prebuild_summary.py \
--log /tmp/aiter-prebuild.log \
Comment on lines +119 to +123
set +e &&
python setup.py bdist_wheel 2>&1 | tee /tmp/aiter-prebuild.log
build_status=${PIPESTATUS[0]}
set -e
prebuild_end=$(date +%s)
Persist the prebuild log and timing metadata so the summary is easy to find in the Actions UI.
Raise the prebuild MAX_JOBS default to 96 to recover compiler parallelism while keeping it below the previous 128 setting.
@gyohuangxin gyohuangxin merged commit 9c50619 into main May 12, 2026
30 checks passed
@gyohuangxin gyohuangxin deleted the ci/aiter-prebuild-max-jobs-summary branch May 12, 2026 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants