Skip to content

[debug] Instrument TLE cluster GEMM CI failure#655

Closed
sunnycase wants to merge 43 commits into
triton_v3.6.xfrom
debug/ci-04-cluster-gemm-649
Closed

[debug] Instrument TLE cluster GEMM CI failure#655
sunnycase wants to merge 43 commits into
triton_v3.6.xfrom
debug/ci-04-cluster-gemm-649

Conversation

@sunnycase

Copy link
Copy Markdown
Collaborator

Temporary CI debug PR for the 04-cluster-gemm.py Hopper failure in PR #649.

Changes:

  • Print runtime/cache/autotune-candidate diagnostics when FLAGTREE_CI_DEBUG_CLUSTER_GEMM=1.
  • Run the suspected BK=256 remote config under CUDA_LAUNCH_BLOCKING=1 before the original full default run in the triton_v3.6.x Hopper workflow.

This PR is intended for diagnostics only and should be closed after collecting logs.

@sunnycase

Copy link
Copy Markdown
Collaborator Author

CI debug summary for PR #649:

  • Original PR [TLE] Feature/extrac tile strides new #649 Hopper rerun still fails in python/tutorials/tle/04-cluster-gemm.py with CUDA error: misaligned address at _autotune_config -> torch.cuda.synchronize() immediately after baseline triton_gemm autotune.
  • Debug run with CUDA_LAUNCH_BLOCKING=1 passed. Candidate 1 for remote autotune was KernelConfig(bm=32, bn=512, bk=256, num_warps=4, num_stages=2) and completed launch/sync/bench successfully.
  • Debug run without CUDA_LAUNCH_BLOCKING, but with per-candidate print(..., flush=True), also passed; all 16 remote candidates completed.

Current inference: the failure is timing-sensitive / host-side perturbation-sensitive, not a deterministic bad config. The first remote candidate remains the likely failing launch in the uninstrumented script, but adding prints is enough to hide it on the Hopper runner.

@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 09:11
@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 10:59
@sunnycase sunnycase marked this pull request as draft June 5, 2026 11:08
@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 11:08
@sunnycase sunnycase marked this pull request as draft June 5, 2026 11:17
@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 11:18
@sunnycase sunnycase marked this pull request as draft June 5, 2026 11:27
@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 11:28
@sunnycase sunnycase marked this pull request as draft June 5, 2026 11:37
@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 11:37
@sunnycase sunnycase marked this pull request as draft June 5, 2026 11:47
@sunnycase sunnycase marked this pull request as ready for review June 5, 2026 11:48
@sunnycase sunnycase marked this pull request as draft June 5, 2026 12:10
@sunnycase sunnycase marked this pull request as ready for review June 8, 2026 02:35
@sunnycase sunnycase marked this pull request as draft June 8, 2026 02:56
@sunnycase sunnycase marked this pull request as ready for review June 8, 2026 03:02
@github-actions github-actions Bot added the nvidia label Jun 8, 2026
@sunnycase sunnycase closed this Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants