Skip to content

[Dev] Fix full CUDA graph capture reverted by pull main#4792

Open
Victarry wants to merge 1 commit into
NVIDIA:devfrom
Victarry:denliu/fix-cu-seqlens-cuda-graph-capture
Open

[Dev] Fix full CUDA graph capture reverted by pull main#4792
Victarry wants to merge 1 commit into
NVIDIA:devfrom
Victarry:denliu/fix-cu-seqlens-cuda-graph-capture

Conversation

@Victarry
Copy link
Copy Markdown
Contributor

@Victarry Victarry commented May 14, 2026

Summary

  • Restore the source-rank cu_seqlens length broadcast allocation to torch.empty(1, dtype=torch.int64, device=dev).fill_(n).
  • Avoid torch.tensor(n, dtype=torch.int64, device=dev), which can perform a CPU-to-CUDA copy and fails during full-iteration CUDA graph capture when the CPU source is not pinned.
  • This restores the earlier dev behavior from f9f9fe415 (Add minor tweaks to support full-iter CG capture). The nightly pull-main chore: nightly sync main into dev (10_05_2026) #4716 post-merge commit cd2001de7 (chore: post-merge fixes for nightly sync main into dev (28_04_2026)) took main's megatron/training/utils.py and reverted this line back to torch.tensor(...).

Test plan

  • python -m py_compile megatron/training/utils.py
  • CHECK_ONLY=true BASE_REF=dev uv run --project /Users/denliu/Projects/repos/agentic-mcore-dev --extra mcore-lint -- bash tools/autoformat.sh
  • Copyright check: no megatron/core or tests Python files changed.
  • Ran paged-stash dev config on Computelab B300 (umb-b300-003) with local/b300-cudnnfe1221:latest; reached iteration 100, printed validation loss, and no longer hit the CUDA graph capture error from torch.tensor(n, device=dev).

@Victarry Victarry requested review from a team as code owners May 14, 2026 02:48
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 14, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Victarry Victarry changed the title [Dev] Fix cu_seqlens broadcast CUDA graph capture [Dev] Fix full CUDA graph capture reverted by pull main May 14, 2026
@Victarry
Copy link
Copy Markdown
Contributor Author

/ok to test 05fc772

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants