[Graph] Improve AMD Turin no-XML ring graph selection on AMD Turin#2127
[Graph] Improve AMD Turin no-XML ring graph selection on AMD Turin#2127voipmonitor wants to merge 3 commits into
Conversation
On AMD multi-socket systems, GPUs on the same NUMA node connect through separate PCIe root complexes under the same PCIe Host Bridge (PATH_PHB). The default P2P level (PATH_PXB) disables P2P for these paths, forcing shared memory transport with 24-42% bandwidth loss. Extend the existing AMD P2P exception to allow PHB-level P2P for configurations with more than 2 GPUs. The original SYS-level P2P for ≤2 GPU configurations is preserved. Benchmarked on dual-socket AMD EPYC 9575F (Turin) with 4x RTX PRO 6000 on the same socket (NCCL 2.29.7+cuda13.2): Transport change: SHM/direct/direct -> P2P/direct pointer Throughput: +24-42% across 256K-128M message sizes Latency: up to 19% lower at 128K Signed-off-by: Martin Vit <martin@voipmonitor.org>
NCCL uses a single flat bandwidth value (16 GB/s) for all AMD CPUs, while Intel has 4 model-specific tiers. This leads to suboptimal topology decisions on modern AMD platforms where inter-socket bandwidth is significantly higher. This patch: 1. Adds per-generation AMD CPU model detection using CPUID family IDs: - Zen 1/2 (Naples/Rome, family 23): 16 GB/s (unchanged) - Zen 3/4 (Milan/Genoa, family 25): 24 GB/s - Zen 5 (Turin, family 26): 32 GB/s 2. Fixes the CPUID familyId/modelId computation in xml.cc per x86 spec: - familyId: add extFamilyId only when base familyId == 15 - modelId: add extModelId << 4 only when familyId is 6 or 15 Bandwidth values were validated on dual-socket AMD EPYC 9575F (Turin) with cross-socket GPU P2P measurements averaging 32.18 GB/s. Zen 3/4 value (24 GB/s) is a conservative estimate pending hardware validation. Signed-off-by: Martin Vit <martin@voipmonitor.org>
Signed-off-by: Martin Vit <martin@voipmonitor.org>
|
Great work! I tested this PR in my environment, and it can solve my problem. But I found that if I don't limit the maximum number of channels with Another question, Zen5 intra-node bandwidth is at least 192GB/s, Why do you set it to 40.0GB/s instead? |
|
++ @thomasgillis @marksantesson to review and mirror once it's ready. |
|
@voipmonitor , can you please rebase this onto current Also, I'm concerned that this seems to be a fix targeted to your environment, not a general fix. In particular, I'd rather the algorithm found the right result rather than patching it up at the end. I'm happy to work with you to find out why it isn't getting the right solution normally after you've rebased. Thanks! |
4 participants
Summary
This draft PR improves no-XML intra-node ring graph selection on dual-socket AMD Turin systems.
It addresses a pathological case where NCCL auto graph selection falls back to a low-bandwidth 8-GPU ring (
Pattern 4,bw 15/15,type SYS/PIX,nChannels 2) unlessNCCL_GRAPH_FILEis used.Motivation
On an 8x RTX PRO 6000 Blackwell server with dual-socket AMD Turin, the application-level cold-prefill path was much slower without
NCCL_GRAPH_FILE, even though transport stayed onP2P/CUMEM.The problematic graph shape was reproducible in NCCL debug logs and consistently showed the same low-bandwidth full-8-GPU ring choice.
Environment notes for the reproducer:
VLLM_ENABLE_PCIE_ALLREDUCE=1)NCCL_P2P_LEVEL=SYSWhat this changes
This stack does three things:
32.0to40.0.The promotion path is intentionally gated to a very specific case:
nChannels == 2typeIntra == PATH_SYStypeInter == PATH_PIXbwIntra <= 15.0bwInter <= 15.0Reproducer
The application-level reproducer is a single cold prefill request through vLLM using:
moonshotai/Kimi-K2.5lightseekorg/kimi-k2.5-eagle3-mlaTRITON_MLADCP=4The measurement is TTFT to the first streamed token on the first request only.
Results
Fair rerun from the same build environment:
ttft=1.073736s,8005prompt tokens,7455.28 tok/sttft=1.070525s,8005prompt tokens,7477.64 tok/sThese are effectively at parity.
For reference, the broken no-XML path before the fix was much slower on the same reproducer:
ttft=9.138378s,8005prompt tokens,875.98 tok/sNon-goal / additional validation
Forcing
8channels was tested separately and was not the fix. It produced only noise at the app level and could be worse on synthetic collective microbenchmarks.Dependency note
This draft stacks on the existing AMD work:
Those are still required for the full setup used here.