Skip to content

Commit 45e8f55

Browse files
authored
Remove NCCL_NET_GDR_LEVEL=LOC from VERL/OpenRLHF Dockerfiles (#1106)
NCCL_NET_GDR_LEVEL=LOC disables GPU Direct RDMA over the network — every cross-node GPU↔GPU transfer is staged through host memory, defeating the purpose of EFA on p5/p5e/p5en/p6. Removing the override lets the aws-ofi-nccl plugin (bundled with the EFA installer, built with --enable-platform-aws) select the appropriate GDR level via the per-instance NCCL_TOPO_FILE it auto-loads.
1 parent 0c1f068 commit 45e8f55

3 files changed

Lines changed: 0 additions & 3 deletions

File tree

3.test_cases/pytorch/openrlhf/Dockerfile

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,6 @@ ENV FI_EFA_USE_DEVICE_RDMA=1
6363
ENV FI_EFA_FORK_SAFE=1
6464
ENV FI_EFA_ENABLE_SHM_TRANSFER=1
6565
ENV NCCL_PROTO=simple
66-
ENV NCCL_NET_GDR_LEVEL=LOC
6766
ENV NCCL_SOCKET_IFNAME=^docker,lo,veth
6867
ENV NCCL_TUNER_PLUGIN=/opt/amazon/ofi-nccl/lib/x86_64-linux-gnu/libnccl-ofi-tuner.so
6968
ENV PMIX_MCA_gds=hash

3.test_cases/pytorch/verl/hyperpod-eks/rlvr/Dockerfile

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,6 @@ ENV FI_EFA_USE_DEVICE_RDMA=1
7878
ENV FI_EFA_FORK_SAFE=1
7979
ENV FI_EFA_ENABLE_SHM_TRANSFER=1
8080
ENV NCCL_PROTO=simple
81-
ENV NCCL_NET_GDR_LEVEL=LOC
8281
ENV NCCL_SOCKET_IFNAME=^docker,lo,veth
8382
ENV NCCL_TUNER_PLUGIN=/opt/amazon/ofi-nccl/lib/x86_64-linux-gnu/libnccl-ofi-tuner.so
8483
ENV PMIX_MCA_gds=hash

3.test_cases/pytorch/verl/kubernetes/rlvr/Dockerfile

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,6 @@ ENV FI_EFA_USE_DEVICE_RDMA=1
7979
ENV FI_EFA_FORK_SAFE=1
8080
ENV FI_EFA_ENABLE_SHM_TRANSFER=1
8181
ENV NCCL_PROTO=simple
82-
ENV NCCL_NET_GDR_LEVEL=LOC
8382
ENV NCCL_SOCKET_IFNAME=^docker,lo,veth
8483
ENV NCCL_TUNER_PLUGIN=/opt/amazon/ofi-nccl/lib/x86_64-linux-gnu/libnccl-ofi-tuner.so
8584
ENV PMIX_MCA_gds=hash

0 commit comments

Comments
 (0)