UCP/RNDV: Use four read paths on ConnectX-9 XDR by roiedanino · Pull Request #11625 · openucx/ucx

roiedanino · 2026-07-06T13:17:35Z

Summary

Detect ConnectX-9 XDR interfaces and expose four transport paths for RDMA READ operations.

Rendezvous GET ZCOPY registers both the existing two-path protocol and a four-path protocol, allowing protocol selection to choose by message size. Rendezvous PUT and AM preserve their existing automatic two-path behavior, and explicit lane settings remain authoritative.

Design

Hardware detection uses the ConnectX-9 PCI identifier and XDR link speed rather than the CPU model. UCX_MAX_RNDV_RAILS=auto keeps a two-device baseline while allowing transports to expose additional read paths on a single device.

roiedanino · 2026-07-06T14:16:53Z

@svc-nvidia-pr-review

svc-nvidia-pr-review · 2026-07-06T14:17:05Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-07-06T14:20:42Z

    int                                    adaptive_progress;
    /** Eager-am multi-lane support */
    unsigned                               max_eager_lanes;
+    /** Raw configured rendezvous multi-lane limit before auto-resolution;


minor: the field can also hold UCS_ULUNITS_INF (rejected in resolve); pls mention it so the comment matches the handled cases.

svc-nvidia-pr-review · 2026-07-06T14:20:42Z

+    if (config->max_rndv_lanes_config == UCS_ULUNITS_AUTO) {
+        config->max_rndv_lanes =
+                ucp_context_default_max_rndv_lanes(cpu_model);
+    } else if ((config->max_rndv_lanes_config == UCS_ULUNITS_INF) ||


minor: On ILP32 platforms unsigned long is 32-bit, so max_rndv_lanes_config > UINT_MAX can never be true, and a user value equal to 0xFFFFFFFE would alias UCS_ULUNITS_AUTO. This is inherent to ULUNITS semantics and matches other config fields, so it is only worth noting; no change required unless you want to guard the 32-bit case.

evgeny-leksikov

According to description "Vera Rubin systems expose four local XDR devices" I would expect to query NICs and their number but not CPU model. IMO something wrong here.

tvegas1 · 2026-07-08T07:49:33Z

-
-The devices to use are selected according to best network speed, PCI bandwidth, 
+By default UCX picks the best network devices and splits large messages across
+them. On NVIDIA Vera platforms the default is 4 rails; on all other platforms


I would not mention Vera explicitly but rather that the default can be platform specific

tvegas1 · 2026-07-08T07:49:50Z


-Yes, by setting `UCX_MAX_RNDV_RAILS=<num-rails>`. Currently up to 4 are supported.
+Yes, by setting `UCX_MAX_RNDV_RAILS=<num-rails>`. Up to 4 rails are supported.
+The default is 4 on NVIDIA Vera platforms and 2 on all other platforms.


tvegas1 · 2026-07-08T07:51:09Z

+  {"MAX_RNDV_RAILS", "auto",
+   "Maximal number of devices on which a rendezvous operation may be executed\n"
+   "in parallel.\n"
+   "The default is 4 on NVIDIA Vera platforms and 2 otherwise.",


I am not sure that all Vera CPU platforms will always have 4 HCAs?

tvegas1 · 2026-07-08T07:52:27Z

+
+    if (config->max_rndv_lanes_config == UCS_ULUNITS_AUTO) {
+        config->max_rndv_lanes =
+                ucp_context_default_max_rndv_lanes(cpu_model);


can we instead add that config in ucx.conf? ideally we would take that information from HCA/GPU topologies rather that from CPU itself?

tvegas1 · 2026-07-08T07:54:02Z

+
+    if (config->min_rndv_chunk_size == UCS_MEMUNITS_AUTO) {
+        config->min_rndv_chunk_size =
+                ucp_context_default_min_rndv_chunk_size(cpu_model);


same here maybe add min rndv chunk size in ucx.conf under Vera section, this would allow to remove most of the changes here?

gleon99 · 2026-07-09T13:56:59Z

+        config->max_rndv_lanes =
+                ucp_context_default_max_rndv_lanes(cpu_model);
+    } else if ((config->max_rndv_lanes_config == UCS_ULUNITS_INF) ||
+               (config->max_rndv_lanes_config > UINT_MAX)) {


The new resolver only rejects values above UINT_MAX, but max_rndv_lanes is subsequently assigned to ucp_proto_multi_init_params_t::max_lanes, whose type is ucp_lane_index_t (uint8_t), and ucp_proto_multi_init asserts it is <= UCP_PROTO_MAX_LANES (64). Consequently 65–255 can abort assertion-enabled builds, while 256 truncates to zero before the assertion and disables the protocol. Please reject values above the actual supported limit during ucp_init and add finite boundary tests.

(see also svc comment above)

gleon99 · 2026-07-09T13:58:49Z

    EXPECT_EQ(2u, ucp_reg_devices_count(e->ucph()->config.ext.max_hca_per_gpu));
 }

+UCS_TEST_P(test_ucp_context, rndv_platform_defaults)


So the Vera auto-resolution is not actually tested in CI?

evgeny-leksikov · 2026-07-09T15:34:22Z

   UCT_IB_DEVICE_FLAG_DC_V2, 80},
  {"ConnectX-9", {0x15b3, 4133},
   UCT_IB_DEVICE_FLAG_MELLANOX | UCT_IB_DEVICE_FLAG_MLX5_PRM |
-   UCT_IB_DEVICE_FLAG_DC_V2, 90},


can we extend https://github.com/openucx/ucx/blob/master/config/ucx.conf with hints per NICs in addition to CPU , models?

tvegas1 · 2026-07-09T16:06:46Z


+        dev_spec = uct_ib_device_spec(uct_ib_iface_device(iface));
+        if (uct_ib_iface_port_is_xdr(iface) &&
+            (dev_spec->flags & UCT_IB_DEVICE_FLAG_XDR_READ_4_PATHS)) {


are we sure 4 paths are needed and two are no enough? ideally we would do it on READ side, not sure if it is easy to do..

tvegas1 · 2026-07-09T16:09:20Z

+        if (uct_ib_iface_port_is_xdr(iface) &&
+            (dev_spec->flags & UCT_IB_DEVICE_FLAG_XDR_READ_4_PATHS)) {
+            iface->num_paths = ucs_max(iface->num_paths,
+                                       UCT_IB_XDR_READ_NUM_PATHS);


do we need to bounce also the MAX_RMA_RAILS?

roiedanino marked this pull request as ready for review July 6, 2026 13:39

roiedanino changed the title ~~[codex] Use four rendezvous lanes on NVIDIA Vera~~ Use four rendezvous lanes on NVIDIA Vera Jul 6, 2026

roiedanino changed the title ~~Use four rendezvous lanes on NVIDIA Vera~~ UCP/CORE: Use four rendezvous lanes on NVIDIA Vera Jul 6, 2026

svc-nvidia-pr-review reviewed Jul 6, 2026

View reviewed changes

roiedanino self-assigned this Jul 7, 2026

roiedanino requested review from evgeny-leksikov and gleon99 July 7, 2026 08:12

evgeny-leksikov requested changes Jul 7, 2026

View reviewed changes

tvegas1 reviewed Jul 8, 2026

View reviewed changes

gleon99 reviewed Jul 9, 2026

View reviewed changes

roiedanino closed this Jul 9, 2026

roiedanino deleted the codex/ucp-vera-rndv-lanes branch July 9, 2026 14:09

roiedanino mentioned this pull request Jul 9, 2026

UCP/RNDV: Use four read paths on ConnectX-9 XDR #11643

Closed

roiedanino reopened this Jul 9, 2026

roiedanino force-pushed the codex/ucp-vera-rndv-lanes branch from fbbe131 to c71cab0 Compare July 9, 2026 14:32

roiedanino changed the title ~~UCP/CORE: Use four rendezvous lanes on NVIDIA Vera~~ UCP/RNDV: Use four read paths on ConnectX-9 XDR Jul 9, 2026

roiedanino force-pushed the codex/ucp-vera-rndv-lanes branch from c71cab0 to 9953434 Compare July 9, 2026 15:03

UCP/RNDV: Use four read paths on ConnectX-9 XDR

0365062

roiedanino force-pushed the codex/ucp-vera-rndv-lanes branch from 9953434 to 0365062 Compare July 9, 2026 15:25

evgeny-leksikov reviewed Jul 9, 2026

View reviewed changes

tvegas1 reviewed Jul 9, 2026

View reviewed changes

Uh oh!

Conversation

roiedanino commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

Uh oh!

roiedanino commented Jul 6, 2026

Uh oh!

svc-nvidia-pr-review commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evgeny-leksikov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

roiedanino commented Jul 6, 2026 •

edited

Loading