UCT/PLUGIN: ib plugin for token query by jeynmann · Pull Request #11277 · openucx/ucx

jeynmann · 2026-03-20T04:45:35Z

What?

Add plugin support for token query(capability, rx token, tx token, tx token lengh, rx token length).
This is the plugin part of PR#11234

How?

Add weak symbols in uct.
The implementation will be provided by plugin lib.

roiedanino · 2026-03-22T11:25:19Z

+}
+
+ucs_status_t __attribute__((weak))
+uct_ib_plugin_ep_query(uct_ep_h ep, uct_ep_attr_t *ep_attr)


Will it be possible to change this to:

uct_ib_plugin_query_qp()
?

roiedanino · 2026-03-25T11:15:05Z

 	dv/ib_mlx5_dv.c \
-	dv/ib_mlx5dv_md.c
+	dv/ib_mlx5dv_md.c \
+	../plugin/uct_ib_plugin_stubs.c


Add a Makefile.am in /plugin and the stub there
(we should also add the .h file to the noinst_HEADERS list)

roiedanino · 2026-03-25T11:16:50Z

+pluginincludedir = $(includedir)/uct/ib/plugin
+plugininclude_HEADERS = plugin/uct_ib_plugin.h


should be added in plugin/Makefile.am

roiedanino · 2026-03-25T11:19:35Z

+uct_ib_plugin_qp_query(const uct_ib_plugin_qp_query_params_t *params,
+                       uct_ib_plugin_qp_query_attr_t *attr)
+{


Where is it going to be called from?

It will be invoked by uct_iface_query_v2 and uct_ep_query.
The UCT implementation will be submitted in a separate PR.

roiedanino · 2026-03-25T11:32:18Z

+ * @brief QP query input parameters.
+ */
+typedef struct uct_ib_plugin_qp_query_params {
+    uint64_t           field_mask;


should also define an enum for the field_mask

evgeny-leksikov · 2026-04-02T19:59:16Z

+        /* IB plugin capability */
+#define UCT_IFACE_FLAG_IB_PLUGIN_QUERY_TOKEN UCS_BIT(47) /**< IB plugin for token query is available */


need to getrid of "IB" and "plugin" in UCT API, just "UCT_IFACE_FLAG_QUERY_TOKEN", not implemented/supported => no capability

evgeny-leksikov · 2026-04-02T20:04:19Z

+/**
+ * @brief Field mask bits for @ref uct_ib_plugin_qp_query_attr_t.
+ * Selects which output fields to populate.
+ */
+enum uct_ib_plugin_qp_query_attr_field {
+    UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_TX_TOKEN_LEN = UCS_BIT(0),
+    UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_RX_TOKEN_LEN = UCS_BIT(1),
+    UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_TX_TOKEN     = UCS_BIT(2),
+    UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_RX_TOKEN     = UCS_BIT(3)
+};


I think we'll have similar bits for uct_ep_query, so why dont pass the same semantic to the "uct_ib_plugin_ep_query", cast to correct ep_type and access QP.
Then uct_ib_plugin_qp_query_params_t becomes redundant.

evgeny-leksikov · 2026-04-02T20:06:54Z

+/**
+ * @brief Return plugin-contributed iface capability flags.
+ *
+ * @return Bitmask of UCT_IFACE_FLAG_* contributed by the plugin.
+ */
+uint64_t uct_ib_plugin_iface_flags(void);


👍 for plugin contribution to defined UCT API

evgeny-leksikov · 2026-04-02T20:09:54Z

+ * @return UCS_OK on success, error code otherwise.
+ */
+ucs_status_t
+uct_ib_plugin_qp_query(const uct_ib_plugin_qp_query_params_t *params,


uct_ib_plugin_ep_query?

@roiedanino suggests renaming it to uct_ib_plugin_query_qp. We now rely on mlx5dv_devx_obj and ibv_qp rather than the internal UCT structure, which might provide a cleaner and simpler approach.

Will it be possible to change this to: uct_ib_plugin_query_qp() ?

IMO it adds inconsistency and overcomplication. Why does iface_flags return native UCT API level flags, but ep_query requires conversion of ep -> qp and qp_attr -> ep_attr converting in base (non-plugin) UCT/IB code?

evgeny-leksikov · 2026-04-07T11:37:02Z

+    UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_TX_TOKEN_LEN = UCS_BIT(0),
+    UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_RX_TOKEN_LEN = UCS_BIT(1),


in #11234 these are defined as iface attributes

roiedanino · 2026-04-19T12:04:19Z

Example code for plugin side:

static ucs_status_t uct_ib_plugin_iface_flags(void) {
    ucs_info("executed uct_ib_plugin_iface_flags successfully");
    return UCS_OK;

}
static ucs_status_t
uct_ib_plugin_qp_query(const uct_ib_mlx5_ext_qp_query_attr_t *params,
                       uct_ib_mlx5_ext_qp_query_attr_t *attr) {
    ucs_info("executed uct_ib_plugin_qp_query successfully %p %p", params, attr);
    return UCS_OK;
}

static uct_ib_mlx5_ext_ops_t ucx_ext_plugin_ops = {
    .iface_flags = uct_ib_plugin_iface_flags,
    .qp_query = uct_ib_plugin_qp_query,
};

UCS_MODULE_INIT()
{
   // ...
    uct_ib_mlx5_ext_register(&ucx_ext_plugin_ops);

    return UCS_OK;
}

yosefe

pls fix in next

yosefe · 2026-05-03T07:40:26Z

                                                       are no outstanding send operations */

+        /* Token query capability */
+#define UCT_IFACE_FLAG_QUERY_TOKEN UCS_BIT(47) /**< Interface supports token query */


let's add this to uct_iface_query_v2

do we need to separate to TX_TOKEN and RX_TOKEN? we need to use TX_TOKEN on 2iface transports like DC

yosefe · 2026-05-03T07:40:56Z

+    ucs_assertv(ops->iface_flags != NULL, "ext_resgiter: iface_flags function is NULL");
+    ucs_assertv(ops->qp_query != NULL, "ext_resgiter: qp_query function is NULL");


yosefe · 2026-05-03T07:41:05Z

+    ucs_assertv(ops->iface_flags != NULL, "ext_resgiter: iface_flags function is NULL");
+    ucs_assertv(ops->qp_query != NULL, "ext_resgiter: qp_query function is NULL");
+    uct_ib_mlx5_ext_ops = *ops;
+    ucs_info("ib mlx5: registered external ops successfully");


make it ucs_debug + details

yosefe · 2026-05-03T07:41:14Z

+#include <ucs/type/status.h>
+#include <ucs/sys/stubs.h>
+
+


remove spaceline

yosefe · 2026-05-03T07:41:42Z

+struct ibv_qp;
+struct mlx5dv_devx_obj;


we can include ib / mlx5dv headers directly

yosefe · 2026-05-03T07:42:24Z

+    uint64_t field_mask;
+    size_t   tx_token_len;
+    size_t   rx_token_len;
+    void     *tx_token;
+    void     *rx_token;
+    uint32_t qp_num;


pls add documentation

yosefe · 2026-05-03T07:42:33Z

+typedef ucs_status_t (*uct_ib_mlx5_ext_iface_flags_func_t)(uint64_t *flags);
+typedef ucs_status_t (*uct_ib_mlx5_ext_qp_query_func_t)(
+        struct ibv_qp *qp, struct mlx5dv_devx_obj *devx_obj,
+        uct_ib_mlx5_ext_qp_query_attr_t *attr);


pls add documentation

yosefe · 2026-05-03T07:45:11Z

+{
+    ucs_assertv(ops->iface_flags != NULL, "ext_resgiter: iface_flags function is NULL");
+    ucs_assertv(ops->qp_query != NULL, "ext_resgiter: qp_query function is NULL");
+    uct_ib_mlx5_ext_ops = *ops;


we probably want a list of providers, move uct_ib_mlx5_ext_ops only to C file, and add function that search for the first non-NULL operation in the list and calls the operation or returns NOT_SUPPORTED if not found. maybe adde default provider that returns 0 from iface_flags .

Signed-off-by: Roie Danino <rdanino@nvidia.com>

…x5_ext_register during UCS_MODULE_INIT Signed-off-by: Roie Danino <rdanino@nvidia.com>

Signed-off-by: Roie Danino <rdanino@nvidia.com>

fteng-NV mentioned this pull request Mar 20, 2026

UCT/PLUGIN: UCT API for op uct_iface_query_v2 #11278

Closed

roiedanino reviewed Mar 22, 2026

View reviewed changes

roiedanino reviewed Mar 25, 2026

View reviewed changes

jeynmann mentioned this pull request Mar 30, 2026

NIXL_EP: update num_channels to num_experts_per_rank ai-dynamo/nixl#1467

Open

fteng-NV mentioned this pull request Apr 1, 2026

UCT/PLUGIN: add definition in uct_iface_attr_v2_t for uct_iface_query_v2 #11316

Open

gleon99 requested a review from evgeny-leksikov April 2, 2026 15:48

evgeny-leksikov reviewed Apr 2, 2026

View reviewed changes

evgeny-leksikov reviewed Apr 7, 2026

View reviewed changes

roiedanino requested review from evgeny-leksikov and yosefe May 3, 2026 07:09

roiedanino previously approved these changes May 3, 2026

View reviewed changes

yosefe reviewed May 3, 2026

View reviewed changes

JeynmannZ and others added 14 commits May 3, 2026 12:08

UCT/PLUGIN: ib plugin for token query

6b17991

UCT/IB: revert uct definations

329f383

UCT/PLUGIN: update plugin

15285f0

UCT/PLUGIN: fix according to comments

79fa50d

UCT/PLUGIN: cleanup comments

819748d

UCT/PLUGIN: add qpn field

df3ebe7

UCT/PLUGIN: fix according to comments

034eafb

UCT/IB/PLUGIN: added runtime function resolve in stubs

57294e4

Signed-off-by: Roie Danino <rdanino@nvidia.com>

UCT/IB/MLX5: plugin API - plugin will register itself using uct_ib_ml…

81cbe65

…x5_ext_register during UCS_MODULE_INIT Signed-off-by: Roie Danino <rdanino@nvidia.com>

UCT/PLUGIN: fix according to comments

f20fc9c

UCT/PLUGIN: fix according to comments

9cf57d2

UCT/PLUGIN: fix according to comments

cca6182

UCT/DEVICE: fix ci check

e3ae85a

UCT/API: Moved iface query flag to v2, minor fixes

a2e4020

Signed-off-by: Roie Danino <rdanino@nvidia.com>

roiedanino dismissed their stale review via a2e4020 May 3, 2026 10:08

roiedanino force-pushed the plugin_query_token branch from f48724e to a2e4020 Compare May 3, 2026 10:08

roiedanino requested a review from yosefe May 3, 2026 10:08

openucx deleted a comment from svc-nixl May 4, 2026

		pluginincludedir = $(includedir)/uct/ib/plugin
		plugininclude_HEADERS = plugin/uct_ib_plugin.h

		/* IB plugin capability */
		#define UCT_IFACE_FLAG_IB_PLUGIN_QUERY_TOKEN UCS_BIT(47) /*< IB plugin for token query is available /

		UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_TX_TOKEN_LEN = UCS_BIT(0),
		UCT_IB_PLUGIN_QP_QUERY_ATTR_FIELD_RX_TOKEN_LEN = UCS_BIT(1),

		ucs_assertv(ops->iface_flags != NULL, "ext_resgiter: iface_flags function is NULL");
		ucs_assertv(ops->qp_query != NULL, "ext_resgiter: qp_query function is NULL");

		#include <ucs/type/status.h>
		#include <ucs/sys/stubs.h>

Conversation

jeynmann commented Mar 20, 2026 • edited by yosefe Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

How?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roiedanino Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeynmann Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roiedanino commented Apr 19, 2026

Uh oh!

yosefe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jeynmann commented Mar 20, 2026 •

edited by yosefe

Loading

roiedanino Mar 25, 2026 •

edited

Loading

jeynmann Apr 3, 2026 •

edited

Loading