[XPU][NIXL] Add GPUDirect RDMA support for XPU

[zhenwei-intel]

Purpose

Add GPUDirect RDMA support for XPU in NIXL connector.

Requirements：

• UCX must include the fix from UCT/IB/ZE: Enable GPUDirect RDMA for Intel Xe devices openucx/ucx#11187.

Limitations:

• Must be set UCX_NET_DEVICES manually for better performance until UCT/ZE: Add device topology registration openucx/ucx#11180 is merged.
• Currently ze-ipc is not supported. UCT/ZE/ZE_IPC: enable level zero ipc support for Intel GPUs openucx/ucx#11218

Test Plan

Performance data of Llama3.3-70B int4 model with fp8 kvcache on 8xB60, ISL=1500, OSL=150
2P1D vs Non-PD under SLO TTFT<5s, ITL<100ms

• Serve more requests: under SLO, 2P1D achieved a request throughput of 1.06, compared to 0.64 for the Non-PD — 1.65x improvement.

PD commands

prefill

export UCX_TLS=ib,rc,ze_copy

export ZE_AFFINITY_MASK=2,3
export model_name=ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4
export tp_size=2


VLLM_USE_V1=1 VLLM_NIXL_SIDE_CHANNEL_HOST=localhost VLLM_NIXL_SIDE_CHANNEL_PORT=5577 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ENABLE_V1_MULTIPROCESSING=1 vllm serve $model_name -tp $tp_size --host localhost --port 7101 --seed 42 --enforce-eager --dtype float16 --gpu-memory-utilization 0.9 --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"xpu"}' --max-model-len 8192 --block-size 64 --no-enable-prefix-caching --kv-cache-dtype fp8

prefill2

export UCX_TLS=ib,rc,ze_copy

export ZE_AFFINITY_MASK=4,5
export model_name=ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4
export tp_size=2


VLLM_USE_V1=1 VLLM_NIXL_SIDE_CHANNEL_HOST=localhost VLLM_NIXL_SIDE_CHANNEL_PORT=5377 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ENABLE_V1_MULTIPROCESSING=1 vllm serve $model_name -tp $tp_size --host localhost --port 7102 --seed 42 --enforce-eager --dtype float16 --gpu-memory-utilization 0.9 --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"xpu"}' --max-model-len 8192 --block-size 64 --no-enable-prefix-caching--kv-cache-dtype fp8

decode

export UCX_TLS=ib,rc,ze_copy

export ZE_AFFINITY_MASK=0,1
export model_name=ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4
export tp_size=2


VLLM_USE_V1=1 VLLM_NIXL_SIDE_CHANNEL_HOST=localhost VLLM_NIXL_SIDE_CHANNEL_PORT=5177 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ENABLE_V1_MULTIPROCESSING=1 vllm serve $model_name -tp $tp_size --host localhost --port 7201 --seed 42 --enforce-eager --dtype float16 --gpu-memory-utilization 0.9 --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"xpu"}' --max-model-len 8192 --block-size 64 --no-enable-prefix-caching --kv-cache-dtype fp8

proxy

python3 tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --prefiller-hosts localhost  localhost --prefiller-ports 7101 7102 --decoder-host localhost --decoder-port 7201 --host localhost --port 7300

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[zhenwei-intel]

cc. @xuechendi, @rogerxfeng8, @yma11

[gemini-code-assist]

Code Review

This pull request adds GPUDirect RDMA support for XPU in the NIXL connector. The changes involve updating the UCX build script to a specific commit that includes the necessary fixes, enabling Level Zero support, and modifying the NIXL connector and XPU platform code to support XPU device memory for KV transfer. The changes are generally correct and well-targeted. I have one suggestion to improve the precision of a workaround to avoid potential side effects.

[xuechendi]

@zhenwei-intel, --kv-cache-dtype fp8 , we won't be able to transfer scale at this moment, so the accuracy is impact. Might not significant in simple text.

[zhenwei-intel]

@zhenwei-intel, --kv-cache-dtype fp8 , we won't be able to transfer scale at this moment, so the accuracy is impact. Might not significant in simple text.

The PR of FP8 KV Cache hasn't been upstreamed yet. On the upstream branch, BF16 KV Cache can be used.

The current performance testing is based on FP8 KV Cache (cherry-picked https://github.com/intel-innersource/applications.ai.gpu.vllm-xpu/pull/57), with a scale of 1.0.

[hshen14]

@zhenwei-intel, --kv-cache-dtype fp8 , we won't be able to transfer scale at this moment, so the accuracy is impact. Might not significant in simple text.

The PR of FP8 KV Cache hasn't been upstreamed yet. On the upstream branch, BF16 KV Cache can be used.

The current performance testing is based on FP8 KV Cache (cherry-picked https://github.com/intel-innersource/applications.ai.gpu.vllm-xpu/pull/57), with a scale of 1.0.

No need to transfer the scale as both the models for P and D instances should have such info.

[xuechendi]

LGTM, @NickLucche , may you help to review

[NickLucche]

this looks ok on my side, nit on buffer naming

[NickLucche]

synced with @xuechendi

[zhenwei-intel]

@jikunshang @1643661061leo I updated the dockerfile.xpu, please take another look.

[Copilot]

Pull request overview

Adds GPUDirect RDMA support for XPU in the NIXL connector by enabling XPU VRAM KV-buffer paths, applying a UCX workaround for memtype misdetection, and updating the XPU Docker image to build UCX+NIXL from source with RDMA dependencies.

Changes:

• Enable "xpu" as a KV buffer device for the NIXL connector and map it to VRAM memory type.
• Apply a UCX environment workaround on XPU to avoid memtype-cache misdetection.
• Update Dockerfile.xpu to build UCX (pinned commit) and NIXL from source and install RDMA tooling/libs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.

File	Description
vllm/platforms/xpu.py	Sets UCX env var to avoid UCX memtype-cache misdetection on XPU.
vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py	Allows XPU KV buffers and maps XPU device to VRAM in NIXL.
docker/Dockerfile.xpu	Builds UCX/NIXL from source and installs RDMA dependencies for XPU images.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jikunshang]

merge as ci passed. thanks for your contribution!