CollectiveX: add UCCL-EP as a third pluggable EP backend (#2288)

Author: Oseltamivir

.github/workflows/collectivex-sweep.yml

@@ -10,7 +10,7 @@ on:
         description: "EP library to sweep — 'all' runs every EP backend in one matrix"
         type: choice
         default: all
-        options: [all, deepep-v2, mori]
+        options: [all, deepep-v2, mori, uccl-ep]
       only_sku:
         description: Restrict to one GHA runner pool; blank = all
         type: string
@@ -84,10 +84,12 @@ jobs:
     if: ${{ fromJSON(needs.setup.outputs.n) > 0 }}
     strategy:
       fail-fast: false
-      # Fixed global cap; real throttling is each SKU pool's runner count and its
-      # cluster's Slurm partition. setup interleaves shards across SKUs, so the
-      # first jobs under this cap spread over pools instead of queuing on one.
-      max-parallel: 10
+      # Global cap; real throttling is each SKU pool's runner count and its cluster's
+      # Slurm partition. setup interleaves shards across SKUs so the cap spreads over
+      # pools instead of queuing on one. Kept above the total per-pool runner count so a
+      # saturated pool's dispatched-but-waiting legs cannot starve pools that have free
+      # runners (10 was too low when the shared fleet is busy with concurrent sweeps).
+      max-parallel: 30
       matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}
     runs-on: ${{ matrix.sku }}
     timeout-minutes: 350

experimental/CollectiveX/.gitignore

@@ -5,6 +5,7 @@ unsupported/
 .shards/
 .collx_workloads/
 .collx_backend/
+.collx_sources/
 /matrix_full.json
 gpucore.*
 

experimental/CollectiveX/README.md

@@ -13,18 +13,22 @@ responsibility. The full measurement methodology is in [docs/methodology.md](doc
 
 The workload uses packed placement and one pinned `fixed-profile` resource configuration per
 backend/topology; there is no tuning sweep. Combine is always BF16; dispatch precision is a swept
-dimension — a BF16 control plus a caller-prequantized FP8 dispatch on every backend. Coverage is
-uniform routing only. Cases run in one of two modes:
+dimension — a BF16 control plus an FP8 dispatch on every backend, caller-prequantized in `normal`
+mode (in `low-latency` the DeepEP and UCCL-EP kernels quantize internally from BF16; MoRI stays
+caller-prequantized). Coverage is uniform routing only. Cases run in one of two modes:
 
 - `normal` uses `layout-and-dispatch-v1`, rank-deduplicated token payloads, and activation-only,
   unweighted rank-sum combine. It runs the full decode and prefill ladders.
 - `low-latency` uses each backend's decode-optimized kernel family: on DeepEP the legacy
   `deep_ep.Buffer` IBGDA `low_latency_dispatch`/`low_latency_combine` (a per-expert padded receive
-  and a source-side gate-weighted combine); on MoRI the `IntraNodeLL` kernel (single-call,
+  and a source-side gate-weighted combine); on UCCL-EP the same legacy `Buffer` low-latency kernels
+  over its CPU-proxy transport; on MoRI the `IntraNodeLL` kernel (single-call,
   pure-intranode, same compact layout and unweighted rank-sum combine as `IntraNode`). It is a
   decode-phase-only, per-SKU-capability-gated addition whose runnable set differs from `normal`'s, so
-  it is enabled from each SKU's `ll_backends` registry entry (currently DeepEP V2 EP8 on H100/H200/B200
-  and MoRI EP8 on MI300X/MI325X/MI355X). Scoped single-node EP8 runs over the intra-node NVLink/XGMI
+  it is enabled from each SKU's `ll_backends` registry entry (currently DeepEP V2 EP8 on H100/H200/B200,
+  MoRI EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel
+  trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL).
+  Scoped single-node EP8 runs over the intra-node NVLink/XGMI
   low-latency path (no `/dev/gdrdrv` needed — validated on H200 with it absent); NVSHMEM/IBGDA on the
   wire is only a multi-node scale-out (EP16) concern.
 
@@ -63,6 +67,7 @@ scale-up domain.
 |---|---|
 | DeepEP V2 | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only |
 | MoRI | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch |
+| UCCL-EP | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now |
 
 DeepEP V2 means the `ElasticBuffer` implementation introduced by
 [DeepEP PR #605](https://github.com/deepseek-ai/DeepEP/pull/605), not a newer legacy `Buffer` build.