[TENT] Opt-in deadline-aware NIC bandwidth arbitration (RFC #2792)#2794
Conversation
…i#2792) Within a priority tier, when several flows contend for one NIC the bandwidth is split blindly/equally today, with no way to give a flow about to miss its deadline a larger share. This adds an opt-in, deadline-aware arbitration. The ordering acts at the per-NIC-path slice-post point: after slices are grouped by (local NIC -> remote NIC), they are sorted most-urgent-first by predicted MLU (predicted transfer time / remaining deadline window, reusing the MLU notion from kvcache-ai#2618) before submitSlices() hands them to the QP budget. That is the point where same-tier flows actually contend for the shared NIC; an earlier prototype that reordered at the priority-queue pop had no effect because the QP budget, not the tier queue, is the contention point. * bw_arbitration.h: pure OrderByUrgency() policy (unit-testable, no RDMA deps) + PredictedMlu(). bw<=0 or no-deadline degrade to original order. * workers.cpp: opt-in wiring via transports/rdma/deadline_bw_arbitration (default false = byte-identical FIFO / equal split). * transfer_engine_bench: --deadline_us / --deadline_tight_threads to tag N threads as tight-deadline flows + per-flow tight/loose throughput report. * bw_arbitration_test: 7 cases (tighter-first, no-deadline-last, past-due, FIFO ties, zero-bandwidth no-op, size-weighting). Measured (H20/ConnectX 200G RoCE, 2 nodes, 16 threads = 4 tight + 12 loose, 1 MB): tight flows 12.1 -> 44.6 GB/s (+269%), loose yield, total throughput unchanged (48.4 GB/s). At light load it is a no-op, as intended. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements deadline-aware NIC bandwidth arbitration within a priority tier (RFC #2792) to prioritize flows nearing their deadlines under NIC contention. It introduces the arbitration logic, integrates it as an opt-in feature in the RDMA transport worker's send path, and adds corresponding benchmark options and unit tests. Feedback on the changes highlights a performance concern in the hot path of asyncPostSend(), where multiple dynamic memory allocations are performed; using a thread_local scratch buffer is suggested to sort slices in-place and avoid heap fragmentation.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
Validation that the effect is the arbitration (not a thread/NIC-placement artifact) — swept the number of tight-deadline threads, arbitration on, 16 threads total, 1 MB:
The share tracks which threads are tagged tight (the Honest detail: a single tight flow tops out at ~11 GB/s (per-flow QP throughput ceiling), so tight=1 takes 11 and leaves the rest to loose rather than monopolizing — priority, not free bandwidth. tight≥4 saturates the NIC. |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Good contribution! Additionally, we prefer to use |
|
Addressed the requested follow-ups:
The PR body now contains both normal-QP and deliberately constrained-QP results. Under normal synchronous batch=1 load there is no accumulated slice backlog and therefore no material difference; under |
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
|
Merged current main and resolved the CMake conflict in |
|
@alogfans This has been addressed. The deadline workload was migrated from the old custom
The updated PR body contains the tebench validation results and the caveat for the constrained-QP run. |
# Conflicts: # mooncake-transfer-engine/benchmark/tent_backend.cpp
|
Merged latest Cluster validation:
|
3 participants
Implements the deadline-aware NIC bandwidth arbitration proposed in #2792.
Problem
TENT's QoS is vertical: SL/TC and priority tiers separate business classes. When flows in the same tier contend for one NIC, bandwidth was split without deadline awareness.
Design
Ordering acts at the per-NIC-path slice-post point: after slices are grouped by
(local NIC → remote NIC), they are sorted most-urgent-first by predicted MLU immediately beforesubmitSlices()reaches the QP budget.bw_arbitration.h: pure, unit-testable urgency policy.workers.cpp: opt-in wiring viatransports/rdma/deadline_bw_arbitration(defaultfalse). The hot path reuses athread_localscratch vector, so capacity is retained across calls rather than allocated for every arbitration.tebench: native--deadline_us,--deadline_tight_threads, and--deadline_bw_arbitrationflags, with aggregate tight/loose throughput reporting. The old customtransfer_engine_benchflags were removed.Validation
bw_arbitration_test: 7/7 passed (deadline ordering, no-deadline, past-due, FIFO ties, zero-bandwidth, size weighting).Two-node H20 / ConnectX RoCE,
mlx5_bond_0, RDMA READ, 16 threads (4 tight + 12 loose), 1 MiB, three repeats:Normal QP budget (
MC_MAX_WR=256, batch=1)No material change is expected here: synchronous batch=1 does not accumulate enough pending slices for arbitration.
Deliberate QP pressure (
MC_MAX_WR=16, batch=32)Across all three repeats, tight aggregate throughput rose by about 300%; total throughput did not regress. The total-throughput increase under this artificial low-WR setting indicates that urgency ordering also avoids QP under-utilization in this workload, so this result should not be interpreted as a general 2x bandwidth claim.
Notes
default_bandwidth_gbps; live per-NIC EWMA can follow separately.Relates to #2792.