[TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation by chienchunhung · Pull Request #15238 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-06-11T00:25:35Z

Summary

This PR adds gated deadline enforcement and in-flight KV-transfer cancellation for the C++ NIXL-backed disaggregated cache transceiver.

Set TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 to opt into the experimental behavior. The flag defaults to off, and focused regression tests protect the existing timeout and backend-selection paths when it is unset. Opt-in is currently accepted only for the C++ NIXL transceiver using the UCX plugin, which is also the default when TRTLLM_NIXL_KVCACHE_BACKEND is unset; Python/V2, direct UCX, MPI, Mooncake, explicitly selected NIXL Libfabric, and ambiguous legacy backend selection fail before a transceiver is constructed.

On the qualified path, timed-out context and generation transfers request cancellation, while cleanup is deferred until transfer status, including cross-rank consensus where required, reaches a terminal outcome. Transfer handles are then released, and buffers that may still be touched by an unquiesced transfer are poisoned fail-closed.

Cancellation flow

The executor detects that a KV-transfer deadline has expired.
The C++ transceiver records a cancellation request for the affected transfer.
NIXL agent polling remains bounded and observes the cancellation request, allowing the executor to resume scheduling and transfer progress between checks.
Generation timeout and error handling is synchronized across ranks before request cleanup proceeds.
Request, KV-block, and transfer-buffer cleanup remains deferred until the transfer reaches a terminal or quiesced state; the backend transfer handle is then released through releaseXferReq.
If quiescence cannot be established, the affected transfer buffer is poisoned and the executor fails closed, requiring process restart.

Scope and limitations

Area	Coverage
Feature disabled (default)	Preserves legacy context cleanup, generation timeout handling, backend normalization, Python/V2 construction, and NIXL Libfabric selection.
C++ NIXL with UCX plugin	Implements deadline enforcement, rank-consistent cancellation, deferred cleanup, transfer-handle release, and fail-closed buffer poisoning behind the opt-in flag.
Python/V2 transceiver	Existing behavior is unchanged while the flag is unset. Explicit opt-in is rejected before construction.
`blockAll` / `nullopt` wait-all paths	Not covered. Explicit wait-all calls can still block indefinitely and require a follow-up implementation.
Direct UCX, MPI, Mooncake, and NIXL Libfabric	Not supported. Explicit opt-in is rejected; each backend requires its own cancellation, quiescence, and resource-lifetime contract.
Production default-on rollout	Deferred. Requires the distributed intent, peer-quiescence, shutdown-handshake, and stress-test gates described in the design follow-up.

All ranks and both disaggregated services must use the same feature setting and qualified transport configuration. #15238 itself does not add cross-service capability negotiation; that default-on hardening is tracked by #15798, with peer cancellation and bounded terminal retirement designed in #15799. Neither follow-up blocks merging #15238 with the feature disabled by default.

Dependency graph

Arrows point from prerequisite to dependent. #15238 and #15737 are parallel, independently mergeable inputs to #15794. All PRs target NVIDIA/main; stacked child diffs are cumulative until their parents merge.

#15798 and #15799 are production/default-on hardening follow-ups. They do not block the default-off merges of #15238, #15794, or #15795. #15798 is required for #15738. #15799 may be replaced only by an explicitly approved bounded fail/restart policy that satisfies the same memory-safety requirements.

flowchart LR
    PR15238["#15238<br/>gated cancellation<br/>(default off)"]
    PR15737["#15737<br/>CacheSender lost-wakeup fix"]
    PR15794["#15794<br/>C++ protocol and buffer safety<br/>(default off)"]
    PR15795["#15795<br/>PyExecutor lifecycle safety<br/>(default off)"]
    PR15798["#15798<br/>CTX↔GEN protocol/mode negotiation"]
    PR15799["#15799<br/>peer cancel/terminal protocol<br/>(design scaffold)"]
    PR15738["#15738<br/>qualified default-on policy"]

    PR15238 --> PR15794
    PR15737 --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15795 --> PR15738
    PR15798 --> PR15738
    PR15799 -.->|or bounded fail/restart policy| PR15738

Validation

Rebased onto upstream main at 8e7b98c09b; merged prerequisites, including [TRTLLM-12721][fix] Bound disagg transfer polling and admission #15356, are inherited from main.
History condensed to two DCO-signed commits: one implementation commit and one test/CI-registration commit.
git diff --check passed.
Python syntax compilation passed for the modified PyExecutor modules and focused tests.
Changed-file pre-commit hooks passed, including clang-format, YAPF, Ruff, codespell, and test-list validation.
Added focused CPU regression coverage for:
- flag-unset short-circuiting and legacy context/generation timeout cleanup;
- ADP rank-uniform generation error handling and vote-before-cleanup ordering;
- unsupported capability warning behavior;
- default-off Python/V2, Libfabric, and legacy backend-selector compatibility;
- strict opt-in qualification for C++ NIXL with the UCX plugin, including default-unset plugin resolution.
Exact-head CI is required for compiled C++ and full regression validation.

Summary by CodeRabbit

New Features
- Added optional in-flight cancellation support for disaggregated KV transfers, with new timeout and retry controls.
- Exposed transfer-buffer health checks to Python so apps can detect poisoned transfer state.
Bug Fixes
- Improved transfer handling under timeouts and failures to avoid stuck requests and clean up resources more reliably.
- Added stricter configuration validation for disaggregated transfer modes.
- Expanded test coverage, including a new stress test for C++ NIXL synchronous KV transfer flows.

chienchunhung · 2026-06-11T00:26:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-11T00:33:11Z

PR_Github #53406 [ run ] triggered by Bot. Commit: d16f28c Link to invocation

tensorrt-cicd · 2026-06-11T16:25:36Z

PR_Github #53406 [ run ] completed with state SUCCESS. Commit: d16f28c
/LLM/main/L0_MergeRequest_PR pipeline #42578 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-06-11T17:40:00Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-11T17:45:45Z

PR_Github #53628 [ run ] triggered by Bot. Commit: 631cd67 Link to invocation

tensorrt-cicd · 2026-06-12T02:57:16Z

PR_Github #53628 [ run ] completed with state SUCCESS. Commit: 631cd67
/LLM/main/L0_MergeRequest_PR pipeline #42773 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-06-12T04:54:26Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-12T05:00:48Z

PR_Github #53788 [ run ] triggered by Bot. Commit: 631cd67 Link to invocation

tensorrt-cicd · 2026-06-12T05:41:53Z

PR_Github #53788 [ run ] completed with state FAILURE. Commit: 631cd67
/LLM/main/L0_MergeRequest_PR pipeline #42906 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-06-12T17:30:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-12T17:36:36Z

PR_Github #53928 [ run ] triggered by Bot. Commit: 36f9ee5 Link to invocation

tensorrt-cicd · 2026-06-13T08:11:36Z

PR_Github #53928 [ run ] completed with state SUCCESS. Commit: 36f9ee5
/LLM/main/L0_MergeRequest_PR pipeline #43021 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-06-14T17:24:36Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-14T17:30:36Z

PR_Github #54148 [ run ] triggered by Bot. Commit: ff622ff Link to invocation

chienchunhung · 2026-06-14T19:23:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-14T19:29:21Z

PR_Github #54155 [ run ] triggered by Bot. Commit: 6954acf Link to invocation

tensorrt-cicd · 2026-06-14T19:32:51Z

PR_Github #54148 [ run ] completed with state ABORTED. Commit: ff622ff

Link to invocation

tensorrt-cicd · 2026-06-15T05:11:53Z

PR_Github #54155 [ run ] completed with state SUCCESS. Commit: 6954acf
/LLM/main/L0_MergeRequest_PR pipeline #43238 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-06-15T17:10:26Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-15T17:16:12Z

PR_Github #54336 [ run ] triggered by Bot. Commit: 24eb554 Link to invocation

chienchunhung · 2026-06-15T19:55:27Z

/bot run

tensorrt-cicd · 2026-06-15T20:01:22Z

PR_Github #54356 [ run ] triggered by Bot. Commit: 24eb554 Link to invocation

tensorrt-cicd · 2026-06-15T20:06:11Z

PR_Github #54336 [ run ] completed with state ABORTED. Commit: 24eb554

Link to invocation

tensorrt-cicd · 2026-07-07T11:45:33Z

PR_Github #57814 [ run ] completed with state SUCCESS. Commit: 853afb4
/LLM/main/L0_MergeRequest_PR pipeline #46514 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-07T17:00:29Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-07T17:13:07Z

PR_Github #58035 [ run ] triggered by Bot. Commit: 86a83e9 Link to invocation

tensorrt-cicd · 2026-07-07T23:50:59Z

PR_Github #58035 [ run ] completed with state SUCCESS. Commit: 86a83e9
/LLM/main/L0_MergeRequest_PR pipeline #46705 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-08T00:37:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-08T00:43:58Z

PR_Github #58102 [ run ] triggered by Bot. Commit: 86a83e9 Link to invocation

tensorrt-cicd · 2026-07-08T03:07:30Z

PR_Github #58102 [ run ] completed with state SUCCESS. Commit: 86a83e9
/LLM/main/L0_MergeRequest_PR pipeline #46765 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-08T05:45:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-08T05:52:33Z

PR_Github #58165 [ run ] triggered by Bot. Commit: 86a83e9 Link to invocation

tensorrt-cicd · 2026-07-08T08:51:05Z

PR_Github #58165 [ run ] completed with state FAILURE. Commit: 86a83e9
/LLM/main/L0_MergeRequest_PR pipeline #46812 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-08T15:19:37Z

/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-4,DGX_H100-2_GPUs-PyTorch-Others-2,DGX_H100-4_GPUs-PyTorch-DeepSeek-1,DGX_H100-4_GPUs-PyTorch-Others-2"

tensorrt-cicd · 2026-07-08T15:26:00Z

PR_Github #58258 [ run ] triggered by Bot. Commit: 86a83e9 Link to invocation

tensorrt-cicd · 2026-07-08T17:00:43Z

PR_Github #58258 [ run ] completed with state FAILURE. Commit: 86a83e9
/LLM/main/L0_MergeRequest_PR pipeline #46898 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-08T18:34:15Z

/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-4,DGX_H100-2_GPUs-PyTorch-Others-2,DGX_H100-4_GPUs-PyTorch-DeepSeek-1,DGX_H100-4_GPUs-PyTorch-Others-2"

tensorrt-cicd · 2026-07-08T18:40:36Z

PR_Github #58282 [ run ] triggered by Bot. Commit: 86a83e9 Link to invocation

tensorrt-cicd · 2026-07-08T20:22:37Z

PR_Github #58282 [ run ] completed with state SUCCESS. Commit: 86a83e9
/LLM/main/L0_MergeRequest_PR pipeline #46918 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-08T22:56:09Z

/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-4,DGX_H100-2_GPUs-PyTorch-Others-2,DGX_H100-4_GPUs-PyTorch-DeepSeek-1,DGX_H100-4_GPUs-PyTorch-Others-2"

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-07-09T22:18:02Z

/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-4,DGX_H100-2_GPUs-PyTorch-Others-2,DGX_H100-4_GPUs-PyTorch-DeepSeek-1,DGX_H100-4_GPUs-PyTorch-Others-2"

tensorrt-cicd · 2026-07-09T22:29:07Z

PR_Github #58517 [ run ] triggered by Bot. Commit: cb13601 Link to invocation

tensorrt-cicd · 2026-07-10T04:09:43Z

PR_Github #58517 [ run ] completed with state SUCCESS. Commit: cb13601
/LLM/main/L0_MergeRequest_PR pipeline #47124 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chienchunhung · 2026-07-10T04:11:13Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-10T04:33:46Z

PR_Github #58587 [ run ] triggered by Bot. Commit: cb13601 Link to invocation

github-actions Bot assigned chienchunhung Jun 11, 2026

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from d16f28c to 631cd67 Compare June 11, 2026 17:14

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from 631cd67 to 36f9ee5 Compare June 12, 2026 17:30

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from 36f9ee5 to ff622ff Compare June 14, 2026 17:23

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from ff622ff to 6954acf Compare June 14, 2026 19:23

chienchunhung mentioned this pull request Jun 14, 2026

[TRTLLM-12721][fix] Bound disagg transfer polling and admission #15356

Merged

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from 6954acf to 24eb554 Compare June 15, 2026 17:10

chuangz0 reviewed Jul 7, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/kv_cache_transceiver.py Outdated

chienchunhung requested a review from chuangz0 July 7, 2026 16:49

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from 44595e4 to 86a83e9 Compare July 7, 2026 16:59

chienchunhung changed the title ~~[TRTLLM-12721][fix] Add gated C++ disagg in-flight cancellation~~ [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation Jul 7, 2026

chienchunhung added 2 commits July 9, 2026 15:14

[TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation

154118c

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[TRTLLM-12721][test] Cover disagg in-flight cancellation

cb13601

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/disagg-deadline-inflight-cancel branch from 86a83e9 to cb13601 Compare July 9, 2026 22:17

chuangz0 approved these changes Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

chienchunhung commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Cancellation flow

Scope and limitations

Dependency graph

Validation

Summary by CodeRabbit

Uh oh!

chienchunhung commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

chienchunhung commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

chienchunhung commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

chienchunhung commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

chienchunhung commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

chienchunhung commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

chienchunhung commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

chienchunhung commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

chienchunhung commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

chienchunhung commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

chienchunhung commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

chienchunhung commented Jul 8, 2026

Uh oh!

chienchunhung commented Jun 11, 2026 •

edited

Loading