feat: AutoFlipController + heal path + bvar metrics for CP<->DP switching.#1969
Open
cchh05 wants to merge 2 commits into
Open
feat: AutoFlipController + heal path + bvar metrics for CP<->DP switching.#1969cchh05 wants to merge 2 commits into
cchh05 wants to merge 2 commits into
Conversation
cchh05
requested review from
Clement-Wang26,
DongheJin,
DragonFive,
JimHsiung,
Kang-Meng,
RobbieLeung,
XuZhang99,
liujinguang0125,
liutongxuan,
walsonyang,
xiao-yu-chen,
yingxudeng,
yq33victor and
zhang-minchao
as code owners
July 17, 2026 08:12
cchh05
force-pushed
the
pr2-cp-dp-autoflip-heal-on-pr1
branch
2 times, most recently
from
July 17, 2026 10:37
7b9cb24 to
a22f815
Compare
16 tasks
Add support for flipping worker layout between CP_PREFILL (cp=N, dp=1) and DP_DECODE (cp=1, dp=N) on a live instance in ~500 ms, without a service restart. Traffic patterns rarely stay in one regime forever; this lets a single deployment adapt across the day rather than being statically sized for either lane. Design ------ * Startup builds BOTH ATB graphs (dual-mode) when `--enable_runtime_cp_dp_switch=true`, so a flip is a graph handle swap, not a rebuild. * Flip flow: scheduler drains in-flight requests -> pause workers -> swap the active `ParallelArgs` (cp/dp) and block-manager pool atomically -> resume. The block-manager pool is guarded by a shared-ptr for lockless reads on the hot path. * Worker-side call sites read the live `parallel_args_.cp_size()` (not the immutable `options_.cp_size()`), so already-scheduled requests observe the new layout after resume. New surfaces ------------ * `mode_switch.proto` + `ModeSwitchService`: RPC surface for flipping, exposed via the xLLM server. * `AutoFlipController` scaffolding for autoscaling callers. * `docs/en/features/runtime_cp_dp_switch.md`: user-facing design doc. Testing ------- * Unit: `dual_parallel_args_test`, `atomic_pool_handle_test`, `lopsided_defer_gate_test`. * E2E on 8*A3: `verify_switch` regression, 18/18 flips stable across 61 model layers; lm_eval CP == DP within tolerance; 10-min soak on 32-proc load did not regress HBM watermark. Related issue: xLLM-AI#1967
…hing.
Follow-on to the runtime CP<->DP mode switching feature. Adds:
1. AutoFlipController — a scheduler-facing controller that watches
traffic stats and issues SwitchMode RPCs, so an instance can flip
layout autonomously instead of relying on an external caller.
2. Heal path — after a switch, if a small tail of in-flight requests
fails to converge (e.g. long-prefill sequences straddling the flip),
a bounded heal window (default 4 rounds) drains them safely without
holding the whole cluster.
3. CP -> DP pending gate — refuses a CP -> DP flip while
the CP side still has queued long-prefill sequences that would be
silently split apart by the DP layout.
4. 7 bvar metrics on `/vars`:
xllm_mode_flip_total
xllm_mode_flip_success_total
xllm_mode_flip_target_cp_total
xllm_mode_flip_target_dp_total
xllm_mode_flip_latency_ms
xllm_mode_flip_heal_rounds
xllm_mode_flip_pending_gate_reject_total
Testing
-------
* Unit: `auto_flip_decide_target_test` (278 lines, covers stat window,
minimum flip interval, target selection, hysteresis).
* E2E: heal path exercised by the same `verify_switch` regression,
which was extended to inject a long prefill straddling the flip;
842 ms average tail drain, 18/18 flips still convergent.
* Soak: 10 min @ 32 procs with periodic auto-flips, HBM stable.
Depends on xLLM-AI#1968 (feat: runtime CP<->DP mode switching).
cchh05
force-pushed
the
pr2-cp-dp-autoflip-heal-on-pr1
branch
from
July 17, 2026 13:23
a22f815 to
c4b45a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-on to #1968. Adds autonomous flipping and post-flip drain safety on top of the runtime CP<->DP switching feature.
AutoFlipController— scheduler-side controller that watches traffic stats and issuesSwitchModeRPCs autonomously. Config knobs: minimum inter-flip interval, stat window, hysteresis.Heal path — after a flip, if a small tail of in-flight requests fails to converge (long-prefill sequences straddling the flip), a bounded heal window (default 4 rounds) drains them safely instead of holding the whole cluster.
CP -> DP pending gate — refuses a
CP -> DPflip while the CP side still has queued long-prefill sequences that would be silently split apart by the DP layout.7 bvar metrics on
/vars:xllm_mode_flip_totalxllm_mode_flip_success_totalxllm_mode_flip_target_cp_totalxllm_mode_flip_target_dp_totalxllm_mode_flip_latency_msxllm_mode_flip_heal_roundsxllm_mode_flip_pending_gate_reject_totalFully gated by the same
--enable_runtime_cp_dp_switchflag introduced in #1968.Related Issues
Part of #1967. Depends on #1968 — please merge that first.
Change Type
Pull Request Checklist
PR Title and Commit Messages
<type>: <subject>.Pre-commit Checks
clang-formatwas run against all touched C++ files (verified via--dry-run --Werror, 0 violations).pre-commit run --all-files— not run locally due to environment gap on my dev machine; happy to run whatever the CI does and iterate on any reported issue.Self Review
.agents/skills/code-review/references/custom-code-style.md.52eca61a, immediate pre-PR#1938 base). Same rationale as feat: runtime CP<->DP mode switching for xLLM workers. #1968 for not rebasing ontomainHEAD.Build and Test Coverage
auto_flip_decide_target_test(278 lines, covers stat window, min flip interval, target selection, hysteresis).python setup.py build test: not run — feature is NPU-only in this PR. Behavior on CUDA is unchanged (feature gated off).verify_switchextended with a long-prefill injection straddling the flip; 842 ms average tail drain, 18/18 flips still convergent. 10-min soak @ 32 procs with periodic auto-flips, HBM watermark stable.Reviewer Notes
pr1-cp-dp-switch-on-52eca61a). Merging order should be: feat: runtime CP<->DP mode switching for xLLM workers. #1968 first, then this one. If maintainers prefer a single combined PR I can squash the two branches, but I split them so feat: runtime CP<->DP mode switching for xLLM workers. #1968 stays reviewable.auto_flip_stats.his the pure-data stat struct;AutoFlipControllerowns the decision logic. The split lets the unit test exercise decision logic without spinning workers.pending_gate_reject_totalmetric surfaces bad configurations quickly — if it ticks in production, either flip cadence is too aggressive or the CP queue is too deep for the current DP fanout.verify_switchruns. Happy to make this a gflag if you'd like.