feat: add native vLLM gRPC sidecar backend - #11804
Conversation
GitHub Actions
Inactive
GitHub Actions
Inactive
github-actions
Bot
added
external-contribution
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| os.environ[VLLM_ENABLE_INFERENCE_V1_GENERATE_ENV] = ( | ||
| "1" if config.enable_engine_apis else "0" | ||
| ) |
There was a problem hiding this comment.
🟡 Frontend writes a parsed setting back into an environment variable instead of passing it explicitly
The engine-APIs setting is written back into a process environment variable (os.environ[VLLM_ENABLE_INFERENCE_V1_GENERATE_ENV] = ... at components/src/dynamo/frontend/main.py:183-185) after being parsed into the frontend config, so the configuration flows in a cycle from Python out to the environment and back into the Rust layer rather than being passed directly.
Impact: This violates the frontend's stated configuration-boundary rule and reintroduces the exact drift risk (a re-read environment value diverging from the parsed setting) the rule exists to prevent.
Boundary rule and consumer path
components/src/dynamo/frontend/CLAUDE.md:7-9 explicitly states: "Do not write parsed FrontendConfig values back to os.environ for Rust to re-read. That creates a cyclic Python -> env -> Rust contract where CLI overrides can diverge from Rust behavior." It further requires (CLAUDE.md:10-11) passing such values "through explicit PyO3 binding parameters or config structs."
Here config.enable_engine_apis is parsed from CLI/env in frontend_args.py (env var DYN_VLLM_ENABLE_INFERENCE_V1_GENERATE), then written back into os.environ under that same name. The Rust builder re-reads it via env_is_truthy(VLLM_ENABLE_INFERENCE_V1_GENERATE_ENV) at lib/llm/src/http/service/service_v2.rs:897-898. The PyO3 HttpService binding (lib/bindings/python/rust/http.rs:27-36) exposes no enable_engine_apis parameter, so the required explicit transport path does not exist and the env write-back is the only mechanism used — the pattern the rule prohibits.
Prompt for agents
The frontend must not write the parsed config.enable_engine_apis value back to os.environ for the Rust HTTP service to re-read, per components/src/dynamo/frontend/CLAUDE.md. Instead, add an explicit transport path: expose an enable_engine_apis parameter on the PyO3 HttpService binding (lib/bindings/python/rust/http.rs) that forwards to the HttpServiceConfigBuilder.enable_engine_apis(...) method (lib/llm/src/http/service/service_v2.rs), then pass config.enable_engine_apis through that explicit binding when the frontend constructs the HTTP service, and remove the os.environ write in main.py:183-185.
Was this helpful? React with 👍 or 👎 to provide feedback.
| sampling_params.extra_args = {} | ||
| # `do_remote_decode` is prefill's to own; merge caller-supplied | ||
| # values on top of the defaults so explicit overrides win. | ||
| kv_defaults = { | ||
| framework_kv = { | ||
| "do_remote_prefill": False, | ||
| "remote_engine_id": None, | ||
| "remote_block_ids": None, | ||
| "remote_host": None, | ||
| "remote_port": None, | ||
| } | ||
| caller_kv = sampling_params.extra_args.get("kv_transfer_params", {}) | ||
| sampling_params.extra_args["kv_transfer_params"] = { | ||
| **kv_defaults, | ||
| **caller_kv, | ||
| "do_remote_decode": True, | ||
| } | ||
| caller_kv = sampling_params.extra_args.get("kv_transfer_params") | ||
| sampling_params.extra_args["kv_transfer_params"] = merge_kv_transfer_params( | ||
| caller_kv, framework_kv | ||
| ) |
There was a problem hiding this comment.
🔍 Prefill kv_transfer_params merge became strict (collision now raises) versus prior caller-wins merge
In components/src/dynamo/vllm/llm_engine.py:468-480 the prefill dispatch switched from a caller-wins overlay ({**kv_defaults, **caller_kv, "do_remote_decode": True}) to merge_kv_transfer_params(caller_kv, framework_kv). merge_kv_transfer_params (engine_generate.py:92-107) raises ValueError on ANY key overlap between caller and framework. Previously, a caller-supplied do_remote_prefill/remote_engine_id/etc. silently overrode the framework default; now the same input fails the request. The same strict-merge semantics were applied to the decode path and to handlers.py (3127-3131, 3453-3456). This appears intentional (disjoint-namespace contract), and existing internal callers pass an empty/absent kv_transfer_params, but any external caller that historically set one of these keys will now hit a hard error instead of an override. Worth confirming no in-tree caller relies on the old override behavior.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
WalkthroughAdds a Rust vLLM gRPC sidecar, native vLLM generate support, LoRA and lifecycle controls, KV-routing metadata, frontend configuration, deployment examples, and parity/metrics validation tooling. ChangesvLLM serving and native generation
Shared runtime and validation
Estimated code review effort: 5 (Critical) | ~120 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
components/src/dynamo/vllm/llm_engine.py (1)
630-671: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winComplete the native P/D prompt-logprob handoff.
Prompt logprobs must be computed by prefill, transferred to decode, and never recomputed on decode.
components/src/dynamo/vllm/llm_engine.py#L630-L671: includeprompt_logprobs_payloadin prefilldisaggregated_params.components/src/dynamo/vllm/llm_engine.py#L483-L505: read the transferred payload and clearsampling_params.prompt_logprobs.components/src/dynamo/vllm/llm_engine.py#L553-L578: use the transferred payload on decode rather than collecting a new one.components/src/dynamo/vllm/handlers.py#L453-L472: clear decode prompt-logprob computation even when the transferred payload is absent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/vllm/llm_engine.py` around lines 630 - 671, Complete the native P/D prompt-logprob handoff across components/src/dynamo/vllm/llm_engine.py lines 630-671, 483-505, and 553-578, plus components/src/dynamo/vllm/handlers.py lines 453-472: add prompt_logprobs_payload to prefill disaggregated_params, read it on decode and clear sampling_params.prompt_logprobs, reuse the transferred payload instead of collecting decode prompt logprobs, and ensure handlers clear decode prompt-logprob computation even when no payload is transferred.lib/backend-common/src/worker.rs (1)
866-881: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winUnregister the model when LoRA-controller initialization fails.
LoraController::new(...).await?runs afterlocal_model.attach(). If listing or reconciling existing adapters fails, execution returns beforeorchestrator_steps(), while the outer safety net only cleans up the engine. This leaves a stale discovery entry routing traffic to a failed worker.Wrap post-attach initialization with rollback that calls
endpoint.unregister_endpoint_instance()before returning the error.Also applies to: 898-906
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/backend-common/src/worker.rs` around lines 866 - 881, Wrap the post-attach initialization, including LoraController::new and the related setup before orchestrator_steps(), with rollback handling. If any initialization step fails, call endpoint.unregister_endpoint_instance() before propagating the original error; preserve the existing successful path and ensure cleanup also covers the additional referenced range.lib/llm/src/kv_router/publisher/zmq_listener.rs (1)
18-22: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winPropagate the revised
start_zmq_listenercontract consistently. The new parameter leaves two test call paths uncompilable.
lib/llm/src/kv_router/publisher/zmq_listener.rs#L18-L22: update the remaining call inlib/llm/src/kv_router/publisher/tests.rsat Line 1191 by insertingNoneafterworker_id.lib/llm/src/kv_router/publisher/tests.rs#L1345-L1427: useUnboundedSender<Vec<PlacementEvent>>, then extract the received event from the batch.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/llm/src/kv_router/publisher/zmq_listener.rs` around lines 18 - 22, The revised start_zmq_listener contract requires updating both test paths: in lib/llm/src/kv_router/publisher/tests.rs at line 1191, pass None immediately after worker_id; in the test covering lines 1345-1427, change the channel to UnboundedSender<Vec<PlacementEvent>> and extract the expected event from the received batch before asserting it.
🟡 Minor comments (9)
lib/vllm-sidecar/src/client.rs-48-91 (1)
48-91: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winFail fast on malformed endpoints.
connect_channelretries deterministicEndpoint::from_sharedfailures untilcfg.deadline. Even afternormalize_endpoint(), malformed inputs can still reach this path and stall bootstrap for the full timeout. Parse once before the retry loop and returninvalid_argimmediately.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/vllm-sidecar/src/client.rs` around lines 48 - 91, Update connect_channel to validate the endpoint once before entering its retry loop, using Endpoint::from_shared and returning invalid_arg immediately for malformed URIs. Refactor try_connect_once to reuse the validated endpoint or otherwise avoid reparsing it on each attempt, while preserving retries for transient connection failures.lib/vllm-sidecar/src/discovery.rs-141-144 (1)
141-144: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winPreserve
0as a valid data-parallel start rank.data_parallel_start_rankcan be0, so this check drops the first DP group’s metadata. Gate it ondata_parallel_size != 0instead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/vllm-sidecar/src/discovery.rs` around lines 141 - 144, Update the data_parallel_start_rank assignment in the discovery metadata construction to gate on parallelism.data_parallel_size != 0, rather than checking data_parallel_start_rank != 0, so rank 0 remains preserved whenever data parallelism is configured.components/src/dynamo/vllm/CLAUDE.md-33-37 (1)
33-37: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winCorrect the entrypoint guidance.
This PR changes
handlers.pyandmain.pyfor native generate, so saying sidecar work never touches Python and that this path “stays untouched” is already false. Describe ownership without prohibiting necessary cross-layer changes.Also applies to: 65-71
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/vllm/CLAUDE.md` around lines 33 - 37, Update the sidecar ownership guidance in CLAUDE.md to remove the absolute claim that sidecar work never touches Python or that the native path must remain untouched. State that the sidecar is separate from the in-process VllmLLMEngine path while allowing necessary coordinated changes across Rust and Python components.components/src/dynamo/vllm/engine_generate.py-29-41 (1)
29-41: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDo not silently discard routed-expert serialization failures.
except Exceptionconverts serialization, allocation, and programming failures into missing client metadata. Remove the catch, or catch only expected conversion failures and re-raise unexpected errors.Proposed fail-fast change
def serialize_routed_experts(routed_experts: Any) -> str | None: """Encode routed experts using vLLM's base64-of-NumPy wire format.""" if routed_experts is None: return None - try: - buffer = io.BytesIO() - np.save(buffer, routed_experts) - return base64.b64encode(buffer.getvalue()).decode("ascii") - except Exception: - logger.warning( - "Unable to encode routed_experts for generate API", exc_info=True - ) - return None + buffer = io.BytesIO() + np.save(buffer, routed_experts) + return base64.b64encode(buffer.getvalue()).decode("ascii")As per coding guidelines and path instructions, broad
Exceptionhandlers must not hide failures.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/vllm/engine_generate.py` around lines 29 - 41, Update serialize_routed_experts so serialization failures are not converted into a None result: remove the broad Exception handler, or restrict handling to explicitly expected conversion failures and re-raise all unexpected errors. Preserve the existing None input behavior and successful base64 encoding path.Sources: Coding guidelines, Path instructions, Linters/SAST tools
components/src/dynamo/vllm/tests/test_vllm_delta_streaming.py-474-476 (1)
474-476: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert the preserved prompt-logprob payload, not only the key.
The final response supplies
prompt_logprobs=[], so this passes even if the earlier non-empty payload is overwritten or discarded. Compare the finalengine_data["prompt_logprobs"]with the expected serialized values.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/vllm/tests/test_vllm_delta_streaming.py` around lines 474 - 476, Update the assertions in the vLLM delta streaming test to validate the contents of the final engine_data["prompt_logprobs"] payload, comparing it with the expected serialized non-empty values rather than only checking key presence. Preserve the existing chunk and log_probs assertions.examples/backends/vllm/launch/sidecar_agg.sh-25-26 (1)
25-26: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winHonor the standard
MODELenvironment override consistently.
examples/backends/vllm/launch/sidecar_agg.sh#L25-L26: initialize with${MODEL:-Qwen/Qwen3-0.6B}.examples/backends/vllm/launch/sidecar_agg_kvbm.sh#L26-L27: initialize with${MODEL:-Qwen/Qwen3-0.6B}.examples/backends/vllm/launch/sidecar_agg_multimodal.sh#L28-L33: prioritizeMODEL, retainingDYN_MODEL_NAMEas a compatibility fallback if required.As per coding guidelines, launch scripts must expose
MODELas an environment variable with a sensible default.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/backends/vllm/launch/sidecar_agg.sh` around lines 25 - 26, Update model initialization in examples/backends/vllm/launch/sidecar_agg.sh (lines 25-26) and examples/backends/vllm/launch/sidecar_agg_kvbm.sh (lines 26-27) to use MODEL when set, defaulting to Qwen/Qwen3-0.6B. In examples/backends/vllm/launch/sidecar_agg_multimodal.sh (lines 28-33), prioritize MODEL while retaining DYN_MODEL_NAME as the compatibility fallback if needed.Source: Coding guidelines
tests/serve/tito_parity/run_pd_three_way.py-522-522 (1)
522-522: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winReplace runtime assertions with explicit failures.
Under
python -O, these checks disappear and nullable setup state can flow into request execution.Proposed fix
- assert groups is not None + if groups is None: + raise RuntimeError("request groups were not initialized") ... - assert groups is not None - assert requests_by_key is not None + if groups is None or requests_by_key is None: + raise RuntimeError("upstream P/D setup did not initialize request state")As per coding guidelines, never use
assertfor runtime validation.Also applies to: 533-534
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/serve/tito_parity/run_pd_three_way.py` at line 522, Replace the runtime assert checks around groups and the related setup state with explicit validation that raises an appropriate failure when the values are None. Update all referenced checks, including the checks at the later indicated lines, so validation remains active under python -O before request execution.Sources: Coding guidelines, Path instructions
tests/serve/tito_parity/verify_generate_metrics.py-355-375 (1)
355-375: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winVerify that P/D worker metrics were populated, not merely registered.
These checks pass when matching series already exist with zero or stale values. Compare against
before, or require positive, finite values for the selected prefill/decode workers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/serve/tito_parity/verify_generate_metrics.py` around lines 355 - 375, Update the disaggregated validation in the metrics verification flow around worker_ttft, worker_input, and worker_itl so it confirms the selected prefill/decode worker metrics were populated after the test, not merely present. Compare the matching values from before and after, or validate that the after values are positive and finite, while preserving the existing missing-series assertion context.tests/serve/tito_parity/README.md-3-5 (1)
3-5: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAlign the vLLM version in
tests/serve/tito_parity/README.md:3The repo pinsvllm==0.24.0inpyproject.toml, but this README still says0.23.0. Update the version callout or remove it if the runner isn’t meant to be version-specific.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/serve/tito_parity/README.md` around lines 3 - 5, Update the upstream vLLM version callout in the parity runner README from 0.23.0 to the repository-pinned 0.24.0, keeping the description otherwise unchanged.
🧹 Nitpick comments (2)
examples/backends/vllm/launch/sidecar_disagg.sh (1)
25-25: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake the model configurable.
This script hardcodes
MODEL, unlike the multimodal variant. Provide an environment override so users need not edit the launch script.Proposed fix
-MODEL="Qwen/Qwen3-0.6B" +MODEL="${DYN_MODEL_NAME:-Qwen/Qwen3-0.6B}"As per coding guidelines, “Expose tunable parameters (MODEL, ...) as environment variables with sensible defaults.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/backends/vllm/launch/sidecar_disagg.sh` at line 25, Update the MODEL assignment in the sidecar launch script to read from an environment variable while retaining Qwen/Qwen3-0.6B as the default when unset, matching the configurability pattern used by the multimodal variant.Source: Coding guidelines
tests/serve/tito_parity/run_parity.py (1)
625-664: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winGate
READY_FILEon router readiness
tests/serve/tito_parity/launch_dynamo_disagg.sh: replace the fixedsleep 5with an explicit check that the frontend has switched to the active PrefillRouter before touchingREADY_FILE.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/serve/tito_parity/run_parity.py` around lines 625 - 664, The requested change applies to tests/serve/tito_parity/launch_dynamo_disagg.sh lines 64-66: replace the fixed sleep before touching READY_FILE with an explicit readiness check that waits until the frontend has switched to the active PrefillRouter, then create or update READY_FILE. No direct change is required in tests/serve/tito_parity/run_parity.py lines 625-664; its run_server readiness flow remains unchanged.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/src/dynamo/vllm/engine_generate.py`:
- Around line 44-61: Update _native_generate_api to import decode_mm_kwargs_item
and GenerateRequest directly from the vLLM 0.24.0 vllm.entrypoints.serve.disagg
modules at module scope, remove the lazy-loading wrapper and try/except
compatibility fallback, and retain the existing symbols for callers.
In `@components/src/dynamo/vllm/main.py`:
- Around line 75-78: Update should_publish_generate_capability to return true
only for WorkerType.Aggregated or WorkerType.Decode and non-Embedding models,
excluding WorkerType.Encode explicitly. Extend the related test matrix with a
ModelType.Chat and WorkerType.Encode case asserting the capability is not
published.
In `@components/src/dynamo/vllm/tests/test_vllm_engine_generate.py`:
- Around line 9-10: Guard the module-level vLLM import in
test_vllm_engine_generate.py with pytest.importorskip("vllm.sampling_params") so
collection is skipped when vLLM is unavailable, while preserving access to
RequestOutputKind and SamplingParams when the dependency exists.
In `@examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml`:
- Around line 51-53: Remove the workingDir setting from the mainContainer in
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml#L51-L53 and
examples/backends/vllm/deploy/sidecar_agg.yaml#L56-L58, and from both decode and
prefill sidecars in examples/backends/vllm/deploy/sidecar_disagg.yaml#L57-L59
and `#L94-L96`. Leave the existing absolute sidecar commands and paths unchanged.
- Around line 63-67: The v1alpha1 sidecar manifests must atomically replace
generated worker arguments when overriding the command. Update the args override
contract at examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml:63-67,
examples/backends/vllm/deploy/sidecar_agg.yaml:59-63, and both decode and
prefill sites in examples/backends/vllm/deploy/sidecar_disagg.yaml:60-64 and
:97-101 so only the explicitly defined sidecar arguments are passed.
In `@examples/backends/vllm/launch/lora/sidecar_agg_lora.sh`:
- Around line 8-10: Source gpu_utils.sh and thread GPU_MEM_ARGS into the vllm-rs
serve launch flow in examples/backends/vllm/launch/lora/sidecar_agg_lora.sh
(lines 8-10), examples/backends/vllm/launch/sidecar_agg.sh (lines 22-23),
examples/backends/vllm/launch/sidecar_agg_kvbm.sh (lines 23-24), and
examples/backends/vllm/launch/sidecar_agg_multimodal.sh (lines 25-26), ensuring
_PROFILE_OVERRIDE_VLLM_KV_CACHE_BYTES reaches each sidecar launcher.
In `@lib/bindings/python/rust/backend.rs`:
- Around line 820-821: Preserve LoRA metadata across both conversion paths in
lib/bindings/python/rust/backend.rs:820-821 and
lib/bindings/python/rust/backend.rs:157-158. Update the duck-typed .llm
extraction to read supports_lora and max_loras with backward-compatible defaults
instead of forcing false/None, and extend the Python LlmRegistration constructor
and getters to store and expose both fields.
In `@lib/vllm-sidecar/src/tests/lifecycle.rs`:
- Around line 188-200: Update the TransportConfig used to construct
VllmSidecarEngine so it sets the RPC deadline to the required short duration
instead of only setting connect_timeout. Keep the existing test_transport
configuration and ensure list_loras() observes that RPC deadline while the
channel is already connected.
In `@lib/vllm-sidecar/src/wire.rs`:
- Around line 67-77: Update the terminal-event handling around the is_prefill
branch so GenerateEvent::PrefillReady is emitted only for explicitly successful
prefill reasons. When the terminal reason is Aborted, propagate
cancellation/error instead of performing the KV-parameter conversion or handoff,
while preserving the existing validation for successful responses.
- Around line 108-114: Update json_to_prost_value and its corresponding
number_to_json conversion to preserve KV-transfer numeric values without lossy
f64 conversion. Reject numbers that cannot be represented exactly rather than
rounding or saturating them, or consistently encode 64-bit identifiers as
strings across the contract; ensure large integers do not become incorrect
values such as i64::MAX.
In `@tests/serve/tito_parity/launch_dynamo_disagg.sh`:
- Around line 5-19: Replace raw process-group cleanup with the shared helpers:
in tests/serve/tito_parity/launch_dynamo_disagg.sh lines 5-19, validate
arguments and source launch_utils.sh before registering cleanup, then capture
the exit status in a cleanup function that removes READY_FILE and invokes
dynamo_reap_and_exit; in examples/backends/vllm/launch/sidecar_disagg.sh line 36
and examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh line 42,
replace the raw EXIT traps with trap dynamo_exit_trap EXIT.
In `@tests/serve/tito_parity/run_parity.py`:
- Around line 29-35: The parity runners currently use fixed ports, preventing
concurrent execution. In tests/serve/tito_parity/run_parity.py (lines 29-35),
use allocate_port()/allocate_ports() for upstream and Dynamo ports and propagate
them through commands and environment; in
tests/serve/tito_parity/run_pd_three_way.py (lines 19-20), allocate prefill and
decode ports instead of 8100/8200. Update
tests/serve/tito_parity/launch_dynamo_disagg.sh (lines 36-62) to accept injected
system, NIXL, event, and health ports, and update
examples/backends/vllm/launch/sidecar_disagg.sh (lines 68-73) plus
examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh (lines 95-100) to
read NIXL side-channel and KV-event ports from environment variables.
In `@tests/serve/tito_parity/verify_generate_metrics.py`:
- Around line 193-228: Update the request orchestration around the
ThreadPoolExecutor and metrics polling to saturate worker capacity, then submit
an additional Generate request before asserting queued_requests. Ensure the
extra request remains pending long enough to observe
dynamo_frontend_queued_requests above baseline, while preserving the existing
active and inflight observations and cleaning up all submitted futures.
---
Outside diff comments:
In `@components/src/dynamo/vllm/llm_engine.py`:
- Around line 630-671: Complete the native P/D prompt-logprob handoff across
components/src/dynamo/vllm/llm_engine.py lines 630-671, 483-505, and 553-578,
plus components/src/dynamo/vllm/handlers.py lines 453-472: add
prompt_logprobs_payload to prefill disaggregated_params, read it on decode and
clear sampling_params.prompt_logprobs, reuse the transferred payload instead of
collecting decode prompt logprobs, and ensure handlers clear decode
prompt-logprob computation even when no payload is transferred.
In `@lib/backend-common/src/worker.rs`:
- Around line 866-881: Wrap the post-attach initialization, including
LoraController::new and the related setup before orchestrator_steps(), with
rollback handling. If any initialization step fails, call
endpoint.unregister_endpoint_instance() before propagating the original error;
preserve the existing successful path and ensure cleanup also covers the
additional referenced range.
In `@lib/llm/src/kv_router/publisher/zmq_listener.rs`:
- Around line 18-22: The revised start_zmq_listener contract requires updating
both test paths: in lib/llm/src/kv_router/publisher/tests.rs at line 1191, pass
None immediately after worker_id; in the test covering lines 1345-1427, change
the channel to UnboundedSender<Vec<PlacementEvent>> and extract the expected
event from the received batch before asserting it.
---
Minor comments:
In `@components/src/dynamo/vllm/CLAUDE.md`:
- Around line 33-37: Update the sidecar ownership guidance in CLAUDE.md to
remove the absolute claim that sidecar work never touches Python or that the
native path must remain untouched. State that the sidecar is separate from the
in-process VllmLLMEngine path while allowing necessary coordinated changes
across Rust and Python components.
In `@components/src/dynamo/vllm/engine_generate.py`:
- Around line 29-41: Update serialize_routed_experts so serialization failures
are not converted into a None result: remove the broad Exception handler, or
restrict handling to explicitly expected conversion failures and re-raise all
unexpected errors. Preserve the existing None input behavior and successful
base64 encoding path.
In `@components/src/dynamo/vllm/tests/test_vllm_delta_streaming.py`:
- Around line 474-476: Update the assertions in the vLLM delta streaming test to
validate the contents of the final engine_data["prompt_logprobs"] payload,
comparing it with the expected serialized non-empty values rather than only
checking key presence. Preserve the existing chunk and log_probs assertions.
In `@examples/backends/vllm/launch/sidecar_agg.sh`:
- Around line 25-26: Update model initialization in
examples/backends/vllm/launch/sidecar_agg.sh (lines 25-26) and
examples/backends/vllm/launch/sidecar_agg_kvbm.sh (lines 26-27) to use MODEL
when set, defaulting to Qwen/Qwen3-0.6B. In
examples/backends/vllm/launch/sidecar_agg_multimodal.sh (lines 28-33),
prioritize MODEL while retaining DYN_MODEL_NAME as the compatibility fallback if
needed.
In `@lib/vllm-sidecar/src/client.rs`:
- Around line 48-91: Update connect_channel to validate the endpoint once before
entering its retry loop, using Endpoint::from_shared and returning invalid_arg
immediately for malformed URIs. Refactor try_connect_once to reuse the validated
endpoint or otherwise avoid reparsing it on each attempt, while preserving
retries for transient connection failures.
In `@lib/vllm-sidecar/src/discovery.rs`:
- Around line 141-144: Update the data_parallel_start_rank assignment in the
discovery metadata construction to gate on parallelism.data_parallel_size != 0,
rather than checking data_parallel_start_rank != 0, so rank 0 remains preserved
whenever data parallelism is configured.
In `@tests/serve/tito_parity/README.md`:
- Around line 3-5: Update the upstream vLLM version callout in the parity runner
README from 0.23.0 to the repository-pinned 0.24.0, keeping the description
otherwise unchanged.
In `@tests/serve/tito_parity/run_pd_three_way.py`:
- Line 522: Replace the runtime assert checks around groups and the related
setup state with explicit validation that raises an appropriate failure when the
values are None. Update all referenced checks, including the checks at the later
indicated lines, so validation remains active under python -O before request
execution.
In `@tests/serve/tito_parity/verify_generate_metrics.py`:
- Around line 355-375: Update the disaggregated validation in the metrics
verification flow around worker_ttft, worker_input, and worker_itl so it
confirms the selected prefill/decode worker metrics were populated after the
test, not merely present. Compare the matching values from before and after, or
validate that the after values are positive and finite, while preserving the
existing missing-series assertion context.
---
Nitpick comments:
In `@examples/backends/vllm/launch/sidecar_disagg.sh`:
- Line 25: Update the MODEL assignment in the sidecar launch script to read from
an environment variable while retaining Qwen/Qwen3-0.6B as the default when
unset, matching the configurability pattern used by the multimodal variant.
In `@tests/serve/tito_parity/run_parity.py`:
- Around line 625-664: The requested change applies to
tests/serve/tito_parity/launch_dynamo_disagg.sh lines 64-66: replace the fixed
sleep before touching READY_FILE with an explicit readiness check that waits
until the frontend has switched to the active PrefillRouter, then create or
update READY_FILE. No direct change is required in
tests/serve/tito_parity/run_parity.py lines 625-664; its run_server readiness
flow remains unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bbfc4fbb-f970-4172-92b9-16881b15f884
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (81)
Cargo.tomlcomponents/src/dynamo/common/backend/publisher.pycomponents/src/dynamo/common/backend/tests/test_publisher.pycomponents/src/dynamo/frontend/frontend_args.pycomponents/src/dynamo/frontend/main.pycomponents/src/dynamo/frontend/tests/test_frontend_args.pycomponents/src/dynamo/vllm/CLAUDE.mdcomponents/src/dynamo/vllm/args.pycomponents/src/dynamo/vllm/backend_args.pycomponents/src/dynamo/vllm/constants.pycomponents/src/dynamo/vllm/engine_generate.pycomponents/src/dynamo/vllm/handlers.pycomponents/src/dynamo/vllm/kv_event_utils.pycomponents/src/dynamo/vllm/llm_engine.pycomponents/src/dynamo/vllm/main.pycomponents/src/dynamo/vllm/tests/test_runtime_metadata.pycomponents/src/dynamo/vllm/tests/test_vllm_delta_streaming.pycomponents/src/dynamo/vllm/tests/test_vllm_engine_generate.pycomponents/src/dynamo/vllm/tests/test_vllm_unit.pyexamples/backends/vllm/deploy/README.mdexamples/backends/vllm/deploy/lora/sidecar_agg_lora.yamlexamples/backends/vllm/deploy/sidecar_agg.yamlexamples/backends/vllm/deploy/sidecar_disagg.yamlexamples/backends/vllm/launch/lora/README.mdexamples/backends/vllm/launch/lora/sidecar_agg_lora.shexamples/backends/vllm/launch/sidecar_agg.shexamples/backends/vllm/launch/sidecar_agg_kvbm.shexamples/backends/vllm/launch/sidecar_agg_multimodal.shexamples/backends/vllm/launch/sidecar_disagg.shexamples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.shlib/backend-common/Cargo.tomllib/backend-common/examples/mocker/src/engine.rslib/backend-common/src/drain.rslib/backend-common/src/engine.rslib/backend-common/src/lib.rslib/backend-common/src/lora.rslib/backend-common/src/publisher.rslib/backend-common/src/worker.rslib/bindings/python/rust/backend.rslib/bindings/python/rust/llm/kv.rslib/kv-router/src/zmq_wire/deserialize.rslib/kv-router/src/zmq_wire/extra_keys.rslib/kv-router/src/zmq_wire/mod.rslib/llm/src/discovery/model.rslib/llm/src/discovery/model_manager.rslib/llm/src/http/service/generate.rslib/llm/src/kv_router/publisher/mod.rslib/llm/src/kv_router/publisher/tests.rslib/llm/src/kv_router/publisher/zmq_listener.rslib/llm/src/local_model.rslib/llm/src/mocker.rslib/llm/src/protocols/openai/generate.rslib/runtime/src/slug.rslib/runtime/src/storage/kv/file.rslib/sglang-sidecar/src/engine.rslib/vllm-rs-backend/src/backend.rslib/vllm-sidecar/Cargo.tomllib/vllm-sidecar/build.rslib/vllm-sidecar/proto/vllm_grpc.protolib/vllm-sidecar/src/args.rslib/vllm-sidecar/src/client.rslib/vllm-sidecar/src/discovery.rslib/vllm-sidecar/src/engine.rslib/vllm-sidecar/src/lib.rslib/vllm-sidecar/src/main.rslib/vllm-sidecar/src/proto.rslib/vllm-sidecar/src/request.rslib/vllm-sidecar/src/tests.rslib/vllm-sidecar/src/tests/bootstrap.rslib/vllm-sidecar/src/tests/conversion.rslib/vllm-sidecar/src/tests/disaggregation.rslib/vllm-sidecar/src/tests/generation.rslib/vllm-sidecar/src/tests/lifecycle.rslib/vllm-sidecar/src/wire.rstests/report_pytest_markers.pytests/serve/tito_parity/README.mdtests/serve/tito_parity/launch_dynamo_disagg.shtests/serve/tito_parity/run_parity.pytests/serve/tito_parity/run_pd_three_way.pytests/serve/tito_parity/run_seeded_rl_parity.pytests/serve/tito_parity/verify_generate_metrics.py
| @lru_cache(maxsize=1) | ||
| def _native_generate_api() -> tuple[Any, Any]: | ||
| """Load the vLLM-native endpoint adapter only when `/generate` is used.""" | ||
| try: | ||
| from vllm.entrypoints.scale_out.token_in_token_out.mm_serde import ( | ||
| decode_mm_kwargs_item, | ||
| ) | ||
| from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( | ||
| GenerateRequest, | ||
| ) | ||
| except ModuleNotFoundError as exc: | ||
| expected_module = "vllm.entrypoints.scale_out.token_in_token_out" | ||
| if exc.name is None or not expected_module.startswith(exc.name): | ||
| raise | ||
| from vllm.entrypoints.serve.disagg.mm_serde import decode_mm_kwargs_item | ||
| from vllm.entrypoints.serve.disagg.protocol import GenerateRequest | ||
|
|
||
| return decode_mm_kwargs_item, GenerateRequest |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify the repository's declared vLLM version and all endpoint-module imports.
fd -H -t f '^(pyproject\.toml|uv\.lock|requirements.*\.txt)$' . -0 |
xargs -0 -r rg -n -C2 '\bvllm\b'
rg -n -C3 \
'entrypoints\.(scale_out\.token_in_token_out|serve\.disagg)' \
components/src/dynamoRepository: ai-dynamo/dynamo
Length of output: 6724
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the file and nearby call sites to understand whether the lazy imports are required.
ast-grep outline components/src/dynamo/vllm/engine_generate.py --view expanded || true
echo '--- engine_generate.py (relevant slice) ---'
sed -n '1,220p' components/src/dynamo/vllm/engine_generate.py
echo '--- search for _native_generate_api usages ---'
rg -n -C3 '_native_generate_api|decode_mm_kwargs_item|GenerateRequest' components/src/dynamo/vllm components/src/dynamo | sed -n '1,220p'Repository: ai-dynamo/dynamo
Length of output: 26205
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check whether the vLLM 0.24.0 package layout in this repo expects the token_in_token_out path.
rg -n -C2 'scale_out\.token_in_token_out|serve\.disagg|generate' pyproject.toml components/src/dynamo/vllm container/deps/requirements.vllm.txt | sed -n '1,260p'Repository: ai-dynamo/dynamo
Length of output: 20420
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Look for repository docs or code comments explaining the vLLM endpoint import split.
rg -n -C2 'token_in_token_out|disagg|scale_out|native generate|/generate' .ai components/src/dynamo | sed -n '1,260p'Repository: ai-dynamo/dynamo
Length of output: 21184
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Resolve whether the repository treats vLLM imports as required or as backend-specific/optional in config.
sed -n '150,210p' pyproject.tomlRepository: ai-dynamo/dynamo
Length of output: 2849
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search the repo for any mention of the two vLLM package layouts and version-specific compatibility notes.
rg -n -C3 'vllm-openai|0\.24\.0|token_in_token_out|serve\.disagg|scale_out' . | sed -n '1,260p'Repository: ai-dynamo/dynamo
Length of output: 23272
🌐 Web query:
vLLM 0.24.0 token_in_token_out serve.disagg mm_serde GenerateRequest
💡 Result:
In vLLM 0.24.0, the components you referenced relate to the library's advanced serving infrastructure, specifically for disaggregated architectures and specialized communication protocols [1][2][3]. 1. serve.disagg: This refers to the vLLM namespace (vllm.entrypoints.serve.disagg) designed for disaggregated serving, a deployment pattern where prefill (prompt processing) and decode (generation) tasks are decoupled across different instances or clusters [2][3][4]. This architecture optimizes throughput and latency by scaling prefill and decode resources independently [4]. 2. GenerateRequest: This is a Pydantic BaseModel defined within vllm.entrypoints.serve.disagg.protocol.py [5][6]. It acts as the structured schema for inference requests handled by the disaggregated serving system [5]. Key fields in GenerateRequest include token_ids, multimodal features, sampling parameters, request ID, and metadata for KV cache transfer between instances [5][6]. 3. mm_serde: Located within the disagg and scale_out entrypoint modules, mm_serde provides utility functions for the serialization and deserialization (serde) of multimodal tensors and associated field metadata [7][3]. It ensures that complex multimodal inputs are correctly packaged and transmitted within the vLLM ecosystem [7]. 4. token_in_token_out: While often mentioned in the same context as disaggregated serving, this typically refers to a distinct entrypoint (vllm.entrypoints.scale_out.token_in_token_out) designed to provide specific token-level interfaces for external clients, enabling granular control over input token processing and output token generation in distributed environments [7][8][9]. vLLM 0.24.0 was released on June 29, 2026, and introduced significant improvements to model support (e.g., MiniMax-M3) and internal engine performance [1][10]. The modules you identified are part of the supporting infrastructure for these advanced deployment capabilities [2][3].
Citations:
- 1: https://github.com/vllm-project/vllm/releases/tag/v0.24.0
- 2: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/serve/disagg/
- 3: https://docs.vllm.ai/en/stable/api/vllm/entrypoints/serve/disagg/
- 4: https://docs.vllm.ai/en/stable/examples/disaggregated/disaggregated_serving/
- 5: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/serve/disagg/protocol/
- 6: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/serve/disagg/protocol.py
- 7: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/scale_out/token_in_token_out/
- 8: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/scale_out/token_in_token_out/serving/
- 9: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/scale_out/token_in_token_out/protocol/
- 10: https://freedom.tech/posts/2026-06-29-vllm-0-24-0/
🌐 Web query:
site:github.com/vllm-project/vllm "token_in_token_out" "serve.disagg" vllm 0.24.0
💡 Result:
In vLLM 0.24.0, disaggregated serving (often referred to as Prefill/Decode or P/D splitting) is a feature that allows running separate vLLM instances for prefill and decoding tasks, connected by a KV cache transfer mechanism [1][2]. Regarding your query about "token_in_token_out serve.disagg," the codebase includes an internal module vllm.entrypoints.serve.disagg [3][4] that facilitates handling requests in this distributed architecture. Key technical aspects as of version 0.24.0 include: 1. Architecture: The system functions by running a "KV producer" (prefill instance) and a "KV consumer" (decode instance) [1][5]. A connector is used to transfer KV caches between these instances [1][6]. 2. Connectors: vLLM supports various connectors for KV transfer, such as the Mooncake connector [7][2]. Note that the P2pNcclConnector was removed in version 0.24.0 [2]. 3. Configuration: Disaggregated serving is configured using the --kv-transfer-config flag during vLLM server startup [8][7][9]. This flag accepts a JSON string specifying the connector type, the instance role (e.g., kv_producer or kv_consumer), and additional necessary parameters [8][5][7]. 4. Proxying: Because the prefill and decode stages are separate, a proxy server is typically required to route incoming user requests to the prefill instance first, and then manage the transition or hand-off to the decode instance once prefill is complete [5][7][10]. For exact implementations and current best practices, you should refer to the examples/disaggregated/ directory in the vLLM repository, which provides scripts for setting up these multi-instance deployments [5][7][10]. The internal vllm/entrypoints/serve/disagg/ path is primarily intended for system-level request handling and internal testing [3][4].
Citations:
- 1: https://github.com/vllm-project/vllm/blob/3fd9d2d3/docs/features/disagg_prefill.md
- 2: https://github.com/vllm-project/vllm/releases/tag/v0.24.0
- 3: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/serve/disagg/protocol.py
- 4: https://github.com/vllm-project/vllm/blob/3e49479c/tests/entrypoints/serve/disagg/test_generate_stream.py
- 5: https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/disaggregated%5Fprefill.sh
- 6: https://github.com/vllm-project/vllm/blob/main/docs/features/disagg_encoder.md
- 7: https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/mooncake_connector/run_mooncake_connector.sh
- 8: https://github.com/vllm-project/vllm/blob/e9f331d7/examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh
- 9: https://github.com/vllm-project/vllm/blob/v0.19.0/docs/serving/expert_parallel_deployment.md
- 10: https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/p2p%5Fnccl%5Fxpyd/disagg%5Fexample%5Fp2p%5Fnccl%5Fxpyd.sh
🌐 Web query:
vllm 0.24.0 "vllm.entrypoints.scale_out.token_in_token_out" "vllm.entrypoints.serve.disagg" exact package path
💡 Result:
In vLLM version 0.24.0, the requested namespaces correspond to the following file system paths: 1. vllm.entrypoints.serve.disagg: The exact package path is vllm/entrypoints/serve/disagg/ [1][2]. 2. vllm.entrypoints.scale_out.token_in_token_out: While you requested this path for v0.24.0, authoritative documentation indicates this module is located at vllm/entrypoints/scale_out/token_in_token_out/ [3][4][5]. Note that serving components within this namespace are found at vllm/entrypoints/scale_out/token_in_token_out/serving.py [4]. It is important to note that the vLLM codebase structure evolves across versions; documentation for v0.24.0 specifically lists the disaggregated serving modules under vllm/entrypoints/serve/disagg/ [1]. If you are looking for specific functionality such as the serving logic, it is typically located in a serving.py file within those respective directories [4][6][7].
Citations:
- 1: https://docs.vllm.ai/en/v0.24.0/api/vllm/entrypoints/serve/disagg/
- 2: https://docs.vllm.ai/en/v0.22.1/api/vllm/entrypoints/serve/disagg/
- 3: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/scale_out/token_in_token_out/
- 4: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/scale_out/token_in_token_out/serving/
- 5: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/scale_out/token_in_token_out/protocol/
- 6: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/serve/disagg/protocol/
- 7: https://docs.vllm.ai/en/v0.23.0/api/vllm/entrypoints/serve/disagg/serving/
🌐 Web query:
site:github.com/vllm-project/vllm "v0.24.0" "token_in_token_out" "serve.disagg" "mm_serde"
💡 Result:
In vLLM v0.24.0, the components you referenced relate to the "Disaggregated Everything" architecture, which decouples request rendering from token-level inference [1][2][3]. The token_in_token_out directory structure, specifically under vllm/entrypoints/scale_out/token_in_token_out/, consolidates the entrypoints for this disaggregated serving flow [4]. This architecture enables a separation of duties where a "Renderer" node handles chat/prompt processing (producing token IDs and features) and a "Generate" node handles core model inference [5][2][3]. Key components in this flow include: 1. serve.disagg: Contains the logic for the disaggregated inference service. This includes the ServingTokens class (which manages the /generate endpoint) and protocol.py, which defines the GenerateRequest and GenerateResponse structures used for passing tokens and multimodal features between nodes [6][2][7]. 2. mm_serde: Short for "multimodal serialization/deserialization," this module is used to handle the efficient transmission of multimodal features (such as pixel_values or other tensor-based data) within the disaggregated request payloads [8][9]. Multimodal data is often serialized using base64-encoded MessagePack to be passed between the rendering and inference nodes [8]. In this workflow, the /v1/chat/completions/render endpoint processes requests into token IDs and serialized multimodal features (via mm_serde), which are then submitted to the /inference/v1/generate endpoint for model execution [5][8]. This design allows vLLM to scale components independently and move computationally intensive rendering tasks away from the primary inference engine [2][3]. Top results: 1, 2, 3, 5, 8
Citations:
- 1: https://github.com/vllm-project/vllm/releases/tag/v0.24.0
- 2: [RFC]: Disaggregated Everything - Token In <> Token Out API Server vllm-project/vllm#22817
- 3: [DisaggEverything] Tokens in<>out
/generateendpoint vllm-project/vllm#24261 - 4: vllm-project/vllm@3483240
- 5: https://github.com/vllm-project/vllm/blob/main/examples/scale_out/example_mm_serve.py
- 6: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/serve/disagg/protocol.py
- 7: https://github.com/vllm-project/vllm/blob/3e49479c/tests/entrypoints/serve/disagg/test_generate_stream.py
- 8: [RFC]: Reduce multimodal payload in token-in/token-out by deferring
pixel_valuespreprocessing to/generatevllm-project/vllm#46722 - 9: https://github.com/vllm-project/vllm/blob/3e49479c/vllm/entrypoints/serve/render/serving.py
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '40,90p' components/src/dynamo/vllm/CLAUDE.mdRepository: ai-dynamo/dynamo
Length of output: 2447
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the exact import usage in the engine code to see whether any module-scope alternative exists.
sed -n '1,120p' components/src/dynamo/vllm/engine_generate.pyRepository: ai-dynamo/dynamo
Length of output: 4391
🌐 Web query:
site:docs.vllm.ai/en/v0.24.0 "token_in_token_out" vllm
💡 Result:
There is no specific technical feature, metric, or configuration option named "token_in_token_out" documented in the official vLLM framework [1][2][3][4][5]. It is possible that this term is a colloquial reference to standard request metrics—typically reported as prompt_tokens and completion_tokens (or output_tokens)—that describe the volume of data entering and exiting the inference engine during a generation task. If you are looking for information related to token handling in vLLM, the official documentation focuses on areas such as: 1. Token Embeddings and Token Classification: vLLM provides utilities for tasks like token-level embedding and classification, which are distinct from generative request metrics [1][3][5]. 2. Speculative Decoding and MTP: Advanced features like Multi-Token Prediction (MTP) manage how tokens are predicted and processed in batches [2]. 3. Structured Output: vLLM APIs include backends (such as Outlines) that process lists of tokens to advance finite state machines for structured generation [4]. If you are encountering this term in logs or external tools used alongside vLLM (such as Prometheus exporters or monitoring dashboards), it likely represents custom naming conventions for tracking input and output token usage, rather than a native vLLM internal command or configuration parameter.
Citations:
- 1: https://docs.vllm.ai/en/v0.24.0/models/pooling_models/token_embed/
- 2: https://docs.vllm.ai/en/v0.24.0/features/speculative_decoding/mtp/
- 3: https://docs.vllm.ai/en/v0.24.0/examples/pooling/token_embed/
- 4: https://docs.vllm.ai/en/v0.24.0/api/vllm/v1/structured_output/backend_outlines/
- 5: https://docs.vllm.ai/en/v0.24.0/examples/pooling/token_classify/
🌐 Web query:
site:docs.vllm.ai/en/v0.24.0 "serve.disagg" vllm
💡 Result:
In vLLM, vllm.entrypoints.serve.disagg refers to the module supporting disaggregated serving [1]. Disaggregated serving is a deployment architecture that separates different stages of the model inference process—specifically the prefill (prompt processing), decode (token generation), and encoding phases—into distinct, independently scalable components [2][3][4]. Key aspects include: - Scalability: By disaggregating these stages, users can allocate resources (such as different types of GPU hardware) more efficiently, as prefill and decode tasks have different computational intensity and memory requirements [2][3]. - Module Structure: The vllm.entrypoints.serve.disagg namespace contains the necessary protocol, serialization, and serving utilities required to coordinate these disaggregated components [1][5]. - Use Cases: This approach is particularly useful in production environments where optimizing throughput and latency for long-context or high-traffic LLM workloads is required [3][6]. For detailed implementation and configuration, refer to the official vLLM examples for disaggregated serving, which include guides on setting up disaggregated prefill and encoders [3][6][4].
Citations:
- 1: https://docs.vllm.ai/en/v0.24.0/api/vllm/entrypoints/serve/disagg/
- 2: https://docs.vllm.ai/en/v0.24.0/
- 3: https://docs.vllm.ai/en/v0.24.0/examples/disaggregated/disaggregated_serving/
- 4: https://docs.vllm.ai/en/v0.24.0/examples/disaggregated/disaggregated_encoder/
- 5: https://docs.vllm.ai/en/v0.24.0/api/vllm/entrypoints/serve/disagg/protocol/
- 6: https://docs.vllm.ai/en/v0.24.0/examples/disaggregated/example_connector/
Import the vLLM-native adapter at module scope
components/src/dynamo/vllm/engine_generate.py:45-61
The repo is pinned to vLLM 0.24.0, which exposes vllm.entrypoints.serve.disagg; import that path directly here and drop the try/except ModuleNotFoundError shim.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/src/dynamo/vllm/engine_generate.py` around lines 44 - 61, Update
_native_generate_api to import decode_mm_kwargs_item and GenerateRequest
directly from the vLLM 0.24.0 vllm.entrypoints.serve.disagg modules at module
scope, remove the lazy-loading wrapper and try/except compatibility fallback,
and retain the existing symbols for callers.
Sources: Coding guidelines, Path instructions
| def should_publish_generate_capability( | ||
| model_type: ModelType, worker_type: WorkerType | ||
| ) -> bool: | ||
| return worker_type != WorkerType.Prefill and model_type != ModelType.Embedding |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not advertise generate capability from Encode workers.
WorkerType.Encode passes the current negative checks even though it is not a generative worker. Use an explicit allowlist for Aggregated and Decode, and add an Encode case to the test matrix.
Proposed fix
def should_publish_generate_capability(
model_type: ModelType, worker_type: WorkerType
) -> bool:
- return worker_type != WorkerType.Prefill and model_type != ModelType.Embedding
+ return (
+ worker_type in (WorkerType.Aggregated, WorkerType.Decode)
+ and model_type != ModelType.Embedding
+ )assert not should_publish_generate_capability(
ModelType.Chat, WorkerType.Encode
)Also applies to: 675-677
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/src/dynamo/vllm/main.py` around lines 75 - 78, Update
should_publish_generate_capability to return true only for WorkerType.Aggregated
or WorkerType.Decode and non-Embedding models, excluding WorkerType.Encode
explicitly. Extend the related test matrix with a ModelType.Chat and
WorkerType.Encode case asserting the capability is not published.
| import pytest | ||
| from vllm.sampling_params import RequestOutputKind, SamplingParams |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== target file ==\n'
cat -n components/src/dynamo/vllm/tests/test_vllm_engine_generate.py
printf '\n== nearby vllm tests using importorskip ==\n'
rg -n "importorskip|vllm.sampling_params|pytestmark" components/src/dynamo/vllm/tests -g '*.py'
printf '\n== test file list ==\n'
git ls-files components/src/dynamo/vllm/testsRepository: ai-dynamo/dynamo
Length of output: 21086
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,220p' components/src/dynamo/vllm/tests/test_vllm_engine_generate.py
echo
find components/src/dynamo/vllm/tests -maxdepth 1 -name '*.py' -print0 | xargs -0 -n1 sh -c 'echo "---- $1"; rg -n "importorskip|vllm\.sampling_params|pytestmark" "$1"' sh
echo
rg -n '"vllm"|vllm' pyproject.toml components/src/dynamo -g 'pyproject.toml' -g '*.toml' -g '*.yaml' -g '*.yml'Repository: ai-dynamo/dynamo
Length of output: 11111
Guard the vLLM import at module scope.
from vllm.sampling_params import RequestOutputKind, SamplingParams makes this test file fail collection in environments without vLLM. Move it behind pytest.importorskip("vllm.sampling_params") or another module-level guard.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/src/dynamo/vllm/tests/test_vllm_engine_generate.py` around lines 9
- 10, Guard the module-level vLLM import in test_vllm_engine_generate.py with
pytest.importorskip("vllm.sampling_params") so collection is skipped when vLLM
is unavailable, while preserving access to RequestOutputKind and SamplingParams
when the dependency exists.
Source: Learnings
| mainContainer: | ||
| image: my-registry/vllm-sidecar-runtime:my-tag | ||
| workingDir: /workspace/examples/backends/vllm |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
The manifests require a working directory absent from the documented image build. Remove it because all sidecar commands and paths are absolute.
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml#L51-L53: removeworkingDir.examples/backends/vllm/deploy/sidecar_agg.yaml#L56-L58: removeworkingDir.examples/backends/vllm/deploy/sidecar_disagg.yaml#L57-L59: remove it from the decode sidecar.examples/backends/vllm/deploy/sidecar_disagg.yaml#L94-L96: remove it from the prefill sidecar.
📍 Affects 3 files
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml#L51-L53(this comment)examples/backends/vllm/deploy/sidecar_agg.yaml#L56-L58examples/backends/vllm/deploy/sidecar_disagg.yaml#L57-L59examples/backends/vllm/deploy/sidecar_disagg.yaml#L94-L96
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml` around lines 51 -
53, Remove the workingDir setting from the mainContainer in
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml#L51-L53 and
examples/backends/vllm/deploy/sidecar_agg.yaml#L56-L58, and from both decode and
prefill sidecars in examples/backends/vllm/deploy/sidecar_disagg.yaml#L57-L59
and `#L94-L96`. Leave the existing absolute sidecar commands and paths unchanged.
| command: | ||
| - dynamo-vllm-sidecar | ||
| args: | ||
| - --grpc-endpoint | ||
| - 127.0.0.1:50051 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
The v1alpha1 manifests append sidecar arguments to generated worker arguments. Because each manifest replaces the worker command, the Rust binary may receive unrelated generated options and exit during parsing.
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml#L63-L67: use an override contract that atomically replacesargs.examples/backends/vllm/deploy/sidecar_agg.yaml#L59-L63: replace rather than append the generated worker arguments.examples/backends/vllm/deploy/sidecar_disagg.yaml#L60-L64: replace the decode worker’s generated arguments.examples/backends/vllm/deploy/sidecar_disagg.yaml#L97-L101: replace the prefill worker’s generated arguments.
📍 Affects 3 files
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml#L63-L67(this comment)examples/backends/vllm/deploy/sidecar_agg.yaml#L59-L63examples/backends/vllm/deploy/sidecar_disagg.yaml#L60-L64examples/backends/vllm/deploy/sidecar_disagg.yaml#L97-L101
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml` around lines 63 -
67, The v1alpha1 sidecar manifests must atomically replace generated worker
arguments when overriding the command. Update the args override contract at
examples/backends/vllm/deploy/lora/sidecar_agg_lora.yaml:63-67,
examples/backends/vllm/deploy/sidecar_agg.yaml:59-63, and both decode and
prefill sites in examples/backends/vllm/deploy/sidecar_disagg.yaml:60-64 and
:97-101 so only the explicitly defined sidecar arguments are passed.
| if is_prefill { | ||
| let params = finish | ||
| .kv_transfer_params | ||
| .as_ref() | ||
| .filter(|params| !params.fields.is_empty()) | ||
| .ok_or_else(|| { | ||
| client::protocol_error("prefill response is missing kv_transfer_params") | ||
| })?; | ||
| events.push(GenerateEvent::PrefillReady(prost_struct_to_json(params))); | ||
| } else { | ||
| events.push(GenerateEvent::Finished(reason)); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not convert an aborted prefill into a successful KV handoff.
Any terminal reason with KV parameters currently becomes PrefillReady, including Aborted. Explicitly accept successful prefill reasons and propagate aborted responses as cancellation/error to prevent decode from consuming incomplete KV state.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/vllm-sidecar/src/wire.rs` around lines 67 - 77, Update the terminal-event
handling around the is_prefill branch so GenerateEvent::PrefillReady is emitted
only for explicitly successful prefill reasons. When the terminal reason is
Aborted, propagate cancellation/error instead of performing the KV-parameter
conversion or handoff, while preserving the existing validation for successful
responses.
| fn json_to_prost_value(value: &serde_json::Value) -> prost_types::Value { | ||
| use prost_types::value::Kind; | ||
| let kind = match value { | ||
| serde_json::Value::Null => Kind::NullValue(prost_types::NullValue::NullValue as i32), | ||
| serde_json::Value::Bool(value) => Kind::BoolValue(*value), | ||
| serde_json::Value::Number(value) => Kind::NumberValue(value.as_f64().unwrap_or(0.0)), | ||
| serde_json::Value::String(value) => Kind::StringValue(value.clone()), |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect every affected conversion and KV-transfer call site.
ast-grep outline lib/vllm-sidecar/src/wire.rs --items all
rg -n -C3 '\b(json_to_prost_struct|prost_struct_to_json|kv_transfer_params)\b' lib/vllm-sidecarRepository: ai-dynamo/dynamo
Length of output: 11731
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '96,170p' lib/vllm-sidecar/src/wire.rs
printf '\n--- tests/conversion.rs ---\n'
sed -n '1,220p' lib/vllm-sidecar/src/tests/conversion.rs
printf '\n--- tests/disaggregation.rs ---\n'
sed -n '1,140p' lib/vllm-sidecar/src/tests/disaggregation.rsRepository: ai-dynamo/dynamo
Length of output: 12627
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
from decimal import Decimal
# Probe the concrete edge cases mentioned in the review:
# 1) integers beyond 2^53 when represented as f64
# 2) reverse conversion of large numeric values back to JSON integers
cases = [
2**53 - 1,
2**53,
2**53 + 1,
2**63 - 1,
2**63,
]
for n in cases:
f = float(n)
print(n, repr(f), int(f) if f.is_integer() else None, f == n)
print("--- json round-trip examples ---")
for s in ["9007199254740991", "9007199254740992", "9007199254740993", "9223372036854775808"]:
n = json.loads(s)
f = float(n)
print(s, "->", repr(f), "->", json.dumps(int(f)) if f.is_integer() else "non-int")
PYRepository: ai-dynamo/dynamo
Length of output: 742
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C3 '\bkv_transfer_params\b|\bdisaggregated_params\b|remote_block_ids|remote_port|remote_engine_id|tp_size|cache_salt' lib/vllm-sidecar
printf '\n--- proto excerpts ---\n'
sed -n '100,220p' lib/vllm-sidecar/proto/vllm_grpc.proto
printf '\n--- request.rs excerpts ---\n'
sed -n '120,180p' lib/vllm-sidecar/src/request.rsRepository: ai-dynamo/dynamo
Length of output: 22063
Avoid lossy number conversion in KV-transfer Structs. json_to_prost_value rounds integers above 2^53, and number_to_json can turn large float-backed values back into i64::MAX. Reject non-exact numbers here, or carry 64-bit IDs as strings in the KV-transfer contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/vllm-sidecar/src/wire.rs` around lines 108 - 114, Update
json_to_prost_value and its corresponding number_to_json conversion to preserve
KV-transfer numeric values without lossy f64 conversion. Reject numbers that
cannot be represented exactly rather than rounding or saturating them, or
consistently encode 64-bit identifiers as strings across the contract; ensure
large integers do not become incorrect values such as i64::MAX.
| set -euo pipefail | ||
| trap 'rm -f "${READY_FILE:-}"; kill 0' EXIT | ||
|
|
||
| if [[ $# -ne 4 ]]; then | ||
| echo "Usage: $0 MODEL ENGINE_CONFIG STARTUP_TIMEOUT READY_FILE" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| MODEL=$1 | ||
| ENGINE_CONFIG=$2 | ||
| STARTUP_TIMEOUT=$3 | ||
| READY_FILE=$4 | ||
|
|
||
| SCRIPT_DIR=$(dirname "$(readlink -f "$0")") | ||
| source "$SCRIPT_DIR/../../../examples/common/launch_utils.sh" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Use the shared process-group cleanup helper instead of raw kill 0. Raw EXIT traps do not preserve the failing status or explicitly reap children; the test launcher also arms its trap before validating arguments, so an invalid invocation can terminate its caller’s process group.
tests/serve/tito_parity/launch_dynamo_disagg.sh#L5-L19: validate arguments and source utilities first, then install a cleanup function that removesREADY_FILEand callsdynamo_reap_and_exitwith the captured status.examples/backends/vllm/launch/sidecar_disagg.sh#L36-L36: replace the raw trap withtrap dynamo_exit_trap EXIT.examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh#L42-L42: replace the raw trap withtrap dynamo_exit_trap EXIT.
Based on learnings, deferred trap registration is preferred when using dynamo_exit_trap; the shared helper preserves status and reaps the process group.
🧰 Tools
🪛 Shellcheck (0.11.0)
[info] 19-19: Not following: ./../../../examples/common/launch_utils.sh was not specified as input (see shellcheck -x).
(SC1091)
📍 Affects 3 files
tests/serve/tito_parity/launch_dynamo_disagg.sh#L5-L19(this comment)examples/backends/vllm/launch/sidecar_disagg.sh#L36-L36examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh#L42-L42
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/serve/tito_parity/launch_dynamo_disagg.sh` around lines 5 - 19, Replace
raw process-group cleanup with the shared helpers: in
tests/serve/tito_parity/launch_dynamo_disagg.sh lines 5-19, validate arguments
and source launch_utils.sh before registering cleanup, then capture the exit
status in a cleanup function that removes READY_FILE and invokes
dynamo_reap_and_exit; in examples/backends/vllm/launch/sidecar_disagg.sh line 36
and examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh line 42,
replace the raw EXIT traps with trap dynamo_exit_trap EXIT.
Source: Learnings
| REPO_ROOT = Path(__file__).resolve().parents[3] | ||
| GENERATE_PATH = "/inference/v1/generate" | ||
| UPSTREAM_PORT = 8000 | ||
| DYNAMO_PORT = 8000 | ||
| MODEL_MAX_LEN = 4096 | ||
| MAX_NUM_SEQS = 4 | ||
| KV_CACHE_BYTES = 4_294_967_296 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Allocate or inject every port to prevent cross-run collisions. The parity runners cannot execute concurrently and may fail when common local ports are already occupied.
tests/serve/tito_parity/run_parity.py#L29-L35: allocate upstream and Dynamo ports with the repository port utilities and pass them through commands and environment.tests/serve/tito_parity/run_pd_three_way.py#L19-L20: allocate the prefill and decode ports instead of fixing them at 8100/8200.tests/serve/tito_parity/launch_dynamo_disagg.sh#L36-L62: accept injected system, NIXL, event, and health ports from the Python harness.examples/backends/vllm/launch/sidecar_disagg.sh#L68-L73: expose the NIXL side-channel and KV-event ports as environment variables.examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh#L95-L100: expose the NIXL side-channel and KV-event ports as environment variables.
As per coding guidelines, tests must use allocate_port()/allocate_ports(), while launch-script ports must be injectable. Based on learnings, static script ports are portability hazards.
📍 Affects 5 files
tests/serve/tito_parity/run_parity.py#L29-L35(this comment)tests/serve/tito_parity/run_pd_three_way.py#L19-L20tests/serve/tito_parity/launch_dynamo_disagg.sh#L36-L62examples/backends/vllm/launch/sidecar_disagg.sh#L68-L73examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh#L95-L100
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/serve/tito_parity/run_parity.py` around lines 29 - 35, The parity
runners currently use fixed ports, preventing concurrent execution. In
tests/serve/tito_parity/run_parity.py (lines 29-35), use
allocate_port()/allocate_ports() for upstream and Dynamo ports and propagate
them through commands and environment; in
tests/serve/tito_parity/run_pd_three_way.py (lines 19-20), allocate prefill and
decode ports instead of 8100/8200. Update
tests/serve/tito_parity/launch_dynamo_disagg.sh (lines 36-62) to accept injected
system, NIXL, event, and health ports, and update
examples/backends/vllm/launch/sidecar_disagg.sh (lines 68-73) plus
examples/backends/vllm/launch/sidecar_disagg_multimodal_p_d.sh (lines 95-100) to
read NIXL side-channel and KV-event ports from environment variables.
Sources: Coding guidelines, Path instructions, Learnings
| with ThreadPoolExecutor(max_workers=1) as executor: | ||
| future = executor.submit( | ||
| parity.post_json, | ||
| parity.DYNAMO_PORT, | ||
| parity.GENERATE_PATH, | ||
| request, | ||
| timeout, | ||
| ) | ||
| deadline = time.monotonic() + timeout | ||
| while not future.done() and time.monotonic() < deadline: | ||
| live_text, live = scrape_metrics(parity.DYNAMO_PORT) | ||
| active = metric_value_or_zero( | ||
| live, "dynamo_frontend_active_requests", {"model": model} | ||
| ) | ||
| inflight = metric_value_or_zero( | ||
| live, "dynamo_frontend_inflight_requests", {"model": model} | ||
| ) | ||
| queued = metric_value_or_zero( | ||
| live, "dynamo_frontend_queued_requests", {"model": model} | ||
| ) | ||
| active_observed |= active > baseline_active | ||
| inflight_observed |= inflight > baseline_inflight | ||
| queue_observed |= queued > baseline_queue | ||
| if active_observed and inflight_observed and queue_observed: | ||
| break | ||
| time.sleep(0.005) | ||
| result = future.result(timeout=timeout) | ||
|
|
||
| (output_dir / "metrics-live.prom").write_text(live_text, encoding="utf-8") | ||
| if not active_observed: | ||
| raise AssertionError("active_requests never rose while Generate was running") | ||
| if not inflight_observed: | ||
| raise AssertionError("inflight_requests never rose while Generate was running") | ||
| if not queue_observed: | ||
| raise AssertionError("queued_requests never rose before the first output token") | ||
| return result |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Create deterministic queue pressure before asserting queued_requests.
A single request may leave the queue before the first metrics scrape, causing healthy deployments to fail nondeterministically. Saturate worker capacity and submit an additional request before observing the queue gauge.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/serve/tito_parity/verify_generate_metrics.py` around lines 193 - 228,
Update the request orchestration around the ThreadPoolExecutor and metrics
polling to saturate worker capacity, then submit an additional Generate request
before asserting queued_requests. Ensure the extra request remains pending long
enough to observe dynamo_frontend_queued_requests above baseline, while
preserving the existing active and inflight observations and cleaning up all
submitted futures.
|
/ok to test 517260c |
GitHub Actions
Inactive
|
/ok to test 1a5f1e9 |
1a5f1e9 to
836fe81
Compare
GitHub Actions
Inactive
|
/ok to test 836fe81 |
GitHub Actions
Inactive
|
/ok to test fc00f8e |
GitHub Actions
Inactive
|
/ok to test b0b51b2 |
biswapanda
changed the title
…atible routes alias
b0b51b2 to
8b5acf8
Compare
GitHub Actions
Inactive
|
/ok to test 8b5acf8 |
2 participants
Overview
Add an out-of-process vLLM backend that connects Dynamo to vLLM's existing native Rust gRPC/token-in-token-out service. The sidecar supports aggregated and disaggregated generation, multimodal inputs, prompt/completion logprobs, KV handoff, runtime LoRA, metrics, session affinity, and opt-in worker discovery for external RL trainers.
Companion PRs:
Published, cross-repository tested heads:
b0b51b2783346a6ff6566fcaecc7d35fcd2ee80b5a8827cee5021a6021707517c93a3d9c61e3cd012d2bfbe3765cfb5c05567ad8b2338564faedf5e3Out-of-process native gRPC backend
dynamo-vllm-sidecaras a standalone Rust backend connected to a separately managed vLLM EngineCore/Rust server.This is a sibling of the in-process Python
VllmLLMEngine; it does not replace that backend.Request/response parity
The sidecar maps Dynamo's generate contract to the native vLLM protocol, including:
Unsupported or malformed fields fail explicitly instead of being silently discarded. KV-transfer merge collisions also fail explicitly: framework-owned P/D handoff keys cannot be overridden by a caller because accepting two owners for routing metadata is unsafe.
Disaggregated KV serving
Structnumber representation exactly.RL worker discovery
Discovery is opt-in and served on a separate listener:
GET /v1/rl/workersreturns one record per engine endpoint:{ "protocol_version": 1, "namespace": "training", "workers": [ { "component": "backend", "instance_id": 10, "model": "Qwen/Qwen3-0.6B", "admin_base_url": "http://worker:8120", "system_url": "http://worker:8181", "world_size": 2, "system_routes": ["update/load_lora"], "error": null } ] }component=backendis the aggregate/decode cohort; disaggregated prefill engines publishcomponent=prefill. Each record represents one engine endpoint, not one GPU.admin_base_urlidentifies vLLM's direct HTTP control endpoint.world_sizeis the number of inference ranks owned by that endpoint.system_urlandsystem_routesdescribe Dynamo-owned mutations such as LoRA loading.Protocol v1 intentionally does not publish trainer-session-specific
rank_startvalues or claim an atomic topology epoch. Prime computes cumulative offsets from one accepted snapshot and fences the total against its configured inference world size.Session-aware routing
Prime forwards each trajectory ID as
X-Dynamo-Session-ID. When frontend session affinity is enabled, Dynamo keeps successive R2E turns on the same decode worker for the configured idle TTL, while the normal router remains the fallback for requests without a session key.A configurable compatibility header may be selected, but the canonical
X-Dynamo-Session-IDheader always takes precedence.Runtime LoRA
Review fixes in the current head
Where reviewers should start
lib/vllm-sidecar/src/request.rs— Dynamo → vLLM request conversionlib/vllm-sidecar/src/wire.rs— native protocol boundarylib/vllm-sidecar/src/engine.rs— generation lifecyclelib/vllm-sidecar/src/discovery.rs— engine capability discoverylib/vllm-sidecar/src/tests/— conversion, lifecycle, generation, and P/D testslib/rl/src/lib.rs— frontend/v1/rl/workersAPIlib/backend-common/src/rl.rs— worker metadata endpointlib/llm/src/http/service/generate.rsandgenerate/— frontend generate contractexamples/backends/vllm/deploy/sidecar_agg.yamlexamples/backends/vllm/deploy/sidecar_disagg.yamlValidation
At the published Dynamo head:
Exact-head GPU validation used the three published commits above for the two Qwen gates. The GLM runs used the immutable image lineage described after the list:
world_size=2, rank 0/1 mapping, v0/v1 updates on both ranks, post-v1 generation, PASS.agents.x-k8s.iosandbox: reward 1.0, 35 turns,agent_completed, PASS.The GLM image was built from Prime
abcb05c34c0bdacfcfda3e0d4223f23ac5f88bd2, Dynamo6c688b197512e86e3b5b957f2c9b54d9b223bba3, and vLLM2d2bfbe3765cfb5c05567ad8b2338564faedf5e3. The Prime and Dynamo commits are ancestors of the current published heads; the later review fixes were independently validated by the exact-head Qwen aggregate and disaggregate gates.The exact
b0b51b278ARM64 sidecar binary was built with DIND and used as an on-disk overlay. The validated base runtime image was rebuilt and pushed as:Compatibility and scope
DYN_ENABLE_RLis set.openengine.v1.