test(multimodal): prove + document client-supplied embeddings via the custom-encoder path#11080
Closed
furionw wants to merge 4 commits into
Closed
test(multimodal): prove + document client-supplied embeddings via the custom-encoder path#11080furionw wants to merge 4 commits into
furionw wants to merge 4 commits into
Conversation
github-actions
Bot
added
test
documentation
Contributor
…er (no batcher) + e2e PR2 of the custom vision-encoder series, stacked on PR1. Wires the contract into the aggregated dynamo.vllm worker and proves the mixed-embeds splice path end to end with a trivial backend and a DIRECT serial call — no micro-batcher yet. - AsyncVisionEncoder (serial glue): off-thread preprocess pool + the A5 all-or-nothing preprocess barrier + a single actor thread running build(model_id) / forward_batch / close. Forwards are serialized (no cross-request coalescing) — that lands in PR3. validate() fails fast if the backend's hardcoded image_token_id is unset/non-int. Stable public surface (load / encode / get_image_placeholder_token_id / shutdown). - handlers.py: resolve a VisionEncoderBackend subclass from --custom-encoder-class, wrap in AsyncVisionEncoder, encoder.load(config.model) (no device), assemble the mixed EmbedsPrompt; requires --enable-prompt-embeds; image-only + text fallback. - Examples: QwenVisionEncoderBackend base (hardcodes image_token_id=151655) + HitchhikersVisionEncoder (answers "42"). - agg_custom.sh launcher + a minimal qwen_vl.jinja template. - e2e: the agg_custom topology profile + the "42" semantic payload. Tests: serial-glue unit tests (encode, A5 barrier, single actor thread, NO coalescing, fail-fast load + thread reaping). Integrated "42" e2e validated on the series' integration branch. Note: hand-set GPU util (reservation is a later milestone). Based on the series' validated merge-base; rebase onto main before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…coder (cross-request batching) PR3 of the custom vision-encoder series, stacked on PR2. Replaces the serial direct-call glue with cross-request batching to the author's max_batch_cost token budget (scalar cost, one-dimensional packing — no shape buckets, no graph ladder; that lands in PR4). handlers.py is unchanged — the AsyncVisionEncoder public surface is the stable boundary. - ThreadedMicroBatcher (generic, torch-free): one pinned non-daemon actor thread running on_start (build) / every fn(items) / on_stop (close); eager drain-on-completion (no timer; max_wait_ms is an opt-in knob); coalesce by scalar cost up to max_batch_cost (None = pass-through). Per-item idempotent finalizer; cancellation + repeat-cancel retirement; worker supervisor; optional max_outstanding_cost admission; cross-loop-safe shutdown. submit(items, costs). - AsyncVisionEncoder now routes preprocessed items through the batcher (was a single-worker serial executor); the A5 preprocess barrier is unchanged; build is build(model_id) and the hardcoded image_token_id is validated at load. Tests: batcher concurrency suite (eager-drain, cost budget, pass-through, cancellation/retirement, supervisor, lifecycle, on_stop) + batcher-backed glue. Note: based on the series' validated merge-base; rebase onto main before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…) + Qwen3-VL ViT example PR4 of the custom vision-encoder series, stacked on PR3. Introduces bucket-wise graph support on the Dynamo side and a real graphed encoder example. - ThreadedMicroBatcher: optional `buckets` ladder. When set, the batcher rounds a batch's packed sum(cost) UP to the nearest rung and passes it as `target_bucket` to fn(items, target_bucket). With max_batch_cost=None it derives the ceiling as max(buckets); buckets=None stays eager / pass-through (PR3 behavior unchanged). - AsyncVisionEncoder passes backend.buckets to the batcher. - Qwen3VLViTEncoder example: loads the real Qwen3-VL vision tower (build(model_id), picks its own device), captures one CUDA graph per rung via torch.compile(reduce-overhead), and in forward_batch pads sum(cost) up to target_bucket, replays, slices the real images back out (CPU). Hardcodes image_token_id via the Qwen base. The author owns padding; Dynamo only picks the rung. `buckets`/`target_bucket` were forward-compat in the contract since PR1; this PR makes them live. Tests: target_bucket rounding (boundary + eager None + buckets-derive + ladder-covers-budget). Graph smoke validated on the integration branch. Deepstack limitation: Qwen3-VL deepstack features are not carried by the one-tensor-per-image contract — this exercises the mechanism, not accuracy. Note: based on the series' validated merge-base; rebase onto main before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
furionw
force-pushed
the
qiwa/vision-encoder-cuda-graphs
branch
from
f853fa4 to
aa90a19
Compare
… custom-encoder path The in-process custom-encoder path already carries client-supplied per-image embeddings with NO new runtime code: RawT is generic, the handler passes the (data:) URL string to preprocess verbatim, and build_mixed_embeds splices whatever forward_batch returns. A client sends embeddings inline as a safetensors data: URI on image_url and writes a VisionEncoderBackend that decodes (preprocess) + projects (forward_batch). This PR proves and documents that path — no feature code. - tests/utils/embeds_passthrough_encoder.py: a TEST-ONLY stub EmbedsPassthroughEncoder (decode the safetensors data: URI, identity passthrough) + encode/decode_embeds_data_uri helpers (the client wire format). Hardcodes the image token id via the Qwen base. - tests/.../test_vllm_embeds_passthrough.py: encode/decode round-trip; non-data URI rejected (A5 barrier); AsyncVisionEncoder.encode passthrough; build_mixed_embeds splice. CPU, no LM. - examples/custom_encoder/README.md: the client contract (safetensors data: URI on image_url) + a projector-backend skeleton (decode in preprocess, project in forward_batch, hardcode image_token_id). Notes the planned image_embeds field. - report_pytest_markers.py: stub safetensors for the marker hook. Verified: 9 unit tests; e2e on Qwen3-VL-2B — client computes embed_tokens(phrase) (11x2048) -> safetensors data: URI -> stub decode+passthrough -> splice -> LM -> "42". Note: based on the series' validated merge-base; rebase onto main before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
furionw
force-pushed
the
qiwa/vision-encoder-embeds
branch
from
59ade3d to
d3f8bc2
Compare
furionw
force-pushed
the
qiwa/vision-encoder-cuda-graphs
branch
from
aa90a19 to
fc9b5d2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR6 of the custom vision-encoder series, stacked on PR4 #11039. Lets a
customer run
dynamo.vllmwith a stock text-only LM and send pre-computedper-image embeddings in the request, where their own small projector maps
embeddings → LM hidden dim and Dynamo splices the result in — no ViT in Dynamo.
Key point: this already works with no new runtime code.
RawTon theVisionEncoderBackendcontract is generic, the handler passes theimage_urlstring to
preprocessverbatim, andbuild_mixed_embedssplices whateverforward_batchreturns. So a client sends each embedding inline as a safetensorsdata:URI on a normalimage_urlcontent part (sending directly, not a URLfetch), and writes a backend that decodes it in
preprocessand projectsit in
forward_batch. The projector is the client's code. This PR only provesand documents that path.
What's here (test + docs only — no feature/contract/handler/frontend change)
tests/utils/embeds_passthrough_encoder.py— a test-only stubEmbedsPassthroughEncoder(decodes the safetensorsdata:URI, identitypassthrough — a real client projects here) +
encode_embeds_data_uri/decode_embeds_data_urihelpers (the client wire format).test_vllm_embeds_passthrough.py— unit smoke (CPU, no LM): encode/decoderound-trip preserves shape+dtype; a non-
data:URI is rejected (A5 barrier →clean request error);
AsyncVisionEncoder.encodepassthrough;build_mixed_embedssplices the decoded embeds at the placeholder rows.
examples/custom_encoder/README.md— the client contract: encode embeddingsas a safetensors
data:URI onimage_url, plus a ~12-line projector-backendskeleton (decode in
preprocess, project inforward_batch).report_pytest_markers.py— stubsafetensorsfor the marker hook.Test plan
embed_tokens(" the Ultimate Question … ")((11, 2048)), encodes it as a safetensorsdata:URI, sends it asimage_url; the stub decodes + identity-passes-through → splice → the LM reads theclient-supplied embeds → answers "42" (
EMBEDS_PATH_OK). This is a scriptedcheck rather than a committed serve profile: building the payload needs the LM's
embed table client-side, which would force a multi-hundred-MB load at pytest
collection. No Rust → no Graham pass.
Follow-up — PR7: first-class
image_embedsfieldThe documented OpenAI-style contract: an
ImageEmbedscontent part inlib/protocols/.../chat.rs, rendered inlib/llm/src/preprocessor.rs→multi_modal_data["image_embeds"], with a worker branch decoding the same bytes.Heavier (Rust + bindings rebuild + Graham review); reuses this PR's helpers.
Note
Based on the series' validated merge-base; rebase onto
mainbefore merge.🤖 Generated with Claude Code