Fix WebGPU inference crash in embedding and multi-modal feature allocation#2163
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a WebGPU-specific heap corruption/crash by ensuring tensors that are fed into the embedding/decoder path are allocated using the model’s “inputs device” allocator (p_device_inputs_) rather than the main device allocator (p_device_). This aligns embedding-related buffers with the codebase’s existing device split (notably WebGPU without graph-capture using CPU memory for inputs).
Changes:
- Allocate embedding input tensors (
Embeddings::Mode::Input) usingmodel_.p_device_inputs_->GetAllocator()(constructor + sequence-length reallocation). - Allocate empty multi-modal feature tensors (used as embedding inputs) using
model_.p_device_inputs_->GetAllocator()in bothUpdate()andAllocateEmptyFeatures(). - Leave vision/speech output allocations on
p_device_unchanged.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/models/embeddings.cpp |
Switches embedding input tensor allocation/reallocation to the inputs allocator to avoid CPU writing into WebGPU-allocated buffers. |
src/models/multi_modal_features.cpp |
Switches empty feature-tensor allocation (embedding inputs) to the inputs allocator to match embedding session execution/memory expectations. |
…nputs Embeddings and multi-modal feature tensors used as inputs to the embedding/decoder session must be allocated on p_device_inputs_ (CPU for WebGPU without graph capture), not p_device_ (WebGPU). The mismatch caused heap corruption when the CPU-based session wrote to GPU memory. This is a no-op for CUDA/DML/RyzenAI where p_device_inputs_ == p_device_. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
2f9e4ed to
8aa0c4a
Compare
baijumeswani
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
p_device_inputs_) for tensors that serve as inputs to the embedding/decoder sessionembeddings.cppandmulti_modal_features.cppallocated these tensors onp_device_(WebGPU GPU memory), but the embedding session runs on CPU for WebGPU (without graph capture), causing the CPU session to write into GPU memory.Changes
src/models/embeddings.cpp(lines 24, 52):model_.p_device_->GetAllocator()→model_.p_device_inputs_->GetAllocator()for embedding input tensor allocation and reallocation during sequence length updatessrc/models/multi_modal_features.cpp(lines 66, 85):model_.p_device_->GetAllocator()→model_.p_device_inputs_->GetAllocator()for empty feature tensors inUpdate()andAllocateEmptyFeatures()— these are inputs to the embedding sessionp_device_Impact
p_device_inputs_ == p_device_on these providersp_device_inputs_ == p_device_on CPUTest plan