Skip to content

Fix WebGPU inference crash in embedding and multi-modal feature allocation#2163

Merged
baijumeswani merged 1 commit into
mainfrom
user/feich/webgpu-embedding-allocator-fix
May 18, 2026
Merged

Fix WebGPU inference crash in embedding and multi-modal feature allocation#2163
baijumeswani merged 1 commit into
mainfrom
user/feich/webgpu-embedding-allocator-fix

Conversation

@feich-ms

@feich-ms feich-ms commented May 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fix heap corruption crash during WebGPU inference by using the correct allocator (p_device_inputs_) for tensors that serve as inputs to the embedding/decoder session
  • Previously, embeddings.cpp and multi_modal_features.cpp allocated these tensors on p_device_ (WebGPU GPU memory), but the embedding session runs on CPU for WebGPU (without graph capture), causing the CPU session to write into GPU memory.
  • This is a pair change for this pr [WebGPU] Support models with opset 24 ops and KV-shared decoder layers (Gemma 4) onnxruntime#28501.

Changes

src/models/embeddings.cpp (lines 24, 52):

  • Changed model_.p_device_->GetAllocator()model_.p_device_inputs_->GetAllocator() for embedding input tensor allocation and reallocation during sequence length updates

src/models/multi_modal_features.cpp (lines 66, 85):

  • Changed model_.p_device_->GetAllocator()model_.p_device_inputs_->GetAllocator() for empty feature tensors in Update() and AllocateEmptyFeatures() — these are inputs to the embedding session
  • Vision/speech model output allocations (lines 42, 98) correctly remain on p_device_

Impact

  • WebGPU: Fixes crash (heap corruption 0xc0000374) during inference
  • CUDA/DML/RyzenAI: No-op — p_device_inputs_ == p_device_ on these providers
  • CPU: No-op — p_device_inputs_ == p_device_ on CPU

Test plan

  • End-to-end WebGPU inference with Gemma 4 E2B (109 tokens generated correctly)
  • CPU inference still works after changes
  • Output matches between CPU and WebGPU execution

Copilot AI review requested due to automatic review settings May 14, 2026 03:19
@feich-ms feich-ms requested a review from a team as a code owner May 14, 2026 03:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a WebGPU-specific heap corruption/crash by ensuring tensors that are fed into the embedding/decoder path are allocated using the model’s “inputs device” allocator (p_device_inputs_) rather than the main device allocator (p_device_). This aligns embedding-related buffers with the codebase’s existing device split (notably WebGPU without graph-capture using CPU memory for inputs).

Changes:

  • Allocate embedding input tensors (Embeddings::Mode::Input) using model_.p_device_inputs_->GetAllocator() (constructor + sequence-length reallocation).
  • Allocate empty multi-modal feature tensors (used as embedding inputs) using model_.p_device_inputs_->GetAllocator() in both Update() and AllocateEmptyFeatures().
  • Leave vision/speech output allocations on p_device_ unchanged.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/models/embeddings.cpp Switches embedding input tensor allocation/reallocation to the inputs allocator to avoid CPU writing into WebGPU-allocated buffers.
src/models/multi_modal_features.cpp Switches empty feature-tensor allocation (embedding inputs) to the inputs allocator to match embedding session execution/memory expectations.

@feich-ms feich-ms marked this pull request as draft May 14, 2026 08:14
@feich-ms feich-ms marked this pull request as ready for review May 18, 2026 01:43
…nputs

Embeddings and multi-modal feature tensors used as inputs to the
embedding/decoder session must be allocated on p_device_inputs_ (CPU for
WebGPU without graph capture), not p_device_ (WebGPU). The mismatch
caused heap corruption when the CPU-based session wrote to GPU memory.

This is a no-op for CUDA/DML/RyzenAI where p_device_inputs_ == p_device_.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@feich-ms feich-ms force-pushed the user/feich/webgpu-embedding-allocator-fix branch from 2f9e4ed to 8aa0c4a Compare May 18, 2026 01:46
@feich-ms feich-ms requested review from qjia7 and xiaofeihan1 May 18, 2026 05:15
@baijumeswani baijumeswani merged commit 5466a7b into main May 18, 2026
16 of 17 checks passed
@baijumeswani baijumeswani deleted the user/feich/webgpu-embedding-allocator-fix branch May 18, 2026 16:18
@feich-ms feich-ms restored the user/feich/webgpu-embedding-allocator-fix branch May 26, 2026 09:47
@feich-ms feich-ms deleted the user/feich/webgpu-embedding-allocator-fix branch May 26, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants