Backmerging pr #1193
Merged
Merged
Conversation
## Summary Promote the CUDA Plugin EP to use the canonical CUDA provider identity and packaging surface. CUDA plugin builds now advertise `CUDAExecutionProvider`, produce the canonical `onnxruntime_providers_cuda` native library name, and can be auto-registered from bundled Python wheels while retaining explicit registration support for standalone plugin packages and native applications. This also aligns CUDA plugin provider/session options with the `ep.cuda.*` namespace, updates allocator ownership so external GPU allocator callbacks remain session-scoped, and refreshes tests, docs, and CI coverage for the promoted plugin path. ## Key Changes | Area | Changes | | --- | --- | | Provider identity | Rename the plugin EP registration/name surface from `CudaPluginExecutionProvider` to `CUDAExecutionProvider`; keep compatibility aliases for historical option keys. | | Native library/package layout | Build and package the plugin as `onnxruntime_providers_cuda.*`, matching the legacy CUDA EP filename; update Python, C#, Java, wheel, NuGet, and test helpers accordingly. | | Python registration | Auto-register the bundled CUDA plugin provider from `onnxruntime/capi` when build info reports `cuda-plugin-ep=1`; retry after `preload_dlls()` and warn on unexpected registration failures. | | Allocators | Keep internal arena/mempool allocators in factory device-cache state, but create external GPU allocator wrappers per EP/session from that session's callbacks so unrelated sessions do not inherit external allocator configuration. | | Provider options | Normalize CUDA plugin config keys under `ep.cuda.*` while preserving legacy aliases; update session/provider option extraction and reporting. | | Tests and docs | Update CUDA plugin C++/Python/C# tests, add allocator isolation coverage, refresh CUDA plugin docs, and update CI/plugin parity wiring. | ## Testing Notes Validated locally: - Focused `-fsyntax-only` compile checks for `cuda_allocator_plugin.cc`, `cuda_ep.cc`, `cuda_ep_factory.cc`, and `cuda_plugin_arena_test.cc` using the existing `cu130_plugin_no_cudnn` build flags. - `python3 -m py_compile onnxruntime/__init__.py`. - `git diff --check` for the edited source and documentation files. - `git rebase origin/main` reported the branch was up to date. Attempted: - `lintrunner -a` was attempted, but local ruff/ruff-format/clangformat adapter execution failed before producing actionable file diagnostics. No tracked file changes were left by that attempt.
### Description Fixes split heuristic edge cases in Flash Attention and Lean Attention when the SM count is reported as zero or when there are no key tiles. The change clamps the SM count used by the heuristics, prevents divide-by-zero paths, and returns stable buffer sizing for empty tile cases. ### Changes - Guard Flash Attention split heuristic against zero SM count and degenerate split counts. - Guard Lean Attention split/buffer sizing against zero SM count, zero key tiles, and single-tile division cases. - Add CUDA provider regression tests for zero SM count and zero key-tile inputs. ### Testing - Added `attention_split_heuristic_test.cc` coverage for Flash Attention and Lean Attention split heuristics. Fixes microsoft#29550
… an empty graph (microsoft#29457) ### Description <!-- Describe your changes. --> InferenceSession::Initialize() validates, for any EP that has graph capture enabled and uses the ALLOW_CPU_FOR_SHAPES node-assignment policy (DirectML, WebGPU), that the partitioned graph is eligible for capture via AreAllComputeNodesAssignedToEpOrCpu(). That helper currently requires at least one node to be assigned to the EP: return has_node_on_provider && !HasMemcpyNodes(graph); For a graph with no nodes, like a model that is fully consumed by the EP's runtime graph fusion, or that trivially folds away (like a lone Constant node), has_node_on_provider is false, so the session throws: This session cannot use the graph capture feature as requested by the user as all compute graph nodes have not been partitioned to the DmlExecutionProvider An empty graph contains nothing that violates the "all compute on the EP or CPU, no Memcpy" requirement, so it should be allowed. This change permits the empty-graph case: return (has_node_on_provider || graph.NumberOfNodes() == 0) && !HasMemcpyNodes(graph); Behavior is unchanged for all non-empty graphs (the expression collapses to the original has_node_on_provider). Adds CApiTest.DmlGraphCaptureEmptyGraph : creates a DML session with ep.dml.enable_graph_capture=1 on a single Constant model (folds to an empty graph) and asserts session creation does not throw. The test fails without this fix and passes with it. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> microsoft#27958 refactored the graph-capture selection logic and replaced AreAllComputeNodesAssignedToCudaOrJsOrDmlEpWebGpuEp() (which returned true for an empty graph) with AreAllComputeNodesAssignedToEpOrCpu() and its new has_node_on_provider requirement. The behavior verified with a DLL-swap: onnxruntime-directml 1.24.x/1.25.2 create the session successfully; 1.27 throws. The failure surfaces through ONNX Runtime GenAI on DirectML: GenAI creates a per-device "allocator-initialization" session on a Constant-only model with DML graph capture enabled to obtain a device allocator (model.cpp). Under ORT 1.27 that session throws at construction, which breaks every GroupQueryAttention-based GenAI LLM on the DirectML EP (observed as WinML EP certification failures across DeepSeek/Llama/Qwen/Phi models --------- Co-authored-by: Aditya Rastogi <adityar@ntdev.microsoft.com>
…#29444) This pull request strengthens the safety of arithmetic operations in the convolution implementation by using `SafeInt` to prevent integer overflows, especially in cases where tensor shapes may be attacker-controlled. It also removes some compiler-specific warning pragmas that are no longer needed due to these changes. **Enhanced integer safety in convolution calculations:** * Replaced raw `size_t` multiplications with `SafeInt<size_t>` for calculating input/output sizes and kernel dimensions, ensuring that any arithmetic overflow is caught and handled appropriately. This is particularly important for preventing security vulnerabilities from attacker-controlled tensor shapes. [[1]](diffhunk://#diff-3f8429fb413f2d0edd2e6b08b2a516744293e5ba092ab9c9da7b34ed0b28948dL1563-R1566) [[2]](diffhunk://#diff-3f8429fb413f2d0edd2e6b08b2a516744293e5ba092ab9c9da7b34ed0b28948dL1581-R1594) * Used `SafeInt<size_t>` for all working buffer size calculations, including those involving thread counts and batch/group products, to prevent buffer overflows and ensure correct memory allocation. [[1]](diffhunk://#diff-3f8429fb413f2d0edd2e6b08b2a516744293e5ba092ab9c9da7b34ed0b28948dL1678-R1686) [[2]](diffhunk://#diff-3f8429fb413f2d0edd2e6b08b2a516744293e5ba092ab9c9da7b34ed0b28948dL1777-L1791) **Code cleanup and maintenance:** * Removed MSVC-specific warning suppression pragmas since `SafeInt` now guards against arithmetic overflow, making these warnings obsolete. [[1]](diffhunk://#diff-3f8429fb413f2d0edd2e6b08b2a516744293e5ba092ab9c9da7b34ed0b28948dL1331-L1335) [[2]](diffhunk://#diff-3f8429fb413f2d0edd2e6b08b2a516744293e5ba092ab9c9da7b34ed0b28948dL1777-L1791) * Added the `core/common/safeint.h` include to provide access to the `SafeInt` functionality.
…yml. (microsoft#29575) ### Description <!-- Describe your changes. --> Don't echo command when setting VSO variable in mac-cpu-packing-jobs.yml. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> If the command is echoed again with `set -x`, the VSO variable may end up with a trailing `'`, which is invalid. Whether this actually happens is intermittent and probably dependent on the output ordering.
## Description Adds a CUDA-focused `CudaQuantizer` helper to ONNX Runtime's Python quantization tools and exports it from `onnxruntime.quantization`. The helper centralizes the CUDA quantization paths needed by ONNX Runtime tests and `onnxruntime-genai` model building, covering both per-channel QMoE weight packing and MatMulNBits blockwise formats without duplicating that logic in test code. This also updates the CUDA MoE/QMoE transformer tests to consume the shared helper, including full-range symmetric offset storage and CUTLASS mixed-GEMM prepacked layouts. ## Summary of Changes ### Quantization API | File | Change | |------|--------| | `onnxruntime/python/tools/quantization/cuda_quantizer.py` | Adds `CudaQuantizer` with per-channel INT4/INT8 quantization, CUDA mixed-GEMM prepacking, MatMulNBits blockwise quantization, CUTLASS prepacked blockwise quantization, and a pure-PyTorch symmetric blockwise reference path. | | `onnxruntime/python/tools/quantization/__init__.py` | Exports `CudaQuantizer` from `onnxruntime.quantization`. | | `onnxruntime/python/tools/quantization/qmoe_quantizer.py` | Replaced by the more general CUDA quantizer implementation. | ### Test Updates | File | Change | |------|--------| | `onnxruntime/test/python/transformers/test_moe_cuda.py` | Replaces duplicated blockwise quantization/prepacking logic with `CudaQuantizer` calls. | | `onnxruntime/test/python/transformers/test_qmoe_cuda.py` | Uses `CudaQuantizer` for per-channel and blockwise QMoE test setup, including full-range unsigned offset storage coverage and default prepacked layout smoke tests. | ## Testing - Built the CUDA Release configuration. - `pytest transformers/test_qmoe_cuda.py::TestQMoEIntPrePackSmoke` - 5 passed, 2 subtests passed. - `pytest transformers/test_qmoe_cuda.py` - 97 passed, 12 skipped, 2 subtests passed. - Validated without CUDA that importing the quantization module does not require `torch`, and that torch-backed methods raise a clear error only when used. - `python -m py_compile onnxruntime/python/tools/quantization/cuda_quantizer.py` ## Checklist - [x] Tests added/updated - [x] No breaking changes to released APIs - [ ] CI passes --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
### Description
<!-- Describe your changes. -->
Gather is missing from the transpose optimizer's handler map. This PR
adds HandleGather, which pushes a Transpose past a Gather so the
transpose can cancel or fuse with neighboring transposes.
The rewrite is:
`data -> Transpose(perm) -> Gather(axis=k, indices=scalar_const)`
becomes
`data -> Gather(axis=perm[k], indices=scalar_const) ->
Transpose(SqueezePerm({perm[k]}, perm))`
Scope is intentionally narrow — only the scalar (0-D) constant-indices
case, which is structurally identical to a Squeeze along the gathered
axis and reuses SqueezePerm for the post-rewrite output perm. Non-scalar
or non-constant indices are left untouched. The framework's
DefaultCostCheck still gates the rewrite on profitability, so
unprofitable cases are rejected before the handler runs.
Tests added in transpose_optimizer_test.cc cover the positive
scalar-indices case, negative-axis normalization, rank-1-indices
fall-through, and dynamic-indices fall-through.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
We've observed that the Transpose → Gather → Transpose pattern produces
sub-optimal inference performance on the QNN EP. With this
handler in place, the surrounding transposes can fold or cancel,
eliminating layout-conversion overhead around Gather.
---------
Signed-off-by: Mu-Chein Hsu <quic_muchhsu@quicinc.com>
…9499) ## Summary This PR adds CUDA `MatMulNBits` support for weights that have already been packed into the fpA_intB SM80 weight-only layout. It lets offline tooling provide prepacked int4/int8 weights while preserving the existing runtime-prepack path for standard MatMulNBits initializers. ## Key Changes - Adds the `weight_prepacked` attribute to `com.microsoft::MatMulNBits`: - `0`: standard MatMulNBits packed layout, runtime-prepacked by CUDA fpA_intB when enabled. - `1`: already prepacked in the CUDA SM80 fpA_intB layout. - `2`: reserved SM90 layout, rejected for now. - Routes SM90 mixed FP16/BF16 activation + int4/int8 weight fpA_intB dispatch through the SM80 CUTLASS layout/kernel path for this operator. - Enforces strict gating for prepacked weights: - requires `onnxruntime_USE_FPA_INTB_GEMM=ON` at build time, - requires `ORT_FPA_INTB_GEMM` at runtime, - requires FP16/BF16 input `A`, - rejects unsupported/reserved prepacked formats. - Reduces GPU memory usage for device-resident prepacked weights by keeping the original CUDA `B` tensor alive and passing it directly to fpA_intB instead of copying it into a second GPU buffer. - Documents the CUDA prepacked-weight contract and adds focused positive/error tests. ## Testing - `PATH=/home/tianlei/git/onnxruntime/.venv/bin:$PATH lintrunner -a` - `cmake --build build/cu130_fp4_bench/Release --target onnxruntime_providers_cuda --parallel 16` - `cd /tmp && PYTHONPATH=/home/tianlei/git/onnxruntime/build/cu130_fp4_bench/Release:/home/tianlei/git/onnxruntime/onnxruntime/test/python/quantization LD_LIBRARY_PATH=/home/tianlei/git/onnxruntime/build/cu130_fp4_bench/Release:/home/tianlei/git/onnxruntime/build/cu130_fp4_bench/Release/onnxruntime/capi:/home/tianlei/cuda13.0/lib64:/home/tianlei/cudnn_9.23_cuda13/lib64:/home/tianlei/cudnn_9.23_cuda13/lib:${LD_LIBRARY_PATH:-} /home/tianlei/git/onnxruntime/.venv/bin/python /home/tianlei/git/onnxruntime/onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py -v` C++ unit tests were not run because the local build tree was configured with `onnxruntime_BUILD_UNIT_TESTS=OFF`. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
### Description Fixes the CUDA packaging pipeline build error (with CUDA arch `52-real`) that was introduced by microsoft#29451. ### Root cause The small-M batched GEMV path added in microsoft#29451 defines `half` helpers for the batched kernel — the `Acc2<half>` accumulator trait and the `PackNatural` / `DotAccum` / `HorizontalAdd` overloads — inside a single `#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ >= 530)` guard that compiles the *entire* declaration out for archs below `sm_53`. However, `MatMulFloat4BatchedKernel<half>` is launched from host code and is therefore instantiated for **every** target architecture, including `sm_52`. During the `sm_52` device pass those `half` helpers no longer exist, so `nvcc` reported (100 errors, e.g.): - `incomplete type "Acc2<half>" is not allowed` - `no suitable user-defined conversion from "WPack<half>" to "const WPack<nv_bfloat16>" exists` (the compiler fell back to the `nv_bfloat16` overloads) ### Fix Mirror the existing `nv_bfloat16` convention in the same file: always define the type and function **signatures**, and gate only the arch-specific (half2-intrinsic) **bodies**. Now `Acc2<half>` is always a complete type and the `half` overloads always exist, so `MatMulFloat4BatchedKernel<half>` compiles for `sm_52`. On archs below `sm_53` the bodies compile to no-ops (returning zeros), which matches the already-shipped `nv_bfloat16` behavior for archs below `sm_80`. ### Motivation and Context The CUDA packaging pipeline builds with `CMAKE_CUDA_ARCHITECTURES=52-real;61-real;75-real;86-real;89-real;90-virtual`, so the `sm_52` device pass is required and the pipeline was broken by microsoft#29451.
### Description Move the `requests` imports in `convert_tf_models_to_pytorch.py` into the two download paths that actually use them (`download_compressed_file` and the non-archive path in `download_tf_checkpoint`). This lets the archive-extraction tests import the module without requiring `requests` to be installed. ### Motivation The transformer conversion tests import the converter module at pytest **collection** time (`from convert_tf_models_to_pytorch import safe_extract_archive`) purely to exercise `safe_extract_archive`. However, `requests` is **not** a declared dependency of the transformers tests — `onnxruntime/python/tools/transformers/requirements.txt` lists `onnx`, `numpy`, `transformers`, `torch`, etc., but not `requests` (it is normally present only transitively via `transformers`). When `requests` is absent, the module-level `import requests` raises `ModuleNotFoundError` at collection time, which fails the `safe_extract_archive` traversal/symlink tests even though those tests never touch the download paths that use `requests`. Deferring the import into the download functions decouples the archive-safety tests from this undeclared optional dependency. Note: this is about dependency *availability*, not import speed — importing `requests` is cheap (its own module code is ~1 ms; the full transitive tree is a one-time ~0.25 s cold, mostly stdlib). Adding `requests` to the transformers test requirements would be an equivalent fix; the lazy import is just the smaller, more localized change. ### Testing - Reproduced the original failure in a venv without `requests`: importing the module raises `ModuleNotFoundError: No module named 'requests'`. After moving the imports, `safe_extract_archive` imports and runs without `requests`. - Direct `safe_extract_archive` tar/zip traversal checks passed with `.\.venv\Scripts\python.exe`. - `.\.venv\Scripts\python.exe -m pytest onnxruntime\test\python\transformers\test_convert_tf_models_to_pytorch.py -q` was attempted locally but collection requires `torch`, which is not installed in this venv. Co-authored-by: Gopalakrishnan Nallasamy <gnallasamy@microsoft.com>
… flag to support Gemma4 (microsoft#29392) ## Summary These four ops were forcing CPU fallback when inputs were INT64, preventing WebGPU graph capture. Converted from static macro registration to the factory function pattern (same as Cast/Unsqueeze/Expand/Range) so INT64 type constraints are included when `enable_int64=true`. `enable_int64` is always `true` when `enable_graph_capture=true` — `GetKernelRegistry` calls `RegisterKernels(true, true)` unconditionally in that path — so this fix has no effect on the default (non-graph-capture) execution path. ## Changed ops | Op | Versions | |---|---| | `Equal` | 7–10, 11–12, 13–18, 19+ | | `Sub` | 7–12, 13, 14+ | | `Where` | 9–15, 16+ | | `ReduceSum` | 1–10, 11–12, 13+ | ## Approach Each op follows the established factory function pattern: - `CreateXVersionedKernelInfo<Start, End>(bool enable_int64)` / `CreateXKernelInfo<Since>(bool enable_int64)` - Removed from static `build_kernel_create_info_function_table[]` - Registered in `RegisterKernels()` alongside Cast/Unsqueeze/Expand/Range `ReduceSum` version 13+ retains `InputMemoryType(OrtMemTypeCPUInput, 1)` (axes input must be on CPU) — unchanged from the original registration. `Where` factory functions delegate to `GetOpTypeConstraints(enable_int64, /*enable_bool=*/true)` rather than a local duplicate type list. All three files (`where.cc`, `binary_elementwise_ops.cc`, `reduction_ops.cc`) use stack-allocated `KernelDefBuilder()` consistent with other WebGPU factory functions, and drop the unused `webgpu_execution_provider.h` include. ## Runtime fixes for INT64 inference `BinaryElementwise` (`Sub`, `Equal`) and `Where` required additional shader-level fixes to correctly handle INT64 tensors at inference time: - **Vectorization disabled for INT64**: INT64 is stored as `vec2<u32>` (8 bytes/element) and has no vec4 representation. Vectorization is now disabled when either input is INT64, and `component=1` is used for correct buffer binding. - **Correct dispatch size**: `vec_size` is computed as element count (not rounded-up vec4 count) for INT64 outputs. - **Scalar input path**: The scalar broadcast path no longer applies an extra `.x` dereference on top of `GetByOffset`, which already extracts the `i32` element for INT64. - **Where non-broadcast path**: Non-broadcast INT64 `Where` now uses the element-per-thread shader path instead of the vec4 fast path that would truncate values to i32. - **Cache key**: `is_int64` is included in the `Where` cache hint, preventing float and INT64 shaders from incorrectly sharing a cached entry. - **Shape indices for INT64**: Shape indices are registered for non-broadcast INT64 `Where` so `BroadcastedIndicesToOffset` has the required helpers available. - **Equal INT64 with bool output**: The element-wise path now explicitly reads 4 consecutive INT64 elements per thread and packs them into a `vec4<bool>` before writing to the bool output buffer. Previously, only element 0 was read and broadcast across all 4 positions of the comparison, giving wrong results. ## Testing - All 1174 nodes of a Gemma4 WebGPU decoder model assigned to `WebGpuExecutionProvider` (previously ~20 nodes fell back to CPU due to INT64 Equal/Sub/Where/ReduceSum) - End-to-end graph capture inference: ~87 tok/s on Gemma4-E2B-it INT4 - `USE_WEBGPU`-guarded unit tests added for all four ops with `kEnableInt64=1`: `Sub_int64_webgpu`, `Equal_int64_webgpu`, `Where_int64_webgpu`, `ReduceSum_int64_webgpu` ## Related - Indirect dispatch fix (prerequisite for graph capture): microsoft#29236 - onnxruntime/mobius#380 --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…icrosoft#29545) ### Description Official Windows DLLs embed the wrong version in their VersionInfo resource, for example File version 1.27.26.615 for the 1.27.0 release. The cmake fix from microsoft#24606 is intact but only covers local builds; official packages are produced by the ADO pipelines, which take a different path in tools/ci_build/build.py. When Build_BuildNumber and Build_SourceVersion are set, that branch stamps VERSION_BUILD_PART from the last two digits of the build year and drops the patch version from the version string entirely. I checked the shipped 1.22.0, 1.23.0, 1.26.0, and 1.27.0 Windows packages and all four carry the wrong version, so this path was never fixed. This change parses major.minor.patch from VERSION_NUMBER, stamps the real ORT version in the numeric FILEVERSION (keeping MMDD as the fourth part for build traceability; it fits a 16-bit WORD), and includes the patch in the version string, which becomes for example 1.27.0.20260615.4.8f0278c. Local builds are unaffected. One open question for reviewers: whether you would rather have the fourth numeric part be the pipeline revision or zero instead of MMDD. ### Motivation and Context Fixes microsoft#29536. Anyone inspecting DLL properties, or tooling that reads VersionInfo to identify the installed ONNX Runtime version, currently sees a build-date artifact instead of the actual release version. Verified by executing the pristine and patched stamping logic with the real 1.27.0 release inputs (the pristine output reproduces the issue exactly), and by compiling onnxruntime.rc with the Windows SDK rc.exe under both define sets and decoding VS_FIXEDFILEINFO from the .res: 1.27.26.615 before, 1.27.0.615 after. A full pipeline build was not run; confirming end to end means building with Build_BuildNumber and Build_SourceVersion set and checking the DLL's FileVersionRaw.
### Description <!-- Describe your changes. --> Enables `GetModel()` in extended minimal builds. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes microsoft#29386. Extended minimal builds with the Core ML Execution Provider fail because getting the model metadata in coreml_execution_provider.cc on L63 requires Graph::GetModel(), which is only available in full (non-minimal) builds.
…wn (microsoft#29149) Recover Conv/ConvTranspose rank from weight when input shape is unknown, enabling layout transformation to NHWC for more nodes. ### Description The layout transformer skips converting a node to NHWC when input[0] has no inferred shape. For Conv and ConvTranspose operators, the data input (input[0]) and the weight (input[1]) always share the same rank. When the input rank is unknown, recover it from the weight. ### Performance Impact Measured on Kokoro-82M-v1.0-ONNX text-to-speech model ([onnx-community/Kokoro-82M-v1.0-ONNX](https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX)) with WebGPU ep, | Platform | Latency reduction | Speedup | |:------------------:|:-----------------:|:-------:| | Intel Wildcat Lake | −32.0% | 1.47× | | Intel Panther Lake | −20.0% | 1.25× | This change yields a 1.2–1.5× speedup on the Kokoro-82M text-to-speech model. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…Bits bias (microsoft#29596) ### Description Fixes a build break in the internal CUDA-EP unit-test module (`onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON`). `TryMatMulNBits` in `matmul_nbits.cuh` is a function template: ```cpp template <class T> bool TryMatMulNBits(int bits, T* output, const T* a_data, const uint8_t* b_data_quant, const T* scales_data, const uint8_t* zero_points, const T* bias_data, int m, int n, int k, int block_size, size_t shared_mem_per_block, cudaStream_t stream); ``` The MoE router bias fusion change (microsoft#29170) added the `const T* bias_data` parameter. The internal CUDA-EP test `fpA_intB_gemm_kernel_test.cc` calls this with a bare `nullptr` for `bias_data`. Since `nullptr` has type `std::nullptr_t` (not a pointer), it fails to match `const T*` during template argument deduction, even though `T` is deducible from the other arguments. GCC reports: ``` error: no matching function for call to 'TryMatMulNBits(...)' note: mismatched types 'const T*' and 'std::nullptr_t' ``` This is compiled only when `onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON`, so the internal CUDA-EP unit-test build leg is currently broken on main. ### Key Changes - Cast the `bias_data` argument at the call site to `static_cast<const AType*>(nullptr)` so `T` deduces correctly and the existing no-bias path is exercised unchanged. ### Motivation and Context Restores the ability to build and run the internal CUDA-EP unit tests (e.g. `./onnxruntime_provider_test --gtest_filter=CUDA_EP_Unittest.*`). ### Validation - Reproduced the exact "no matching function / mismatched types 'const T*' and 'std::nullptr_t'" deduction failure with a minimal template repro; confirmed the typed-nullptr cast compiles cleanly.
…rosoft#29593) ## Summary - Fix an out-of-bounds/data-race in the WebGPU GQA decode split-reduce shader (`flash_attention_decode_qkv.wgsl.template`) that corrupts the upper half of each head when `v_head_size_vec < tile_size_k_vec`. - Correct 16 stale `seqlens_k` values in the WebGPU `GroupQueryAttention` op tests (`group-query-attention.jsonc`). ## Motivation The experimental "Run ort-web tests - WebGPU EP" CI stage had 9 failing `GroupQueryAttention` op tests (tensor mismatches). Root cause is two independent bugs: **1. Kernel out-of-bounds race.** In the fused QKV decode shader, the V-multiply reduction writes `tile_output[m][k + local_idx]` (where `tile_output` is sized `v_head_size_vec`) but is guarded only by `if (local_idx < tile_size_k_vec)` (`tile_size_k_vec = 8`). When `v_head_size_vec < tile_size_k_vec` (e.g. `head_size = 8` → `v_head_size_vec = 2`), threads with `local_idx` in `2..7` compute an out-of-bounds index. WGSL clamps OOB array indices to the last valid element, so those threads all race on `tile_output[m][1]` with a `+=`, corrupting the second head_size vec4. The manifestation is GPU-dependent (zeros on some GPUs, off-by-8 stale values on the CI GPU); the first vec4 stays correct. Fix: add `&& k + local_idx < v_head_size_vec` to the reduction guard in both the TurboQuant and non-TurboQuant branches. **2. Stale test data.** 16 cases in `group-query-attention.jsonc` set `seqlens_k = total_sequence_length`, but the canonical GQA convention is `seqlens_k = total_sequence_length - 1` (0-based past length), as used by the C++ GQA tests and enforced by the CPU kernel bounds check. This produced an off-by-one KV read on WebGPU and `seqlens_k out of range` errors on the CPU/wasm backend. The stored expected outputs were already correct. ## Test plan - [x] Built wasm + WebGPU EP (`--build_wasm --use_webgpu --use_webnn`) and deployed to `js/web/dist`. - [x] Ran `npm test --webgpu-ep -- op group-query-attention -b=webgpu`: all `[webgpu]` and `[wasm]` GQA cases pass (previously 9 `[webgpu]` tensor mismatches + CPU `seqlens_k` errors). - [x] Confirmed the kernel fix alone reduced `[webgpu]` failures 16 → 9, and the test-data fix cleared the remaining off-by-one cases on both backends.
…icrosoft#29576) ### Description `Java_ai_onnxruntime_OrtTrainingSession_evalStep` acquires the output handle array with `GetLongArrayElements(jniEnv, outputHandlesArr, NULL)` but never calls the matching `ReleaseLongArrayElements`, so the pinned array (or its native copy, depending on the JVM) leaks on every call and on all return paths: the `EvalStep` error `goto`, the output-conversion failure, and the normal success return. This adds the release with `JNI_ABORT` immediately after the handle pointers are copied into `outputValues`, before `EvalStep` is invoked, so it covers every subsequent path. `JNI_ABORT` is correct because the handles are only read. The change mirrors the input handling a few lines above in the same function, and the sibling `trainStep()` and `OrtSession.run()` bindings, which already release their output handles this way. Inference results are unaffected, which is why functional tests do not catch the leak. ### Motivation and Context Fixes microsoft#29573. `evalStep()` was the only one of the three equivalent bindings missing the release, so repeated evaluation (for example a validation loop) accumulated native memory and GC pressure.
… attention (microsoft#28963) ## Description The quantized KV-cache flash-attention path for the CPU `GroupQueryAttention` contrib op carried two latent batch-stride bugs that produced incorrect results for `batch_size > 1`. This PR ports the fixes already landed for the FP32 path (PR microsoft#28962) to the quantized path, and adds a regression test that exercises the previously-uncovered scenario. ## Summary of Changes ### Bug fixes | File | Change | |------|--------| | `onnxruntime/core/mlas/inc/mlas_qkv_quant.h` | Add `q_batch_stride` field to `MlasFlashAttentionQuantizedKVArgs` so the kernel uses a caller-supplied Q batch stride instead of assuming the unpacked `num_heads*S*H` layout. | | `onnxruntime/core/mlas/lib/flashattn_qkv.cpp` | Use `args->q_batch_stride` for the Q pointer in both the main tiled kernel and the flash-decoding kernel; compute the attention-bias batch stride from `bias_head_extent = broadcast_head ? 1 : num_heads` (two sites) instead of always using `num_heads`. | | `onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h` | Set `q_batch_stride` to `(num_heads + 2*kv_num_heads)*S*H` for packed QKV (else `num_heads*S*H`) in the unified dispatch; correct the per-batch Q offset and bias slice to use the packed stride and head extent. | ### Tests - Add `test_int8_bias_broadcast_head_multi_batch` to `test_gqa_cpu_quantized.py` covering `[B, 1, S, T]` bias with `batch_size > 1`. The existing `test_int8_bias_broadcast_head` used `batch_size == 1`, which masked the head-broadcast batch-stride bug. ## Testing - `cd onnxruntime/test/python/transformers && python -m pytest test_gqa_cpu_quantized.py -q` → 21 passed, 2 skipped. - Verified the new test catches the bug: temporarily reverting the bias-stride fix makes `test_int8_bias_broadcast_head_multi_batch` fail (mismatches starting at batch index 1) while the `batch_size == 1` variant still passes; restoring the fix makes all tests pass. - `lintrunner -a` → no lint issues. ## Motivation and Context Follow-up to PR microsoft#28962, which fixed the identical two bugs (packed-QKV Q batch stride and attention-bias head-broadcast batch stride) in the non-quantized FP32 CPU GQA path. The quantized path was left untouched there as out of scope; existing quantized parity tests did not cover `batch > 1` with `[B, 1, S, T]` bias, so the bugs went undetected.
…acy directory load path (microsoft#29501) ### Description Two changes to the model package flow, enabled by the file-path external initializers support: **1. Fold `external_data` into `session_options` with path resolution.** - Path-valued session options (an allowlist: `session.model_external_initializers_file_folder_path` and `ep.context_file_path`) are resolved against the package (`sha256:<hex>`, relative, or absolute) at variant-parse time, using the same rules as `model_file`. - The dedicated `external_data` variant field and its special session injection are removed; the model is always loaded from the selected variant path. - On the advanced path (caller supplies their own `OrtSessionOptions`), path-valued options are carried over from the variant for keys the caller did not set, so a model that needs its external-initializers folder still loads. **2. Remove the legacy directory-based `CreateSession(package_dir)` path.** - Removes the `is_directory(package_root)` branch in `CreateSessionAndLoadModelImpl`. Model packages are loaded through the experimental `OrtModelPackageApi` (`CreateModelPackageContext` -> `SelectComponent` -> `CreateSession`). - The three end-to-end tests that exercised the directory path are migrated to the experimental API via a `CreateSessionFromModelPackage` test helper, preserving coverage for factory-based selection, `PREFER_CPU` policy selection, and compiled-model compatibility scoring. ### Motivation and Context The file-path external initializers folder support (microsoft#29459) lets the model package flow drop its workaround of memory-mapping the model and forcing a buffer load: it sets the folder option and loads from the selected variant path directly. Building on that, the dedicated `external_data` field folds into the general session-options mechanism. A variant declares `session.model_external_initializers_file_folder_path` (or other path-valued options) in its `session_options`, and ORT resolves those values against the package at parse time. This removes special-case code and lets other path-valued options such as the EPContext file path be resolved the same way. An allowlist is used rather than value-syntax sniffing because session-option values are arbitrary strings; only known path-valued keys are resolved, so ordinary values pass through untouched. The legacy directory-based `CreateSession(package_dir)` path was meant to be removed when the experimental `OrtModelPackageApi` landed. It only did variant/EP selection and never merged the variant's `session_options`/`provider_options`, so a package loaded via `Ort::Session(env, dir, so)` silently ignored its manifest session options, inconsistent with the experimental API. Removing it leaves a single, consistent way to load a model package. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# PR: Clamp 1D attention mask_index to valid bounds ## Description The CPU attention helper translates a 1D `mask_index` (right-side end position, and optionally a left-side start position) into per-token mask-fill loop bounds. The existing `std::max(0, std::min(...))` clamp was correct but verbose; this PR replaces it with `std::clamp` and adds a regression test that exercises out-of-range mask values. This hardens the kernel against out-of-bounds writes when callers supply mask positions outside `[0, all_sequence_length]`. ## Summary of Changes ### Bounds clamping | File | Change | |------|--------| | `onnxruntime/contrib_ops/cpu/bert/attention_helper.h` | Use `std::clamp(static_cast<int>(mask_index[...]), 0, all_sequence_length)` for both `end_position` and `start_position` in the 1D `mask_index` branch of `PrepareMask`. | ### Test coverage | File | Change | |------|--------| | `onnxruntime/test/contrib_ops/attention_op_test.cc` | Add `AttentionMaskIndex1DClampOOB`, covering an over-large `end_position` (999) and a negative `start_position` (-5). Both clamp cleanly and produce the fully-unmasked output. | ## Testing - Build and run the contrib attention tests: `onnxruntime_test_all --gtest_filter='ContribOpAttentionTest.AttentionMaskIndex1DClampOOB'` - The new test asserts that an out-of-range end position (above `all_sequence_length`) applies no right-side masking, and a negative start position applies no left-side masking — matching a fully-unmasked sequence. - Existing `ContribOpAttentionTest.*` cases continue to pass; behavior for in-range mask values is unchanged. ## Motivation and Context `end_position`/`start_position` drive raw index loops into the mask buffer. An unclamped value past `all_sequence_length` (or a negative start) could otherwise step outside the intended range. `std::clamp` guarantees both stay within `[0, all_sequence_length]`, preventing out-of-bounds writes with no behavior change for valid inputs. ## Checklist - [x] Tests added/updated - [ ] Documentation updated (if applicable) - [x] No breaking changes (or documented in description) - [ ] CI passes
…-overflow (microsoft#29443) This pull request improves the safety of buffer size calculations in the `SamplingState` initialization logic by ensuring that all multiplications involving `batch_size` and `vocab_size` are safely performed using `SafeInt<size_t>`. This prevents potential integer overflow bugs that could lead to under-allocated buffers and memory errors. **Buffer allocation safety improvements:** * All buffer size calculations that multiply `batch_size` and `vocab_size` now use `SafeInt<size_t>` to ensure checked arithmetic, preventing silent integer overflows that could cause heap-buffer-overflow issues. This includes allocations for both CPU and CUDA buffers in `SamplingState`. [[1]](diffhunk://#diff-ad3815054e84321b726b1e4c36d32cf2ab224301f699094f33b6ffd81b91eb64L25-R48) [[2]](diffhunk://#diff-ad3815054e84321b726b1e4c36d32cf2ab224301f699094f33b6ffd81b91eb64L52-R58) * The calculation for the buffer size of `h_sampled_all` now also safely casts `max_iter` to `size_t` before multiplication, further protecting against overflow. These changes make the code more robust and secure, especially when handling large or model-controlled input sizes.
…t#29445) MatMulNBits::PrePack ran at session initialization and called the MLAS pack routines using byte counts derived from the node attributes (N, K, bits, block_size) without ever comparing those attributes to the actual tensor Shape(). A crafted .onnx whose attributes overstate the real B (or scales / zero_points) extent triggered a heap-buffer-overflow READ inside MlasQNBitGemmPackQuantBData / MlasLutGemmPack during OrtApis::CreateSession (no Run() required). The canonical shape check already lives in matmul_nbits_helper::CheckInputs, but is invoked only from Compute() -- after PrePack has already done the OOB read, and by then the original B tensor is replaced with nullptr in the kernel context so the Compute-time check never re-validates it. Fix: at the top of PrePack, after the existing early-return guards and before any tensor.DataRaw() read, validate the incoming initializer's Shape() against the attribute-derived shape: - B -> (N, k_blocks, blob_size) - scales -> (N * k_blocks) or (N, k_blocks) - zero_points -> uint8: (N * zp_blob) or (N, zp_blob); else (N * k_blocks) or (N, k_blocks) A mismatch returns INVALID_ARGUMENT so the session fails to load rather than reading past the buffer.
This pull request improves the validation logic for the RotaryEmbedding operator to prevent out-of-bounds reads when the rotary embedding dimension derived from `cos_cache` exceeds the input tensor's `hidden_size`. It also adds dedicated unit tests to verify that this validation triggers as expected. **Validation improvements:** * Added a check in `rotary_embedding_helper.h` to ensure that the effective rotary embedding dimension (`cos_cache` width × 2) does not exceed `hidden_size` when `rotary_embedding_dim` is 0, returning an error if the condition is violated. **Unit test additions:** * Added `ContribRotaryEmbedding_RejectsCosCacheExceedsHiddenSize` test in `rotary_embedding_op_test.cc` to verify that an invalid configuration is correctly rejected in the contrib op. * Added `RotaryEmbedding_RejectsCosCacheExceedsHiddenSize` test in `rotary_embedding_op_test.cc` (providers/cpu/llm) to verify the same validation in the mainline op.
### Description <!-- Describe your changes. --> Prevents buffer overflow for conv node when input images are too large ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes microsoft#29130 The ReduceMean hypothesis in the issue was wrong and this addresses the actual root cause --------- Co-authored-by: Prathik Rao <prathikrao@microsoft.com>
…icrosoft#29574) ### Description Remove the blanket version check in CustomOpKernel that prevented custom ops compiled against a newer ORT from loading on an older runtime. Instead, cap the version passed to GetApi() at ORT_API_VERSION. This aligns custom ops with the EP plugin ABI pattern where forward compatibility is supported via runtime version detection. Individual newer functions in OrtCustomOp (CreateKernelV2, InferOutputShapeFn, GetMayInplace, etc.) are already gated by per-function version checks throughout custom_ops.cc, making the blanket reject both redundant and harmful. The EP is responsible for not calling API functions newer than the runtime version -- the same contract as the EP plugin interface. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ety (microsoft#29584) ### Description Three related fixes in the QMoE (`contrib_ops/cuda/moe`) weight-only GEMM profiling path that can cause a sticky `CUDA 700` (illegal memory access) surfacing at a later MoE kernel launch: - **`moe_kernels.cu`** — `GemmProfilerBackend::runProfiler` now keys its workspace layout on `mSM >= 90`, the same key used by `getWorkspaceSize` / `prepareRouting` / `prepareQuantParams` / `prepareTmaWsInputs`. Previously it keyed on the per-tactic `is_tma_warp_specialized`, which diverges on SM90 when a non-TMA (Ampere-fallback) tactic is profiled (e.g. INT4 mixed-input GEMM tiles). That divergence drops the `tma_ws_input` region and shifts every subsequent sub-buffer offset, so the profiler reads `expert_first_token_offset` and the GEMM inputs from the wrong (random-filled) locations → OOB read in the grouped GEMM. - **`moe_gemm_profiler`** — `profileTactics`/`runProfiling` accept an optional `timing_stream`; the profiler runs on the caller-supplied compute stream when provided, so its kernels are strictly ordered with surrounding compute-stream work and share the temp allocator's stream context. Profiling on a private side stream races with the stream-aware temp arena, which can hand the same scratch block to a later compute-stream allocation (e.g. the real MoE workspace) while the profiler's grouped-GEMM kernels are still in flight. - **`moe_quantization.cc`** — Skip profiling while the compute stream is being captured into a CUDA graph (profiling launches kernels, records/synchronizes events, and allocates/frees scratch — all illegal during capture) and fall back to a config cached from an earlier non-capturing run, or the runner's default tactic. ### Motivation and Context These are latent correctness/stability bugs in the QMoE tactic profiler that manifest on SM90 (H200) and under CUDA graph capture. They are independent of the fpA_intB MatMulNBits work and are split out as a standalone fix. ### Testing - Built with `USE_FPA_INTB_GEMM=ON` (arch 80;90) on H200. - `test_qmoe_cuda.py` passes (SwiGLU int4/int8, FP16/BF16, multiple batch/seq shapes).
… MatMulNBits (microsoft#29585) ### Description Extends the CUDA fpA_intB weight-only `MatMulNBits` path (FP16/BF16 int4/int8) in three ways. All are gated behind the existing `ORT_FPA_INTB_GEMM` opt-in; default behavior is unchanged. **1. Native SM90 (Hopper) mixed-GEMM kernel** - `cutlass_extensions` `GemmFpAIntB::operator()`: reuse the SM80 (Ampere) mixed-GEMM kernel for `__CUDA_ARCH__ >= 900` (Hopper/Blackwell) instead of stubbing it out, so the SM80-compat dispatch produces a real kernel on Hopper (mirrors `MoeFCGemm::operator()`). - `fpA_intB_launcher_sm90.inl`: gate the host-callable launcher on `COMPILE_HOPPER_TMA_GEMMS` only (not `__CUDA_ARCH__` / `__NV_SASS_VERSION__`, which are unset in the host pass), so the native SM90 TMA/WGMMA launcher symbol is emitted instead of the "recompile with 90a" stub. - `CutlassFpAIntBGemmRunnerInterface`: add `setArch(int)` and `setUseSm90Native(bool)`; the runner routes SM90 to `sm90_dispatch` when native, else the SM80 compat path. - `matmul_nbits`: `FpAIntBPackingSmForKernel()` returns 90 for `weight_prepacked=2` on an SM90 device (else 80); `InitGemmProfiler` opts into the native kernel (sm==90) or forces the runner to SM80 (compat on Hopper) so tactic enumeration and workspace sizing match the dispatched kernel; `GemmIdCore` gains an `sm` field so SM80-compat and SM90-native tactics for the same shape are not confused; the GEMV is launched with the packing arch (not the raw device SM) to match the packed interleave layout. - `weight_prepacked=2` (SM90 layout) is now accepted (was reserved/rejected): requires an SM90 device and `block_size ∈ {64, 128}`. Contrib-op docs updated accordingly. **2. `block_size=32`** - Relax the group-size gates in the GEMV dispatcher, the CUTLASS fine-grained `can_implement` / kernel launcher, and the `MatMulNBits` eligibility check. The SM80 fine-grained kernel supports group size 32 natively. **3. Fused bias** - Drop the `!has_bias_` gate: a fused bias (input 5) is already supported by the GEMV, the SM80/SM90 CUTLASS epilogue, and the tactic profiler, so bias-bearing nodes (e.g. gpt-oss `qkv_proj`/`o_proj`) become eligible. ### Motivation and Context Enables the fpA_intB weight-only path for more shapes and for Hopper-native execution, unblocking int4/int8 models with fused bias and `block_size=32` on H200. Builds on microsoft#29499 (prepacked fpA_intB weights). ### Testing - Built with `USE_FPA_INTB_GEMM=ON` (arch 80;90) on H200. - C++ `onnxruntime_provider_test --gtest_filter='*MatMulNBits*:*FpAIntB*'`: 65 passed, 1 skipped. - `test_op_matmulnbits_prepacked_cuda.py`: 7 passed (int4/int8, SM80/SM90, bias parity vs runtime prepack).
…osoft#29606) Update pinned dependencies in the llama and phi2 transformer model requirements to their patched versions, clearing the flagged 1ES Component Governance alerts. Package changes: - onnx: 1.18.0 -> 1.22.0 (llama, phi2) - protobuf: 4.25.8 -> 6.33.5 (llama) onnx 1.18.0 -> 1.22.0 (patched in 1.21.0) resolves: - CVE-2026-27489: path traversal via symlink (arbitrary file read) - CVE-2026-34445: unsafe setattr in ExternalDataInfo (object property overwrite via crafted model) - CVE-2026-28500: onnx.hub.load() trust-check bypass via silent=True - GHSA-q56x-g2fj-4rj6: TOCTOU arbitrary file read/write in save_external_data protobuf 4.25.8 -> 6.33.5 resolves: - CVE-2026-0994: recursion-depth bypass in json_format.ParseDict() causing DoS (no fix in the 4.25.x line; patched in 5.29.6 / 6.33.5) Versions align with the onnx==1.22.0 / protobuf==6.33.5 pairing already used across the repo's CI and transformers-test requirements. Files: - onnxruntime/python/tools/transformers/models/llama/requirements.txt - onnxruntime/python/tools/transformers/models/phi2/requirements.txt
### Summary Two small, independent CI fixes that unblock currently-failing required pipelines. The NPM packaging pipeline's web e2e consuming test broke when a floating `vite` range pulled the just-released 7.3.x line, and the Python DML pipeline started failing on a MeanVarianceNormalization precision mismatch. Neither change affects runtime code. ### Key Changes | Pipeline | File | Change | Why | |---|---|---|---| | NPM packaging (`web-ci` → e2e) | `js/web/test/e2e/package.json` | Cap `vite` from `^7.1.12` to `>=7.1.12 <7.3.0` | The e2e runner uses a non-deterministic `npm install`, so `^7.1.12` floated onto vite 7.3.6 (the tarball the install aborted on). vite 7.3.0 also bumped esbuild `^0.25.0 → ^0.27.0`, pulling platform binaries not reliably mirrored in the internal feed. Capping below 7.3.0 keeps the bundler smoke test on the known-good line. | | Python DML | `onnxruntime/test/testdata/onnx_backend_test_series_filters.jsonc` | Exclude `^test_mvn_cpu` from the DML EP backend test list | `test_mvn` (MeanVarianceNormalization) fails on DML with 27/27 mismatched elements (max rel diff ~25) — a precision issue, not a functional regression. Filtering it matches how other DML precision/known-issue cases are already handled in this list. | ### Testing Notes - **NPM packaging**: re-run the web CI e2e step (`npm run test:e2e -- --browser=Chrome_default`); `npm install` in `build/js/e2e` now resolves vite to a 7.1/7.2 release instead of 7.3.6, so the install no longer aborts (and the Windows `npm warn cleanup ... EPERM` rollback noise disappears). - **Python DML**: re-run the DML python backend test job; `test_mvn_*` is now skipped for the DML EP alongside the existing excluded cases.
… on Linux (microsoft#29591) On Linux with the WebGPU EP, if `SessionOptionsAppendExecutionProvider` is called when no Vulkan adapter is available, a subsequent `OrtReleaseEnv` on the same thread hangs forever in a futex wait. ### Root causes **1. C++ exceptions thrown inside Dawn `WaitAny` callbacks (primary)** `ORT_ENFORCE` was called directly inside the `RequestAdapter`/`RequestDevice` callback lambdas. Dawn's `WaitAny` does not release its internal `EventManager` mutex on exception, so throwing through it leaves that mutex permanently locked. The deadlock fires later when `Cleanup()` calls `wgpuInstanceRelease` → `EventManager::ShutDown()` → tries to re-acquire the same mutex on the same thread. **2. Zombie `WebGpuContext` left in the factory map (secondary)** When `Initialize()` threw, the `WebGpuContext` entry remained in `contexts_` with `ref_count=1` and no owner — a resource leak that would also re-trigger the deadlock path at `Cleanup()`. ### Changes (`webgpu_context.cc`) - **Adapter callback**: replace the throwing lambda with a non-throwing one that writes into a local `RequestAdapterResult` struct; `ORT_ENFORCE` is moved to *after* `WaitAny` returns: ```cpp // Before — throws inside Dawn's callback dispatch: [](wgpu::RequestAdapterStatus status, wgpu::Adapter adapter, wgpu::StringView message, wgpu::Adapter* ptr) { ORT_ENFORCE(status == wgpu::RequestAdapterStatus::Success, ...); *ptr = std::move(adapter); }, &adapter // After — captures result, throws outside Dawn: [](wgpu::RequestAdapterStatus status, wgpu::Adapter adapter, wgpu::StringView message, RequestAdapterResult* result) { result->status = status; if (status == wgpu::RequestAdapterStatus::Success) result->adapter = std::move(adapter); else result->message = std::string{message}; }, &adapter_result // ORT_ENFORCE(adapter_result.status == ...) called here, after WaitAny ``` - **Device callback**: same pattern applied to `RequestDevice`. - **`CreateContext` cleanup**: wrap `Initialize()` in a `try/catch`; on failure decrement `ref_count` and erase the entry from `contexts_` if it reaches zero. ### Motivation and Context Fixes the hang reported when registering the WebGPU EP with no usable Vulkan adapter and then calling `OrtReleaseEnv`. The process would park on a futex with `__owner` set to its own TID — a non-recursive mutex locked twice on the same thread — and never exit. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
### Description Bump version to 1.29.0. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…osoft#29271) 1. Double-buffer the B tile in workgroup memory when type is float16. 2. Load A as vec4 when K % 4 == 0. ### Description <!-- Describe your changes. --> Optimize the Intel subgroup GEMM/MatMul kernels in the WebGPU EP with two changes, both gated on the Xe-3LPG architecture: vec4 A loads: when K is a multiple of 4, A is loaded from global memory as vec4 (4 consecutive K elements per load) via cooperative subgroup loading, instead of scalar loads. Double-buffered B tile: for fp16 B inputs, the B tile in workgroup memory is double-buffered, prefetching the next tile while computing on the current one. This overlaps global-memory load latency with compute and reduces from 2 workgroupBarriers per K-loop iteration to 1. Both `a_vec4` and `b_is_fp16` are added to the program CacheHint so distinct pipelines are cached per configuration. On other architectures the kernels fall back to scalar A loads and a single B buffer, preserving previous behavior for shared memory limitation and registers pressure. Measured speedups on Xe-3LPG (avg ~12.7%): Model Speedup jina-clip-v1-version-fp16 13.4% sd-v1.5-text-encoder-demo 15.9% florence-2-base-decoder-fp16 13.3% moondream2-vision-encoder-fp16 11.4% sd-turbo-text-encoder-fp16-demo-layernorm 9.6% ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> The Intel subgroup matmul kernels were memory-bound on A loads and incurred two workgroup barriers per K tile. Loading A as vec4 cuts the number of global-memory accesses, and double-buffering the B tile hides load latency behind compute, improving throughput on the fp16 vision/text-encoder workloads above without regressing other architectures.
…path (microsoft#29525) ### Description `com.microsoft.GroupQueryAttention`'s optional `attention_bias` input (input #10) is implemented by the CPU EP (microsoft#23944) and the WebGPU EP (microsoft#25285, microsoft#26769), but the CUDA EP rejects it at runtime. This PR wires it through. No new kernel is needed: the unfused GQA fallback added for microsoft#28195 calls `LaunchUnfusedAttention`, whose kernel already implements an additive bias with dim-0/dim-1 broadcast, per-batch `seqlens_k`, causal/sliding-window masking and softcap — the op just passed `attn_bias=nullptr`. The change is dispatch plumbing: - **`group_query_attention.cc`** — remove the blanket rejection; validate the bias element type; set `broadcast_attn_bias_dim_0/1` from the bias shape (the fields already exist on `AttentionParameters`); add `!has_attention_bias` to the XQA / cuDNN SDPA / flash / flash-fast-decode / MEA eligibility so bias-carrying nodes dispatch to the unfused fallback; set `data.attention_bias`. - **`attention_data.h`** — add the `attention_bias` pointer to `GroupQueryAttentionData`. - **`group_query_attention_impl.cu`** — pass the real pointer and broadcast flags in `UnfusedGqaAttention` (previously hardcoded `nullptr`/`false`). Why each fused path stays disqualified with a bias: - flash / flash fast-decode: `flash_api.h` has no bias parameter (same exclusion as MHA and the ONNX `Attention`-op CUDA kernel). - XQA: no bias parameter. - cuDNN SDPA: GQA's cuDNN path is bottom-right causal, which cuDNN documents as incompatible with a bias (`multihead_attention.cc` has the same restriction). - cutlass MEA: the wrapper computes the bias row stride from `kv_sequence_length`, which GQA sets to the KV-cache capacity (`seqlen_present_kv_cache`) rather than `total_sequence_length`, so rows would be misaligned under past/present buffer sharing. Left for a follow-up (needs an explicit `attn_bias_strideM` in `MemoryEfficientAttentionParams`). Kept `NOT_IMPLEMENTED` (explicit, clear errors instead of the previous blanket rejection): bias × quantized KV cache (unfused requires `T == U`), bias × smooth-softmax/head_sink. ### Tests New `TestGQAAttentionBias` in `test_gqa.py`: prompt and past/decode parity across packed/unpacked QKV, shared/separate KV buffer, rotary, odd head sizes (40/80), and a subsequent multi-token prompt. The bias is **non-zero random** so a kernel that silently ignores the input fails parity (the harness previously modeled a zeros bias). Also fixes the test graph builder declaring the bias input's last dim as the KV-cache capacity instead of `total_sequence_length` (the shape the op validates). Full `test_gqa.py` suite: 476 tests pass, no regressions (SM89, CUDA 12.8). ### Motivation and Context Fixes microsoft#29506. Transformers.js-exported speech models carry non-causal attention patterns as `attention_bias` and currently cannot run on the CUDA EP at all, while running fine on WebGPU and CPU: - [`onnx-community/Voxtral-Mini-4B-Realtime-2602-ONNX`](https://huggingface.co/onnx-community/Voxtral-Mini-4B-Realtime-2602-ONNX) (streaming ASR) - [`onnx-community/cohere-transcribe-03-2026-ONNX`](https://huggingface.co/onnx-community/cohere-transcribe-03-2026-ONNX) End-to-end validation with this patch on an RTX 4070 SUPER: the full Voxtral-Mini-4B-Realtime streaming pipeline (q4f16) transcribes correctly at **RTF 0.23** (31 s clip, 25.6 tok/s sustained decode, 6.4 GB VRAM) — on this workload the unfused-attention path outperforms the same model on the WebGPU EP (RTF 0.26). (While validating, an unrelated pre-existing issue surfaced: `GroupQueryAttentionFusion` breaks graphs whose GQA nodes carry >9 inputs — filed as microsoft#29524.) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.