Sync_msft_10_7_2026 by hdharpure9922 · Pull Request #1195 · intel/onnxruntime

hdharpure9922 · 2026-07-10T06:59:48Z

No description provided.

### Description Reduce per-call heap allocation overhead in the CPU `MatMul<float>` kernel for the small batched-GEMM case. `MatMul<float>::Compute()` builds an array of MLAS batch-parameter structs before calling `MlasGemmBatch` / `MlasSBGemmBatch`. Previously this always used `std::vector`, so even the common single-GEMM path performed one heap allocation per `Compute()` call. This change uses stack-backed `InlinedVector` storage when `helper.OutputOffsets().size() <= 2`, and keeps the existing `std::vector` path for larger batches: - `MLAS_SGEMM_DATA_PARAMS` and `MLAS_SBGEMM_DATA_PARAMS` (aarch64/Linux fast-math path) both use `InlinedVector<..., 2>` for `max_len <= 2`. - Larger batches fall back to `std::vector`, preserving prior behavior (and avoiding the `InlinedVector` overflow path, which I measured to regress for larger batches). The struct is ~64 bytes and trivially copyable, so the net effect is removing exactly **one `malloc`/`free` pair per `Compute()` call** in the small-batch case. ### Motivation and Context Many real CPU MatMul shapes flatten to `helper.OutputOffsets().size() == 1`. `MatMulComputeHelper` special-cases an activation × 2D-weight matrix into a single GEMM (offsets `{0}`), so shapes like the following all hit `max_len == 1`: ```text [M, K] x [K, N] [B, S, K] x [K, N] [B, H, S, K] x [K, N] ``` These appear in transformer projections, MLP layers, and classifier heads. Removing a heap allocation from that path is a small, low-risk latency improvement. ### Performance Measurements Measured on Windows (MSVC, `RelWithDebInfo`) with `onnxruntime_perf_test`, using 256-node **square** MatMul chains (`[N,N]·[N,N]`, which flatten to `max_len == 1` — the case this PR optimizes). 5 trials each; medians shown. `per-node saving` = `(base_p50 − opt_p50) / 256`. | N | base avg (ms) | opt avg (ms) | base p50 (µs) | opt p50 (µs) | per-node saving | |---:|---:|---:|---:|---:|---:| | 1 | 0.178246 | 0.177635 | 159.7 | 153.4 | 24.6 ns | | 8 | 0.189659 | 0.182407 | 167.7 | 158.4 | 36.3 ns | | 16 | 0.209381 | 0.194601 | 178.4 | 168.4 | 39.1 ns | | 32 | 0.307178 | 0.293141 | 279.4 | 269.7 | 37.9 ns | | 64 | 1.389432 | 1.362902 | 1371.2 | 1333.6 | 146.9 ns | | 128 | 3.403018 | 3.145609 | 3305.9 | 3123.3 | 713.3 ns | **Interpretation (honest scope):** - The benefit is a **fixed ~36–39 ns per `Compute()` call**, clearly and consistently visible at N=8/16/32 — exactly one `malloc`/`free` pair removed. - Because the saving is fixed, its true contribution shrinks as the GEMM grows: ~5–6% at N≤16, ~3.5% at N=32, but **well under 1% for N≥64**. - The larger apparent deltas at N=64/128 (147 ns / 713 ns per node) exceed what a fixed allocation can account for and are dominated by threaded-GEMM run-to-run variance (the baseline/optimized ranges overlap), so they should **not** be read as a real ~5% win. In short: this is a targeted micro-optimization that helps **tiny-GEMM-heavy** CPU workloads; for mainstream MatMul sizes the effect is below the measurement noise floor. It is intentionally low-risk rather than a broad throughput win. ### Validation - Built `onnxruntime_perf_test` separately for the baseline (`std::vector`) and optimized (`InlinedVector`) code for an apples-to-apples comparison. - Built `onnxruntime_provider_test` with the final change. - Focused MatMul tests pass: ```text onnxruntime_provider_test.exe --gtest_filter=*MatMul* [==========] 430 tests from 16 test suites ran. [ PASSED ] 430 tests. YOU HAVE 1 DISABLED TEST ``` --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>

…t#29566) ### Description Adds a CPU IOBinding test that makes the no-copy/in-place contract for preallocated outputs explicit. The new test: - builds the existing in-memory MatMul model used by `io_binding_test.cc`; - binds CPU inputs; - allocates a CPU output `OrtValue` and records its raw tensor buffer pointer; - binds that `OrtValue` as output `Y`; - runs the session with `IOBinding`; - verifies the returned output still points to the exact bound buffer and validates the numeric result. This guards the intended behavior that a preallocated CPU output is written in-place rather than ORT allocating a separate output buffer and copying into/returning that. ### Motivation and Context Preallocated I/O binding is one of the local zero-copy/copy-elision paths that can be validated without a GPU or multi-EP setup. Existing tests checked values; this adds a direct buffer-identity assertion so regressions that accidentally allocate/copy to a different buffer are caught. ### Testing Built `onnxruntime_provider_test` (Release, Windows) and ran: ```powershell .\onnxruntime_provider_test.exe --gtest_filter="InferenceSessionTests.TestBindCpu:InferenceSessionTests.TestBindCpuPreallocatedOutputUsesBoundBuffer" ``` Result: ```text [==========] 2 tests from 1 test suite ran. [ PASSED ] 2 tests. ``` Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>

…t#29625) ### Description  Mark every Dawn callback (RequestAdapter, RequestDevice, SetUncapturedErrorCallback, SetDeviceLostCallback, MapAsync, PopErrorScope, CreateComputePipelineAsync) noexcept and avoid throwing from inside them. A C++ exception unwinding out of a Dawn callback runs while Dawn holds internal locks and is undefined behavior, and can leave the EventManager mutex locked so a later instance release self-deadlocks. The MapAsync callbacks in BufferManager::Download and WebGpuContext::CollectProfilingData now write the status and message into a MapAsyncResult struct and the outcome is checked after Wait returns (ORT_THROW_IF_ERROR + ORT_ENFORCE). PopErrorScope builds and returns an onnxruntime::Status for both a failed pop and a validation error instead of enforcing inside the callback. ### Motivation and Context  Fix crashes caused by exceptions crossing the WebGPU callback boundary. Follow up to microsoft#29591 with similar changes for other callbacks. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

### Description Guards the CUDA GridSample coordinate conversion paths for non-finite and extreme grid values, matching the existing CPU behavior for float grids. The CUDA path now sanitizes coordinates before integer conversion, uses wider intermediate indices where needed, and clamps reflected indices before sampling. Per review feedback, `GsReflect` performs the `isfinite` check before computing the reflection range (`x_max - x_min`), so a non-finite coordinate returns early without the extra subtraction. The same reorder is applied to the CPU `GsReflect` (`core/providers/cpu/tensor/grid_sample.cc`) to keep the CPU and CUDA implementations in lockstep; behavior is unchanged. ### Tests - `.\.venv\Scripts\python.exe tools\ci_build\build.py --config RelWithDebInfo --build --parallel --target onnxruntime_provider_test --build_dir build\Windows` - `.\build\Windows\RelWithDebInfo\RelWithDebInfo\onnxruntime_provider_test.exe --gtest_filter=*Grid*` - 180 tests passed - `.\.venv\Scripts\clang-format.exe` on touched C++ files Note: the local build is CPU-only (`onnxruntime_USE_CUDA=OFF`), so CUDA compilation/runtime coverage will come from CUDA CI. ### Test coverage The hardening is applied at shared choke-points: coordinate sanitization runs before interpolation-mode dispatch, and the `int64_t` index widening and reflected-index clamps are single shared branches. So one representative case per {mode, padding, dimensionality} exercises the hardened path rather than the full cross-product. Regression tests use constant-valued images, so the expected output is well-defined regardless of which (now sanitized/clamped) index each adversarial coordinate resolves to; this also sidesteps a pre-existing CPU-vs-CUDA reflected-index difference (double-reflect + round-half-to-even). The new `GridSampleCudaHardeningTest` cases run on CPU and CUDA (and CUDA-NHWC when `ENABLE_CUDA_NHWC_OPS` is enabled); the CUDA kernel is registered for `float` only, so `double` is exercised through the shared templated CPU path. --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>

…ftest tool (--data_shape) (microsoft#29571) ### Description This PR adds a `--data_shape` flag to `onnxruntime_perf_test` app that enables profiling multiple input shapes within a single session. The model is compiled once and then run with each specified shape group in round-robin order, providing per-shape latency statistics (Avg, Min, Max, P50, P90, P95, P99). This is useful for evaluating dynamic-shape model performance across different input dimensions without needing to recompile or restart the session for each shape. Usage: #With -I (generate random default inputs of specified shapes): `onnxruntime_perf_test.exe -v -e cpu -m times -r 10 -I --data_shape "data:[1,3,60,10][1,3,60,20]" "model.onnx"` #Without -I (select matching test data folders): `onnxruntime_perf_test.exe -v -e cpu -m times -r 10 --data_shape "data:[1,3,60,10][1,3,60,20]" "model.onnx"` ### Motivation and Context Addresses the feature request: microsoft#28628 --------- Co-authored-by: n1harika <niharika.sathish@intel.com>

…c to 32 (microsoft#29586) ## Summary - Bump `FlashAttentionDecodeQKVProgram` workgroup size from 64 to 128. - Bump `tile_size_k_vec` from 8 to 32. - `sub_tile_count` is derived from `WorkgroupSizeX() / tile_size_k_vec`, so it goes from 8 to 4 automatically. The shader template expresses all loops in terms of these parameters, so no shader-side changes are needed. ## Motivation `FlashAttentionDecodeQKVProgram` was dispatching with a smaller workgroup / tile_size_k_vec than `MatMulNBitsProgram` on the same class of hardware. Mirroring the MatMulNBits dispatch shape improves occupancy and reduces the per-call cost of the fused QK^T + softmax + V multiply step during token generation. Measured on Qwen3-1.7B-graph-prune (WebGPU EP, D3D12, decode): | Metric | Before | After | |---|---|---| | `GroupQueryAttention\|FlashAttentionDecodeQKV` per-call GPU time | 0.83 ms | 0.47 ms | | Token generation throughput | 257 tps | 282 tps | ## Test plan - [x] Built ORT with WebGPU EP (Release, D3D12) on Windows. - [x] `onnxruntime_provider_test.exe --gtest_filter="GroupQueryAttentionTest.*"` — 52 passed, 12 CUDA-only tests skipped. All WebGPU FlashAttention paths pass, including `BatchedRightPaddedRotaryPrefillFlashAttention_WebGPU`, `BatchedRightPaddedRotaryPrefillFlashAttentionLargeSpread_WebGPU`, and `WebGPU_SharedKV_*` variants. - [x] End-to-end correctness on Qwen3-1.7B-graph-prune (decode). - [x] Profiled Qwen3-1.7B-graph-prune with the WebGPU profiling pipeline; confirmed the per-op time improvement above.

…icrosoft#29505) ### Description - Add xe-2hpg, xe-3lpg, and xe-3lpg-xs to the execution allowlist. - Gate im2col-matmul on fp16 as the kernel is tuned for it. - Fall back to default convolution for fp32 and other data types. ### Motivation and Context See above.

…ion (microsoft#29555) ### Description Adds error-path unit tests for `InferenceSession` validation / early-return branches, which were under-covered (only failure/`kExpectFailure`-style assertions were a small fraction of the suite). All nine target previously-untested branches and are verified passing against a local Release build. | Test | Branch exercised | Asserted message | |---|---|---| | `RunBeforeInitializeReturnsError` | `Run` when not initialized | `Session not initialized` | | `InitializeBeforeLoadReturnsError` | `Initialize` with no model loaded | `Model was not loaded` | | `RunWithInvalidOutputNameReturnsError` | `ValidateOutputs` unknown output | `Invalid output name` | | `RunWithWrongInputTypeReturnsError` | `CheckTypes` (input) | `Unexpected input data type` | | `RunWithWrongInputRankReturnsError` | `CheckShapes` (input rank) | `Invalid rank for input` | | `RunWithMismatchedFeedCountReturnsError` | feed/name count mismatch | `feed names has ...` | | `RunWithWrongOutputTypeReturnsError` | `CheckTypes` (output) | `Unexpected output data type` | | `LoadMalformedModelFromArrayReturnsError` | malformed model bytes | graceful failure | | `LoadNonexistentModelReturnsError` | missing model file | graceful failure | ### Motivation and Context Error/validation paths in `InferenceSession::Run` / `Initialize` / `Load` were thinly tested relative to the numeric happy-path bulk. These tests lock in the failure contracts (both the non-OK `Status` and the message substring, via `ASSERT_STATUS_NOT_OK_AND_HAS_SUBSTR`) so a regression that changes the behavior or the message surfaces clearly. ### Notes - `"Invalid input name"` is already covered by `TestOptionalInputs`, so it is intentionally not duplicated. - Test-only change (`onnxruntime/test/framework/inference_session_test.cc`); no product/runtime code is touched. --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com>

### Summary The ORT-Web JSEP pooling output-shape helper (`PoolConvUtil.computePoolOutputShape` → `computeShapeHelper` → `adjustPadAndReturnShape`) was **floor-only**: it had no `ceilMode` parameter and silently ignored `ceil_mode`. Under `ceil_mode=1` this allocated the output tensor one element too small along each affected spatial axis, producing a wrong output shape (and downstream shape mismatches) — independent of any pooling divisor concern. The same floor-only helper is duplicated in the legacy `js/web/lib/onnxjs/util.ts`; both copies are fixed. ### Fix - Thread a `ceilMode` parameter (default `0`, i.e. floor) through `computePoolOutputShape` → `computeShapeHelper` → `adjustPadAndReturnShape` in **both** `js/web/lib/wasm/jsep/util.ts` and `js/web/lib/onnxjs/util.ts`. - Route the NOTSET / VALID / SAME_UPPER / SAME_LOWER branches through a new `computeOutputSize()` helper that produces results identical to the C++ reference `PoolAttributes::ComputeOutputSize` (`onnxruntime/core/providers/cpu/nn/pool_attributes.h`), including the `ceil_mode` **"shrink the last window if it starts entirely in the trailing padding"** rule (ref: onnx/onnx#5741). - The floor path (`ceilMode` default) is algebraically unchanged, so Conv and `auto_pad` pad-adjustment are unaffected. - `js/web/lib/wasm/jsep/webgpu/ops/pool.ts` now passes `attributes.ceilMode` into the shape computation. ### Scope: SHAPE-only This PR fixes the **output-shape** computation only. End-to-end `ceil_mode` execution remains gated by the existing `throw` guards in `parseAveragePoolAttributes` / `parseMaxPoolAttributes`, because the WebGPU pooling kernel does not yet implement `ceil_mode` trailing-padding handling (and, for AveragePool, the `count_include_pad` divisor). Removing those throws + adding kernel support is a **tracked follow-up**; the now-correct shape path is exercised directly by the added unit tests until then. Comments at the throw sites and in the test header document this. ### Tests Adds `js/web/test/unittests/pool-output-shape.ts` (registered in `unittests/index.ts`), asserting output shapes for **both** implementations (jsep + onnxjs) against the C++ CPU reference ground truth: - `test_maxpool_2d_ceil` → `[1,1,2,2]`, `AveragePool_10_ceil1_2d` → `[1,1,2,3]` - AvgPool `ceil` 1D `[1,2,4]` / 2D `[1,1,3,3]` / 3D `[1,1,2,2,2]` (matching the CPU `pool_op_test.cc` cases) - `ceil_mode` with dilation, `VALID` / `SAME_UPPER` / `SAME_LOWER` auto_pad - a shrink-rule discriminator (naive `ceil()` would give 3; correct = 2) - floor-mode no-regression case ### Related - PyTorch context: pytorch/pytorch#183528 - Related ORT PR: microsoft#16752 ### Post-review fixups - **SAME_UPPER/SAME_LOWER integer division (external-review Major).** The `legacyTargetSize = (inSize + stride - 1) / stride` computation now uses `Math.floor(...)` to match C++ `pool_attributes.h` `ComputeSizePadDilations`, which uses integer division. This fixes a latent float-division divergence that mis-rounded the auto_pad pad distribution. The correction applies to **all** ceil modes (not only `ceil_mode=1`), aligning JSEP/onnxjs with the CPU/CUDA EPs. The floor-path (`ceil_mode=0`) **output shape is unchanged** — only the SAME_* pad split is corrected — so nothing previously-correct regresses. Added `ceil_mode=0` SAME_UPPER/SAME_LOWER non-divisible regression tests that assert the corrected pad out-param (SAME_UPPER → `[0,1]`, SAME_LOWER → `[1,0]`). - **Throw-message / TODO clarity.** The retained `ceil_mode` `throw` statements and the pool op TODO banner were reworded to make explicit that the output **shape** is now computed correctly, while `ceil_mode` **kernel execution** (padding/divisor) remains the pending WebGPU follow-up. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@apsonawane

### Description Follow-up to the merged microsoft#29605, addressing a review nit from @apsonawane on the base-offset index arithmetic in `CropAndResizeForward`. The suggestion was that the `static_cast<int64_t>(...)` around the per-channel base offset looked redundant. It turns out the cast could **not** simply be removed: the offset expression is a `SafeInt<int64_t>`, and `pointer + SafeInt<int64_t>` is ambiguous — `SafeInt` exposes many implicit conversion operators (`char`, `short`, `int`, …), so the operand for pointer arithmetic cannot be resolved unambiguously (and `SafeInt`'s own `operator+(U, SafeInt)` tries to treat the pointer as an integer). The cast was therefore load-bearing. To honor the readability intent without the awkward inline cast, this change hoists the offset into a named `int64_t` local in both the bilinear and nearest branches: ```cpp const int64_t bottom_data_offset = (SafeInt<int64_t>(roi_batch_ind) * channels + c) * height * width; const T* offset_bottom_data = bottom_data + bottom_data_offset; ``` Initializing an `int64_t` directly from the `SafeInt<int64_t>` expression is unambiguous (it selects `operator int64_t()`), so this compiles cleanly, keeps the `SafeInt` overflow checking on the offset computation, and reads more clearly than the inline cast. **No behavior change.** ### Motivation and Context Improves readability of the index arithmetic introduced in microsoft#29605 while keeping the overflow-checked offset computation intact. ### Tests Covered by the existing `CropAndResizeTest` suite — all 10 tests pass: ``` onnxruntime_provider_test --gtest_filter='CropAndResizeTest.*' ``` Signed-off-by: Tita Wang <titaiwang@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…in contrib Range shape inference (microsoft#29265) ### Summary The `com.microsoft` Range operator's shape-inference helper read a fixed number of bytes from an initializer's `raw_data` without first checking the buffer length, and the element-count computation could pass non-finite or out-of-range values to an `int64` cast. This change adds the missing validations and aligns the shape-inference and CPU kernel paths. ### Changes - `GetFirstElement` now checks `raw_data` length is at least the element size before reading, and reads via `std::memcpy` into an aligned local. - `CalcRangeDim` and the CPU kernel `ComputeRange` now reject non-finite computed counts, handle non-positive counts before the `int64` cast, and reject counts that are not representable as `int64` (`>= 2^63`). Both paths use identical messages and semantics. - The output dimension for empty/backward ranges is clamped to 0 in shape inference to match the kernel. ### Tests Added contrib Range model-load regression tests in `range_test.cc` covering: - truncated `raw_data` for `start`, `limit`, and `delta` (double, plus float and int64 element types), - zero delta, - a finite-but-too-large element count, - an exact-size success boundary, - backward-range zero-dimension inference. Tests assert on `Status` (safe for no-exception builds) and are guarded by `#ifndef DISABLE_CONTRIB_OPS` (throwing cases additionally by `!defined(ORT_NO_EXCEPTIONS)`). 21/21 `RangeTest` cases pass locally. ### Follow-up `int16` inputs are currently supported only via `raw_data` (there is no `get_data<int16_t>` specialization for the non-raw-data path); this is left as a separate follow-up. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jatinwadhwa921

LGTM

GopalakrishnanN and others added 12 commits July 8, 2026 15:51

Merge branch 'master' into sync_msft_10_7_2026

988c912

hdharpure9922 requested a review from jatinwadhwa921 July 10, 2026 07:49

jatinwadhwa921 approved these changes Jul 10, 2026

View reviewed changes

jatinwadhwa921 merged commit fab6702 into intel:ovep-develop Jul 10, 2026
4 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Sync_msft_10_7_2026#1195

Sync_msft_10_7_2026#1195
jatinwadhwa921 merged 12 commits into
intel:ovep-developfrom
hdharpure9922:sync_msft_10_7_2026

hdharpure9922 commented Jul 10, 2026

Uh oh!

jatinwadhwa921 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

hdharpure9922 commented Jul 10, 2026

Uh oh!

jatinwadhwa921 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants