Sync with Microsoft ONNX Runtime - 09072026#1194
Merged
Merged
Conversation
…ative (microsoft#29604) ### What Skip constructing the `MinLengthLogitsProcessor` when `eos_token_id` is negative. A negative `eos_token_id` is the "no-EOS" sentinel used by greedy/sampling generation (it defaults to `-1`). With no EOS token there is nothing to demote, so a MinLength processor built with a negative eos can only ever be a guaranteed no-op. This change guards its construction at the list level in `LogitsProcessorInitImpl` (`logits_processor.h`), so we do not build a processor that would do nothing. ```cpp if (parameters.min_length > 0 && parameters.eos_token_id >= 0) { ... // add MinLength processor } ``` ### Why This is a small **defense-in-depth / code-clarity** improvement, not a behavior change and not a correctness or security fix: - For a valid `eos_token_id >= 0`, behavior is unchanged — the processor is still constructed and enforces the minimum length exactly as before. - For a negative eos, the added guard skips a processor that would be a no-op anyway (`SetScore` already ignores negative token ids), so this is a minor performance and clarity optimization. - It mirrors the existing conditional-adds in the same function (e.g. `RepetitionPenalty`), keeping the construction logic consistent. CPU-only: the CUDA path is already inherently a no-op for a negative eos, so no CUDA code is touched. ### Tests Adds 4 unit tests in `min_length_logits_processor_test.cc` (run via `onnxruntime_provider_test --gtest_filter='*MinLengthLogitsProcessorTest*'`): - `SetScoreIgnoresNegativeTokenId` — documents the pre-existing inline backstop that ignores negative token ids. - `ListInitSkipsProcessorForNegativeEosTokenId` — drives `LogitsProcessorList::Init` with a negative eos and confirms a below-min-length run leaves scores unchanged (the processor is skipped as a guaranteed no-op). - `ListInitDemotesEosBelowMinLength` — positive control / enforcement path: with a valid eos the processor is constructed and demotes the eos score below min length. - `ListInitLeavesScoresUnchangedAtMinLength` — at the min length, no demotion. The tests only reference exported (`LogitsProcessorList::Init`/`Process`) and header-inline symbols so they link in both static and shared-library builds. --------- Signed-off-by: Tita Wang <titaiwang@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…soft#29614) ### Description The SM80 fused MoE GEMM launchers were generated as two monolithic translation units — `fused_moe_gemm_sm80_bf16.generated.cu` and `fused_moe_gemm_sm80_f16.generated.cu` — each packing **30 CUTLASS kernel instantiations** (5 tile shapes × 3 stages × 2 epilogues) into a single `.cu` file. nvcc's `--threads` flag only parallelizes across GPU architectures (`80;90`) within a file; it does **not** parallelize template instantiations inside one translation unit. As a result each of these two objects was a long, serial compile on the build's critical path (each ~5–6 min in a Debug `80;90` build), and Ninja could only overlap the two of them. This PR updates the kernel generator to split the SM80 instantiations by tile shape, producing **10 smaller files** (5 tile shapes × 2 dtypes, 6 kernels each) that Ninja compiles in parallel. ### Key Changes - `generate_moe_kernels.py`: the SM80 loop now groups instantiations by `(element_type, tile_shape)` and emits one file per group, named `fused_moe_gemm_sm80_{dtype}_m{M}_n{N}_k{K}.generated.cu`. Added `glob`-based cleanup that removes stale SM80 generated files (the old monolithic per-dtype files and any files from a previous tile configuration) so they are not compiled. - Regenerated the launcher directory: removed the 2 monolithic files and added the 10 split files. Total number of compiled kernel instantiations is unchanged. cmake already discovers these via `GLOB_RECURSE ... CONFIGURE_DEPENDS`, so no cmake changes are needed — the new files are picked up automatically on the next configure. ### Testing - Verified generator idempotency (re-running reports all files up to date) and stale-file cleanup. - Built the new objects in a Debug `CMAKE_CUDA_ARCHITECTURES="80;90"` build: - one 6-kernel split object compiles in ~67s; - all 10 split objects build in ~182s wall-clock in parallel. - Previously the two 30-kernel objects were ~5× the per-file cost each and only 2-way parallel, so this substantially shortens the MoE portion of the critical path on multi-core machines. Total compiled kernels (and thus runtime coverage) is unchanged. To regenerate after future tile-shape changes: ``` python onnxruntime/contrib_ops/cuda/llm/generate_moe_kernels.py -a "80;90" -o onnxruntime/contrib_ops/cuda/llm/moe_gemm/launchers ```
…microsoft#29254) ### Description The `R_zero_point` / `R_scale` (recurrence) quantization-parameter shape validation in `DynamicQuantizeLSTM` was inadvertently checking the `W` (input) quantization parameters instead of the `R` ones. This change validates the `R` parameters' own shapes symmetrically with `W`, so malformed recurrence quantization parameters are rejected with a clear error. ### Changes - Fix the shape checks so `R_zero_point` and `R_scale` are validated against the `R` tensor's expected shape (previously bound to the `W` tensor). - Add two expect-failure unit tests covering inconsistent recurrence zero-point and scale shapes. ### Motivation Improves input validation and error diagnostics for malformed `DynamicQuantizeLSTM` recurrence quantization parameters. CPU-only; no behavior change for valid inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description
The `CropAndResize` CPU contrib op (`com.microsoft::CropAndResize`)
computes bilinear/nearest interpolation indices from the ROI box
coordinates. The bounds guards were written as `if (in_y < 0 || in_y >
height - 1)` / `if (in_x < 0 || in_x > width - 1)`. Because every
comparison involving a NaN is false, a non-finite (NaN or ±inf) ROI
coordinate slipped past both comparisons, skipped the extrapolation
`continue`, and reached the integer index computation
(`(int)floorf(NaN)`) with an invalid value.
This is a robustness/correctness gap: ROI coordinates are a runtime
input, and non-finite values are not handled gracefully.
### Fix
- **NaN-safe bounds guards.** Rewrite both guards into
negated-conjunction form: `if (!(in_y >= 0 && in_y <= height - 1))` /
`if (!(in_x >= 0 && in_x <= width - 1))`. This is logically identical
for all finite coordinates, but is `true` for NaN/±inf, so non-finite
coordinates now take the extrapolation branch and are filled with
`extrapolation_value` (matching the documented behavior for out-of-range
coordinates).
- **crop_size validation.** Add a Status-based
`ORT_RETURN_IF_NOT(crop_height > 0 && crop_width > 0, ...)` check in
`Compute` so non-positive crop sizes are rejected with a clear error
rather than producing degenerate work.
- **Index arithmetic hardening.** Use `SafeInt<int64_t>` for the
interpolation index computation, which allows the now-unnecessary MSVC
26451 (arithmetic-overflow) suppression pragma to be removed.
No behavior change for valid finite ROIs. CPU-only — there is no CUDA
`CropAndResize` kernel.
### Tests
Added to `onnxruntime/test/contrib_ops/crop_and_resize_op_test.cc`:
- NaN ROI coordinate → extrapolation (bilinear height, bilinear width,
and nearest).
- ±inf ROI coordinate → extrapolation.
- Finite boundary values (exact `[0, 1]` identity crop and just-outside
`1.0001`) — no regression.
- Non-positive `crop_size` (`{0, 2}` and `{-1, 2}`) rejected.
- Out-of-range batch index rejected.
Run with:
```
onnxruntime_provider_test --gtest_filter='CropAndResizeTest.*'
```
All 10 CropAndResize tests pass (5 new + 5 pre-existing, no regression).
An AddressSanitizer build can additionally corroborate the index
handling in CI.
### Motivation and Context
Handles non-finite ROI coordinates gracefully and makes the
CropAndResize index arithmetic robust, consistent with how out-of-range
coordinates are already treated (extrapolation).
Signed-off-by: Tita Wang <titaiwang@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on CPU-only Linux (microsoft#29590) ### Description `onnxruntime-gpu` 1.27 introduced a hard `NEEDED libcudart.so.13` entry in `onnxruntime_pybind11_state.so`, causing `ImportError` at `import onnxruntime` on CPU-only Linux machines — before any provider is selected. **Root cause:** `cmake/onnxruntime_python.cmake` was changed to compile `fpA_intB_gemm_adaptor.cu` and `fpA_intB_gemm_preprocessors_impl.cu` directly into `onnxruntime_pybind11_state.so` and link `CUDA::cudart` (dynamic). This embeds a load-time CUDA dependency in the Python module itself. **Fix:** Move the CUDA weight-preprocessing entry point (`pack_weights_for_cuda_mixed_gemm`) out of the main pybind module and into a **standalone extension module**, `onnxruntime_cuda_quant_preprocess`, that links `CUDA::cudart` on its own. The main `onnxruntime_pybind11_state.so` no longer compiles or links any CUDA code, so `import onnxruntime` has no `libcudart` dependency. The new module is imported **lazily** by `onnxruntime/python/tools/quantization/cuda_quantizer.py` only when weight prepacking is actually requested — never at `import onnxruntime` time. These preprocessing APIs are **offline-only** helpers: they are used by quantization tooling and model builders to produce prepacked weight initializers ahead of time, and are not part of the inference runtime hot path. Because nothing in the runtime imports them, isolating them into a separate, on-demand DLL has no runtime cost and cleanly keeps CUDA out of the base `import onnxruntime` path. **Why not the provider bridge:** An earlier iteration routed the call through the `ProviderInfo_CUDA` virtual interface (`TryGetProviderInfo_CUDA()`). That does not work for the CUDA-EP-as-plugin build (`onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`): `cuda_provider_factory.cc` is excluded from the plugin sources and there is no provider bridge, so `TryGetProviderInfo_CUDA()` returns `nullptr` and the call throws. The standalone module has no such dependency and works for **both** the legacy in-tree CUDA EP build and the plugin build. ### Key Changes | File | Change | |---|---| | `onnxruntime/python/onnxruntime_pybind_cuda_quant.cc` | **New.** Self-contained `pack_weights_for_cuda_mixed_gemm` (device malloc + transpose/convert + arch permutation) and a `PYBIND11_MODULE(onnxruntime_cuda_quant_preprocess, …)` entry point. | | `cmake/onnxruntime_python.cmake` | Add the `onnxruntime_cuda_quant_preprocess` module target (built when `onnxruntime_USE_CUDA AND NOT WIN32`, compiling the two `fpA_intB` `.cu` files + `CUDA::cudart` + cutlass, hidden visibility) and copy it into `onnxruntime/capi/`. Main pybind module keeps no CUDA sources/links. | | `onnxruntime/python/onnxruntime_pybind_quant.cc` | Remove the `USE_CUDA` `PackWeightsForMixedGemm` and its registration. The CPU-only `pack_fp4_weights_for_cuda_moe_gemm` stays in the main module. | | `onnxruntime/core/providers/cuda/cuda_provider_factory.{h,cc}` | Revert the `PackWeightsForMixedGemm` `ProviderInfo_CUDA` addition (no longer needed; absent in plugin builds). | | `onnxruntime/python/tools/quantization/cuda_quantizer.py` | `_get_pack_weights_for_cuda_mixed_gemm()` now imports `onnxruntime.capi.onnxruntime_cuda_quant_preprocess` lazily; add `has_cuda_weight_prepacking()` capability helper. | | `setup.py` | Package `onnxruntime_cuda_quant_preprocess.so` in the Linux/macOS wheels. | | `onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py` | Point the prepacked-weight parity test and its skip guard at the new module. | | `docs/contrib_ops/cuda/matmul_nbits.md` | Update the offline-packer code snippets to import the new module. | ### Motivation and Context `import onnxruntime` must succeed on CPU-only machines even when the GPU wheel is installed. CUDA dependency errors should surface only when a CUDA provider is explicitly loaded/selected, or when offline CUDA weight prepacking is explicitly requested. This restores the 1.26 behavior where `onnxruntime_pybind11_state.so` had no `NEEDED libcudart.so.*` entry, and — unlike the provider-bridge approach — it also works in the CUDA-EP-as-plugin build. ### Testing Notes - Built both modules in the CUDA build; `readelf -d onnxruntime_pybind11_state.so` shows **no** `libcudart` `NEEDED` entry, while `onnxruntime_cuda_quant_preprocess.so` has `NEEDED libcudart.so.13`. - `import onnxruntime` and lazy loading of `onnxruntime.capi.onnxruntime_cuda_quant_preprocess` both succeed; `has_cuda_weight_prepacking()` returns `True` on a CUDA machine. - `test_op_matmulnbits_prepacked_cuda.py` passes (INT4/INT8 prepacked-vs-runtime parity), confirming the relocated packer produces byte-identical prepacked weights. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.