Friendly ping on this one. @tianleiwu
Most CI is green; the only failure I see is windows_x64_asan, which appears to be an AddressSanitizer OOM in unrelated tests. Happy to rebase/update if preferred.
and would appreciate any guidance on whether this minimal fix is acceptable, or whether maintainers would prefer folding the follow-up bias-rescaling approach into this PR.

Hi @tianleiwu, this PR is blocked from auto-merge by a single failing check (windows_x64_asan / build_x64).Could you re-run the ASAN job? Thanks!

Hi @tianleiwu, this PR is blocked from auto-merge by a single failing check (windows_x64_asan / build_x64).Could you re-run the ASAN job? Thanks!

Done

Gentle ping - @tianleiwu. this PR is still blocked by a same single check. Could yo re-run this job again?

Gentle ping - @tianleiwu. this PR is still blocked by a same single check. Could yo re-run this job again?

Job re-tried. Hope that it can pass this time.

@tianleiwu Failed again😅. I think there is a st problem with it 🤔 .

@tianleiwu Failed again😅. I think there is a st problem with it 🤔 .

Last retry (failed 5 times). If it does not work, please merge/rebase main.

@tianleiwu Sadly failed at the same point. As you suggested, rebased onto current main. Sory for the trouble

Hi @tianleiwu, sorry to keep bothering you with this. After rebasing to the main, the CI still failed at the same point with the same ASAN OOM, so I suspect the added test coverage might be pushing it over the memory limit. I've reduced the new cases down to the minimal essential ones (keeping the bias + alpha != 1 regression checks). Could you trigger CI once more when you have a moment?

@elwhyjay,

Here is AI analysis of the failed CI job:

The failure is most likely caused by TransposeOptimizerTests.TestDequantizeLinearNoAxis, which crashes under ASan with OOM immediately after starting:

• Failing test in logs: TransposeOptimizerTests.TestDequantizeLinearNoAxis
• Failure: AddressSanitizer: Out of memory. The process has exhausted 8192MB
• Failing file: onnxruntime/test/optimizer/transpose_optimizer_test.cc at the test definition for TestDequantizeLinearNoAxis and its helper RunDequantizeLinearTestCase:
  • helper: RunDequantizeLinearTestCase at onnxruntime/test/optimizer/transpose_optimizer_test.cc:3669
  • failing test: onnxruntime/test/optimizer/transpose_optimizer_test.cc:3760
  • source: https://github.com/microsoft/onnxruntime/blob/bf7bc2d094eb19c144cf59168b2f489d45c79f56/onnxruntime/test/optimizer/transpose_optimizer_test.cc
• Job definition running ASan build+tests: .github/workflows/windows_build_x64_asan.yml:41
  • source: https://github.com/microsoft/onnxruntime/blob/bf7bc2d094eb19c144cf59168b2f489d45c79f56/.github/workflows/windows_build_x64_asan.yml

Root cause

TestDequantizeLinearNoAxis invokes RunDequantizeLinearTestCase for 6 variants:

• ONNX domain: uint8_t, uint16_t, int16_t
• MS domain: uint8_t, uint16_t, int16_t

In RunDequantizeLinearTestCase, all variants use TransformerTester(...) across multiple opsets:

std::vector<int> opsets = {15, 18, 21};
if constexpr (std::is_same_v<QuantType, uint16_t> || std::is_same_v<QuantType, int16_t>) {
  if (q_domain == kOnnxDomain) {
    opsets = std::vector<int>{21};
  }
}

So this single test expands into many ASan-heavy session builds. Since the log shows the crash happens exactly when TestDequantizeLinearNoAxis starts, the issue is test memory pressure, not a product correctness failure.

Best fix

Reduce the matrix for this specific no-axis test under ASan, especially for the 16-bit variants and/or MS-domain duplicates, because the behavior being validated is already covered by neighboring tests:

• TestDequantizeLinearScalarIgnoreAxis
• TestDequantizeLinearVector

A focused change is to keep one representative no-axis case and skip the redundant high-memory variants in ASan builds.

Suggested code change

Update TestDequantizeLinearNoAxis to limit coverage under ASan:

TEST(TransposeOptimizerTests, TestDequantizeLinearNoAxis) {
  std::optional<std::vector<int64_t>> zp_input_shape = std::vector<int64_t>{};
  std::vector<int64_t> zp_value_shape{};
  std::optional<ONNX_NAMESPACE::AttributeProto> no_axis;  // Empty axis value will not be set.

  RunDequantizeLinearTestCase<uint8_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain);

#if !defined(__SANITIZE_ADDRESS__)
  RunDequantizeLinearTestCase<uint16_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain);
  RunDequantizeLinearTestCase<int16_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain);
#if !defined(DISABLE_CONTRIB_OPS)
  RunDequantizeLinearTestCase<uint8_t>(zp_input_shape, zp_value_shape, no_axis, kMSDomain);
  RunDequantizeLinearTestCase<uint16_t>(zp_input_shape, zp_value_shape, no_axis, kMSDomain);
  RunDequantizeLinearTestCase<int16_t>(zp_input_shape, zp_value_shape, no_axis, kMSDomain);
#endif
#endif
}

More portable version for MSVC ASan

Because compiler macros differ on Windows, a safer pattern is to add a small local guard:

#if defined(__SANITIZE_ADDRESS__) || defined(ADDRESS_SANITIZER) || defined(_MSC_VER)
#define ORT_ASAN_BUILD 1
#endif

Then:

TEST(TransposeOptimizerTests, TestDequantizeLinearNoAxis) {
  std::optional<std::vector<int64_t>> zp_input_shape = std::vector<int64_t>{};
  std::vector<int64_t> zp_value_shape{};
  std::optional<ONNX_NAMESPACE::AttributeProto> no_axis;

  RunDequantizeLinearTestCase<uint8_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain);

#if !defined(ORT_ASAN_BUILD)
  RunDequantizeLinearTestCase<uint16_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain);
  RunDequantizeLinearTestCase<int16_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain);
#if !defined(DISABLE_CONTRIB_OPS)
  RunDequantizeLinearTestCase<uint8_t>(zp_input_shape, zp_value_shape, no_axis, kMSDomain);
  RunDequantizeLinearTestCase<uint16_t>(zp_input_shape, zp_value_shape, no_axis, kMSDomain);
  RunDequantizeLinearTestCase<int16_t>(zp_input_shape, zp_value_shape, no_axis, kMSDomain);
#endif
#endif
}

Preferred cleaner fix

If you want to preserve all coverage outside ASan without sprinkling guards in tests, add an optional opsets override or reduced-coverage mode to the helper and use it only here. Example:

template <typename QuantType>
static void RunDequantizeLinearTestCase(const std::optional<std::vector<int64_t>>& zp_input_shape,
                                        const std::vector<int64_t>& zp_value_shape,
                                        std::optional<ONNX_NAMESPACE::AttributeProto> axis,
                                        const std::string& q_domain = "",
                                        std::vector<int> opsets_override = {}) {
  ...
  std::vector<int> opsets = opsets_override.empty() ? std::vector<int>{15, 18, 21} : std::move(opsets_override);

  if (opsets_override.empty()) {
    if constexpr (std::is_same_v<QuantType, uint16_t> || std::is_same_v<QuantType, int16_t>) {
      if (q_domain == kOnnxDomain) {
        opsets = std::vector<int>{21};
      }
    }
  }

  TransformerTester(..., opsets);
}

Then:

TEST(TransposeOptimizerTests, TestDequantizeLinearNoAxis) {
  ...
#if defined(ORT_ASAN_BUILD)
  RunDequantizeLinearTestCase<uint8_t>(zp_input_shape, zp_value_shape, no_axis, kOnnxDomain, {21});
#else
  ...
#endif
}

Why this is the right solution

The workflow is an ASan job (windows_build_x64_asan), and the log shows the process dies from memory exhaustion rather than a semantic assertion failure. The fix should therefore reduce peak memory from this test’s combinatorial coverage, not change runtime code.

Hi @tianleiwu, thank you for the analysis! I went with option 1 (__SANITIZE_ADDRESS__ guard) because adding _MSC_VER would also disable the variants on non-ASan Windows builds, and MSVC 16.9+ defines __SANITIZE_ADDRESS__ under /fsanitize=address anyway. Also restored the alpha_not_one cases I had trimmed earlier.

Option 3 changes the helper signature and other call sites, so I left it out of this PR - but I can add it here if you'd prefer, or send it as a small follow-up. Thank you again!

Hi @tianleiwu, the rerun failed again with the same ASan OOM, but the failing test was TransposeOptimizerTests.TestReduceMaxKeepdimsFalse (line 1265), which runs before TestDequantizeLinearNoAxis (line 3760), so the guard never gets a chance to trigger. Looking at the previous failures too, the OOM consistently happens somewhere in the TransposeOptimizerTests block (lines ~1200–1700).
So it looks like the cumulative memory pressure builds up before reaching the guard. Could you advise on how you'd like to proceed?

Hi @tianleiwu, the rerun failed again with the same ASan OOM, but the failing test was TransposeOptimizerTests.TestReduceMaxKeepdimsFalse (line 1265), which runs before TestDequantizeLinearNoAxis (line 3760), so the guard never gets a chance to trigger. Looking at the previous failures too, the OOM consistently happens somewhere in the TransposeOptimizerTests block (lines ~1200–1700). So it looks like the cumulative memory pressure builds up before reaching the guard. Could you advise on how you'd like to proceed?

@elwhyjay, The solution is either reduce memory usage of the test case, or reduce test parallel in ASan to avoid OOM.

Here is a PR for the later: #28675. You can merge it to your branch and test it.

Hi @tianleiwu, thank you for #28675! I've merged the latest main (which now includes your --test_parallel 4 change) and also reverted the ASan guard I had added to transpose_optimizer_test.cc since it's no longer needed. Could you trigger CI when you have a moment?

@elwhyjay, it seems that Asan pipeline still failed.

HI @tianleiwu ! To reduce cumulative ASan pressure from this PR, I reduced the new alpha_not_one Gemm matrix to one representative uint8 QGemm path with int32 bias, while keeping bias/no-bias and output-q/no-output-q coverage in both default and fastmath tests. This should still exercise the selector behavior this PR changes, while avoiding repeated alpha coverage across every type combination. Hopefully this is a reasonable smaller test surface to try for the ASan pipeline.

@tianleiwu
Thanks again for helping with the CI reruns here. I pushed one more small test-surface reduction in ea615ec.

The previous reduction did not clear the ASan job, but it does look like some progress: compared with the earlier log where ASan ran out around the transpose optimizer area, the latest run got further through onnxruntime_test_all before hitting the same 8GB ASan allocator limit.

This latest commit keeps the new alpha_not_one coverage on the representative uint8/int32-bias QGemm path, and additionally avoids repeating the same selector behavior across the larger shape and contrib-QDQ variants. Hopefully this preserves the regression coverage for the behavior changed in this PR while giving the ASan pipeline a smaller incremental surface to try.