Skip to content

Commit ac74d92

Browse files
committed
fix(conv): gate ASM-GTC fwd NHWC solver off int32-overflow tensors
ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC (the ASM-GTC forward solver from the unmaintained MISA project) indexes global tensor memory with 32-bit element indices and does not implement large-tensor support. Its IsApplicable() relied on AllTensorsDimsFitIntoInt(), which only checks that each individual length/stride fits in int32 -- it never bounds the flattened element count. A tensor whose total element count exceeds INT_MAX therefore overflows the kernel's 32-bit indexing and is silently computed incorrectly (observed forward verification error ~0.37 on gfx942 and gfx950). Gate the solver not-applicable when any tensor's element count exceeds INT_MAX, so a large-tensor-capable solver (e.g. grouped CK xdlops) is selected instead. This is a no-op for every int32-safe shape, so existing workloads are unaffected. Adds a CPU device-applicability unit test that shares geometry across a batch-size boundary so the element count crosses INT_MAX, asserting the solver stays applicable just below the limit and is gated off just above. Related to ROCM-27526.
1 parent 251432d commit ac74d92

3 files changed

Lines changed: 114 additions & 0 deletions

File tree

projects/miopen/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Full documentation for MIOpen is available [here](https://rocm.docs.amd.com/proj
1919
### Resolved Issues
2020
* [RNN] Fix RNN workspace tensor descriptor int overflow
2121
* [Conv] Enabled grouped Composable Kernel (CK) xdlops fwd, bwd, and wrw convolution (2D and 3D) for tensors whose strides exceed the int32 range.
22+
* [Conv] Gated the unmaintained ASM-GTC forward NHWC solver (`ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC`) off tensors whose element count exceeds the int32 range, preventing it from being selected and silently returning incorrect results on large convolutions.
2223
* [Conv] Fixed `miopenStatusInternalError` thrown by Find on depthwise NHWC grouped convolutions under `MIOPEN_FIND_MODE=NORMAL`.
2324
* Fixed a thread-safety issue where concurrent access to the AI-heuristic model caches was not guarded by a mutex.
2425
* [Conv] Fixed Composable Kernel (CK) grouped-convolution solvers not always being registered in host code.

projects/miopen/src/solver/conv/conv_asm_implicit_gemm_gtc_fwd_nhwc.cpp

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -923,6 +923,22 @@ bool ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC::IsApplicable(
923923
if(!problem.AllTensorsDimsFitIntoInt())
924924
return false;
925925

926+
// This solver originates in the (now unmaintained) MISA project and addresses
927+
// global tensor memory with 32-bit element indexing. AllTensorsDimsFitIntoInt()
928+
// above only validates that each individual length/stride fits in int32; it does
929+
// NOT bound the flattened element count. A tensor whose total element count
930+
// exceeds INT_MAX therefore overflows the kernel's 32-bit indexing and silently
931+
// returns wrong results. MISA does not implement large-tensor support, so gate the
932+
// solver off such shapes; a large-tensor-capable solver is selected instead.
933+
{
934+
constexpr std::size_t max_int32 =
935+
static_cast<std::size_t>(std::numeric_limits<int>::max());
936+
if(problem.GetIn().GetElementSize() > max_int32 ||
937+
problem.GetOut().GetElementSize() > max_int32 ||
938+
problem.GetWeights().GetElementSize() > max_int32)
939+
return false;
940+
}
941+
926942
if(!problem.IsFp32() && !problem.IsFp16() &&
927943
!(problem.IsBfp16() &&
928944
(device_name == "gfx90a" || device_name == "gfx942" || StartsWith(device_name, "gfx95"))))
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
/*******************************************************************************
2+
*
3+
* MIT License
4+
*
5+
* Copyright (c) 2026 Advanced Micro Devices, Inc.
6+
*
7+
* Permission is hereby granted, free of charge, to any person obtaining a copy
8+
* of this software and associated documentation files (the "Software"), to deal
9+
* in the Software without restriction, including without limitation the rights
10+
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
11+
* copies of the Software, and to permit persons to whom the Software is
12+
* furnished to do so, subject to the following conditions:
13+
*
14+
* The above copyright notice and this permission notice shall be included in all
15+
* copies or substantial portions of the Software.
16+
*
17+
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
18+
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
19+
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
20+
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
21+
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
22+
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
23+
* SOFTWARE.
24+
*
25+
*******************************************************************************/
26+
27+
#include "unit_conv_solver.hpp"
28+
29+
// ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC (the MISA ASM-GTC forward solver)
30+
// indexes global tensor memory with 32-bit element indices and does not implement
31+
// large-tensor support. IsApplicable() therefore gates the solver off any problem
32+
// whose flattened element count exceeds INT_MAX, so it cannot be selected for a
33+
// shape it would silently compute incorrectly.
34+
//
35+
// The two cases below share geometry and differ only in batch size N, so the
36+
// element count crosses the INT_MAX boundary and isolates the gate:
37+
// C*H*W per sample = 1024*162*92 = 15,261,696 elements
38+
// N=140 -> 2,136,637,440 <= INT_MAX (2,147,483,647): int32-safe, applicable
39+
// N=141 -> 2,151,899,136 > INT_MAX : gated off, not applicable
40+
41+
namespace {
42+
43+
auto GetInRangeConvCase()
44+
{
45+
using miopen::unit_tests::ConvTestCase;
46+
return ConvTestCase{{miopenHalf, miopenTensorNHWC, {140, 1024, 162, 92}},
47+
{miopenHalf, miopenTensorNHWC, {1024, 1024, 3, 3}},
48+
miopenHalf,
49+
{{1, 1}, {1, 1}, {1, 1}}};
50+
}
51+
52+
auto GetOverInt32ConvCase()
53+
{
54+
using miopen::unit_tests::ConvTestCase;
55+
return ConvTestCase{{miopenHalf, miopenTensorNHWC, {141, 1024, 162, 92}},
56+
{miopenHalf, miopenTensorNHWC, {1024, 1024, 3, 3}},
57+
miopenHalf,
58+
{{1, 1}, {1, 1}, {1, 1}}};
59+
}
60+
61+
// int32-safe shape: the solver keeps its usual device applicability (regression guard).
62+
auto GetInRangeParams()
63+
{
64+
auto p = miopen::unit_tests::UnitTestConvSolverParams(Gpu::gfx908 | Gpu::gfx90A | Gpu::gfx94X |
65+
Gpu::gfx950);
66+
p.CheckXnackDisabled();
67+
return p;
68+
}
69+
70+
// element count > INT_MAX: the solver must be applicable on no device (gated off).
71+
auto GetGatedParams()
72+
{
73+
auto p = miopen::unit_tests::UnitTestConvSolverParams(Gpu::None);
74+
p.CheckXnackDisabled();
75+
return p;
76+
}
77+
78+
} // namespace
79+
80+
using CPU_UnitTestConvSolverAsmGTCFwdNHWCLargeTensorDevApplicability_NONE =
81+
CPU_UnitTestConvSolverDevApplicabilityFwd_NONE;
82+
83+
TEST_P(CPU_UnitTestConvSolverAsmGTCFwdNHWCLargeTensorDevApplicability_NONE,
84+
ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC)
85+
{
86+
this->RunTest(miopen::solver::conv::ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC{});
87+
};
88+
89+
INSTANTIATE_TEST_SUITE_P(Int32Safe,
90+
CPU_UnitTestConvSolverAsmGTCFwdNHWCLargeTensorDevApplicability_NONE,
91+
testing::Combine(testing::Values(GetInRangeParams()),
92+
testing::Values(GetInRangeConvCase())));
93+
94+
INSTANTIATE_TEST_SUITE_P(OverInt32ElemCount,
95+
CPU_UnitTestConvSolverAsmGTCFwdNHWCLargeTensorDevApplicability_NONE,
96+
testing::Combine(testing::Values(GetGatedParams()),
97+
testing::Values(GetOverInt32ConvCase())));

0 commit comments

Comments
 (0)