Skip to content

fix(conv): gate ASM-GTC fwd NHWC solver off int32-overflow tensors#9768

Open
SreecharanGundaboluAMD wants to merge 1 commit into
release/rocm-rel-7.0.2.1from
users/SreecharanGundaboluAMD/gate_MISA
Open

fix(conv): gate ASM-GTC fwd NHWC solver off int32-overflow tensors#9768
SreecharanGundaboluAMD wants to merge 1 commit into
release/rocm-rel-7.0.2.1from
users/SreecharanGundaboluAMD/gate_MISA

Conversation

@SreecharanGundaboluAMD

@SreecharanGundaboluAMD SreecharanGundaboluAMD commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Motivation

ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC, the ASM-GTC forward solver from the
unmaintained MISA project, indexes global tensor memory with 32-bit element indices
and has no large-tensor support. On any problem whose flattened element count exceeds
INT_MAX it overflows that index and silently returns wrong results (forward
verification error ~0.37 on gfx942/gfx950; no crash, it runs, reports a time, and
produces incorrect output).

Because find selects a solver by timing alone and does not verify numerics, this
fast-but-wrong solver gets auto-selected over correct ones, and it currently breaks
develop in more than one configuration:

  • gfx950, default find: the heuristic selects MISA → wrong.
  • gfx942, exhaustive find (FIND_MODE=1/ENFORCE=4): MISA out-times CK on the 1024→1024
    shapes and is selected → wrong.

Neither a version bump nor a find-mode choice reliably avoids it. Since MISA is
does not implement large-tensor support, it must be gated
during the applicability checks so it can never be selected for shapes it computes
incorrectly.

Technical Details

  • Added an element-count guard to ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC::IsApplicable():
    returns not-applicable when GetIn()/GetOut()/GetWeights().GetElementSize() > INT_MAX.
  • The existing AllTensorsDimsFitIntoInt() check only validates that each individual
    length/stride fits in int32, it does not bound the flattened element count. For a
    packed NHWC tensor the largest stride ≈ elements/N, so a shape can have every stride fit
    int32 while the element count overflows; the new predicate is what catches these.
  • No-op for every int32-safe shape (the guard is false), so existing workloads take the
    identical code path, zero functional change outside the overflow regime.
  • Complementary to feat(conv): large-tensor grouped CK xdlops conv + split_k workspace backport #9427: that PR keys CK large-tensor instances on
    !AllTensorsDimsFitIntoInt() (stride regime) and does not touch MISA; this gate covers
    the element-count regime and removes the silently-wrong MISA candidate.
  • Scope: forward NHWC ASM-GTC solver. CHANGELOG.md updated. Related to ROCM-27526.

Files: src/solver/conv/conv_asm_implicit_gemm_gtc_fwd_nhwc.cpp,
test/gtest/unit_conv_solver_ConvAsmImplicitGemmGTCDynamicFwdXdlopsNHWC_LargeTensor.cpp,
CHANGELOG.md.

Test Plan

  • New CPU device-applicability gtest (CPU_UnitTestConvSolver...DevApplicability): a
    batch-size boundary pair with identical geometry so the element count crosses INT_MAX,
    asserting the solver stays applicable just below the limit (N=140, 2.14e9 elements) and
    is gated off just above (N=141, 2.15e9). MockHandle-based, runs host-side, no allocation.
  • Empirical end-to-end validation on real hardware from from-source builds: gfx942/MI300X
    and gfx950/MI355X, ROCm 7.0.2.1 and 7.14, MIOpenDriver -t 1 verify.

Test Result

  • Forcing MISA on the >INT_MAX shapes reproduces the silent failure on both arches and both
    stacks (verify error ~0.37, some NaN), confirming it is the solver, not the stack.
  • With the gate: forcing the solver on those shapes returns "no suitable algorithm was found"
    on gfx942 and gfx950; natural find selects grouped CK xdlops and verifies OK on all
    previously-failing shapes; int32-safe shapes are unchanged.
  • No other fast forward solver applies at these sizes (GEMM and Winograd both report
    not-applicable), so CK is the correct target the gate steers find toward.
  • The new gtest is exercised by CI. (It was not run on this host, no GPU/clang-format dev
    toolchain available; the gate code itself compiles, having been built identically in the
    from-source validation images.)

Submission Checklist

@JonathanLichtnerAMD JonathanLichtnerAMD left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a small tweak to the PR message, but LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants