feat(conv): large-tensor grouped CK xdlops conv + split_k workspace backport#9427
Open
JonathanLichtnerAMD wants to merge 3 commits into
Open
Conversation
JonathanLichtnerAMD
marked this pull request as draft
July 14, 2026 22:34
JonathanLichtnerAMD
force-pushed
the
users/jlichtne/ROCM-27526-large-tensor-support
branch
2 times, most recently
from
July 15, 2026 01:13
4f94615 to
f04147c
Compare
1 task
JonathanLichtnerAMD
force-pushed
the
users/jlichtne/ROCM-27526-large-tensor-support
branch
from
July 22, 2026 14:56
f04147c to
8f094e7
Compare
JonathanLichtnerAMD
marked this pull request as ready for review
July 22, 2026 15:16
mattwill-amd
approved these changes
Jul 22, 2026
SreecharanGundaboluAMD
approved these changes
Jul 23, 2026
1 task
JonathanLichtnerAMD
force-pushed
the
users/jlichtne/ROCM-27526-large-tensor-support
branch
from
July 24, 2026 19:54
f0ee75d to
0c9b0c6
Compare
…7526)
Enable the grouped Composable Kernel xdlops convolution solvers (2D and 3D)
for tensors whose element strides exceed the int32 range, backporting the
ROCM-23997 large-tensor work.
- Forward, backward-data, and weight (fwd/bwd/wrw) are all enabled: for
>INT_MAX shapes the solver selects CK's int64 (long_index_t) "Large_Tensor"
instances; sub-INT_MAX shapes keep the int32 path via the narrowing helpers.
bwd/wrw dispatch large-tensor instances unconditionally, exactly like fwd
(IsLargeTensorCKInstance detects them via the "_Large_Tensor" type string).
The bwd/wrw Large_Tensor instances come from the container CK build
(composable_kernel#3755).
- Add ck_grouped_conv_narrow.hpp: ToCKIndexArray int32 narrowing (asserts the
>INT_MAX contract) and NarrowedCKArrays{2,3}D bundles stored as CKArgs
members so CK's captured references stay valid.
- Widen CKArgs length/stride members to int64 so packed-stride products are
computed in 64-bit before they can overflow.
- convolutionocl.cpp: guard the explicit compile / prepare-invoker paths so an
applicable-but-unsatisfiable solver throws cleanly instead of dereferencing
an empty invoker_factory.
Tests:
- API applicability sweeps (2D/3D) bracketing the INT_MAX element-stride
boundary, and a unit_TensorDescriptor case pinning the exactly-INT_MAX vs
one-past boundary.
- Numerical unit tests for the fwd/bwd/wrw large-stride path execute and verify
against the CK Large_Tensor instances.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds NHWC 2D grouped convolution coverage for the grouped CK xdlops solvers across three regimes: sub-INT_MAX element counts, tensors above 2 GB whose element count still fits int32, and tensors whose element count exceeds INT_MAX (which require the CK large-tensor instances). All geometries are 3x3 with SAME padding, exercising the native NHWC path. - conv_api_solution_count_large2d.cpp: fast applicability/compile probe (fwd/bwd/wrw) for all shapes. - unit_conv_solver_ConvHipImplicitGemmGroupXdlops_Large2D.cpp: on-device execute+verify coverage; excluded from the default gtest filter as multi-GB (run explicitly). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lvers Backport of the workspace portion of PR #1426 (ALMIOPEN-706). MIOpen sized the CK workspace for the grouped bwd, grouped wrw, and 3D grouped wrw solvers via GetCKAlphaBetaWorkspace (the alpha/beta backward-weights workspace query), which does not account for CK's split_k reduction workspace. Once a tuned split_k>1 configuration was selected the reduction ran against an under-allocated workspace and could return silently wrong results. - Add NextCKSplitkValue (CK's split_k autodeduce sentinel, -1) and GetCKSplitkWorkspaceSize / GetCKSplitkMaxWorkspaceSize, which query CK for the actual per-instance workspace across the split_k sweep. The max sweep starts at split_k=1 and accumulates in std::size_t to cover >2 GB workspaces. - GetCKSplitkMaxWorkspaceSize applies the RequiresLargeTensorCKInstance filter (matching FillValidKernelsIDs / IsCKArgsSupported / IsCKApplicable) so a >INT_MAX problem never drives a non-large instance through the int32 narrowing path. - GetWorkspaceSize uses the CK workspace size (falling back to GetCKAlphaBetaWorkspace only when CK reports 0), and the NCHW/NHWC invoker factories bind a correctly-sized workspace pointer. - Drop the generic-search assertion that workspace size must be independent of the PerformanceConfig. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JonathanLichtnerAMD
force-pushed
the
users/jlichtne/ROCM-27526-large-tensor-support
branch
from
July 24, 2026 20:30
0c9b0c6 to
11bb026
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID: ROCM-27526
Motivation
MIOpen's grouped Composable Kernel (CK) xdlops convolution solvers
(
ConvHipImplicitGemm{,3D}Group{Fwd,Bwd,Wrw}Xdlops) hard-rejected any problemwhose tensor dimensions or element strides did not fit in
int32(
AllTensorsDimsFitIntoInt()). Large NHWC 2D conv shapes that cross theINT_MAXelement-count boundary therefore fell through to slower fallbacks orhad no applicable CK path at all. This is the ROCM-27526 large-tensor
enablement for the ROCm 7.0.2 backport branch, building on the ROCM-23997
large-tensor work.
Separately, the grouped bwd / wrw / 3D-wrw solvers previously sized the CK
workspace via
GetCKAlphaBetaWorkspace(the alpha/beta backward-weightsworkspace query), which does not account for CK's split_k reduction workspace.
Once a tuned
split_k > 1configuration was selected, the CK reductiontherefore ran against an under-allocated workspace, silently returning wrong
results (ALMIOPEN-2305). This PR backports the workspace portion of PR #1426 /
ALMIOPEN-706 to fix that, and adds test coverage for both the large-tensor and
the split_k-workspace behaviors.
Technical Details
This PR has three parts:
Large-tensor support for grouped CK xdlops conv.
src/include/miopen/solver/ck_grouped_conv_narrow.hpp: shared helpers forthe six inline
CKArgsstructs. DefinesToCKIndexArray(int64→int32narrowing that asserts the value fits — guards the >
INT_MAXcontract), andthe
NarrowedCKArrays{2,3}Dbundles plusMakeNarrowedCKArrays. The bundlesare stored as
mutablemembers of the owningCKArgsbecause CK'sMakeArgumentPointercaptures references into the length/stride arrays; afunction-local temporary would be a stack-use-after-scope.
CKArgsdim member (G,N,K,C,Hi,Wi,...) toint64_tand thelength/stride arrays to
std::arrayin all six solver.cppfiles, so the NCHW packed-stride products (e.g.Hi*Wi*G*C) arecomputed in 64-bit before they can overflow.
implicitgemm_ck_util.hpp:RequiresLargeTensorCKInstance(true when!AllTensorsDimsFitIntoInt()) andIsLargeTensorCKInstance(matches"Large_Tensor"in the CK instance's typestring).
FillValidKernelsIDs,IsCKArgsSupported, andIsCKApplicablenowfilter to large-tensor instances for out-of-range shapes, and
FillValidKernelsIDsmay now legitimately return an empty list (assertsrelaxed accordingly).
PerformanceConfig::Initno longer indexes an emptyvalid_kernels.MakeArgPtrdispatches per instance: large-tensor instances bind CK's int64long_index_toverload with the un-narrowed members; all other instances gothrough
GetNarrowedArrays()and bind the int32 overload. Forward,backward-data, and weight (2D + 3D) are all enabled, dispatching to CK's
grouped-conv
Large_Tensorint64 instances provided by the container CK(composable_kernel PR feat(grouped-conv): port large-tensor / global load-store support (#9258) composable_kernel#3755).
The six solvers dropped their
!AllTensorsDimsFitIntoInt()early-return inIsApplicable.convolutionocl.cpphardening: bothPrepareInvokerandCompileSolutionnow check that a coarsely-applicable solver actually produces a usable
solution / is applicable to the concrete problem, throwing
MIOPEN_THROWinstead of dereferencing an unset
invoker_factory(previously undefinedbehavior / crash for a large-tensor shape with no matching CK instance).
CHANGELOG.mdentry; aunit_TensorDescriptor.cppcase added to pin theexactly-
INT_MAXvs one-past boundary (strict>).Large 2D grouped conv coverage.
unit_conv_solver_ConvHipImplicitGemmGroupXdlops_Large2D.cpp: on-deviceexecute+verify (fwd/bwd/wrw) over large NHWC 2D grouped conv shapes spanning
three regimes — sub-
INT_MAXelement counts; tensors above 2 GB whose elementcount still fits
int32; and tensors whose element count exceedsINT_MAX(which require the CK large-tensor instances). All geometries are 3×3 with
SAME padding, exercising the native NHWC path. Shapes are anonymized.
conv_api_solution_count_large2d.cpp: fast applicability/compile probe overthe same shapes.
CK split_k reduction workspace backport (PR #1426 / ALMIOPEN-706).
implicitgemm_ck_util.hpp: addedNextCKSplitkValue(which now includes CK'ssplit_k = -1autodeduce value,CkSplitkAutoDeduce) andGetCKSplitkMaxWorkspaceSize, which sweeps split_k per instance and queries CKfor the real workspace size. For
>INT_MAXproblems this sweep is filtered tolarge-tensor instances (mirroring
FillValidKernelsIDs/IsCKArgsSupported/IsCKApplicable), so it never drives a non-large instance through the int32narrowing path (
ToCKIndexArray) — which otherwise asserts in a debug build atGetWorkspaceSizefor a>INT_MAX-stride grouped-wrw shape.GetWorkspaceSizeLayoutTransformConvnow acceptsan optional CK workspace size and uses it in the NCHW/NHWC totals (falling
back to
GetCKAlphaBetaWorkspaceonly when CK reports 0); the invokerfactories bind a correctly-sized workspace pointer (
_ck_buff_dessized fromGetCKSplitkWorkspaceSize) instead of the alpha/beta-only estimate.GetCKMaxWorkspaceSizeto the grouped bwd, grouped wrw, and 3D groupedwrw solvers and wired it into their
GetWorkspaceSize(declarations added inconv/solvers.hpp).generic_search.hppthat workspacesize must be independent of the
PerformanceConfig— with split_k itlegitimately varies.
split_k > 1configneeded a reduction workspace (ALMIOPEN-2305).
Risk Assessment
Roughly 3000 lines across 25 files, but ~1900 are new self-contained test
files. Behavioral source changes are concentrated in the six grouped CK xdlops
solvers plus two shared CK utility headers: the large-tensor path (fwd/bwd/wrw)
only activates for shapes that already fail
AllTensorsDimsFitIntoInt()—sub-
INT_MAXshapes keep the identical int32 code path via the narrowinghelpers. The workspace backport is the highest-blast-radius piece because it
changes reported workspace sizes and invoker workspace binding for all grouped
bwd/wrw problems (not just large tensors), and it removes a generic-search
invariant.
Risk Level: 🟡 Medium
Impacted Components
conv_hip_implicit_gemm_{,3d_}grouped_{fwd,bwd,wrw}_xdlops.cpp.implicitgemm_ck_util.hpp(workspace + large-tensordispatch), new
ck_grouped_conv_narrow.hpp.conv/solvers.hpp(newGetCKMaxWorkspaceSizedeclarations).generic_search.hpp(removed workspace-invariant assertion — affects thegeneric tuning loop broadly, not just these solvers).
convolutionocl.cppexplicit-compile / prepare-invoker paths.Potential Side Effects
GetWorkspaceSize(includes theCK reduction workspace); callers that pre-size workspace buffers will see the
increase. This is the intended fix.
CKArgsmembers to int64 is arithmetically transparent forin-range shapes but is a struct-layout change across the six solvers.
future workspace-sizing bug would no longer be caught by that check.
convolutionocl.cppguards convert some previously-UB/crash paths intoclean
miopenStatus*throws — a behavior change for callers that hit aninapplicable/unsatisfiable solver by explicit ID.
Mitigation Steps
INT_MAXshapes are unchanged: they take the narrowing path and bind thesame int32 CK overload as before;
ToCKIndexArrayasserts if the contract isever bypassed.
Result).
*LargeStride*,*Large2D*) are excluded fromthe default gtest filter so pre-checkin cost is unaffected; run explicitly.
Test Plan
INT_MAXelement-stride boundary:conv_api_solution_count_{2d,3d}_large_stride.cppandconv_api_solution_count_large2d.cpp(shared scaffolding inconv_api_solution_count_large_stride_common.hpp), restricted to a CI archallowlist (gfx90A / gfx94X / gfx950).
(
unit_conv_solver_*_LargeStride.cpp) and the large 2D NHWC path(
unit_conv_solver_ConvHipImplicitGemmGroupXdlops_Large2D.cpp) across thethree element-count regimes.
unit_TensorDescriptor.cppboundary cases at exactlyINT_MAXand one past.workspace backport.
Test Result
INT_MAX, and the >2 GB-but-element-count-fits-int32cases passon gfx942-class hardware.
int32-safe shapes, workspace-requiring
split_k > 1CK configurations are nowselected and produce correct results, where before they returned silently
wrong results (ALMIOPEN-2305).
test-harness precision artifact of the reference computation, not a solver
bug.