[Issue] rocBLAS large-batch batched kernels fail on Linux-gfx94X with `hipErrorInvalidConfiguration` at `batch_count=65539`

[chiranjeevipattigidi]

Description:

Large-batch rocBLAS tests are failing in CI on gfx94X,Linux due to an invalid HIP kernel launch configuration.

Failures noticed in CI of rocm-system bump PR: ROCm/TheRock#5117
Commit history: ROCm/rocm-systems@79e85e1...4120a4e

Context:

• Platform: Linux
• Impacted Arch: gfx94x
• Run: https://github.com/ROCm/TheRock/actions/runs/25530109042

Full log:

• https://github.com/ROCm/TheRock/actions/runs/25530109042/job/75004205080?pr=5117
• https://github.com/ROCm/TheRock/actions/runs/25530109042/job/75004205091?pr=5117
• https://github.com/ROCm/TheRock/actions/runs/25530109042/job/75004205093?pr=5117

Failed Tests:

_/hemm_batched.blas3/pre_checkin_hemm_large_batch_f32_c_RL_2_2_2r0i_4_2_2r0i_4_65539
_/hemm_strided_batched.blas3/pre_checkin_hemm_large_batch_f32_c_LL_2_2_2r0i_4_0_2_0_2r0i_4_0_65539
_/herkx_strided_batched.blas3_tensile/pre_checkin_herkx_large_batch_f32_c_LC_2_2_1r0i_2_0_2_0_1_2_0_65539
_/dgmm_strided_batched.blas3/pre_checkin_dgmm_large_batch_f32_r_R_2_2_2_4_1_2_2_4_65539

Test Summary:

[  FAILED  ] _/hemm_batched.blas3/pre_checkin_hemm_large_batch_f32_c_RL_2_2_2r0i_4_2_2r0i_4_65539, where GetParam() = { function: "hemm_batched", name: "hemm_large_batch", category: "pre_checkin", known_bug_platforms: "", alpha: 2.0, beta: 2.0, M: 2, N: 2, lda: 4, ldb: 2, ldc: 4, batch_count: 65539, a_type: f32_c, b_type: f32_c, c_type: f32_c, d_type: f32_c, compute_type: f32_c, initialization: rand_int, gpu_arch: "", flush_batch_count: 1, side: 'R', uplo: 'L' }

[  FAILED  ] _/hemm_strided_batched.blas3/pre_checkin_hemm_large_batch_f32_c_LL_2_2_2r0i_4_0_2_0_2r0i_4_0_65539, where GetParam() = { function: "hemm_strided_batched", name: "hemm_large_batch", category: "pre_checkin", known_bug_platforms: "", alpha: 2.0, beta: 2.0, M: 2, N: 2, lda: 4, ldb: 2, ldc: 4, batch_count: 65539, a_type: f32_c, b_type: f32_c, c_type: f32_c, d_type: f32_c, compute_type: f32_c, initialization: rand_int, gpu_arch: "", flush_batch_count: 1, side: 'L', uplo: 'L' }

[  FAILED  ] _/herkx_strided_batched.blas3_tensile/pre_checkin_herkx_large_batch_f32_c_LC_2_2_1r0i_2_0_2_0_1_2_0_65539, where GetParam() = { function: "herkx_strided_batched", name: "herkx_large_batch", category: "pre_checkin", known_bug_platforms: "", beta: 1.0, N: 2, K: 2, lda: 2, ldb: 2, ldc: 2, batch_count: 65539, a_type: f32_c, b_type: f32_c, c_type: f32_c, d_type: f32_c, compute_type: f32_c, initialization: rand_int, gpu_arch: "", flush_batch_count: 1, transA: 'C', uplo: 'L' }

[  FAILED  ] _/dgmm_strided_batched.blas3/pre_checkin_dgmm_large_batch_f32_r_R_2_2_2_4_1_2_2_4_65539, where GetParam() = { function: "dgmm_strided_batched", name: "dgmm_large_batch", category: "pre_checkin", known_bug_platforms: "", stride_a: 4, stride_c: 4, stride_x: 2, M: 2, N: 2, lda: 2, ldb: 2, ldc: 2, incx: 1, batch_count: 65539, initialization: rand_int, gpu_arch: "", flush_batch_count: 1, side: 'R' }


 4 FAILED TESTS

Impact:

Blocker for promotion of rocm-systems submodule to therock: ROCm/TheRock#5117

[chiranjeevipattigidi]

Hello @TorreZuk
rocblas failures also noticied in Linux also which are different from failures reported for windows in #7149,
Can you take look at this ?

[TorreZuk]

@chiranjeevipattigidi can you make a ROCM jira ticket for this, it sounds like a systems change with big implications.

[JeniferC99]

https://amd-hub.atlassian.net/browse/ROCM-24377 @TorreZuk

[jayhawk-commits]

Seems to have been introduced by ROCm/rocm-systems#5707