Generalize the team size calc for SYCL/CUDA/HIP in Homme#8530
Generalize the team size calc for SYCL/CUDA/HIP in Homme#8530tcclevenger wants to merge 1 commit into
Conversation
Previously num_warps_device was hardcoded, and num_threads_warp was obtained using Kokko::Impl variables which a) could go away and should not be relied on and b) did not have a SYCL equiv. SYCL was also using hard coding num thread/vector lane which were too small and lead to large loss of performance at small work/node runs (e.g., large number of nodes).
tcclevenger
added
HOMME
CI: approved
|
Here's the performance difference on Aurora (SYCL) with these changes: Dycore device kernels
Overall atmosphere
|
|
Do you expect this to change performance on Frontier or PM-GPU ? |
No, it should not change performance at all. I believe I tested in April that it yields the exact same team sizes for HIP and Cuda, but let me sanity check than one more time since it was a few months ago. |
abagusetty
left a comment
abagusetty
left a comment
There was a problem hiding this comment.
LGTM!
The changes shouldnt have an impact to frontier and PM-GPU: since the hardcoded values already map to number returned by kokkos-API now. Just adds more hardware portability
|
Confirmed on Frontier everything is the same. PM-GPU hasn't run yet, but I'm confident it's the same there too. @rljacob Am I ok to merge this today? Each change is ifdef guarded so guaranteed no code is different for any chrysalis config. Ok to wait as well. |
|
lets wait |
3 participants
Previously
num_warps_devicewas hard-coded, andnum_threads_warpwas obtained usingKokko::Implvariables which a) could go away and should not be relied on and b) did not have a SYCL equiv.SYCL was also using hard coding num thread/vector lane which were too small and lead to large loss of performance at small work/node runs (e.g., 2K and 4K nodes for ne1024).
This solutions was found and tested at the ALCF 2026 hackathon with @abagusetty.
[BFB]