Guard known-affected cuSolverMp NCCL versions#7658
Conversation
fcedb4a to
44b5034
Compare
|
Great job, thanks for the detailed diagnosis! @Stardust0831 I developed the multi-GPU RT-TDDFT path, and I have also encountered hangs with cuSOLVERMp/cuBLASMp under certain BLACS process grids and library versions. At the time, however, I did not isolate the problem using NVIDIA’s official samples or verify the affected version range in detail, so this investigation is very helpful. It is especially useful that the same failure can be reproduced independently of ABACUS and that upgrading to cuSOLVERMp 0.9.x resolves the 16-rank RT-TDDFT case. This gives us much stronger evidence that the hang is caused by the upstream library rather than the RT-TDDFT implementation itself. What are your planned next steps for this PR? Are you planning to add the cuSOLVERMp version guard and documentation mentioned in the description, and possibly keep the 4-rank smoke test as regular coverage while using a 16-rank 4×4-grid case as a periodic regression test? |
44b5034 to
5821e69
Compare
Stardust0831
changed the title
|
@AsTonyshment Thanks for the suggestions. I have split the follow-up into two focused PRs. |
2 participants
Summary
ALLOW_KNOWN_AFFECTED_CUSOLVERMP=ONas an explicit risk overridediago_cusolvermp.cppinclude the BLAS connector and title helper it uses directlyRelated issue: #7657
Rationale
NVIDIA documents an STEDC defect through cuSolverMp 0.8.0 affecting non-power-of-two block sizes and certain 2D configurations.
SyevdandSygvduse STEDC internally. The failure was independently reproduced with NVIDIA's officialmp_sygvdsample and with a 16-rank ABACUS RT-TDDFT run; the same ABACUS case completed with cuSolverMp 0.9.0 and cuBLASMp 0.9.1.The direct includes make the cuSolverMp translation unit self-contained; they do not change solver behavior. The remaining changes are configure-time diagnostics and dependency documentation. No INPUT parameter or numerical algorithm changes are introduced. The SAI GPU workflow and multinode Si48 smoke are intentionally handled in a separate Draft PR.
Tests
cmake/tests/test_cusolvermp_version_guard.cmakecovers:The exact
diago_cusolvermp.cppblob in this PR was compiled successfully on SAI with cuSolverMp 0.9.0, cuBLASMp 0.9.1, NCCL 2.29.3, CUDA 12.9.1, and OpenMPI 5.0.10. It then passed the 2-node/16-GPU Si48 RT-TDDFT smoke in runs 29870131642 and 29870141313.The current local environment does not provide a
cmakeexecutable, so the CMake script was not rerun during the final PR split.