Add cuSolverMp RT-TDDFT smoke coverage and version notes

[Stardust0831]

Reminder

• I have linked the reproducing issue.
• I have added an appropriate regression test or explained the CUDA CI limitation.
• I have listed the exact verification commands and results below.
• I have described the user-visible behavior change below.
• I have explained the core-module impact below.

Linked Issue

Related issue: deepmodeling/abacus-develop#7657

Unit Tests and/or Case Tests for my changes

Commands run during diagnosis:

# NVIDIA cuSOLVERMp official sample, old library
mpirun -np 16 ./mp_sygvd -grid_layout R -p 4 -q 4 -m 39 -mbA 10 -nbA 10

# ABACUS RT-TDDFT, old library
sbatch --ntasks=16 --gpus-per-node=16 run_full_rt_tddft.sbatch gpu16 16

# ABACUS RT-TDDFT, cuSolverMp 0.9.0 + cuBLASMp 0.9.1
sbatch --ntasks=16 --gpus-per-node=16 run_full_rt_tddft.sbatch gpu16 16

Results:

• cuSolverMp 0.7.2: official sample and ABACUS case hang; ranks 4--15 emit NCCL WARN CommUserRank : comm argument is NULL.
• cuSolverMp 0.9.0 + cuBLASMp 0.9.1: 3 MD/RT steps complete, exit code 0, no NCCL warnings, final energy -1336.999498209972 eV.
• CPU-only and CUDA CI regression coverage: not run in this environment.

Additional SAI self-hosted validation (4 V100 GPUs):

• Project cuSolverMp 0.9.x + cuBLASMp 0.9.x RT-TDDFT smoke: passed, Slurm job 685177, 109.447 s, all 4 checks passed.
• Full GPU workflow run 29707130597 / Slurm 685147: all PW, NAO, and RT-TDDFT suites passed; one pre-existing stochastic SDFT BPCG reference exceeded the default force/stress tolerances. The workflow now continues to the cuSolverMp smoke and reports the suite status instead of stopping at the first failed layer.

What's changed?

Implemented in the current CI branch

• Add a workflow_dispatch SAI self-hosted GPU workflow.
• Select project or system cuSolverMp/cuBLASMp by absolute prefix.
• Run existing GPU suites with their required MPI widths.
• Add a 4-rank RT-TDDFT ks_solver=cusolvermp smoke test.
• Continue collecting diagnostics after a suite failure, while preserving the final failure status.

Proposed ABACUS follow-up in this PR

This PR proposes a configure-time guard in cmake/modules/SetupCuSolverMp.cmake:

• Preserve the existing CAL selection for versions older than 0.7.0.
• For the NCCL backend, reject or clearly warn about 0.7.0 <= version < 0.9.0, because this range includes the known STEDC/Sygvd defects.
• Emit an actionable message naming the required cuSolverMp and cuBLASMp versions and the NVIDIA release-note rationale.
• Update the CUDA acceleration documentation with the tested minimum version.

The exact policy (fatal configure error versus opt-in legacy override) should be confirmed by maintainers because it changes compatibility for users of older NVHPC installations. The proposed default is to fail early for the NCCL backend, avoiding a runtime hang.

The accompanying SAI workflow is workflow_dispatch only and exposes an mp_version input (project or system) so the cuSolverMp/cuBLASMp runtime is selected by absolute prefix rather than by whichever module happens to be loaded. It also provides a 4-rank RT-TDDFT cuSolverMp smoke case for a future GPU CI job; it does not run automatically for untrusted pull requests.

Governance Notes

• INPUT/docs changes: no INPUT behavior changes; only CUDA dependency
  documentation and CMake diagnostics.
• Core module impact: no numerical kernel or MPI algorithm changes. This is a build-time compatibility guard for Diag_CusolverMP_gvd and the existing NCCL-backed path.
• Exceptions requested: none.
• Follow-up: add an automated multi-rank CUDA case when CI has a supported cuSolverMp installation and sufficient GPUs.

[AsTonyshment]

Great job, thanks for the detailed diagnosis! @Stardust0831 I developed the multi-GPU RT-TDDFT path, and I have also encountered hangs with cuSOLVERMp/cuBLASMp under certain BLACS process grids and library versions. At the time, however, I did not isolate the problem using NVIDIA’s official samples or verify the affected version range in detail, so this investigation is very helpful.

It is especially useful that the same failure can be reproduced independently of ABACUS and that upgrading to cuSOLVERMp 0.9.x resolves the 16-rank RT-TDDFT case. This gives us much stronger evidence that the hang is caused by the upstream library rather than the RT-TDDFT implementation itself.

What are your planned next steps for this PR? Are you planning to add the cuSOLVERMp version guard and documentation mentioned in the description, and possibly keep the 4-rank smoke test as regular coverage while using a 16-rank 4×4-grid case as a periodic regression test?