Add SAI GPU validation and multinode cuSolverMp smoke

[Stardust0831]

Reminder

• I have read AGENTS.md and docs/developers_guide/agent_governance.md.
• I have linked an issue or explained why this PR does not need one.
• I have added adequate unit tests and case tests.
• I have listed the exact verification commands run and their results.
• I have described user-visible behavior changes, including INPUT parameter changes.
• I have explained core-module impact.
• I have requested any needed governance exception below.

Linked Issue

Related to #7657. This PR adds SAI regression coverage and does not close the NVIDIA library issue.

Depends on #7666 for the self-contained cuSolverMp translation-unit includes used by the verified GPU build. Merge #7666 first.

Unit Tests and/or Case Tests for my changes

bash ci/sai/tests/test_sai_ci.sh
find ci/sai -type f \( -name "*.sh" -o -name "*.sbatch" -o -name "*.env.example" \) -print0 | xargs -0 bash -n
shellcheck -S error ci/sai/*.sh ci/sai/*.sbatch ci/sai/tests/test_sai_ci.sh ci/sai/toolchains/*.env.example
actionlint .github/workflows/sai-bootstrap.yml .github/workflows/sai-gpu-full.yml
python3 tools/03_code_analysis/agent_governance_check.py --base upstream/develop --head HEAD --format text
git diff --check upstream/develop...HEAD

Results:

• SAI control regression suite: 21/21 passed.
• Bash syntax, ShellCheck, actionlint, governance, and diff checks passed.
• A real local --probe-only invocation using an existing OpenSSH Host alias reached SAI, verified the configured remote user and canonical HOME, and detected the 16V100 partition without creating a run or submitting a job.
• Concurrent end-to-end runs 29870131642 and 29870141313 each passed 47/48 GPU matrix cases with 0 infrastructure failures. All 16 RT-TDDFT GPU cases and the 2-node/16-GPU Si48 cuSolverMp smoke passed. The only failure was the existing 16_SDFT_GPU/005_PW_SDFT_MALL_BPCG_GPU numerical reference deviation.
• Those full SAI runs predate the local-launcher hardening commit. The new head has component, static, and real probe evidence, but has not launched another complete GPU matrix.

What changed?

• Add GitHub-hosted SSH workflows for a protected manual SAI trigger and a daily default-branch trigger. There is no automatic pull_request trigger.
• Keep the SSH private key in GitHub Environment secrets, pin the SAI host key, and execute the GitHub control plane from the trusted default branch. Approved source commits still execute unsandboxed as abacususer01; the README documents this trust boundary.
• Add a direct local launcher for developers with their own SAI account. It uses a standard OpenSSH config/Host alias, so usernames and identity files remain in the developer's SSH configuration rather than in the repository. Local runs always use the candidate cache role and never advance the daily baseline.
• Materialize local control files from the recorded CONTROL_SHA with git archive. Ignored, untracked, or concurrently modified working-tree files cannot enter the remote control plane. Source payload generation is shared by the GitHub and local entrypoints.
• Address review hardening findings: avoid empty LD_LIBRARY_PATH elements, reject symlinks inside staged GPU cases, and verify the expected remote username without a hard-coded numeric UID.
• Build ABACUS with cuSolverMp 0.9.0, cuBLASMp 0.9.1, SAI NCCL 2.29.3, CUDA 12.9.1, and OpenMPI 5.0.10 without modifying /opt or system modules.
• Submit all 48 existing GPU cases as resource-homogeneous Slurm arrays with concurrency caps 2/8/8. A narrow PMIx startup signature retries once; numerical failures and timeouts do not retry.
• Add a reconstructed Si48 solid case and run it as a separate 2-node/16-GPU cuSolverMp RT-TDDFT functional smoke. It checks completion only and is not a physical or performance reference.
• Allow concurrent manual/local runs in namespaced directories. Source payloads and artifacts are compressed, and a user-level cleanup service removes uploaded run data after 72 hours.

The known SDFT reference failure is intentionally not hidden or relaxed, so the full workflow remains red until that numerical discrepancy is resolved.

Repository layout

GitHub-specific entrypoints remain in .github/workflows/, the Si48 regression fixture remains under the existing tests/ tree, and reusable GitHub/local/SSH/Slurm controls live in ci/sai/. This PR adds one new top-level directory, ci/.

Governance Notes

• INPUT/docs changes: no ABACUS INPUT behavior changes. The new INPUT belongs only to the Si48 regression fixture and is documented in the case README and ci/sai/README.md.
• Core module impact: none; this PR does not modify source/.
• Exceptions requested: none.
• Operator setup: before merge/enablement, maintainers must configure the sai-ssh-scheduled and protected sai-ssh-manual Environments, their variables, and the Environment-only SSH secret as documented in ci/sai/README.md. Local developers instead copy ci/sai/local-run.env.example outside the checkout and reference their own OpenSSH config; private key content is never stored in the repository config.

[AsTonyshment]

I will convert this PR into draft temporarily since it depends on the merge of #7658.

[Stardust0831]

@AsTonyshment Thanks.
I moved the change that PR #7665 actually depends on into the standalone PR #7666, which adds the required direct includes and normalizes the formatting of diago_cusolvermp.cpp; this decouples #7665 from #7658 and its separate discussion about whether cuSolverMp versions below 0.9.0 should be blocked entirely.

[Copilot]

Pull request overview

Adds an SSH-based, GitHub-hosted control-plane workflow to run ABACUS GPU validation on the SAI Slurm cluster (including a new multinode cuSolverMp RT-TDDFT smoke), plus supporting scripts for secure source transfer, job submission, artifact collection, and automated retention cleanup.

Changes:

• Introduce GitHub Actions workflows for scheduled and manually-approved SAI GPU validation runs over pinned-host-key SSH.
• Add a comprehensive ci/sai/ control suite (toolchain setup, Slurm orchestration, source snapshot/delta transfer cache, artifact collection, cleanup cron installation, and regression tests).
• Add a reconstructed Si48 RT-TDDFT cuSolverMp smoke fixture case for 2-node / 16-GPU functional completion checks.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/STRU	New Si48 structure fixture for multinode cuSolverMp smoke.
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/README	Documents provenance and non-reference intent of the Si48 smoke case.
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/KPT	Gamma-only KPT for the Si48 smoke case.
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/INPUT	RT-TDDFT GPU INPUT selecting ks_solver cusolvermp.
ci/sai/toolchains/archive-mp09-sai-nccl2293.env.example	Example SAI module/toolchain loader for CUDA/NVHPC/OpenMPI + MP/NCCL selection.
ci/sai/tests/test_sai_ci.sh	Local regression tests for the SAI control scripts (security and policy checks).
ci/sai/test_gpu.sbatch	2-node Slurm job script for the multinode cuSolverMp RT-TDDFT smoke run.
ci/sai/test_gpu_case.sh	Per-array-task launcher for running one GPU test case with retry metadata.
ci/sai/summarize_gpu_case_matrix.sh	Produces a markdown summary and CI annotations for the 48-case matrix.
ci/sai/source_transfer_cache.sh	Implements locked baseline/candidate source snapshot + delta transfer and verification.
ci/sai/run_slurm_job.sh	Submits and monitors Slurm jobs with cancellation on signal/exit.
ci/sai/run_remote_ci.sh	Remote coordinator entrypoint for preparing deps, building, and running validations.
ci/sai/run_gpu_validation.sh	Coordinates running the GPU arrays and multinode smoke concurrently.
ci/sai/run_gpu_case_matrix.sh	Submits the GPU case matrix as Slurm arrays and records accounting state.
ci/sai/run_gpu_case_attempts.sh	Runs a case with a single allowed PMIx-startup retry and writes TSV metadata.
ci/sai/README.md	Operator/user documentation for trust boundary, triggers, caching, artifacts, cleanup.
ci/sai/probe_remote_sai.sh	Remote capability probe used by workflows before launching the run.
ci/sai/prepare_remote_run.sh	Creates an isolated run directory on SAI with containment/collision checks.
ci/sai/prepare_nvidia_mp.sh	Fetches/verifies/caches NVIDIA MP archives with SHA256 pinning and atomic install.
ci/sai/prepare_gpu_case_matrix.sh	Builds the manifest inventory for the 48 existing GPU cases.
ci/sai/prepare_cusolvermp_smoke.sh	Stages a fixed-input cuSolverMp smoke case with file hash validation.
ci/sai/prepare_cleanup_install.sh	Stages cleanup installer artifacts in a diagnostic directory for bootstrap.
ci/sai/mpirun_with_mapping.sh	Wrapper to apply SAI partition mapping options to mpirun.
ci/sai/mark_artifacts_uploaded.sh	Atomically writes a marker used by retention cleanup after artifact upload.
ci/sai/known_hosts	Pinned SAI SSH host key material for StrictHostKeyChecking.
ci/sai/install_cleanup_cron.sh	Installs cleanup script into user libexec and manages a marked crontab block.
ci/sai/configure_ssh_client.sh	Creates a locked-down SSH config/key directory from an Environment secret.
ci/sai/collect_remote_artifacts.sh	Whitelist-based artifact collector producing a tar.gz stream from the run root.
ci/sai/cleanup_sai_runs.sh	Retention cleanup for uploaded/incomplete runs, diagnostics, and old transfers.
ci/sai/build_gpu.sh	Builds ABACUS with selected CUDA/NCCL/cuSolverMp/cuBLASMp stack and records linkage.
ci/sai/build_gpu.sbatch	Slurm submission script for the build step.
.github/workflows/sai-gpu-full.yml	Main scheduled/manual workflow driving SSH-based rebuild + validation + artifacts.
.github/workflows/sai-bootstrap.yml	Manual bootstrap workflow for probing SAI and installing cleanup cron.