Harden CI for self-hosted SLURM runners #4272

Job	Run time
Lint Gate	1m 19s
Detect File Changes	10s
Github (macos, mpi, no-debug, false)	1h 17m 46s
Github (macos, mpi, debug, false)	1h 26m 53s
Github (ubuntu, mpi, debug, true)	1h 41m 8s
Georgia Tech \| Phoenix (gpu-acc)	11s
Oak Ridge \| Frontier (AMD) (cpu)	44m 54s
Github (ubuntu, mpi, debug, false)	3h 10m 42s
Github (ubuntu, no-mpi, single, no-debug, false)	28m 46s
Github (ubuntu, mpi, no-debug, true)	1h 33m 54s
Oak Ridge \| Frontier (gpu-acc [2/2])	1s
Georgia Tech \| Phoenix (cpu)	11s
Oak Ridge \| Frontier (cpu)	7h 2m 14s
Oak Ridge \| Frontier (gpu-omp [2/2])	7h 2m 14s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1s
Oak Ridge \| Frontier (gpu-acc [1/2])	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	1s
Oak Ridge \| Frontier (gpu-omp [1/2])	1s
Github (ubuntu, mpi, no-debug, false)	2h 15m 37s
Georgia Tech \| Phoenix (gpu-omp)	1h 33m 59s
	1d 4h 20m 3s

Provide feedback