Harden CI for self-hosted SLURM runners #4255

Job	Run time
Lint Gate	1m 18s
Detect File Changes	9s
Github (ubuntu, mpi, debug, true)	2h 12m 38s
Github (ubuntu, mpi, no-debug, true)	1h 33m 32s
Github (ubuntu, mpi, no-debug, false)	2h 19m 43s
Github (ubuntu, mpi, debug, false)	2h 11m 45s
Oak Ridge \| Frontier (gpu-acc [1/2])	1s
Georgia Tech \| Phoenix (cpu)	1h 9m 8s
Oak Ridge \| Frontier (gpu-omp [1/2])	1s
Oak Ridge \| Frontier (cpu)	3h 39m 28s
Georgia Tech \| Phoenix (gpu-acc)	1h 22m 38s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1s
Oak Ridge \| Frontier (AMD) (cpu)	3h 39m 28s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	3h 39m 28s
Github (macos, mpi, no-debug, false)	1h 18m 13s
Github (macos, mpi, debug, false)	1h 8m 28s
Oak Ridge \| Frontier (gpu-omp [2/2])	3h 39m 28s
Github (ubuntu, no-mpi, single, no-debug, false)	28m 26s
Oak Ridge \| Frontier (gpu-acc [2/2])	1s
Georgia Tech \| Phoenix (gpu-omp)	1h 21m 28s
	1d 5h 45m 22s

Provide feedback