Add test sharding, proactive clean, and retry logic for self-hosted CI #4637

Job	Run time
Detect File Changes	8s
Lint Gate	1m 18s
Github (ubuntu, mpi, no-debug, true)	27m 32s
Github (ubuntu, mpi, debug, false)	26m 21s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	1s
Github (ubuntu, mpi, debug, true)	26m 2s
Oak Ridge \| Frontier (AMD) (cpu)	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1s
Github (macos, mpi, debug, false)	31m 41s
Georgia Tech \| Phoenix (cpu)	1s
Georgia Tech \| Phoenix (gpu-acc)	31m 33s
Github (ubuntu, mpi, no-debug, false)	26m 32s
Oak Ridge \| Frontier (gpu-acc [2/2])	31m 33s
Github (ubuntu, no-mpi, single, no-debug, false)	26m 49s
Oak Ridge \| Frontier (gpu-acc [1/2])	31m 33s
Github (macos, mpi, no-debug, false)	31m 54s
Oak Ridge \| Frontier (gpu-omp [2/2])	1s
Georgia Tech \| Phoenix (gpu-omp)	31m 33s
Oak Ridge \| Frontier (cpu)	1s
Oak Ridge \| Frontier (gpu-omp [1/2])	31m 33s
	5h 56m 8s

Provide feedback