Add test sharding, proactive clean, and retry logic for self-hosted CI #4580

Job	Run time
Lint Gate	1m 16s
Detect File Changes	11s
Github (ubuntu, mpi, no-debug, true)	8m 9s
Github (ubuntu, mpi, no-debug, false)	8m 9s
Github (macos, mpi, no-debug, false)	8m 14s
Oak Ridge \| Frontier (gpu-acc [2/2])	7m 4s
Github (ubuntu, mpi, debug, false)	8m 8s
Github (ubuntu, mpi, debug, true)	8m 8s
Github (macos, mpi, debug, false)	1s
Georgia Tech \| Phoenix (cpu)	1s
Github (ubuntu, no-mpi, single, no-debug, false)	8m 8s
Oak Ridge \| Frontier (gpu-omp [1/2])	1s
Oak Ridge \| Frontier (gpu-acc [1/2])	7m 56s
Georgia Tech \| Phoenix (gpu-acc)	1s
Oak Ridge \| Frontier (cpu)	7m 56s
Georgia Tech \| Phoenix (gpu-omp)	7m 56s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	7m 2s
Oak Ridge \| Frontier (AMD) (cpu)	7m 56s
Oak Ridge \| Frontier (gpu-omp [2/2])	7m 56s
	1h 44m 14s

Provide feedback