Add test sharding, proactive clean, and retry logic for self-hosted CI #4338

Job	Run time
Detect File Changes	12s
Lint Gate	1m 57s
Github (macos, mpi, no-debug, false)	1h 32m 41s
Github (ubuntu, mpi, debug, true)	1h 43m 30s
Github (macos, mpi, debug, false)	48m 55s
Github (ubuntu, mpi, debug, false)	1h 52m 25s
Github (ubuntu, mpi, no-debug, false)	1h 52m 24s
Github (ubuntu, mpi, no-debug, true)	1h 33m 31s
Oak Ridge \| Frontier (gpu-acc [2/2])	1s
Georgia Tech \| Phoenix (gpu-acc)	1s
Github (ubuntu, no-mpi, single, no-debug, false)	28m 39s
Oak Ridge \| Frontier (AMD) (cpu)	1h 45m 8s
Georgia Tech \| Phoenix (gpu-omp)	1s
Oak Ridge \| Frontier (gpu-acc [1/2])	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1h 45m 8s
Oak Ridge \| Frontier (gpu-omp [1/2])	1h 9m 17s
Georgia Tech \| Phoenix (cpu)	1s
Oak Ridge \| Frontier (gpu-omp [2/2])	1h 45m 8s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	1s
Oak Ridge \| Frontier (cpu)	1h 45m 8s
	18h 4m 9s

Provide feedback