Add test sharding, proactive clean, and retry logic for self-hosted CI #4568

Job	Run time
Lint Gate	1m 15s
Detect File Changes	10s
Github (macos, mpi, no-debug, false)	1h 42m 11s
Oak Ridge \| Frontier (AMD) (cpu)	1s
Oak Ridge \| Frontier (gpu-acc [1/2])	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1s
Github (ubuntu, mpi, no-debug, true)	1h 33m 31s
Georgia Tech \| Phoenix (cpu)	1h 2m 3s
Github (ubuntu, mpi, debug, true)	1h 40m 32s
Oak Ridge \| Frontier (gpu-omp [2/2])	7h 43m 54s
Github (ubuntu, no-mpi, single, no-debug, false)	28m 46s
Github (ubuntu, mpi, debug, false)	2h 44m 0s
Georgia Tech \| Phoenix (gpu-acc)	1h 4m 41s
Oak Ridge \| Frontier (gpu-acc [2/2])	1s
Github (ubuntu, mpi, no-debug, false)	2h 38m 59s
Georgia Tech \| Phoenix (gpu-omp)	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	7h 43m 54s
Oak Ridge \| Frontier (cpu)	7h 43m 54s
Oak Ridge \| Frontier (gpu-omp [1/2])	1s
Github (macos, mpi, debug, false)	1h 11m 50s
	1d 13h 19m 46s

Provide feedback