Add test sharding, proactive clean, and retry logic for self-hosted CI #4514

Job	Run time
Detect File Changes	9s
Lint Gate	1m 16s
Github (macos, mpi, debug, false)	1h 21m 23s
Github (ubuntu, mpi, no-debug, false)	2h 15m 26s
Github (ubuntu, mpi, debug, true)	2h 24m 51s
Github (ubuntu, mpi, debug, false)	2h 40m 37s
Github (macos, mpi, no-debug, false)	1h 14m 43s
Github (ubuntu, no-mpi, single, no-debug, false)	28m 37s
Georgia Tech \| Phoenix (gpu-acc)	1h 17m 42s
Github (ubuntu, mpi, no-debug, true)	1h 33m 48s
Oak Ridge \| Frontier (AMD) (cpu)	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	3h 48m 9s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	3h 48m 9s
Georgia Tech \| Phoenix (gpu-omp)	1s
Oak Ridge \| Frontier (gpu-acc [1/2])	1s
Oak Ridge \| Frontier (gpu-omp [1/2])	1s
Georgia Tech \| Phoenix (cpu)	35m 18s
Oak Ridge \| Frontier (gpu-acc [2/2])	1s
Oak Ridge \| Frontier (gpu-omp [2/2])	3h 48m 9s
Oak Ridge \| Frontier (cpu)	3h 48m 9s
	1d 5h 6m 31s

Provide feedback