Add test sharding, proactive clean, and retry logic for self-hosted CI #4548

Job	Run time
Lint Gate	1m 17s
Detect File Changes	10s
Github (macos, mpi, no-debug, false)	1h 13m 48s
Github (ubuntu, mpi, debug, false)	5h 15m 53s
Github (ubuntu, mpi, debug, true)	1h 39m 10s
Github (ubuntu, no-mpi, single, no-debug, false)	28m 21s
Github (ubuntu, mpi, no-debug, true)	1h 35m 21s
Github (macos, mpi, debug, false)	1h 10m 31s
Georgia Tech \| Phoenix (gpu-omp)	6h 47m 49s
Github (ubuntu, mpi, no-debug, false)	2h 15m 36s
Oak Ridge \| Frontier (AMD) (gpu-omp [2/2])	6h 47m 49s
Georgia Tech \| Phoenix (gpu-acc)	31m 57s
Oak Ridge \| Frontier (gpu-acc [2/2])	1s
Oak Ridge \| Frontier (AMD) (cpu)	1s
Oak Ridge \| Frontier (cpu)	1s
Oak Ridge \| Frontier (gpu-omp [2/2])	1s
Georgia Tech \| Phoenix (cpu)	1s
Oak Ridge \| Frontier (AMD) (gpu-omp [1/2])	1s
Oak Ridge \| Frontier (gpu-omp [1/2])	6h 47m 49s
Oak Ridge \| Frontier (gpu-acc [1/2])	6h 47m 49s
	1d 17h 23m 26s

Provide feedback