Harden benchmark workflow: retry builds, proactive clean, robust monitoring #4350

Job	Run time
Lint Gate	1m 17s
Detect File Changes	11s
Github (ubuntu, mpi, no-debug, true)	33m 28s
Github (ubuntu, mpi, debug, false)	33m 26s
Oak Ridge \| Frontier (gpu-acc)	4m 41s
Github (macos, mpi, debug, false)	25m 19s
Github (ubuntu, mpi, no-debug, false)	33m 27s
Github (ubuntu, mpi, debug, true)	33m 28s
Github (ubuntu, no-mpi, single, no-debug, false)	29m 14s
Github (macos, mpi, no-debug, false)	33m 40s
Oak Ridge \| Frontier (gpu-omp)	5m 55s
Georgia Tech \| Phoenix (gpu-acc)	4m 24s
Oak Ridge \| Frontier (AMD) (gpu-omp)	1s
Georgia Tech \| Phoenix (gpu-omp)	4m 33s
Oak Ridge \| Frontier (AMD) (cpu)	1s
Oak Ridge \| Frontier (cpu)	1s
Georgia Tech \| Phoenix (cpu)	5m 54s
Georgia Tech \| Phoenix (gpu-acc)	1h 33m 33s
Georgia Tech \| Phoenix (gpu-omp)	1h 3m 54s
Oak Ridge \| Frontier (gpu-omp)	1h 32m 24s
Oak Ridge \| Frontier (AMD) (cpu)	42m 10s
Oak Ridge \| Frontier (cpu)	1h 12m 56s
Oak Ridge \| Frontier (gpu-acc)	5h 6m 39s
Georgia Tech \| Phoenix (cpu)	1h 0m 10s
Oak Ridge \| Frontier (AMD) (gpu-omp)	4h 38m 7s
Lint Gate	1m 17s
Github (macos, mpi, no-debug, false)	1h 19m 9s
Github (macos, mpi, debug, false)	1h 8m 58s
Github (ubuntu, mpi, debug, true)	1h 48m 56s
Github (ubuntu, mpi, debug, false)	3h 4m 14s
Github (ubuntu, mpi, no-debug, false)	2h 15m 40s
Detect File Changes	11s
Github (ubuntu, mpi, no-debug, true)	1h 34m 0s
Github (ubuntu, no-mpi, single, no-debug, false)	29m 14s
	1d 8h 40m 32s

Provide feedback