test_runner: fix suite rerun edge case #5711

Job	Run time
Build slim tarball	2m 19s
aarch64-linux: with shared libraries	22m 3s
x86_64-linux: with shared libraries	27m 29s
x86_64-darwin: with shared libraries	36m 46s
aarch64-darwin: with shared libraries	21m 35s
	1h 50m 12s

Provide feedback