[AutoTuner] Restore resume check

[luarss]

This pull request includes several changes to the AutoTuner tests to re-enable previously disabled tests, improve the test setup, and add new utility methods.

Fixes #3005

Re-enabling tests and improving setup:

• flow/test/test_autotuner.sh: Re-enabled the test_tune_resume test in the AutoTuner script by uncommenting the test execution line.

Codebase simplification and improvements:

• tools/AutoTuner/test/resume_check.py: Removed unnecessary directory changes and imported the glob module to facilitate file pattern matching. [1] [2]
• tools/AutoTuner/test/resume_check.py: Introduced the check_trial_times method to check the modification times of trial iterations, improving the robustness of the test_tune_resume test.

[vvbandeira]

This is a little obfuscated:

1. The function's name does not match the expected return value type/format. A check should return True/False.
2. Returning 9e99 is not clear on what is going on.
3. Creating a folder does not confirm the run status. For this test to be true to its purpose, we need to guarantee that it has not finished, not just started.

We should consider using a get_experiment_status function to check if all iterations have finished running; this function would return at least "RUNNING" and "FINISHED", other states might be helpful but are not required now.

[luarss]

1. The function's name does not match the expected return value type/format. A check should return True/False.

Can possibly change it to get_trial_times

2. Returning 9e99 is not clear on what is going on.

It is just a dummy value, to compare for latest modified time in while loop line 121-128

            # Check if first config is complete
            while True:
                cur_modified_time = self.check_trial_times()
                print(f"Current modified time: {cur_modified_time}")
                print(f"Latest modified time: {latest_modified_time}")
                if abs(cur_modified_time - latest_modified_time) < 1e-3:
                    break
                latest_modified_time = cur_modified_time
                time.sleep(10)

3. Creating a folder does not confirm the run status. For this test to be true to its purpose, we need to guarantee that it has not finished, not just started.

This function returns the latest modified time of a given iteration (folder names are matched using iteration glob) - so if a run is completed the folder should no longer be modified.

4, We should consider using a get_experiment_status function to check if all iterations have finished running; this function would return at least "RUNNING" and "FINISHED", other states might be helpful but are not required now.

get_experiment_status is helpful in general, but might not be too useful in resume_check because we need to check iteration completion, as opposed to experiment completion (or all_iterations completion)

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 21 days if no further activity occurs. Remove the Stale label or comment to keep it open.