Fix 'list index out of range' Error with 'mlflow_logging': True and 'max_iter': 1

[Stickic-cyber]

Why are these changes needed?

This PR handles empty manual_run_ids to prevent 'list index out of range' error.
The fix ensures that if manual_run_ids is empty, best_mlflow_run_id falls back to self.parent_run_id or the active MLflow run ID before attempting to access manual_run_ids.

Related issue number

closes #1416

Checks

• I've used pre-commit to lint the changes in this PR (note the same in integrated in our CI checks).
• I've included any doc changes needed for https://microsoft.github.io/FLAML/. See https://microsoft.github.io/FLAML/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[thinkall]

Thank you @Stickic-cyber for the PR. Could you please provide more details for the PR?

[Stickic-cyber]

Hi @thinkall

I have updated the pull request and resubmitted it with the following changes:

1.Handling empty manual_run_ids:
The original code assumes manual_run_ids always has entries, but if it is empty, an IndexError would occur. The updated code checks if manual_run_ids is empty and, if so, uses the parent_run_id or the current active run ID from MLflow as a fallback.

2.Safe access to _config_history:
The original code assumes _config_history[automl._best_iteration] exists, which could lead to a KeyError if it's missing. The new code first checks if _best_iteration exists in _config_history before attempting to access it, preventing errors.

[thinkall]

I think a better way is to do search for max_iter = 1. See my comments below.

[thinkall]

In current design, max_iter = 1 makes no sense to FLAML. There will be no mlflow run created for it. When max_iter>1, self.mlflow_integration.record_state(self, search_state, estimator) will be called to create mlflow runs to record states.

The key here is that no run is created and appended to manual_run_ids. Set it to parent_run or current active run is not correct.

I see two ways to fix the issue:

1. Don't skip search when max_iter = 1, it's acceptable as one trial won't take a lot of time, thus won't affect the test process (I think people set max_iter to 1 for quick test purpose).

2. Add mlflow_integration.record_state to the logic of max_iter < 2.

Need to test which one is better, i.e., won't introduce new bugs.

[thinkall]

@Stickic-cyber it seems that you missed my detailed comments on your code changes.

[thinkall]

If we do search for max_iter=1, this change won't be needed.

[thinkall]

@Stickic-cyber , please follow https://microsoft.github.io/FLAML/docs/Contribute#pre-commit to fix format issue.

[Stickic-cyber]

Thanks for your review and suggestions!

I see your point regarding max_iter = 1. I'll test both approaches:

1.Running the search even when max_iter = 1.

2.Adding mlflow_integration.record_state when max_iter < 2.

I'll check which one works best without introducing new issues. Also, I'll follow the pre-commit guidelines to fix the formatting.

Thanks again for the feedback! I'll update the PR accordingly.

[Stickic-cyber]

After testing, I believe Option 1 is more suitable (my previous attempt was somewhat redundant). If we were to use Option 2, adding logic to the record_state function would first require modifying the judgment logic in thesearch function within automl.py. However, since this is already functioning correctly (as shown in the pre-commit), I think introducing additional logic to the record_state function at this stage would be inefficient and unnecessarily complex.

[Stickic-cyber]

@Stickic-cyber please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree

[thinkall]

However, since this is already functioning correctly (as shown in the pre-commit)

I don't understand this part.

[thinkall]

If we were to use Option 2, adding logic to the record_state function would first require modifying the judgment logic in thesearch function within automl.py.

I actually think we should go with this solution as it seems to be the safest option.