[Feature Request]: Cross validation and ensemble voting

[mikaelsimard5]

Description

• Tasks in the project template don't include cross validation capabilities - as of now everything is ran once and there is no possibility of doing say 5 fold cross-validation to obtain more robust metrics.
• Ideally to boost model performance you can run say your 5 trained models as part of cross-validation as an ensemble model, at least for classification.

Why do you need the new feature?

• CV is standard practice in ML so the project template should support that. Depending on the project this may or may not be implemented at various levels as training time linearly scales with the number of folds. For instance, hyper parameters may be tuned on a single fold and the final model may be trained as part of a 5 fold cross-validation.
• Ensemble models are useful to maximise model performance, but may not be practical in some time-sensitive applications. Nonetheless, the project template should support that.

How are you going to use this new feature?
As part of normal training/testing loops, or even in deployment.

Solution suggestion

• Config file includes number of cross-validation folds; 1 defaults to the same code as is now. Data handling, training and mlflow logging require new loops/methods to handle various folds.
• mlflow adjusting/model saving may need some work if up to N ensemble models are saved - the architecture of inference has to change i.e. load the N models on mlflow, run all and average out the results.
• For mlflow logging there should be one parent run with N child run (one per fold) and aggregate metrics logged on the parent - those are the only metrics worth comparing.