It would be nice to make modelfree.hyperparams.train_rl a tune.Trainable rather than a function, adding checkpointing support. This would let us use the HyperBand and Population Based Training schedulers. Conceptually this is easy enough: we already supporting saving models via save_callbacks, and can restore using load_path. However, the interfaces don't quite line up: Ray expects _train to perform one small training step, with _save called in between. There's no good way to make Stable Baselines return part-way. We could call it repeatedly with small total_timesteps, but this would make the progress be wrong, breaking annealers.
It would be nice to make
modelfree.hyperparams.train_rla tune.Trainable rather than a function, adding checkpointing support. This would let us use the HyperBand and Population Based Training schedulers. Conceptually this is easy enough: we already supporting saving models viasave_callbacks, and can restore usingload_path. However, the interfaces don't quite line up: Ray expects_trainto perform one small training step, with_savecalled in between. There's no good way to make Stable Baselines return part-way. We could call it repeatedly with smalltotal_timesteps, but this would make the progress be wrong, breaking annealers.