A collection of experimental optimizers implemented according to the Optimisers.jl interface. We intend to use this package as a testing ground for new optimization algorithms, and then possibly get them incorporated into the main Optimisers.jl package. As such, please do not expect much stability from this package.
pkg> add Optimisers
pkg> add https://github.com/MurrellGroup/CannotWaitForTheseOptimisers.jlusing CannotWaitForTheseOptimisers, OptimisersThis package currently includes attempts at implementing:
- Muon which performs an orthogonalization step before parameter update, and seems excellent for training transformers.
- Apollo which tracks low rank moments using a random projection, reducing the memory footprint of the optimizer.
- NormGrowthCap which prevents the norm of the parameters from growing too quickly.