Use `Optim.jl` for optimisation

[yebai]

Turing.jl currently uses optimisation algorithms (eg, l-BFGS) in Optim.jl via Optimization.jl/OptimizationOptimJL.jl, which adds a layer of indirection. Is there any reason for that?

For long-term stability, we may want to reduce the dependency footprint and use Optim.jl directly.

EDIT: another related deps is SciMLBase.

[penelopeysm]

Is there any reason for that?

Yes indeed, using Optimization means that you aren't tied to one backend only and you can use any solver that Optimization has an interface for. That is the entire purpose of Optimization.jl, to provide a common interface to all the backends. However, I would also point out that the use of Optimization.jl predates my time on the project and so perhaps you might also want to consult someone else about it? #2221 #2214

we may want to reduce the dependency footprint and use Optim.jl directly.

Optimization itself has very few deps because all of its code is already loaded in the sub-packages, so I don't think that removing Optimization will lead to any significant difference in the dependency footprint https://github.com/SciML/Optimization.jl/blob/master/Project.toml. Most of the deps that one might want to cut from there are concerned with logging, but AbstractMCMC already pulls them in as well: https://github.com/TuringLang/AbstractMCMC.jl/blob/main/Project.toml

If this is about removing DI, I have to inform you that DI is still a hard dep of Optim.jl (via NLSolversBase.jl). Also, it's one thing to remove it as a dep of Turing but unless you succeed in removing it as a dep from SciML, there'd still be no way anyone can import SciML + Turing together. That means we would have to, for example, remove everything to do with differential equations in the docs.

[penelopeysm]

If this is something you want to do, we could loop in SciML devs on this.

[yebai]

It’s okay for Turing.jl to indirectly depend on DI. DynamicPPL, Bijectors and inference algorithms should avoid DI.

[penelopeysm]

Optimization itself has very few deps because all of its code is already loaded in the sub-packages, so I don't think that removing Optimization will lead to any significant difference in the dependency footprint

To be more concrete, removing Optimization and OptimizationOptimJL, and adding Optim back, gives the following diff in the Manifest:

- [9aa1b823] FastClosures v0.3.2
- [7f7a1694] Optimization v5.5.1
- [bca83a33] OptimizationBase v5.1.1
- [36348300] OptimizationOptimJL v0.4.12
- [9f842d2f] SparseConnectivityTracer v1.2.1
- [0a514795] SparseMatrixColorings v0.4.27

out of 177 packages in the Manifest. On top of removing the ability to use any other solver in Optimization.jl, removing it would also undo a significant amount of work that has gone into this codebase, so I would suggest that it's not worth it.