Batch solver

[klamike]

This PR adds a batched solver to MadIPM. It takes as input implementations of AbstractBatchNLPModel from BatchQuadraticModels.jl.

The solver mimics closely the unbatched MadIPM algorithm, trying to keep the batched and unbatched code as close as possible not only in terms of the computations but also the syntax and code structure. The hope is to eventually (not in this PR) unify many of the functions.

Things I'd like to leave for future PRs:

• Fix the initialization Fix the initialization #93
• Add more batch KKT systems (Normal, Condensed, etc.)
• Unify batched and unbatched functions/kernels (wait for MadCore)
• MOI interface (wait for Add Batched{S} set jump-dev/MathOptInterface.jl#2904)
• MadDiff support Add support for batched JVP/VJP with MadIPM.SparseUniformBatchKKTSystem MadDiff.jl#12
• CUDA graph support (note batch CUDSS cannot be captured, so graph breaks are needed. some prototype here)

[frapac]

@klamike very nice work! I will review this PR in detail.

By any chance, can you share the script you are using to run the benchmark on the DC-OPF instances?

[sshin23]

This is awesome. Congratulations @klamike. Looks like we have strong scaling for the pglib_opf_case14_ieee case

[klamike]

Why a copy-paste ? There cannot be an abstract type that allows sharing code between the two ?

I didn't think about it too deeply, you are right that we can still have some code sharing. Once we have MadCore we can put generic MOI stuff there.

[klamike]

Note that 8baa4a0 also removes device sync for the unbatched solver. It improves the performance (which makes the batch speedup worse, but still ~10x on 1354x64)

[klamike]

Okay, the code should be final now, including the tests. Note MadIPM doesn't have docs.

I'm getting started on the paper writing now. There can be another PR if any changes are needed for the paper benchmarks. And once we have MadCore ~70% of this code can probably be moved there.

[frapac]

I just skimmed through the latest changes, they all look good to me.

I would suggest we move on and merge this PR. We can continue optimizing the code later, in subsequent PRs.

We now depend on BatchQuadraticModels, which is not released. I think it's fine keeping it, as it is specified in the [sources] inside the Project.toml. Long term, we will see if we do our own version of QuadraticModels with batch support.