Provide AD-based defaults for GaussNewton strategy, and default for ContinuousMultivariateLogPdf

[ofSingularMind]

I think my use case may be general/common enough to bake in as the default for projecting Continuous(Uni)MultivariateLogPdf functions using the Bonnet/GaussNewton strategies.

The following is the code I use to project my ContinuousMultivariateLogPdf onto Gaussian:

# using previously defined my_pdf <: ContinuousMultivariateLogPdf
function my_logpdf!(out, x)
    out[1] = my_pdf.logpdf(x)
    return out
end
function my_grad!(out, x)
    out .= ForwardDiff.gradient(my_pdf.logpdf, x)
    return out
end
function my_hess!(out, x)
    out .= ForwardDiff.hessian(my_pdf.logpdf, x)
    return out
end

inplace = ExponentialFamilyProjection.InplaceLogpdfGradHess(my_logpdf!, my_grad!, my_hess!)
params = ProjectionParameters(
    tolerance = 1e-8,
    strategy = ExponentialFamilyProjection.GaussNewton(nsamples = 1), # deterministic
)
prj = ProjectedTo(MvNormalMeanCovariance, D; parameters = params)
result = project_to(prj, inplace) # <--- note inplace

So, it seems to me that what is feasible and good would be to allow the following to be automatically equivalent to the above by default, without providing the three mutation-functions:

# using previously defined my_pdf <: ContinuousMultivariateLogPdf
params = ProjectionParameters(
    tolerance = 1e-8,
    strategy = ExponentialFamilyProjection.GaussNewton(nsamples = 1), # deterministic
)
prj = ProjectedTo(MvNormalMeanCovariance, D; parameters = params)
result = project_to(prj, my_pdf) # <--- note my_pdf is here now, instead of inplace

And so we are saying, okay, if you provide just a pdf with this strategy, we assume you intend AD-based grad/hess.

And sure, if we want a different AD-library (e.g. DifferentiationInterface.jl), we bake that in instead

Good? :)

[gdalle]

Hi! Following @Nimrais's tag, a few remarks on this:

• In large-scale settings, you probably want to use Enzyme instead of ForwardDiff
• If you do use ForwardDiff, you can
  • Pick the in-place versions of gradient! and hessian!
  • Pre-compute the config object to avoid type instabilities and additional allocations
• You can also try DifferentiationInterface, which allows you to switch between backends and whose preparation mechanism abstracts away stuff like ForwardDiff's config