Improve DynamicPPL benchmarks

[yebai]

DynamicPPL’s current benchmarks are fairly noisy: ratios vary substantially across runs. The current setup also compares the same set of models on both main and a PR branch, which makes benchmarks hard to run locally.

It would be useful to improve the benchmark setup in two ways:

1. Reduce run-to-run noise (ie, run all experiments in a single CI job)
2. Introduce an external baseline, such as Stan, and report benchmark ratios relative to Stan’s log density and gradient evaluations rather than relative to main.

Concrete suggestions

• Report primal in absolute time
• Report gradient / primal

Reference: chalk-lab/Mooncake.jl#1163 (comment)

[penelopeysm]

The current setup also compares the same set of models on both main and a PR branch, which makes benchmarks hard to run locally.

You can julia --project=benchmarks benchmarks/benchmarks.jl (https://github.com/TuringLang/DynamicPPL.jl/tree/main/benchmarks). If you want to run comparative benchmarks it is always going to be necessary to do two different checkouts and changing the CI setup won't affect that.

[penelopeysm]

Maybe another way of putting it is that the benchmarks try to answer the question of whether this PR is better or worse than main. Introducing an external benchmark doesn't actually simplify that: you still have to benchmark main against the external, and then benchmark the PR against the external, and then divide through.

[ArpanC6]

Hlw @yebai I'd like to work on this! I've gone through the issue and @penelopeysm's comments and have a clearer picture of the scope.

My plan

1. Run all benchmark experiments in a single CI job to reduce run to run noise
2. Report primal evaluation time in absolute terms
3. Report gradient/primal ratio
4. Introduce Stan as an external baseline for comparison

I understand from @penelopeysm's comment that introducing an external baseline doesn't eliminate the need for two checkouts for comparative benchmarks I'll keep that in mind while designing the solution.

Could you confirm this issue is open for contribution? I'll start with a draft approach and open a PR once I have something to show.