Nutpie with short chains

[potash]

In some problems I am fitting many (e.g. 8) chains but very short (e.g. as little as 100/100 warmup/sampling iters). I do this to quickly test out a model variation especially in conjunction with cross validation. (Later I'll fit long chains when I write things up.)

I mention this because I noticed in my one time using nutpie (this model), that nutpie was way better than stan when I did 1000/1000 warmup/sampling iters but for 100/100 it was basically a tie (nutpie finished first but had slightly lower ESS for most params).

My questions are:

1. Is this expected/typical of the nutpie algorithm?
2. Are there any settings I should change when using nutpie with short chains?

[aseyboldt]

I never systematically experimented with that little tuning, so I'm really not sure. I would expect both to fail quite a lot unless the model is very friendly.
I did run a larger set of models with 400, 600 and 1000 tuning steps, and usually nutpie did better with fewer tuning draws, but it really can go either way. This is also very noisy, if you run it with different seeds you often get quite different results for the same model.
The mass matrix in nutpie will typically converge much faster in terms of number of gradient evaluations, but stan typically uses more gradient evaluations for the same number of warmup draws, that makes it hard to predict what will happen precisely.

Changes to the mass matrix are generally a bit messy if you combine them with nuts, because the number of leapfrog steps are usually a power of two. A small change in the mass matrix can double or half the number of leapfrog steps if the optimal number of steps is close to a power of two, which will then usually change the effective sample size. So I think just looking at a single model won't tell you much about the overall differences.

As to parameters, maybe the window_switch_rate and early_window_switch_rate parameters will have an effect? But that's pure speculation.