helper and docs for minibatching

[odunbar]

Purpose

• Closes Minibatching compatability with CES, internalize flattening operations #436
  See docs here

Content

• add flattening helper, and use it in get_training_points and encoder_kwargs_from. The helper takes pairs (u,g_batched) with for example with g_batched of size 3*dim_y x n_samples and converts it to ( (u,g_1), (u,g_2), (u,g_3)) where g_i is of size dim_y x n_samples. It allows for different sized minibatches
• this helper is now used int get_training_points(...) and encoder_kwargs_from(...)
• add unit tests for the helper
• add docs to explain the behaviour with minibatching. Included a note about increased sensitivity to noise distribution when one has "same-input-different-output" data

MISC

• Created issue docs-wide @ref for menitoned methods and structs. #440 to resolve the @ref in docs

---

• I have read and checked the items on the review checklist.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.04%. Comparing base (185c1a6) to head (4f7d4d3).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #438      +/-   ##
==========================================
+ Coverage   94.97%   95.04%   +0.06%     
==========================================
  Files          12       12              
  Lines        2408     2441      +33     
==========================================
+ Hits         2287     2320      +33     
  Misses        121      121              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[braghiere]

Nice! I hit exactly this (get_training_points couldn't stack different-sized minibatch outputs) calibrating the land model, so glad it's fixed upstream. The noise note is a great call. A few things below.

[braghiere]

The final_samples_out contract (stack as [g_b1; g_b2; …] matching the next cycle's batches) is easy to get wrong, and it's only checked for divisibility. a wrong order silently mis-pairs noise. I avoided this by evaluating the final ensemble on the full obs vector. Could it take the unbatched output and split deterministically, or at least validate the ordering?

[odunbar]

I am not sure i follow. Passing in unbatched g_batch still doesn't help because if the minibatch are for the observations [15,7,2,34] then how are you ensuring that you pass in the four g_batch items in the correct order? Or do you mean that you are running the model to produce the entirity of all minibatches e.g. [1,...,100] and then extracting the 4 you want? Because this kind of sidesteps the entire minibatching procedure.

We also can't check ordering, as g_final is always a matrix, so there is no way to validate generally what order it is.

However i would say it isn't necessarily "easy to get wrong" as you just run "the next iteration" using the gotten minibatch from EKP. As the user has already done this for the iterations in EKP it should just be akin to calling it again (without the update_ensemble! call). to ensure folks are aware of the syntax though I have added this into the docstring:

get_training_points(
      ekp::EnsembleKalmanProcesses.EnsembleKalmanProcess{FT, IT, P},
      train_iterations::Union{AbstractVector{IT}, IT} where IT;
      g_final
  ) -> EnsembleKalmanProcesses.DataContainers.PairedDataContainer
  

  Extract and flatten the training data from an EnsembleKalmanProcess into a PairedDataContainer
  suitable for training an Emulator.

  Arguments
  ≡≡≡≡≡≡≡≡≡

    •  ekp: EnsembleKalmanProcess holding the parameter ensemble and forward-model outputs.
    •  train_iterations: integer n (uses iterations 1:n) or an index vector such as 3:2:9.
    •  g_final (keyword, default nothing): optional AbstractMatrix of forward-model outputs for
       the final parameter ensemble (not yet stored in ekp), sized as get_g(ekp, 1).

  To ensure consistency in g_final, we recommend that, given the ekp::EnsembleKalmanProcess and
  prior::ParameterDistribution used to generate it, the user evaluates the forward map G on the
  parameters returned by get_ϕ_final:

  params = get_ϕ_final(prior, ekp)
  g_final = reduce(hcat, [G(param) for param in eachcol(params)])

  If ekp is using minibatching, G must be evaluated on the same minibatch that ekp is currently on:

  params = get_ϕ_final(prior, ekp)
  batch_final = get_current_minibatch(ekp)
  g_final = reduce(hcat, [G(param, batch_final) for param in eachcol(params)])

  See EnsembleKalmanProcesses.Observations for more on minibatching and ObservationSeries.