Co-authored by @ArneBouillon

Purpose

• Derived from the LIS work of Add likelihood-informed data processors #376
• Closes Add new utilities for the likelihood-informed DataProcessor #370

Content

• Migration of the LIS from Add likelihood-informed data processors #376 under the following simplifications/changes:
  • [change] encoder_kwargs for structure vectors now keep samples_in and samples_out in their respective distributions, and also contain the algorithm time values dt corresponding to the distributions
  • [simplification] we no longer provide the dim_criterion, only a threshold for doing dimension reduction
  • [change] the user provides iters (Vector{Int}) not alphas (Vector{FT}) to indicate which dt values they would like to use for dimension reduction. A message reports the dt of the requested iterations
  • [change] if multiple iters are requested, by default we compute the LIS for each iter, then use a trapezoidal rule to create the final subspace.
  • [simplified] Reduced the complexity of the DimReduction example
  • [change] added a shift by the data mean
  • [change] Use a more robust information measure f(k) = log(1+k)^2 and truncate based on cumsum(f(k))/sum(f(k)) > eps. Leads to cutoff eps that is not "0.99999"
• Extended encoder_kwargs_from(eki,prior; g_final=...) to obtain all kwargs for encoding a typical problem. Additionally, we provide a convenience where the user can provide the "g" ensemble for a final evaluation of "u" in from the eki
• Recreated the DimensionReduction example for a linear problem to assess performance

MISC

• Refactor the boundary between public/private encode/decode functions.
  • API (to Utilities): encode_data, encode_structure_matrix, encode_with_schedule applied to encoder_schedule. NOT applicable to processors.
  • Private: _encode_data _encode_structure_matrix only applicable to exact processors etc.
  • API (to Emulators): encode_data and encode_structure_matrix (and decode counterparts) allow application to emulator in place of encoder_schedule
• add small check for error message
• created Docs for DimensionReduction Example #418 as I have not included a docs page for the new example.
• One concern I have: The noise_injector in the forward map wrapper can induce instability in the MCMC algorithm, so we might consider changing the default here. (this is not for inflating the posterior, but for evaluating the likelihood when providing reduced parameters to the forward map. This may really be a symptom from the need to resample with MC over several forward map evaluations when injecting the noise)
• Closes Counterintuitive value i of get_training_points(ekp, i) #419 Small (breaking) bugfix spotted when resolving codecov

Example DimensionReduction

We just try the DR methods (using the forward map wrapper in place of an emulator) for a linear example in 100D->100D: The following is output

┌ Info: (reduced_dim, error) of posterior mean to whitened "reference" solution,
└  when using the truncation ({for PCA}, {for LI})
5×6 DataFrame
 Row │ reference        PCA-in-out      PCA(mat)-in-out  LI-in(1:1)-out(1:1)  LI-in(1:5)-out(1:1)  truncation       
     │ Tuple…           Tuple…          Tuple…           Tuple…               Tuple…               Tuple…           
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ (100, 0.277785)  (41, 0.713683)  (55, 0.306185)   (50, 0.303924)       (32, 0.29261)        (0.995, 0.99995)
   2 │ (100, 0.319642)  (32, 0.798957)  (38, 0.320583)   (34, 0.312999)       (23, 0.339262)       (0.99, 0.9995)
   3 │ (100, 0.29158)   (23, 0.825192)  (23, 0.344845)   (21, 0.323374)       (17, 0.354786)       (0.98, 0.995)
   4 │ (100, 0.309952)  (12, 0.815737)  (11, 0.484712)   (14, 0.390514)       (12, 0.433838)       (0.95, 0.97)
   5 │ (100, 0.301116)  (7, 0.781519)   (6, 0.542278)    (9, 0.520454)        (8, 0.459082)        (0.9, 0.9)

We see that as you increase truncation, our new LI methods reduce errors comparably, or more more slowly, than PCA alone. We also see that using only PCA from EKP samples (over using the prior and obs. matrices here) lead to poor errors