You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Replacement of Emulator processing with new encoding framework (#367)
* initial gutting of Emulator transformation and replace with new encoders
* emulator bug-fixes
* add new interface
* remove old normalization/standardizations code
* works when no scheduler provided
* trunc -> trunc_val
* update and cleanup example and plots
* gp case
* cycle over cases and encoders
* n=200
* smaller noise and new schedule train
* more flexible regularizer (now it is not always going to be "I"
* rm unnecessary plots from compare_regression
* new message
* converted Ishigami
* user_... to ...
* emulate
* copy -> deepcopy
* G-function update
* utils
* L63
* L63
* sensible default for EKI-produced data
* new encode/decode works with sampling too
* sinusoid
* new encoder methods and exports
* bugfix truncation of variance
* use new method to encode
* fix regularization for non-I matrices
* update Lorenz and Lorenz spatial dep examples
* updated darcy
* updated cloudy example
* remove standardization
* updated EDMF-data
* updated GCM
* typo
* typo
* added more API and ==
* add ! and mor eunit test coverage for emulators
* remove large comment
* more cases
* GP test updated
* use def. enc. for GP tests
* all return types
* output cases
* updated RF tests
* update interfcace to abstractGP
* updated run tests for MCMC
* format
* missed by foramtter
* docs
* docs
* docstrings and comments addressed
* format
* add docs page
* add docs into index
* Reduce number of methods; give data processors separate files
* Pass both structure matrices to PDCPs
* Format code
* Fix bugs
* Adapt tests to new functions
* replace logic for input obs
* format and emulate docs page
* n_iter scalar
* n_iter -> scalar
* remove default structure matrix, extend framework of encoding a nothing, add tests
* behaviour when passing nothing into ML tools
* format
* when not learning noise, regularization is scaled with output matrix
* remove @info
* typo
* missing \lambda in uniform scaling case
* typo
* format
* missed format file
---------
Co-authored-by: Arne Bouillon <arne.bouillon@kuleuven.be>
Co-authored-by: ArneBouillon <45404227+ArneBouillon@users.noreply.github.com>
# [Data Processing and Dimension Reduction](@id data-proc)
2
+
3
+
## Overview
4
+
When working with high-dimensional problems with modest training data pairs, the bottleneck of CES procedure is the training of a competent emulator. It is often necessary to process and dimensionally-reduce the data to ease the learning task of the emulator. We provide a flexible (and extensible!) framework to create encoders and decoders for this purpose. The framework works as follows
5
+
- An `encoder_schedule` defines the type of processing to be applied to input and/or output spaces
6
+
- The `encoder_schedule` is passed into the `Emulator` where it is initialized and stored.
7
+
- The encoder will be used automatically to encode training data and predictions within the Emulate and Sample routines.
8
+
9
+
An external API is also available using the `encode_data`, and `encode_structure_matrix` methods if needed.
10
+
11
+
## Define an encoder schedule
12
+
13
+
The user may provide an encoder schedule to transform data in a useful way. For example, mapping all output data each dimension to be bounded in [0,1]
14
+
```julia
15
+
simple_schedule = (minmax_scale(), "out")
16
+
```
17
+
The unit of the encoder contains a `DataProcessor`, the type of processing, and an string, whether to apply to input data,`"in"`, or output data `"out"`, or both `"in_and_out"`.
18
+
19
+
The encoder schedule can be a vector of several units that apply multiple `DataProcessors` in order:
1. The inputs are decorrelated with their sample mean and covariance (and projected to low dimensional subspace if necessary) i.e PCA
30
+
2. The scaled inputs are then subject to a "Robust" univariate scaling, mapping 1st-3rd quartiles to [0,1]
31
+
3. The outputs are decorrelated using an "output structure matrix" (provided to the emulator `output_structure_matrix=`). Furthermore, apply a dimension-reduction to a space that retains 95% of the total variance.
32
+
4. In the reduced input-output space, a canonical correlation analysis is performed. Data is oriented and reduced (if necessary) maximize the joint correlation between inputs and outputs.
33
+
34
+
!!! note "Default Encoder schedule"
35
+
The current default encoder schedule applies `decorrelate_structure_mat()` if a structure matrix (input or output) is provided, else it applies `decorrelate_sample_cov()`.
36
+
37
+
!!! note "Switch encoding off"
38
+
To ensure that no encoding is happening, the user must pass in an empty schedule `encoder_schedule = []`
39
+
40
+
41
+
## Creating an emulator with a schedule
42
+
43
+
The schedule is then passed into the Emulator, along with the data and desired structure matrices
44
+
```julia
45
+
emulator =Emulator(
46
+
machine_learning_tool,
47
+
input_output_pairs;
48
+
output_structure_matrix = obs_noise_cov,
49
+
encoder_schedule = complex_schedule,
50
+
)
51
+
```
52
+
Note that due to the item `(decorrelate_structure_mat(retain_var=0.95), "out")` in the schedule, we must provide the `output_structure_matrix`.
53
+
54
+
# Types of data processors
55
+
56
+
We currently provide two main types of data processing: the `DataContainerProcessor` and `PairedDataContainerProcessor`.
57
+
58
+
The `DataContainerProcessor` encodes "input" data agnostic of the "output" data, and vice versa, examples of current implementations are:
59
+
-`UnivariateAffineScaling`: such as `quartile_scale()`, `minmax_scale()`, and `zscore_scale()`, which apply some basic univariate scaling to the data in each dimension
60
+
-`Decorrelator`: such as `decorrelate_structure_mat()` and `decorrelate_sample_cov()`, or `decorrelate()` which perform [(truncated-)PCA](https://en.wikipedia.org/wiki/Singular_value_decomposition) using either the sample-estimated or user-provided covariance matrices (or their sum).
61
+
62
+
The `PairedDataContainerProcessor` encodes inputs (or outputs) using information of the both inputs and outputs in pairs
63
+
-`CanonicalCorrelation` - constructed with `canonical_correlation()`, which performs [canonical correlation analysis](https://en.wikipedia.org/wiki/Canonical_correlation) to process the pairs. In effect this performs PCA on the cross-correlation from input and output samples.
64
+
-[Coming soon]`LikelihoodInformed` - this will use the data or likelihood from the inverse problem at hand to build diagnostic matrices that are used to find informative directions for dimension reduction (e.g., [Cui, Zahm 2021](http://doi.org/10.1088/1361-6420/abeafb), [Baptista, Marzouk, Zahm 2022](https://arxiv.org/abs/2207.08670))
65
+
66
+
This is an extensible framework, and so new data processors can be added to this library.
67
+
68
+
!!! note "Some experiments"
69
+
Some effects of the data processing (with older API) are outlined in a practical setting in the results and appendices of [Howland, Dunbar, Schneider, (2022)](https://doi.org/10.1029/2021MS002735).
Copy file name to clipboardExpand all lines: docs/src/emulate.md
+3-34Lines changed: 3 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,14 +17,11 @@ Wrapping a predefined machine learning tool, e.g. a Gaussian process `gauss_proc
17
17
emulator =Emulator(
18
18
gauss_proc,
19
19
input_output_pairs; # optional arguments after this
20
-
obs_noise_cov = Γy,
21
-
normalize_inputs =true,
22
-
standardize_outputs =true,
23
-
standardize_outputs_factors = factor_vector,
24
-
retained_svd_frac =0.95,
20
+
output_structure_matrix = Γy,
21
+
encoder_schedule = encoder_schedule,
25
22
)
26
23
```
27
-
The optional arguments above relate to the data processing.
24
+
The optional arguments above relate to the data processing, which is described [here](@ref data-proc)
28
25
29
26
### Emulator Training
30
27
@@ -40,34 +37,6 @@ y, cov = Emulator.predict(emulator, new_inputs)
40
37
```
41
38
This returns both a mean value and a covariance.
42
39
43
-
44
-
## Data processing
45
-
46
-
Some effects of the following are outlined in a practical setting in the results and appendices of [Howland, Dunbar, Schneider, (2022)](https://doi.org/10.1029/2021MS002735).
47
-
48
-
### Diagonalization and output dimension reduction
49
-
50
-
This arises from the optional arguments
51
-
-`obs_noise_cov = Γy` (default: `nothing`)
52
-
We always use singular value decomposition to diagonalize the output space, requiring output covariance `Γy`. *Why?* If we need to train a $$\mathbb{R}^{10} \to \mathbb{R}^{100}$$ emulator, diagonalization allows us to instead train 100 $$\mathbb{R}^{10} \to \mathbb{R}^{1}$$ emulators (far cheaper).
53
-
-`retained_svd_frac = 0.95` (default `1.0`)
54
-
Performance is increased further by throwing away less informative output dimensions, if 95% of the information (i.e., variance) is in the first 40 diagonalized output dimensions then setting `retained_svd_frac=0.95` will train only 40 emulators.
55
-
56
-
!!! note
57
-
Diagonalization is an approximation. It is however a good approximation when the observational covariance varies slowly in the parameter space.
58
-
!!! warn
59
-
Severe approximation errors can occur if `obs_noise_cov` is not provided.
60
-
61
-
62
-
### Normalization and standardization
63
-
64
-
This arises from the optional arguments
65
-
-`normalize_inputs = true` (default: `true`)
66
-
We normalize the input data in a standard way by centering, and scaling with the empirical covariance
To help with poor conditioning of the covariance matrix, users can also standardize each output dimension with by a multiplicative factor given by the elements of `factor_vector`.
70
-
71
40
## [Modular interface](@id modular-interface)
72
41
73
42
Developers may contribute new tools by performing the following
0 commit comments