Documentation update (#26)

* Cleaned up README.md to be a bit more clean and clear (namely, removing newlines which simulated soft-wrap poorly)

* Cleaned up README.md to be a bit more clean and clear (namely, removing newlines which simulated soft-wrap poorly)

* Added docstring for 'registered_data_hook'

* Updated docstrings for the ABCs for data hooks

* Updated the docstrings of data-encoding hooks

* Updated DocStrings for the 'feature_selection.py' data hooks. Also fixes a slight error in the SampleNullityDrop hook, where it was evaluating checking the threshold against sample count (rather than feature count)

* [Minor] Corrected indentation of use-case to match the rest of the doc-strings

* Added the docstring for the Imputation data hooks (of which there is currently only one: SimpleImputation)

* Added the docstring for the Standardization data hooks (of which there is currently only one: StandardScaling)

* Added missing docstring for the "evaluate_param" function, used by SciKit-Learn's root ModelManager

* Updated docstrings for the SciKit-Learn Ensemble models (mostly adding example usage)

* Updated docstring for the Linear models provided by this tool. Note the extended discussion of how this implementation works around SciKit-Learn's triple-variant approach!

* Fixed incorrect indentation in the use-case examples for Ensemble models. Oops

* Extended the docstring of KNeighborsClassifierManager to include example usage. Woohoo

* Added example usage to the SVC docstring.

* Extended the docstring of the tuning utility classes, to clarify their use in the broader context of the framework.

* Update data/hooks/encoding.py

Addendum suggested by valosekj

Co-authored-by: Jan Valosek <39456460+valosekj@users.noreply.github.com>

* Update data/hooks/encoding.py

Added additional (common) parameter, as suggested by Jan

Co-authored-by: Jan Valosek <39456460+valosekj@users.noreply.github.com>

* Update README.md

Swapped to GitHub Block formatting, which is a lot better at drawing the eye of the user to this note. Thanks Jan for pointing this out!

Co-authored-by: Jan Valosek <39456460+valosekj@users.noreply.github.com>

* Update data/hooks/encoding.py

Modified suggestion by Jan

* Initial Sphinx docs addition. Still WIP, be gentle

* Further improves to the documentation, mostly restructuring and clarification

* Minor formatting correction to the FAQ page; its header should now actuall say what it is

* Initial commit of the Getting Started page

* Initial addition of the "walkthrough" for MOOP; still very much WIP

* Extended the data config documentation to discuss data hooks.

* Initial addition of model configuration + parameter tuning walkthrough

* [Minor] Grammar correction in model config docs

* Added model config template for easy user reference

* Added the template file for the study config before I forget to later

* Initial commit of the study config documentation (walk-through); last step is documenting how to interpret it!

* Some clean-up and re-wiring of cross-references in docs. Should be a bit nicer to navigate now

* Trimmed the titles of the walkthrough, as they were too elaborate for the context of a tutorial

* Added missing comma in the model config tutorial JSON example; oops

* Yet another missing comma aaaaa

* Slight correction to the parameters for the output path in the study config tutorial

* Initial results interpretation discussion added to docs

* Added missing results.csv, using in result interpretation docs

* First addition to results, showing common plots which can be generated from a MOOP analysis

* Extended results documentation, showing how to compare two different MOOP runs to one another via plotting

* Added warning about an edge case bug, where the data is erroneously coded as an "object" type if the database write is interrupted during a MOOPs run

* Initial addition of statistical analyses in the walkthrough; still very much WIP

* Swapped to PyData theme; should help with eye strain via its dark mode.

* [Minor] Corrected misleading comment in statistics documentation. Oops

* Added section on calculating statistics for single runs, place before multi-run comparison

* Added a section detailing some common statistical analyses that can be run using MOOP's results

* Fixed ".. codeblock" appearing erroneously in some doc pages

* Clarified an example of the dataset in question

* Initial ReadTheDocs commit!

* Clarified null-like data formatting

* Changed header of "index" section to remove duplicate adjacent headers

* Fixed strategy used in docs

---------

Kalum Ost

Author: valosekj

.gitignore

@@ -3,6 +3,7 @@
 /testing/output/output.db
 /testing/private/*
 /.idea/*
+/docs/build/
 *__pycache__/
 .vscode/
 .DS_Store

.readthedocs.yaml

@@ -0,0 +1,13 @@
+version: "2"
+
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.12"
+
+python:
+  install:
+    - requirements: docs/requirements.txt
+
+sphinx:
+  configuration: docs/source/conf.py

README.md

@@ -15,27 +15,23 @@ it can be easily extended to allow for the analysis of any tabular dataset.
    * `conda activate modular_optuna_ml`
    * `mamba activate modular_optuna_ml`
 4. Done!
-5. This only sets up the tool to be run; you will still need to create the configuration files for the analyses you want to run (see `testing` for an example).
+
+> [!NOTE]
+> This only sets up the tool to be run; you will still need to create the configuration files for the analyses you want to run (see the `testing` directory for some examples).
 
 ## Running the Program
 
 Four files are needed to run an analysis
 
-* A tabular dataset, containing the metrics you want to run the analysis on
-  * Should contain at least 1 independent and 1 target metric; unsupervised analyses are currently 
-    not supported
+* A tabular dataset (usually a `.tsv` file), containing the metrics you want to run the analysis on
+  * Should contain at least 1 independent and 1 target metric; unsupervised analyses are currently not supported
 * A data configuration file; this defines where a dataset is and what pre-processing methods
-should be applied to its contents. An example, alongside the dataset it manages, can be found 
-in `testing/iris_data/`
+should be applied to its contents. An example, alongside the dataset it manages, can be found in `testing/iris_data/`
 * A model configuration file; this defines which ML model to test, which hyper-parameters to tune,
 and how to tune them. A few examples are available in `testing/model_configs/`
-* A study configuration file; this defines which metrics to evaluated throughout the runtime of the
-analysis, and where to save the results (currently only supports an SQLite DB output format). An
-example is provided in `testing/testing_study_config.json`
+* A study configuration file; this defines which metrics to evaluated throughout the runtime of the analysis, and where to save the results (currently only supports an SQLite DB output format). An example is provided in `testing/testing_study_config.json`
 
-Once all three have been created, and you have installed all dependencies (detailed in 
-`environment.yml`) simply run the following command (replacing the values within the 
-curly brackets with the corresponding file name):
+Once all three have been created, and you have installed all dependencies (detailed in `environment.yml`) simply run the following command (replacing the values within the curly brackets with the corresponding file name):
 
 `python run_ml_analysis.py -d {data_config} -m {model_config} -s {study_config}`
 
@@ -51,48 +47,30 @@ The overall structure of the analysis can be broken down into the following broa
 
 1. **Configuration Loading:** All configuration files are loaded and checked for validity. 
 2. **Dataset Loading:** The tabular dataset designated in the data configuration file is loaded
-   * If a target column is specified, it is split off the dataset at this point to isolate it from 
-   pre-processing (see below)
-3. **Study Initialization:** An Optuna study is initialized, set up to run `n_trials` trials as specified 
-in the study config file.
-   * All steps past this point occur per-trial, sampling from the corresponding `Trial` instance to 
-   determine the hyperparameters to use.
-   * Configuration files denote a parameter as being "trial tunable" by placing a dictionary in the 
-   place of a constant; an example of this can be seen in the `penalty` parameter for the 
+   * If a target column is specified, it is split off the dataset at this point to isolate it from pre-processing (see below)
+3. **Study Initialization:** An Optuna study is initialized, set up to run `n_trials` trials as specified in the study config file.
+   * All steps past this point occur per-trial, sampling from the corresponding `Trial` instance to determine the hyperparameters to use.
+   * Configuration files denote a parameter as being "trial tunable" by placing a dictionary in the place of a constant; an example of this can be seen in the `penalty` parameter for the 
    `testing/model_configs/log_reg.json` file.
-   * Details on how hyper-parameters are sampled via Optuna Trials can be found 
-   (here)[https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html].
-4. **Universal Pre-Processing:** Any data processing hooks for which `"run_per_replicate": true` are 
-run on the dataset in its entirety
+   * Details on how hyper-parameters are sampled via Optuna Trials can be found (here)[https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html].
+4. **Universal Pre-Processing:** Any data processing hooks for which `"run_per_replicate": true` are run on the dataset in its entirety
    * If a data processing hook does not specify a `run_per_replicate` value, it defaults to `true`.
-5. **In-Out Splits:** The dataset is split via a stratified k-fold split into in- and out-groups,
-`n_replicates` times.
+5. **In-Out Splits:** The dataset is split via a stratified k-fold split into in- and out-groups, `n_replicates` times.
    * As the parameter name implies, each of these splits will make up an analytical "replicate"
    * Any post-split hooks for which `"run_per_replicate": true` will also run here, fitting to the 
    in-dataset and transforming both the in- and out-dataset if possible 
    * If a data processing hook does not specify a `run_per_replicate` value, it defaults to `true`.
-   * NOTE: Despite this occurring per-trial, the RNG state being fixed prior to study start ensures that
-   the in-out datasets are the same for all trials, so long as universal pre-processing did not delete
-   and samples during its run-time
-6. **Replicate Pre-Processing:** For each in-dataset, any data processing hooks for which 
-`"run_per_cross": true` are run on the in-dataset.
+   * NOTE: Despite this occurring per-trial, the RNG state being fixed prior to study start ensures that the in-out datasets are the same for all trials, so long as universal pre-processing did not delete any samples during its run-time
+6. **Replicate Pre-Processing:** For each in-dataset, any data processing hooks for which "run_per_cross": true` are run on the in-dataset.
    * If a data processing hook does not specify a `run_per_cross` value, it defaults to `false`.
-7. **Train-Test Splits:** The validation dataset is split via a stratified k-fold split into 
-`n_crosses` splits, as defined in the study configuration file.
+7. **Train-Test Splits:** The validation dataset is split via a stratified k-fold split into `n_crosses` splits, as defined in the study configuration file.
    * As the parameter name implies, each of these splits will make up an analytical "cross"
-   * Any post-split hooks for which `"run_per_cross": true` will also run here, fitting to the 
-   train dataset and transforming both the train and test set if possible
+   * Any post-split hooks for which `"run_per_cross": true` will also run here, fitting to the train dataset and transforming both the train and test set if possible
    * If a data processing hook does not specify a `run_per_cross` value, it defaults to `false`.
-8. **Cross-Validate Performance Reported:** Any metrics that the user requested be tracked are 
-calculated. These metrics are defined in the study config like so.
-   * `train`: Evaluate the metric on a model which has been trained on the training set, evaluating the 
-   metric from the model itself, or from the model's output when applied to the test set.
-     * As a result of this being run once per cross, each metric specified at this hook will result in
-        `n_crosses` values being output (each denoted as `{metric_name} [{cross_idx}]`)
-   * `validate`: Evaluate the metric on a model which has been trained on the in-dataset, evaluating the 
-   metric from the model itself, or from the model's output when applied to the in-dataset.
-   * `test`: Evaluate the metric on a model which has been trained on the in-dataset, evaluating the 
-   metric from the model itself, or from the model's output when applied to the out-dataset.
-   * `objective`: Evaluated identically to the `train` hook, but reported as an average both to you 
-   and the study instance (allowing the study to guide the hyperparameter sampling in future trials)
+8. **Cross-Validate Performance Reported:** Any metrics that the user requested be tracked are calculated. These metrics are defined in the study config like so.
+   * `train`: Evaluate the metric on a model which has been trained on the training set, evaluating the metric from the model itself, or from the model's output when applied to the test set.
+     * As a result of this being run once per cross, each metric specified at this hook will result in `n_crosses` values being output (each denoted as `{metric_name} [{cross_idx}]`)
+   * `validate`: Evaluate the metric on a model which has been trained on the in-dataset, evaluating the metric from the model itself, or from the model's output when applied to the in-dataset.
+   * `test`: Evaluate the metric on a model which has been trained on the in-dataset, evaluating the metric from the model itself, or from the model's output when applied to the out-dataset.
+   * `objective`: Evaluated identically to the `train` hook, but reported as an average both to you and the study instance (allowing the study to guide the hyperparameter sampling in future trials)
      * Currently, only one `objective` metric can be defined due to this averaging.

config_templates/data_config_template.json

@@ -0,0 +1,37 @@
+{
+  "label": "data_label",
+  "data_source": "path/to/your/data.csv",
+  "format": "tabular",
+  "separator": ",",
+  "index": "column_to_label_samples_with",
+  "pre_split_hooks": [
+    {
+      "type": "hook_type",
+      "param1": 1,
+      "param2": "b"
+    }, {
+      "type": "tunable_hook_type",
+      "tunable_param": {
+        "label": "param_label_in_db",
+        "type": "int",
+        "low": 1,
+        "high": 10
+      }
+    }
+  ],
+  "post_split_hooks": [
+    {
+      "type": "fitted_hook_type",
+      "param3": "For the Greater Good"
+    }, {
+      "type": "fitted_and_tunable_hook_type",
+      "tunable_param": {
+        "label": "fitted_param_label_in_db",
+        "type": "float",
+        "log": true,
+        "low": 0.1,
+        "high": 10.0
+      }
+    }
+  ]
+}

config_templates/model_config_template.json

@@ -0,0 +1,18 @@
+{
+  "label": "model_label",
+  "model": "ModelType",
+  "parameters": {
+    "non_tuned_param": "value",
+    "tuned_float_param": {
+      "label": "float_param_label",
+      "type": "float",
+      "low": 1.0,
+      "high": 2.0
+    },
+    "tuned_choice_param": {
+      "label": "cat_label",
+      "type": "category",
+      "choices": ["A", "B", 3]
+    }
+  }
+}

config_templates/study_config_template.json

@@ -0,0 +1,22 @@
+{
+  "label": "StudyLabel",
+  "random_seed": 12345,
+  "no_replicates": 10,
+  "no_crosses": 10,
+  "no_trials": 10,
+  "target": "metric_label",
+  "objective": "metric_function",
+  "metrics": {
+    "train": [
+      "metric_function"
+    ],
+    "validate": [
+      "metric_function"
+    ],
+    "test": [
+      "metric_function"
+    ]
+  },
+  "track_params": true,
+  "output_path": "path/to/output.db"
+}

data/hooks/__init__.py

@@ -8,6 +8,16 @@
 
 # Decorator to allow for registry key to be kept alongside the class of interest
 def registered_data_hook(key: str):
+    """
+    Decorator for registering a data hook.
+
+    :param key: The label that this data hook will be registered as. This will be the string that the user provides
+        in the 'type' argument within the data configuration file to request the corresponding data hook be used.
+
+    NOTE: Currently, for a data hook to be registered, the package it is part of must be imported in this file
+    specifically. We are looking into a more elegant solution to this, but for now, add new imports to the set
+    placed below this class to ensure the data hooks in that module are registered correctly.
+    """
     def _decorator(cls: Type[DataHook]):
         # Decorator which registers a data manager under a specific key automatically
         if key in DATA_HOOKS.keys():

data/hooks/base.py

@@ -10,8 +10,18 @@ class DataHook(ABC):
     """
     Basic implementation for functions which can be called as data hooks.
 
-    These should be configured during program init (to allow for fail-faster checking),
-    after which they will be run at the user-specified points within the dataset
+    Data hook use is split into two stages: initialization and application.
+
+    During initialization, any configuration settings the user provided are parsed and used to initialize the data hook
+        instance. This is done before any ML analyses are run (to follow the fail-faster paradigm), and is where you
+        should parse any configuration options and check their validity. Anything that needs to be run once at the
+        beginning, before the data hook is applied to any data, should be done here as well. This code is usually placed
+        into the `from_config` function.
+
+    During application, the hook is applied to the data provided to it (in `BaseDataManager` form). At minimum, the
+        data hook will receive a data manager `x` containing feature/regressor values `x`. If the analysis is a
+        supervised one, can also receive a data manager containing the target values `y` as well. See `run` for further
+        details.
     """
     def __init__(self, config: dict, logger: Logger = Logger.root):
         # Basic init implementation which tracks attributes shared with all data hooks
@@ -40,18 +50,27 @@ def from_config(cls, config: dict, logger: Logger = Logger.root) -> Self:
     @abstractmethod
     def run(self, x: BaseDataManager, y: Optional[BaseDataManager] = None) -> BaseDataManager:
         """
-        Run this hook's process on a given DataManager in its entirely.
+        Run this hook's process on a given DataManager in its entirety.
+
+        To avoid potentially propagating data modifications across replicates and/or cross-validation splits, this
+            function should return a modified copy of the original input data, rather than modifying the input
+            DataManager(s) directly!
+
         :param x: The data to process
         :param y: The target metric to use, if the data hook needs it.
-        :return: The data manager, post-processing. For safety, it should generally be its own (copied) instance
+        :return: A copy of the original data manager `x`, with this data hook applied to it.
         """
         ...
 
 
 class FittedDataHook(DataHook, ABC):
     """
-    Data hook which "fits" itself to a set of training data, and uses that fit to
-    inform how it will be applied to other datasets
+    An extended data hook which allows for the hook to be "fit" to a training dataset, then applied to said training
+        dataset AND another testing dataset. This should be used for data hooks which manage data transformations which,
+        if they were applied to the entire dataset indiscriminately, would result in data leakage.
+
+    Like `run` before it, this function should return modified copies of its input data, rather than modifying the input
+        DataManager(s) directly!
     """
     @abstractmethod
     def run_fitted(self,
@@ -62,11 +81,12 @@ def run_fitted(self,
         ) -> (BaseDataManager, BaseDataManager):
         """
         Run this hook's process on a pair of DataManagers, fitting on the training input applying to both
-        :param x_train: The data which should be used to "fit" the hook to, before it is applied to both
-        :param x_test: A dataset which will have the hook applied to it, but not fit to it.
-        :param y_train: The target metric to use during fitting, if the data hook needs it.
-        :param y_test: The target metric to use during application to testing, if the data hook needs it.
+
+        :param x_train: A dataset which will be used to "train" the hook. Post-training, the hook is applied to it as well
+        :param x_test: A dataset which will have the hook applied to it only, but will not affect how the hook is "trained".
+        :param y_train: The target metric associated with `x_train` for each of its samples, should the analyses be a supervised one.
+        :param y_test: The target metric associated with `x_test` for each of its samples, should the analyses be a supervised one.
             NOTE: 'y_test' is here solely for standardization, and should probably never be used to avoid overfitting!
-        :return: The modified versions of x_train and x_test, after the fit has been applied to them
+        :return: The modified copies of x_train and x_test, after the fit has been applied to them.
         """
         ...