Drift correction (#58)

* Adding metabolomics filtering feature

* Metabolomics example data set

* Adding test for metabolomics filtering function

* Filter functions added.

* Updating and cleaning up filtering docstrings

* Filtering example api jupyter notebook added.

* Changed filtering function name.

* ✨ Feat: Add script and docs for QC-based LOESS drift correction module

* ✏️ Fix: prevent division by zero, edit documentation

* Adding metabolomics example data for drift correction function testing

* Editing example aradopsis metabolomics data (removing faulty row).

* 📝 Adding api example notebook for drift correction

* Adding jupytext file for drift correction api example.

* ✨ Start collecting dataframe schemas for input and output DataFrames (#29)

* 🚧 start collecting dataframe schemas with enrichment analysis result

* ✨ ensure valid numeric omics data functionality

* 🚧 add schema validation to examples

- can be added to function call using `pa.check_types`

* 🐛 add missing dependency

* 🐛 bool and boolean with NANs are not the same for pandera.

* 🐛 fix tests (convert dtypes expect bool)

* 🚧 Add and align differential analysis schema

- ANOVA descriptive statistis seems to have errors
- should missing values for p-values be allowed?

* 🐛 leave strings as they are

* 🚧 build schema on the fly for numeric data

- could be used to give specific feedback on user provided columns

* 🎨 add pandera to mapped types of other libraries

* 🚧 add Schemas to functions

- anova has two return types depending on number of groups...

* 🚧 add two schemas to anova function

Schemas should be unified eventually

* 📝 all commands for local execution of docs

... as integration test

* 🐛 remove testing shell script

to find errors in action runs (for automation)

* ✨ construct types for exploratory analysis

- create a separate PR to refactor exploratory analysis module

* 🎨 add Sebastians hint on BaseModel visualization

* 🎨 clean-up types

- formatting was not done properly in docs
- not used as a composite type as of now

* ✨ custom PR template for new module (#39)

* ✨ custom PR template for new module

* 🎨 highlight only folder name

* 📝 propose to use built-in virtual environment

* 📝 some more design hints

* 📝 any docstring type for functions is fine

* 🎨 remove intermediate heading

* 📝 add hint on using example data for api examples

* Update .github/workflows/PULL_REQUEST_TEMPLATE/module.md

---------

Co-authored-by: feliciaschulz <112621625+feliciaschulz@users.noreply.github.com>

* 🔧 Update CI (#45)

* 🔧 remove constraint on numpy as latest inmoose does nto require it anymore

* 🔧 align github actions to python package template

... as for other librarier

* 🎨 format toml with even better toml

* 🔧 isort and ruff configuration

- turn ruff configuration on in a separate PR

* 🎨 format src folder

* 🎨 apply ruff check autofixes

* 🔥 remove tox artefacts

* 🚚 rename actions yaml file

* 🎨 isort tests

* 🚚 separate batch correction from normalization (#46)

* 🚚 separate batch correction from normalization

- fix Move batch correction outside Normalisation 💄
Fixes #22

* 🐛 sync normalization nb and update test imports

* 🎨 format and remove unused import

* 🐛 sync nb

* 🐛 add batch correction to website index

* 🎨 link module api

* Documentation updates (#47)

* 🎨 move from rst to md

* 🚧 Close clean up conf.py
Fixes #21

- remove duplication
- sort code as in other packages
- double check if additional entries wer needed

* 🎨 consistent scaling and naming (#48)

* 🎨 consistent scaling and naming

- apply standardization before PCA (leads to minor change in axis)
- talk about batch correction, not normalization

* 🎨 shorten titles

* 🎨 order more meaningfully

* 📝 add Contributing.md as reference to PR template (#49)

* 📝 add Contributing.md as reference to PR template

* 🐛 make it an url

* Metabolomics example data (#50)

* 🚧 process metabolomics example data

* 🎨 format

* ✨ tested data with ANOVA, finish processing of data

* Adding function for visualising example of the LOESS drift correction.

* Adding functions for alternative drift correction method: CPCA

* Adding CPCA drift correction to api examples and refining code.

* Adding example data for metabolomics

* Creating jupytext file for drift correction api example

* 🎨 format tutorials and code (black and isort)

* 🎨 apply ruff suggestions

* docs: update markdown reference

* 🎨 Updating index.md to add drift correction api example.

* 🎨 put drift correction plotting functions from src to api examples

* 🎨 implement logger in drift correction functions to replace print statements

* remove excel lock file

* adding synced jupytext drift_correction.py

* Changing import order for ruff checks

* docs: update markdown reference

* 🎨 removing unused imports

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 Typo in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 Typo in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 Typo in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 fixing error in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* docs: update markdown reference

* 🎨 ensuring x_qc is sorted so CubicSpline works correctly

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 fixing linting errors from ruff

* 🎨 filtering issue: opposite functionality to what the documentation explained. fixed.

* docs: update markdown reference

* 🎨 implementing changes suggested by copilot: NaN value handling, removing temp name column before returning df, raising ValueError with wrong QC threshold

* 🎨 isort on multiple files to improve order of inputs.

* 🎨 format notebooks a bit more and add filtering to index

- hide long source cells
- few typos found by copilot
- split long lines using rewrap tool

* 🎨 clear outputs and hide installation panel output

* 🎨 remove unnamed column on integer indices in display

* 🎨 Changing example data for loess-based drift correction api example. Minor other changes in documentation of notebook.

* 🎨 fixing black errors

* 🎨 improvements in visibility

* 🐛 set defaults for parameters

- anyway not optional, so just set defaults

* docs: update markdown reference

* 🎨 saveguard string and switch to logging

* 🎨 format

* 🐛 remove outdated parameter from call

---------

Co-authored-by: Henry Webel <heweb@dtu.dk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: 4 people

docs/api_examples/batch_correction.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "27b8d53e",
+   "id": "2b2c0d3a",
    "metadata": {
     "lines_to_next_cell": 2
    },
@@ -19,7 +19,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "dcc1505c",
+   "id": "e452352c",
    "metadata": {
     "tags": [
      "hide-output"
@@ -33,7 +33,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "563697dc",
+   "id": "c7399c16",
    "metadata": {
     "tags": [
      "hide-input"
@@ -107,7 +107,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60ee225e",
+   "id": "cd6853a6",
    "metadata": {},
    "source": [
     "\n",
@@ -117,7 +117,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a32698b4",
+   "id": "cf489def",
    "metadata": {
     "tags": [
      "parameters"
@@ -141,7 +141,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9bd7687c",
+   "id": "14907bce",
    "metadata": {},
    "source": [
     "## Data loading\n",
@@ -151,7 +151,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ced30460",
+   "id": "fde7c5dd",
    "metadata": {
     "tags": [
      "hide-input"
@@ -169,7 +169,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "55b42bc2",
+   "id": "2113619e",
    "metadata": {},
    "source": [
     "Metadata here is of type integer. All floats are proteomics measurements."
@@ -178,7 +178,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "857b0522",
+   "id": "afd00cfd",
    "metadata": {
     "tags": [
      "hide-input"
@@ -192,7 +192,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "23657991",
+   "id": "29a05170",
    "metadata": {
     "tags": [
      "hide-input"
@@ -206,7 +206,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8bef0df5",
+   "id": "86a0d548",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -216,7 +216,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "713a7b35",
+   "id": "7bf84ee6",
    "metadata": {},
    "source": [
     "## Before batch correction\n",
@@ -226,7 +226,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "986f7cab",
+   "id": "0b0c854c",
    "metadata": {
     "tags": [
      "hide-input"
@@ -242,7 +242,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2bf8ca74",
+   "id": "426ee866",
    "metadata": {
     "lines_to_next_cell": 0
    },
@@ -258,7 +258,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1cf6f58b",
+   "id": "f9ba51bd",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -273,7 +273,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ca46af9e",
+   "id": "1aa67514",
    "metadata": {},
    "source": [
     "Plot PCA and UMAP after batch correction on standard normalized data"
@@ -282,7 +282,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "cdc4afa2",
+   "id": "907ed6c7",
    "metadata": {
     "tags": [
      "hide-input"
@@ -296,7 +296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9fda9a50",
+   "id": "a530c6ba",
    "metadata": {},
    "source": [
     "See change by substracting combat corrected data from original data.\n",
@@ -306,7 +306,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e9cecdd8",
+   "id": "88ab0d76",
    "metadata": {
     "tags": [
      "hide-input"
@@ -319,7 +319,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d27e6f10",
+   "id": "84ad3f0d",
    "metadata": {},
    "source": [
     "Done."