Kegg ks (#77)

* 🚧 start implementing KEGG retrival

* 🚧 kegg related functionality

* Drift correction (#58)

* Adding metabolomics filtering feature

* Metabolomics example data set

* Adding test for metabolomics filtering function

* Filter functions added.

* Updating and cleaning up filtering docstrings

* Filtering example api jupyter notebook added.

* Changed filtering function name.

* ✨ Feat: Add script and docs for QC-based LOESS drift correction module

* ✏️ Fix: prevent division by zero, edit documentation

* Adding metabolomics example data for drift correction function testing

* Editing example aradopsis metabolomics data (removing faulty row).

* 📝 Adding api example notebook for drift correction

* Adding jupytext file for drift correction api example.

* ✨ Start collecting dataframe schemas for input and output DataFrames (#29)

* 🚧 start collecting dataframe schemas with enrichment analysis result

* ✨ ensure valid numeric omics data functionality

* 🚧 add schema validation to examples

- can be added to function call using `pa.check_types`

* 🐛 add missing dependency

* 🐛 bool and boolean with NANs are not the same for pandera.

* 🐛 fix tests (convert dtypes expect bool)

* 🚧 Add and align differential analysis schema

- ANOVA descriptive statistis seems to have errors
- should missing values for p-values be allowed?

* 🐛 leave strings as they are

* 🚧 build schema on the fly for numeric data

- could be used to give specific feedback on user provided columns

* 🎨 add pandera to mapped types of other libraries

* 🚧 add Schemas to functions

- anova has two return types depending on number of groups...

* 🚧 add two schemas to anova function

Schemas should be unified eventually

* 📝 all commands for local execution of docs

... as integration test

* 🐛 remove testing shell script

to find errors in action runs (for automation)

* ✨ construct types for exploratory analysis

- create a separate PR to refactor exploratory analysis module

* 🎨 add Sebastians hint on BaseModel visualization

* 🎨 clean-up types

- formatting was not done properly in docs
- not used as a composite type as of now

* ✨ custom PR template for new module (#39)

* ✨ custom PR template for new module

* 🎨 highlight only folder name

* 📝 propose to use built-in virtual environment

* 📝 some more design hints

* 📝 any docstring type for functions is fine

* 🎨 remove intermediate heading

* 📝 add hint on using example data for api examples

* Update .github/workflows/PULL_REQUEST_TEMPLATE/module.md

---------

Co-authored-by: feliciaschulz <112621625+feliciaschulz@users.noreply.github.com>

* 🔧 Update CI (#45)

* 🔧 remove constraint on numpy as latest inmoose does nto require it anymore

* 🔧 align github actions to python package template

... as for other librarier

* 🎨 format toml with even better toml

* 🔧 isort and ruff configuration

- turn ruff configuration on in a separate PR

* 🎨 format src folder

* 🎨 apply ruff check autofixes

* 🔥 remove tox artefacts

* 🚚 rename actions yaml file

* 🎨 isort tests

* 🚚 separate batch correction from normalization (#46)

* 🚚 separate batch correction from normalization

- fix Move batch correction outside Normalisation 💄
Fixes #22

* 🐛 sync normalization nb and update test imports

* 🎨 format and remove unused import

* 🐛 sync nb

* 🐛 add batch correction to website index

* 🎨 link module api

* Documentation updates (#47)

* 🎨 move from rst to md

* 🚧 Close clean up conf.py
Fixes #21

- remove duplication
- sort code as in other packages
- double check if additional entries wer needed

* 🎨 consistent scaling and naming (#48)

* 🎨 consistent scaling and naming

- apply standardization before PCA (leads to minor change in axis)
- talk about batch correction, not normalization

* 🎨 shorten titles

* 🎨 order more meaningfully

* 📝 add Contributing.md as reference to PR template (#49)

* 📝 add Contributing.md as reference to PR template

* 🐛 make it an url

* Metabolomics example data (#50)

* 🚧 process metabolomics example data

* 🎨 format

* ✨ tested data with ANOVA, finish processing of data

* Adding function for visualising example of the LOESS drift correction.

* Adding functions for alternative drift correction method: CPCA

* Adding CPCA drift correction to api examples and refining code.

* Adding example data for metabolomics

* Creating jupytext file for drift correction api example

* 🎨 format tutorials and code (black and isort)

* 🎨 apply ruff suggestions

* docs: update markdown reference

* 🎨 Updating index.md to add drift correction api example.

* 🎨 put drift correction plotting functions from src to api examples

* 🎨 implement logger in drift correction functions to replace print statements

* remove excel lock file

* adding synced jupytext drift_correction.py

* Changing import order for ruff checks

* docs: update markdown reference

* 🎨 removing unused imports

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 Typo in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 Typo in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 Typo in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 fixing error in documentation

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* docs: update markdown reference

* 🎨 ensuring x_qc is sorted so CubicSpline works correctly

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* 🎨 fixing linting errors from ruff

* 🎨 filtering issue: opposite functionality to what the documentation explained. fixed.

* docs: update markdown reference

* 🎨 implementing changes suggested by copilot: NaN value handling, removing temp name column before returning df, raising ValueError with wrong QC threshold

* 🎨 isort on multiple files to improve order of inputs.

* 🎨 format notebooks a bit more and add filtering to index

- hide long source cells
- few typos found by copilot
- split long lines using rewrap tool

* 🎨 clear outputs and hide installation panel output

* 🎨 remove unnamed column on integer indices in display

* 🎨 Changing example data for loess-based drift correction api example. Minor other changes in documentation of notebook.

* 🎨 fixing black errors

* 🎨 improvements in visibility

* 🐛 set defaults for parameters

- anyway not optional, so just set defaults

* docs: update markdown reference

* 🎨 saveguard string and switch to logging

* 🎨 format

* 🐛 remove outdated parameter from call

---------

Co-authored-by: Henry Webel <heweb@dtu.dk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Filter metabolomics revised (#72)

* 🔥 Removing old metabolomics filtering scripts

* 🎨 Adding new filtering functions for metabolomics analyses.

* 🔥 Removing old filtering api example synced .py file.

* 🚧 Starting new api example notebook for filtering function in metabolomics.
Work in progress.

* 🗃️ Adding new data set for metabolomics data. Raw version of previously existing data set, unfiltered and unimputed.

* 🎨 Finishing first version of revised api example notebook for the metabolomics filtering functions.

* 🎨 Fixing ruff, black, isort checks on filtering api examples.

* docs: update markdown reference

* 🎨 Addressing codex comments: type annotations & raising error if only one QC sample is provided in filter_cv().

* docs: update markdown reference

* 🎨 Fixing ruff issues (import order).

* docs: update markdown reference

* 🔧 increase per cell timeout limit to 4mins

* 🎨 style notebook

- formatting long lines
- combining some cells

* 🔧 remove out-of-life Python Version 3.9

* 🔧 add Python 3.14, remove 3.9

* 🎨 format

* 🚧 add some basic tests

* 🔥 remove old code

* 📝 just add hint on relation to scipy.stats.variation

* ✅ add equality tests to tests of new filtering methods

* 🎨 add mechanics for modified filtering in using groups

* 🎨 format

* 🐛 add copilot suggestions

* docs: update markdown reference

* 🎨 Adding option of using string to represent column name to parameter 'groups' in missingness filtering function.

* docs: update markdown reference

* ✅ test filter by missingness with column for groups

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Henry Webel <heweb@dtu.dk>

* Imputation metabolomics (#73)

* 🔥 Removing old filtering api example synced .py file.

* 🚧 Starting new api example notebook for filtering function in metabolomics.
Work in progress.

* 🎨 Finishing first version of revised api example notebook for the metabolomics filtering functions.

* 🎨 Fixing ruff, black, isort checks on filtering api examples.

* ✨ Adding new metabolomics-specific imputation functions to existing imputation module.

* 🎨 Adding functionality for choosing sub-groups in rows for imputation (so that controls can be excluded, for example)

* ✏️ Fixing typos in documentation in metabolomics MS api example.

* 🎨 Finished version of the imputation api example notebook specialised for metabolomics data analysis.

* 🎨 Creating synced jupytext file for metabolomics imputation api example.

* 🗃️ Adding filtered metabolomics data set.

* 🔧 Add imputation metabolomics api example to index.md

* 🎨 Fixing ruff, isort, black checks.

* docs: update markdown reference

* ⏪ to main branch version

* 🔧 sync notebooks

* 🎨 raise error if non-numeric columns are selected

better to be explicit about it. just warn in case data is not specified

* 🎨 formatting and ensuring numeric column type

- first drop metadata, than transpose to ensure numeric data

* 🎨 address copilot comments

* docs: update markdown reference

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Henry Webel <heweb@dtu.dk>

* 🚧 add pubchem lookup for KEGG

* 🚚 move file to more meaningful name

* ✨ add dump for fast lookup of pubchem ids to kegg ids

* 🚧 pubchem.py

* ✨ add parsing of kegg api returns

* docs: update markdown reference

* 🔥 remove pubchem custom calls

- replace using pubchempy package

* docs: update markdown reference

* Apply suggestions from code review

- minor hints

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* docs: update markdown reference

* try to use native function

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: feliciaschulz <112621625+feliciaschulz@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: 4 people

MANIFEST.in

@@ -3,6 +3,7 @@ include CONTRIBUTING.rst
 include HISTORY.rst
 include LICENSE
 include README.rst
+include src/acore/io/pubchem_to_kegg_ids.csv
 
 recursive-include tests *
 recursive-exclude * __pycache__

docs/api_examples/enrichment_analysis.py

@@ -112,6 +112,10 @@
 # %% [markdown]
 # # Download functional annotations, here pathways, for the protein groups
 # in our selection of the dataset.
+#
+# Check the `acore.io.uniprot` module for more details on how to fetch and process
+# annotations from UniProt, especially check the link given there on all the available
+# fields: [return_fields](https://www.uniprot.org/help/return_fields)
 
 # %%
 from acore.io.uniprot import fetch_annotations, process_annotations
@@ -129,7 +133,6 @@
     # cache the annotations
     fname.parent.mkdir(exist_ok=True, parents=True)
     annotations.to_csv(fname, index=True)
-
 annotations
 
 # %% [markdown]

docs/markdown_ref/acore.io.md

@@ -50,19 +50,76 @@ Download a file from the internet.
 
 ## Submodules
 
+## acore.io.download module
+
+Download files from the internet.
+
+### download_file(url: [str](https://docs.python.org/3/library/stdtypes.html#str), local_filename: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)
+
+Download a file from the internet.
+
 ## acore.io.ftp module
 
 ### download_from_ftp(ftp_url: [str](https://docs.python.org/3/library/stdtypes.html#str), user: [str](https://docs.python.org/3/library/stdtypes.html#str), password: [str](https://docs.python.org/3/library/stdtypes.html#str), to: [str](https://docs.python.org/3/library/stdtypes.html#str), file_name) → [str](https://docs.python.org/3/library/stdtypes.html#str)
 
 Download a file from an FTP server.
 
-## acore.io.http module
+## acore.io.kegg module
 
-Download files from the internet.
+### link_kegg_batch(target_db: [str](https://docs.python.org/3/library/stdtypes.html#str), gene_ids: [Iterable](https://docs.python.org/3/library/typing.html#typing.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str)]) → [str](https://docs.python.org/3/library/stdtypes.html#str)
 
-### download_file(url: [str](https://docs.python.org/3/library/stdtypes.html#str), local_filename: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)
+Fetch from KEGG in batches informations.
 
-Download a file from the internet.
+Docs: [https://www.kegg.jp/kegg/rest/keggapi.html](https://www.kegg.jp/kegg/rest/keggapi.html)
+
+* **Parameters:**
+  * **target_db** ([*str*](https://docs.python.org/3/library/stdtypes.html#str)) – Target endpoint of KEGG API, e.g. “ko” or “pathway”.
+  * **gene_ids** ([*list*](https://docs.python.org/3/library/stdtypes.html#list) *of* [*str*](https://docs.python.org/3/library/stdtypes.html#str)) – List of KEGG gene IDs to query.
+* **Returns:**
+  A list of strings containing the fetched information.
+* **Return type:**
+  [str](https://docs.python.org/3/library/stdtypes.html#str)
+
+### fetch_kegg_ko_descriptions(ko_terms: [Iterable](https://docs.python.org/3/library/typing.html#typing.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str)], timeout: [float](https://docs.python.org/3/library/functions.html#float) = 30.0) → DataFrame
+
+Fetch common descriptions for KEGG KO terms.
+
+* **Parameters:**
+  * **ko_terms** (*Iterable* *[*[*str*](https://docs.python.org/3/library/stdtypes.html#str) *]*) – KEGG KO identifiers such as `K03007` or `ko:K03007`.
+  * **timeout** ([*float*](https://docs.python.org/3/library/functions.html#float) *,* *optional*) – Timeout in seconds for each KEGG API request.
+* **Returns:**
+  A DataFrame with columns `ko_term`, `symbol` and
+  `common_description`.
+* **Return type:**
+  pd.DataFrame
+
+### Notes
+
+The KEGG API accepts up to 10 entry identifiers per request. This helper
+batches larger inputs automatically.
+
+### cid_to_kegg_id(pubchem_cid: [int](https://docs.python.org/3/library/functions.html#int)) → [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)
+
+Convert a single PubChem CID to a KEGG compound ID via KEGG conv API.
+
+### parse_compound_pathway_mapping(raw_mapping: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)]]
+
+Parse tab-delimited KEGG-style compound/pathway mappings into a dictionary.
+
+### parse_kegg_name_description(raw_text: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [str](https://docs.python.org/3/library/stdtypes.html#str)]]
+
+Parse KEGG pathway entries into ENTRY -> {NAME, DESCRIPTION}.
+
+### lookup_cid_to_kegg_id(pubchem_cid: [Iterable](https://docs.python.org/3/library/typing.html#typing.Iterable)[[int](https://docs.python.org/3/library/functions.html#int)]) → Series | [None](https://docs.python.org/3/library/constants.html#None)
+
+Look up KEGG IDs for a list of PubChem CIDs using a pre-downloaded mapping file.
+
+* **Parameters:**
+  **pubchem_cid** (*Iterable* *[*[*int*](https://docs.python.org/3/library/functions.html#int) *]*) – A list of PubChem CIDs to look up.
+* **Returns:**
+  A Series mapping PubChem CIDs to KEGG IDs, or None if no matches are found.
+* **Return type:**
+  pd.Series | None
 
 ## acore.io.pride module
 

docs/markdown_ref/acore.md

@@ -110,10 +110,17 @@
       * [Submodules](acore.io.uniprot.md#submodules)
       * [acore.io.uniprot.uniprot module](acore.io.uniprot.md#module-acore.io.uniprot.uniprot)
   * [Submodules](acore.io.md#submodules)
+  * [acore.io.download module](acore.io.md#module-acore.io.download)
+    * [`download_file()`](acore.io.md#acore.io.download.download_file)
   * [acore.io.ftp module](acore.io.md#module-acore.io.ftp)
     * [`download_from_ftp()`](acore.io.md#acore.io.ftp.download_from_ftp)
-  * [acore.io.http module](acore.io.md#module-acore.io.http)
-    * [`download_file()`](acore.io.md#acore.io.http.download_file)
+  * [acore.io.kegg module](acore.io.md#module-acore.io.kegg)
+    * [`link_kegg_batch()`](acore.io.md#acore.io.kegg.link_kegg_batch)
+    * [`fetch_kegg_ko_descriptions()`](acore.io.md#acore.io.kegg.fetch_kegg_ko_descriptions)
+    * [`cid_to_kegg_id()`](acore.io.md#acore.io.kegg.cid_to_kegg_id)
+    * [`parse_compound_pathway_mapping()`](acore.io.md#acore.io.kegg.parse_compound_pathway_mapping)
+    * [`parse_kegg_name_description()`](acore.io.md#acore.io.kegg.parse_kegg_name_description)
+    * [`lookup_cid_to_kegg_id()`](acore.io.md#acore.io.kegg.lookup_cid_to_kegg_id)
   * [acore.io.pride module](acore.io.md#module-acore.io.pride)
     * [`download_PRIDE_data()`](acore.io.md#acore.io.pride.download_PRIDE_data)
   * [acore.io.uncompress module](acore.io.md#module-acore.io.uncompress)

docs/markdown_ref/modules.md

@@ -73,8 +73,9 @@
       * [`download_file()`](acore.io.md#acore.io.download_file)
       * [Subpackages](acore.io.md#subpackages)
       * [Submodules](acore.io.md#submodules)
+      * [acore.io.download module](acore.io.md#module-acore.io.download)
       * [acore.io.ftp module](acore.io.md#module-acore.io.ftp)
-      * [acore.io.http module](acore.io.md#module-acore.io.http)
+      * [acore.io.kegg module](acore.io.md#module-acore.io.kegg)
       * [acore.io.pride module](acore.io.md#module-acore.io.pride)
       * [acore.io.uncompress module](acore.io.md#module-acore.io.uncompress)
     * [acore.kaplan_meier_analysis package](acore.kaplan_meier_analysis.md)

src/acore/io/__init__.py

@@ -1,4 +1,4 @@
-from .http import download_file
+from .download import download_file
 from .pride import download_PRIDE_data
 from .uncompress import unrar