Docs + cross-language CI parity gates (phase 7)

Closes the porting plan: top-level README reflects the now-functional
Python contribution path, code/python/README.md gets a getting-started
block + API map, and the Python CI workflow grows two new required
parity gates.

README updates
--------------
- Top-level README: replaced the "Contribution via python is not yet
  functional" paragraph with a description of the yeastgem +
  raven-python split. Updated the load/save example to use
  read_yeast_model / commit_yeast_model (with the saveYeastModel ->
  commitYeastModel rename note). Removed the obsolete .env
  setup-step language (yeastgem auto-detects the repo root).
- code/python/README.md: rewritten as an API map covering the seven
  modules (io, compare, conditions, biomass, missing_fields,
  model_tests, curation) with one-liner descriptions and links into
  the source. Add the dev / pytest / ruff workflows.

CI workflow (.github/workflows/python.yml)
------------------------------------------
- test (matrix Python 3.10/3.11/3.12, ruff + pytest) — unchanged.
- parity-level-1-round-trip (new) — runs
  code/python/tests/ci/check_round_trip.py: load the committed
  model/yeast-GEM.xml via cobrapy, write it to a temp file, reload,
  diff via raven_python.comparison.diff_models. Catches SBML library
  regressions, annotation losses, and accidental id rewrites.
- parity-level-2-metrics (new) — runs
  code/python/tests/ci/check_metrics.py: compute growth R²,
  essential-gene accuracy / sensitivity / specificity / MCC +
  confusion matrix, and anaerobic flux R² on the committed model;
  diff against the MATLAB-produced reference at
  code/python/tests/reference/metrics.json within tolerance.
- The matlab-reference-compare placeholder job is gone — its work is
  now done by the two real parity gates.
- Workflow path filters extended to trigger on changes to
  data/yeastgem/, data/conditions/, data/essentialGenes/, and
  data/physiology/ (everything the CI reads).

Reference metrics
-----------------
code/python/tests/reference/metrics.json seeds the level-2 gate with
the values measured during phase 5 verification (MATLAB R2024b +
Gurobi 13.0 + RAVEN feat/yeast-gem-shared on commit b4d3769):

  growth_r2                0.906164
  essential_genes accuracy 0.902439  (tp/tn/fp/fn 934/65/94/14)
  anaerobic_flux_r2        0.904765

Tolerances absorb the known Gurobi-vs-HiGHS drift around the 1e-6
growth-ratio threshold: gene counts ±2, R² ±5e-3, MCC ±5e-2.

Verified locally:
- parity-level-1-round-trip → Models are semantically equal.
- parity-level-2-metrics → All metric-parity checks passed.

Prerequisite for the CI to pass on GitHub: raven-python's
feat/yeast-gem-shared branch must be pushed so the
``raven-python @ git+https://...@feat/yeast-gem-shared`` URL in
pyproject.toml resolves. Once that branch lands on a tagged release,
the pin can switch to a version constraint.

PORTING_PLAN.md status table marks phase 7 done; the porting plan is
complete.

Author: edkerk

.github/workflows/python.yml

@@ -6,20 +6,31 @@ on:
     paths:
       - 'code/python/**'
       - 'code/io.py'
+      - 'data/yeastgem/**'
+      - 'data/conditions/**'
+      - 'data/essentialGenes/**'
+      - 'data/physiology/**'
       - 'model/**'
       - '.github/workflows/python.yml'
   pull_request:
     branches: [main, develop]
     paths:
       - 'code/python/**'
       - 'code/io.py'
+      - 'data/yeastgem/**'
+      - 'data/conditions/**'
+      - 'data/essentialGenes/**'
+      - 'data/physiology/**'
       - 'model/**'
       - '.github/workflows/python.yml'
 
 jobs:
+  # Unit tests + lint across the supported Python matrix. Fast (~5 min
+  # wall clock per Python version after caches warm).
   test:
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
         python-version: ['3.10', '3.11', '3.12']
 
@@ -44,12 +55,11 @@ jobs:
         working-directory: code/python
         run: pytest -v
 
-  # Level-1 semantic-equality gate vs. the committed MATLAB reference
-  # artifact. Skipped until the reference bundle is seeded (see
-  # code/python/tests/reference/README.md). When enabled, this becomes
-  # a required check per the lock-step parity policy.
-  matlab-reference-compare:
-    if: false  # enable once reference bundle is committed
+  # Level-1 parity — Python SBML read+write of the committed
+  # model/yeast-GEM.xml must round-trip to a semantically-equal model.
+  # Catches SBML library regressions, annotation losses, and
+  # accidental id rewrites.
+  parity-level-1-round-trip:
     runs-on: ubuntu-latest
     needs: test
     steps:
@@ -60,12 +70,37 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: '3.12'
+          cache: pip
+          cache-dependency-path: code/python/pyproject.toml
+
+      - name: Install yeastgem
+        run: pip install -e code/python/
+
+      - name: SBML round-trip preserves model
+        run: python code/python/tests/ci/check_round_trip.py
+
+  # Level-2 parity — Python validation metrics must match the
+  # committed MATLAB-produced reference within tolerance. Tolerances
+  # account for Gurobi-vs-HiGHS solver drift around the essential-gene
+  # 1e-6 growth-ratio threshold. Regenerate the reference via
+  # code/python/tests/reference/runPhase5Metrics.m when the metrics
+  # shift legitimately.
+  parity-level-2-metrics:
+    runs-on: ubuntu-latest
+    needs: test
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+          cache-dependency-path: code/python/pyproject.toml
 
       - name: Install yeastgem
         run: pip install -e code/python/
 
-      - name: Compare Python-loaded model against MATLAB reference
-        run: |
-          python -m yeastgem.compare \
-            model/yeast-GEM.xml \
-            code/python/tests/reference/yeast-GEM.xml
+      - name: Validation metrics match the committed reference
+        run: python code/python/tests/ci/check_metrics.py

README.md

@@ -71,14 +71,26 @@ Please see the installation instructions for each software package.
   * [RAVEN Toolbox](https://github.com/SysBioChalmers/RAVEN) version 2.8.3 or later
 
 * Python-based  
-  Contribution via python (cobrapy) is not yet functional. In essence, if you can retain the same format of the model files, you can still contribute to the development of yeast-GEM. However, you cannot use the MATLAB functions.
-
-  If you want to use any of the [provided](https://github.com/SysBioChalmers/yeast-GEM/tree/main/code) Python functions, you may create an environment with all requirements:
+  Contribution via Python is supported through the `yeastgem` package
+  under [code/python/](code/python/) and its
+  [PORTING_PLAN.md](code/python/PORTING_PLAN.md). The package builds
+  on [cobrapy](https://github.com/opencobra/cobrapy) and
+  [raven-python](https://github.com/SysBioChalmers/raven-python) (the
+  Python port of RAVEN) — the latter provides the generic GEM
+  utilities (`diff_models`, `add_sbo_terms`, condition / biomass /
+  curation helpers) that `yeastgem` configures with the yeast-specific
+  data files under [data/](data/).
+
+  Install from a checkout:
   ```bash
-  pip install -r code/requirements/requirements.txt  # install all dependencies
-  touch .env # create a .env file for locating the root
+  pip install -e code/python/[dev]
   ```
 
+  The release pipeline equivalent to the MATLAB `commitYeastModel`
+  is `yeastgem.commit_yeast_model`. The historical
+  [code/io.py](code/io.py) is kept as a deprecated forwarding shim
+  that re-exports from the new package.
+
 If you want to locally run `memote run` or `memote report history`, you should also install [git lfs](https://git-lfs.github.com/), as `results.db` (the database that stores all memote results) is tracked with git lfs.
 
 ## Model usage
@@ -87,21 +99,26 @@ Make sure to load/save the model with the corresponding wrapper functions:
 * In Matlab:
   ```matlab
   cd ./code
-  model = loadYeastModel(); % loading
-  saveYeastModel(model);    % saving
+  model = loadYeastModel();   % loading
+  commitYeastModel(model);    % saving — release pipeline (was saveYeastModel)
   ```
   * If RAVEN is not installed, you can also use COBRA-native functions (`readCbModel`, `writeCbModel`), but these model-files cannot be committed back to the GitHub repository.
-* In Python:  
-Before opening Python, the following command should (once) be run in the yeast-GEM root folder:  
-  ```bash
-  touch .env # create a .env file for locating the root
-  ```
-  Afterwards, the model can be loaded in Python with:
+  * `saveYeastModel` is kept as a deprecated shim that forwards to `commitYeastModel`; it emits a deprecation warning.
+* In Python (after `pip install -e code/python/`):
   ```python
-  import code.io as io
-  model = io.read_yeast_model() # loading
-  io.write_yeast_model(model)   # saving
+  from yeastgem import read_yeast_model, commit_yeast_model
+  model = read_yeast_model()    # loading
+  commit_yeast_model(model)     # saving — release pipeline (validates,
+                                # applies canonical state, writes SBML +
+                                # ΔG CSVs, updates README)
   ```
+  The Python release pipeline currently writes the `.xml` artifact and
+  the ΔG side-car CSVs; the `.yml` / `.txt` companion exports still
+  require running the MATLAB `commitYeastModel`. Anaerobic growth and
+  the model_tests benchmarks are wired in
+  [`yeastgem.model_tests`](code/python/yeastgem/model_tests/); batch
+  curation from TSV inputs is available via
+  [`yeastgem.curation`](code/python/yeastgem/curation.py).
 
 ### Online visualization
 

code/python/PORTING_PLAN.md

@@ -17,7 +17,7 @@ saved and validated entirely from Python.
 | 4. Tier 2 — biomass + conditions in Python | **done (core)** | Biomass subsystem moved upstream as `raven_python.biomass` (`BiomassConfig` + `BiomassComponent` + `sum_biomass` / `scale_biomass` / `rescale_pseudoreaction` / `set_gam`; 19 new tests on synthetic models). yeast-GEM ids.yml gained a `biomass_components` section; `yeastgem.biomass` exposes `sum_biomass`, `scale_biomass`, `rescale_pseudoreaction` (with the yeast `lipid` → backbone+chain aggregation), `set_gam` (auto-locates the NGAM reaction by name), and `change_amino_acid_ratio` (reads `data/physiology/aminoAcid_Bjorkeroth2020.tsv`). `yeastgem.conditions.apply` now handles `amino_acid_ratio` before delegating to upstream; `yeastgem.io.commit_yeast_model` runs the anaerobic growth check on a copy. **Verified** end-to-end on the real model: Python `conditions.apply('anaerobic')` produces SBML semantically equal to MATLAB `applyYeastCondition('anaerobic')`; Python `commit_yeast_model` (with anaerobic check active) produces SBML semantically equal to MATLAB `commitYeastModel`. 54 yeast-GEM tests + 38 new raven-python tests passing. **Deferred:** chemostat sweep + `fit_gam` (analysis/calibration, not part of the commit pipeline; tracked in UPSTREAM_CANDIDATES.md). |
 | 5. Tier 3 — test suite | **done** | Ported the four ``code/modelTests/`` routines to ``yeastgem.model_tests``: ``growth`` (Tobias 2013 chemostat R² across 4 conditions), ``essential_genes`` (cobrapy ``single_gene_deletion`` + Stanford KO collection, returns ``EssentialGeneResult`` dataclass with accuracy / sensitivity / specificity / MCC), ``anaerobic_flux_predictions`` (Jouhten 2008 + Frick & Wittmann flux R² + mean relative error), ``plot_anaerobic`` (fermentation-product bar plot), ``find_duplicated_rxns`` (wrapper over the new ``raven_python.manipulation.find_duplicate_reactions``). Stanford ORF lists extracted from ``essentialGenes.m`` to ``data/essentialGenes/{inviable,verified}_orfs.txt`` so both languages read the same source. 7 new yeast-GEM tests + 6 new raven-python tests; full Python suite 61/61 passing. Verified vs MATLAB on the real model (`runPhase5Metrics.m`): growth R² matches at 1e-7; anaerobic flux R² and essential-gene accuracy/MCC match within 5e-3; single 1-gene difference in the essential-gene confusion matrix is a Gurobi/HiGHS solver-tolerance borderline at the 1e-6 ratio threshold. |
 | 6. Tier 4 — curation framework | **done** | Generic `curateModelFromTables` engine moved to RAVEN (with `metPrefix` / `rxnPrefix` parameters defaulted to BiGG `M_`/`R_`); equivalent `raven_python.curation.{batch_curate, batch_curate_from_tsv}` in raven-python with the same schema (DataFrames + a `from_tsv` convenience). yeast-GEM keeps the user-facing `curateMetsRxnsGenes` MATLAB function as a 50-line shim that pins yeast's `s_`/`r_` prefixes and forwards upstream; the historical v8_*/v9_* curation scripts and `TEMPLATEcuration` keep working without change. New `yeastgem.curation.curate_mets_rxns_genes` Python entry point with the same prefix pinning. "Everything after the listed core columns is MIRIAM" — yeast-GEM's existing TSVs (12+10+9 MIRIAM columns) work unchanged. 13 new raven-python tests + 4 new yeast-GEM tests; full Python suite 65/65 passing. **MATLAB shim verified** to forward correctly (no-op call leaves the model unchanged). Direct MATLAB-vs-Python end-to-end parity check is blocked by pre-existing flakiness in the legacy `curateMetsRxnsGenes` (errors on the v8_6_3 VolPolyP schema and the v8_7_0 DBnewRxns pack); the Python implementation is more permissive than the legacy MATLAB on these edge cases. |
-| 7. Docs + CI | partial | Python CI workflow added; README "not yet functional" note still to update. |
+| 7. Docs + CI | **done** | Top-level README updated: "Contribution via Python is supported" section explaining the `yeastgem` + `raven-python` split + `saveYeastModel` → `commitYeastModel` rename. `code/python/README.md` rewritten with a getting-started block and an API map across the seven modules. CI workflow has three required jobs: `test` (matrix Python 3.10/3.11/3.12 + ruff + pytest), `parity-level-1-round-trip` (Python SBML read+write must round-trip the committed model semantically equal — `tests/ci/check_round_trip.py`), and `parity-level-2-metrics` (Python validation metrics must match the committed MATLAB reference within tolerance — `tests/ci/check_metrics.py` against `tests/reference/metrics.json`). Reference tolerances absorb the known Gurobi-vs-HiGHS solver drift (1 gene on the essential-gene confusion matrix, ≤ 5e-3 on R² metrics). Both parity scripts pass locally. **Prerequisite for CI to pass**: `raven-python`'s `feat/yeast-gem-shared` branch must be pushed to GitHub so the `pip install` URL dep resolves. |
 
 ## Design principles
 

code/python/README.md

@@ -1,46 +1,102 @@
-# yeastgem — Python port of the yeast-GEM functions
+# yeastgem — Python interface to yeast-GEM
 
-This directory hosts the in-development Python interface to yeast-GEM. It
-is the Python counterpart to the MATLAB code under [../](..).
+Python counterpart to the MATLAB code under [../](..). Builds on
+[cobrapy](https://github.com/opencobra/cobrapy) and
+[raven-python](https://github.com/SysBioChalmers/raven-python) — the
+latter hosts the generic GEM utilities (model diffing, SBO term
+assignment, condition / biomass / curation engines) that `yeastgem`
+configures with the yeast-specific data files under
+[../../data/](../../data/).
 
-## Status
-
-Early scaffolding. See [PORTING_PLAN.md](PORTING_PLAN.md) for the full
-plan and [UPSTREAM_CANDIDATES.md](UPSTREAM_CANDIDATES.md) for the
+See [PORTING_PLAN.md](PORTING_PLAN.md) for the porting history and
+[UPSTREAM_CANDIDATES.md](UPSTREAM_CANDIDATES.md) for the
 function-level upstream tracking.
 
 ## Install (development)
 
-From a yeast-GEM checkout:
-
 ```bash
 pip install -e code/python/[dev]
 ```
 
-This installs the `yeastgem` package (cobrapy-based; no ravengem
-dependency by design).
+`raven-python` is pinned via a `git+` URL in
+[pyproject.toml](pyproject.toml) to the
+`feat/yeast-gem-shared` branch; once that branch is on a release tag
+the pin will switch to a version constraint.
 
 ## Quick start
 
 ```python
-from yeastgem import read_yeast_model
+from yeastgem import read_yeast_model, commit_yeast_model
 
-model = read_yeast_model()           # cobra.Model
-print(model.optimize().objective_value)
+model = read_yeast_model()
+print(model.optimize().objective_value)        # → ~0.088 / h on the default media
+
+# Make some changes …
+commit_yeast_model(model)                       # full release pipeline
 ```
 
-`yeastgem` auto-detects the repo root via the package location, the
-`YEAST_GEM_PATH` environment variable, or a `.env` file at the repo
-root (historical convention) — in that order.
+`yeastgem` auto-locates the repo root via the package install path,
+the `YEAST_GEM_PATH` environment variable, or a legacy `.env` file —
+in that order. No additional setup needed for the common case.
+
+## API map
+
+| Area | Module | Highlights |
+|---|---|---|
+| **I/O** | [`yeastgem.io`](yeastgem/io.py) | `read_yeast_model`, `commit_yeast_model` (release pipeline: canonical state → SBML validity → aerobic + anaerobic growth → write `.xml` + ΔG CSVs → update README). `write_yeast_model` is a deprecated forwarding shim. |
+| **Comparison** | [`yeastgem.compare`](yeastgem/compare.py) | `compare_models` / `ComparisonReport` re-exported from `raven_python.comparison.diff_models`. Use for cross-toolchain semantic-equality checks. |
+| **Conditions** | [`yeastgem.conditions`](yeastgem/conditions.py) | `apply(model, name)` — minimal_Y6, anaerobic, glycine_nitrogen, nitrogen_limitation. Files under [`data/conditions/`](../../data/conditions/). |
+| **Biomass** | [`yeastgem.biomass`](yeastgem/biomass.py) | `sum_biomass`, `scale_biomass`, `rescale_pseudoreaction`, `set_gam`, `change_amino_acid_ratio`. Configured from [`data/yeastgem/ids.yml`](../../data/yeastgem/ids.yml). |
+| **Annotations** | [`yeastgem.missing_fields`](yeastgem/missing_fields.py) | `add_sbo_terms`, `load_delta_g`, `save_delta_g`. |
+| **Model tests** | [`yeastgem.model_tests`](yeastgem/model_tests/) | `growth` (Tobias 2013 chemostat R²), `essential_genes` (Stanford KO collection), `anaerobic_flux_predictions`, `plot_anaerobic`, `find_duplicated_rxns`. |
+| **Curation** | [`yeastgem.curation`](yeastgem/curation.py) | `curate_mets_rxns_genes` / `..._from_tsv` — batch curation from data tables with the yeast `s_`/`r_` id prefixes. |
 
 ## Layout
 
 ```
 code/python/
-  yeastgem/          # the package
-    io.py            # read/write the model (commit_yeast_model lands later)
-  tests/             # pytest unit tests
+  yeastgem/                # the package
+    io.py                  # read_yeast_model + commit_yeast_model
+    compare.py             # backwards-compat shim → raven_python.comparison
+    config.py              # YeastIDs loader (data/yeastgem/ids.yml)
+    conditions.py          # apply(model, name)
+    biomass.py             # sum_biomass / scale_biomass / set_gam / AA-ratio
+    missing_fields.py      # add_sbo_terms, ΔG CSV persistence
+    curation.py            # batch curation wrapper
+    model_tests/           # Tier-3 benchmarks (growth, essential genes, …)
+  tests/                   # pytest suite (65 tests across the package)
+    reference/             # MATLAB-produced verification artefacts +
+                           #   the runPhase*.m drivers
   pyproject.toml
   PORTING_PLAN.md
   UPSTREAM_CANDIDATES.md
 ```
+
+## Running the tests
+
+```bash
+cd code/python
+pytest -q
+```
+
+Tests load the real model once per session (~2 min) and exercise every
+public function on it. ruff is the linter:
+
+```bash
+ruff check code/python
+```
+
+The CI workflow under
+[`.github/workflows/python.yml`](../../.github/workflows/python.yml)
+runs the same checks across Python 3.10 / 3.11 / 3.12, plus two
+cross-language parity gates (level-1 SBML round-trip vs the committed
+model, level-2 metric parity vs the committed reference values).
+
+## Where work happens
+
+Code under [`yeastgem/`](yeastgem/) is *only* yeast-specific
+configuration and orchestration. Anything generic — model diff,
+condition application, biomass scaling, curation, annotation — lives
+in [raven-python](https://github.com/SysBioChalmers/raven-python).
+Functions tracked for future upstreaming are in
+[UPSTREAM_CANDIDATES.md](UPSTREAM_CANDIDATES.md).