Skip to content

predictlocalization_comparison.md

Latest commit

 

History

History
104 lines (83 loc) · 6.14 KB

predictlocalization_comparison.md

File metadata and controls

104 lines (83 loc) · 6.14 KB
Raw

RAVEN predictLocalization vs the deterministic MILP (head-to-head)

The new compartment-assignment MILP is a deterministic successor to RAVEN's predictLocalization (Agren et al., pcbi.1002980; RAVEN 2.0, Wang et al., pcbi.1006541). Both optimise the same objective — sum of per-gene localisation scores minus inter-compartment transport cost — but predictLocalization solves it by stochastic simulated annealing (greedy hill-climbing in the shipped code, no random seed, wall-clock budget) while ours solves it to a deterministic global optimum by MILP. This is the controlled head-to-head: same model, same scores, same default compartment.

  • Driver: scripts/compare_predictlocalization.py (--prep/--score) + scripts/run_predictlocalization.m (the actual RAVEN function, MATLAB R2024b).
  • Model: yeast-GEM; scores: the slow DeepLoc 2.1 run, normalised, mapped to yeast compartment ids (identical to the MILP's input). 1143 genes.
  • Metric: gene-level agreement with curated yeast-GEM (a gene is correct if its assigned compartment is in the collapsed compartment set of its reactions) — the native output of predictLocalization and the common denominator across methods.

Two setup findings (needed to run predictLocalization fairly)

  1. transportCost must be dialled down for soft scores. Its default (0.5) is sized for clean integer-style scores and overwhelms DeepLoc's 0-1 probabilities: at 0.5, 86-97% of genes are stranded in the default compartment (moving a gene out costs ~0.5 per transported metabolite but gains <=1.0 in score). At transportCost=0.05 the assignment distributes properly. This is the same soft-score calibration lesson as the yeast localization benchmark; the MILP arm uses the matched 0.05.
  2. A crash-on-timeout bug. When the wall-clock loop ends mid-iteration, predictLocalization raises an undefined-variable error in its epilogue (geneScore at line ~590), losing the whole run. The harness guards each run with try/catch so one bad draw does not kill a batch.

Gene-level agreement

transportCost/transport_cost = 0.05 for both arms; predictLocalization run 5x at maxTime=5 min.

method n accuracy notes
argmax (DeepLoc top, no network) 1143 74.5% naive floor
MILP (deterministic) 867 84.0% one global optimum, 90 s
predictLocalization (SA, mean of 5) 911 75.2% min 74.6%, max 75.9%

Strict common gene set (806 genes scored by every method)

Removes the differing-coverage confound — the cleanest comparison.

method accuracy
argmax 83.9%
MILP (deterministic) 83.9%
predictLocalization (mean of 5) 76.8% (min 76.3%, max 77.4%)

The determinism gap (the decisive result)

Across 5 runs at identical settings, predictLocalization assigned 34.8% of genes (317/911) to different compartments between runs, with accuracy wandering over a 1.2 pp band. The MILP returns one reproducible global optimum — no seed, no variance, no "run several and pick".

Is predictLocalization just under-converged? (no)

The simulated annealing starts with every gene in the cytosol and moves one gene at a time, so its accuracy keeps climbing with the wall-clock budget — but it converges slowly and plateaus below the MILP, which reaches its optimum in 90 s:

predictLocalization budget accuracy geneScore
0.5 min ~68% 755
5 min (mean of 5) 75.2% ~951
15 min 78.7% 991 (still rising)
MILP (90 s, optimal) 84.0% —

Even at 3x the budget predictLocalization reaches only 78.7% and is still improving — it does not catch the deterministic MILP, and every run is a different model.

What this shows

  • Against the actual prior method, the MILP wins on every axis. On the common gene set it is ~7 pp more accurate (83.9% vs 76.8%), deterministic (vs 35% of genes flipping between runs), and faster (90 s to optimality vs a multi-minute wall-clock budget that it still needs — at 0.5 min it scores only ~68%). predictLocalization underperforms because simulated annealing is a heuristic that does not reach the optimum and is stochastic.
  • Network-aware does not beat naive at the gene level here — but that cuts against predictLocalization, not us. The MILP matches argmax (both 83.9%), i.e. the gene-level metric is near-saturated by simply taking each gene's top DeepLoc call. Yet predictLocalization, despite being network-aware, scores below argmax (76.8%): its stochastic search actively loses ground. The MILP recovers the argmax-level agreement and does the network-aware part (reaction-level placement, inter-compartment transports, optional flux functionality) deterministically — value the gene-level metric does not capture but the reaction-level benchmark does.
  • Determinism is the headline for a paper. Reproducibility, not a few points of agreement, is the clean differentiator: the same inputs always give the same model, which matters for curation and for anyone building on the output.

For the contemporary, mechanistically-different competitor (carve-a-universal-model rather than assign-with-transport-minimisation), see the CarveFungi analysis.

Reproducing

python scripts/compare_predictlocalization.py --prep --out .research_tmp/pl
# MATLAB: matlab -batch "gssFile='.../deeploc_gss_yeast.csv'; outDir='.../pl'; N=5; maxT=5; \
#                        transCost=0.05; run('scripts/run_predictlocalization.m')"
python scripts/compare_predictlocalization.py --score --out .research_tmp/pl --transport-cost 0.05 \
    --doc /tmp/pl_compare.md   # scratch; this page is curated by hand