Skip to content

NEWS.md

Latest commit

 

History

History
119 lines (111 loc) · 7.01 KB

NEWS.md

File metadata and controls

119 lines (111 loc) · 7.01 KB
Raw

SelectSim (Python) 0.1.0

Initial feature-complete Python port of the SelectSim R package.

  • Ported the core algorithm: selectX() end-to-end pipeline, AlterationLandscape, template-matrix generation (template.py), TMB-based weighting (weights.py), the permutation null model (null_model.py), and all overlap / effect-size / FDR statistics (stats.py).
  • Added an optional accelerated backend for null_model_parallel (backend="jax", via the selectsim[fast] extra) alongside the default NumPy backend, and an optional on-disk Zarr store for large null-model runs (store="zarr", via the selectsim[storage] extra) so simulations no longer need to be held entirely in memory.
  • Added selectsim.gam: a full port of R/gam_utils.r's MAF-ingestion pipeline (filter_maf_*, stat_maf_*, maf_to_gam, TCGA_maf_schema, GENIE_maf_schema, mutation_type) for building Gene Alteration Matrices (GAMs) directly from raw MAF files.
  • Added selectsim.plotting: matplotlib/seaborn ports of the R package's visualization helpers -- theme_publication, obs_exp_scatter, overlap_pair_extract, ridge_plot_ed, ridge_plot_ed_compare.
  • Added selectsim.io: Parquet read/write helpers and Zarr null-model store utilities.
  • Bundled real TCGA LUAD example data (tests/data/*.parquet, exported from the R package's luad_run_data/luad_result/luad_maf datasets) and added an R-parity test suite (TestSelectXWithRData in tests/test_selectsim.py) validating the Python port's output against a real SelectSim::selectX() R reference run.
  • 126 tests passing across algorithm, statistics, GAM ingestion, plotting, and I/O modules.
  • Added Sphinx documentation (docs/) with a full API reference, an "Overview" quick-start, and two tutorial notebooks (introduction, data_processing) mirroring the R package's vignettes, plus standard open-source project scaffolding (LICENSE, CITATION.cff, CI workflows, issue templates).
  • Supports Python 3.10-3.12 (requires-python = ">=3.10,<3.13"). The upper bound is intentional, not a typo: numpy/scipy/jax/zarr don't all publish prebuilt wheels for 3.13+ yet, and installing on an unsupported interpreter can silently trigger a slow (or failing) from-source build of numpy. 3.10/3.11/3.12 were each verified to resolve prebuilt wheels and pass the full test suite.

SelectSim (Python) 0.2.0

  • Fixed several correctness bugs found in code review: filter_maf_column computed a union instead of an intersection when excluding rows matching several substring values (inclusive=False, fixed=False); store="zarr" fully materialized all permutations in memory before writing them to disk, defeating its own purpose (now streams in bounded batches, ~5x lower peak memory); the jax backend recompiled from scratch on every call instead of reusing JAX's compilation cache; n_cores was silently ignored with backend="jax" (now warns); ridge_plot_ed crashed on an empty null model; and retrieve_outliers's n_sim parameter was dead code and its 90th-percentile threshold was off-by-one relative to the R original.
  • This round of work (the fixes above, plus the earlier module ports, documentation, and uv migration) was developed with the assistance of Claude Code (Anthropic), directed and verified by Arvind Iyer throughout; see git log for detailed commit history.

SelectSim (Python) 0.3.0

  • Performance: the default numpy null-model backend (null_model_parallel, store="memory" or "zarr") no longer loops over genes in Python to pick each row's top-k residual columns; it now uses the same vectorized double-argsort rank trick the (now removed) jax backend used, with no per-gene Python loop. Benchmarked as bit-identical to the old implementation (full R-parity test suite still passes) and consistently faster, especially at higher n_cores.
  • Performance: store="zarr" null-model runs now chunk the on-disk Zarr array to match the internal write-batch size instead of defaulting to one chunk per permutation; writing in batches of 200 (the default) previously forced 200 separate compress/write calls per batch. Benchmarked at ~2.5x faster zarr writes in isolation.
  • Performance: estimate_p_val's gene_names: List[str] parameter is now gene_index: Dict[str, int] -- callers looking up many pairs (as estimate_pairwise_p does, used when estimate_pairwise=True) now build the name-to-index mapping once instead of doing an O(n_genes) list scan per gene per pair.
  • Removed: the jax null-model backend (backend="jax" on null_model_parallel, the selectsim[fast] extra). Benchmarked against the numpy backend on real LUAD-sized data on CPU (no GPU): jax was consistently 5-10x slower, dominated by per-call JIT dispatch overhead that this workload's array sizes never amortize. If GPU-backed acceleration is needed in the future, it should be re-evaluated against the current (vectorized) numpy backend on representative GPU hardware, not reintroduced by default.
  • Fixed: significance_heatmap was implemented and tested but never exported from the top-level selectsim package (only reachable via selectsim.plotting.significance_heatmap); it's now ss.significance_heatmap.
  • Added null_model_parallel/selectX store="auto": estimates the null model's memory footprint from cohort size and n_permut and picks "memory" or "zarr" automatically against a detected (or configured, memory_budget_gb) RAM budget, so one call scales safely across machines without the caller needing to know cohort dimensions up front.
  • Added oncoprint and oncoprint_pair: new (not present in the R package) multi-gene and two-gene co-mutation visualizations comparing the observed alteration pattern to a retained null-model simulation, with a per-sample TMB track.
  • Extended the introduction tutorial notebook with significance_heatmap, oncoprint, oncoprint_pair, and ridge_plot_ed_compare (previously only obs_exp_scatter and ridge_plot_ed were demonstrated).
  • This round of work was developed with the assistance of Claude Code (Anthropic), directed and verified by Arvind Iyer throughout; see git log for detailed commit history.

SelectSim (Python) 0.3.1

  • Performance: import selectsim dropped from ~1.5s to ~0.6s. The top-level package eagerly imported selectsim.plotting, which pulls in matplotlib and scipy (used nowhere outside plotting.py), even for callers who only ever call selectX() and never plot anything. Plotting functions (obs_exp_scatter, ridge_plot_ed, oncoprint, etc.) are now resolved lazily on first access (PEP 562 module __getattr__) instead -- same ss.<name> usage, just deferred. On the bundled LUAD benchmark this cut whole-process wall time by ~23% and peak memory by ~21%.
  • Removed: the seaborn dependency. It was listed in pyproject.toml and mentioned in a couple of docstrings, but never actually imported or used anywhere in the codebase.