Skip to content

create-pyenv.md

Latest commit

 

History

History
380 lines (266 loc) · 13.8 KB

create-pyenv.md

File metadata and controls

380 lines (266 loc) · 13.8 KB

Create the Python runtime environment

NOTE: Docker is much easier

Before you begin all the steps below to install the Python runtime for the Whole Cell Model, consider how much easier it is to run it within a Docker container.

The script cloud/build-containers-locally.sh will build a Docker Image on your computer if you have Docker Engine or Docker Desktop installed. The script cloud/build.sh will do the same using Google Artifact Registry if you have a Google Cloud project set up.

Either way, you can run the Whole Cell Model (WCM) in a Docker Container, isolated from your computer's operating system, Python versions, binary libraries, and everything else installed.

See docs/README.

Background

The requirements.txt file lists the Python packages needed for the Python runtime environment, with terse setup instructions.

This page goes through the Python environment setup steps in more detail and with more options, but requirements.txt gets updated more frequently since that file specifies the current package versions.

NOTE: While you can create virtual environments using virtualenv or venv in place of pyenv, be sure to put the environment outside the wcEcoli/ directory. Otherwise make clean will break it!

Sherlock: Sherlock is the Stanford scientific computing cluster. Outside the Covert lab, just skip our Sherlock notes. Inside the lab, look in $GROUP_HOME/downloads/ and $GROUP_HOME/installation_notes/ for downloaded software packages and notes on recompiling them as needed to install new packages, new libraries, and new Python releases for the team.

Prerequisites

  • Install the software tools as described in dev-tools, including
    • pyenv and pyenv-virtualenv
    • initializing pyenv in your shell profile
    • gcc or llvm
    • git
    • a programming editor such as PyCharm, Sublime Text, or Visual Studio Code
  • Set up Git and GitHub including Connecting to GitHub with SSH.
  • Clone the repo wcEcoli git to a local directory like ~/dev/wcEcoli/. See About remote repositories. It's probably better to use the SSH URL git@github.com:CovertLab/wcEcoli.git but it's also possible to use an HTTPS URL.

Install native libraries

  1. Use your package manager to install the needed libraries [see the requirements.txt file for the latest list] or compile them from source.

    Most of this list comes from pyenv's requirements to install Python releases, so check the pyenv wiki for the latest list of libraries.

    We no longer recommend installing openblas library using a package manager or by compiling from source. Instead we let NumPy and SciPy install their own copy and let Aesara find NumPy's copy.

    You can optionally install openblas via package manager or source code, but be sure to get at least release v0.3.9.

    On macOS

    brew install cmake glpk openssl readline swig suite-sparse xz

    Use the cmake native package rather than the cmake pip, which can't build osqp and qdldl.

    On Ubuntu

    sudo apt install -y glpk-utils libglpk-dev glpk-doc libssl-dev libreadline-dev \
  libncurses5-dev libncursesw5-dev libffi-dev zlib1g-dev libbz2-dev xz-utils \
  libsqlite3-dev tk-dev

    For Ubuntu, you might also need to find and install the proprietary package python-glpk.

    Don't use apt-get to install libopenblas-dev until that package repository updates to a recent release of OpenBLAS such as the one that's embedded in numpy and scipy package wheels. On Ubuntu, use apt-cache policy libopenblas-dev to check the candidate version. To see which version of OpenBLAS is embedded in numpy, see openblas_support.py in the relevant numpy release tag. Recommendation: Let numpy and scipy install their embedded copies of OpenBLAS (see below).

    On Sherlock

    The needed packages are already installed. Set up your bash profile to locate the group environment modules, load the git and python modules, and initialize pyenv. You'll need these newer git modules since they use a compatible version of libressl.

    export PI_HOME=$GROUP_HOME

##### Add group-wide path settings #####
if [ -f "${PI_HOME}/etc/bash_profile" ]; then
    . "${PI_HOME}/etc/bash_profile"
fi

module load git/2.45.1 git-lfs/2.11.
module load wcEcoli/python3

export PYENV_ROOT="${PI_HOME}/pyenv"

if [ -d "${PYENV_ROOT}" ]; then
    export PATH="${PYENV_ROOT}/bin:${PATH}"
    eval "$(pyenv init -)"
    eval "$(pyenv virtualenv-init -)"
fi

  2. Optional: Download and install other packages according to their instructions or take a wait-and-see approach with them.

    • CPLEX from IBM (free for students)

Install Python

On Sherlock

  1. Install Python 3 in a shared pyenv for the team if it needs updating.

    See $GROUP_HOME/installation_notes/python3.txt.

    If you need to update binary libraries like libressl, readline, or libffi, see their $GROUP_HOME/installation_notes/*.txt files.

    Each of these libraries and tools needs an environment module. We module load the module to make it accessible via environment variable paths like CPPFLAGS. See for example $GROUP_HOME/modules/xz/5.2.5.lua.

On Ubuntu and other Linux environments

  1. Install Python using pyenv, enabling Python to be loaded as a shared library.

    PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.11.3

Anywhere else

Note: If running Parca gets an error message that "the loader can't load libpython" or an error related to "shared objects" and "needing to compile with -fPIC", use the above command that has --enable-shared.

Note: If pyenv install produces an error message like No module named _lizma in WSL on Windows, try running sudo apt install liblzma_dev, then uninstall and reinstall Python with pyenv.

  1. Use pyenv.

    pyenv install 3.11.3

Create the "wcEcoli3" python virtual environment

On Sherlock: Skip this section and instead run pyenv local wcEcoli3 to set up your project directory unless you need to update or rebuild the team's shared virtualenv.

  1. Create a pyenv virtualenv and select it in your project directory.

    cd ~/dev/wcEcoli  # or wherever you cloned the `wcEcoli` project to
pyenv virtualenv 3.11.3 wcEcoli3 && pyenv local wcEcoli3

  2. Upgrade this virtual environment's installers.

    pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel

  3. CONDITIONAL: Link numpy and scipy to the BLAS linear algebra library of your choice.

    See Issue #931. There are several degrees of freedom for installing a BLAS library, numpy, and scipy that might change the computed results. We do not know how to set up environments to get consistent cross-platform results. Unfortunately, that means WCM simulation results and bugs can vary across platforms.

    The simplest setup is to pip install numpy and scipy from precompiled binary "wheels", that is, without the --no-binary numpy,scipy option. This uses their default BLAS library -- usually an embedded copy of OpenBLAS.

    You can use a package manager to install a specific version of the OpenBLAS library to help with consistency or bug fixes.

    TODO: Document how to install numpy 1.26+ and scipy 1.11+ using --no-binary and specified linear algebra libraries, e.g., -Csetup-args=-Dblas=accelerate -Csetup-args=-Dlapack=accelerate. (The ~/.numpy-site.cfg file is obsolete.)

    To use an OpenBLAS library, you need to know where it is.

    • Brew on macOS installs OpenBLAS in brew --prefix openblas.
    • For other package managers, find out where they installed lib/libopenblas*.
    • Compiling OpenBLAS from source in Ubuntu goes into /opt/OpenBLAS/ by default.
    • Compiling from source with make FC=gfortran && make PREFIX=/XYZ install installs it in that specified /XYZ PREFIX directory.
    • On Sherlock, it's installed in $GROUP_HOME/downloads-sherlock2/compiled/openblas. (Using an environment module to load the OpenBLAS when installing numpy and scipy works if the same environment module is loaded at runtime.)

  4. Install NumPy.

    Install numpy before installing scipy and stochastic-arrow to avoid installation errors.

    pip install numpy==1.26.3  # see requirements.txt for the right version

    Or:

    pip install numpy==1.26.3 --no-binary numpy  # see requirements.txt for the right version

  5. Install the packages listed in requirements.txt.

    pip install -r requirements.txt && pyenv rehash

    Or:

    LDFLAGS="-shared $LDFLAGS" pip install -r requirements.txt --no-binary numpy,scipy && pyenv rehash

    The LDFLAGS="-shared $LDFLAGS" preamble fixes dozens of scipy build errors starting with
    In function _start (.text+0x20): undefined reference to main and
    undefined reference to PyFloat_FromDouble.

  6. Prerequisite for the following Python steps: Set the PYTHONPATH environment variable, e.g.

    export PYTHONPATH=$PWD

    or run the ppath alias recommended in dev-tools.md.

  7. Test the NumPy and SciPy installation.

    python runscripts/debug/summarize_environment.py

    It should print entries like the ones below for numpy and scipy showing which BLAS library they link to. library_dirs = ['/usr/local/opt/openblas/lib'] is a path to source openblas while library_dirs = ['/usr/local/lib'] indicates numpy's embedded openblas.

    lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/opt/openblas/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c

    or this:

    lapack_opt_info:
   libraries = ['openblas', 'openblas']
   library_dirs = ['/usr/local/lib']
   language = c
   define_macros = [('HAVE_CBLAS', None)]

    (NumPy's embedded OpenBLAS library is in the virtualenv's site-packages/numpy.libs/ on Linux, or site-packages/numpy/.dylibs/ on macOS, or site-packages/numpy/.libs/ on Windows.)

  8. Required: Add the following line to your shell profile and run it in your current shell. This gets more consistent results from OpenBLAS and it improves performance significantly, especially when called from multiple processes.

    export OPENBLAS_NUM_THREADS=1

  9. Time the NumPy and SciPy libraries

    runscripts/debug/time_libraries.sh

    (It might fail some timing expectations with a message like "AssertionError: 0.xxxxx not less than or equal to 0.4.")

  10. Test Aesara:

    python -c 'import aesara; print([aesara.config.blas.ldflags, aesara.config.device, aesara.config.floatX])'

    It should print something like this (with variations if you compiled OpenBLAS from source):

    ['-lblas', 'cpu', 'float64']

  11. Compile the project's native code.

    make clean compile

    (Yes, it's expected to print deprecation warnings.)

  12. Run the unit tests and runParca.py.

    pytest

ppath
python runscripts/manual/runParca.py

    If the unit tests or runParca.py fail with an error message saying the loader can't load ...libpython... or an error related to "shared objects" and "needing to compile with -fPIC", that means you need to --enable-shared when installing python. Go back to that step, run

    PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.11.3 --force

    then delete and recreate the virtualenv wcEcoli3. Delete it via the command pyenv virtualenv-delete wcEcoli3 or pyenv uninstall wcEcoli3.

  13. Remember to copy the git hooks from the repo (see git hooks) to your .git directory to maintain an up to date environment while doing development:

    cp runscripts/git_hooks/*[^.md] .git/hooks/

  14. If you're using PyCharm, be sure to select the project's Python interpreter so PyCharm understands the version of Python and its installed libraries. This enables code completion, usage documentation in context, visual debugging, warnings about code problems, click-through to library source code, etc.

    PyCharm >
    Preferences >
    Project: wcEcoli >
    Project Interpreter >
    gear ⚙️ >
    Add... >
    Virtualenv Environment >
    Existing environment >
    Interpreter >
    [run pyenv which python in a shell to find the python location, something like /usr/local/var/pyenv/versions/wcEcoli3/python, and paste that path into the text box or navigate there].

Sherlock SCRATCH directory setup

  1. Make sure the model's output goes to the $SCRATCH filesystem (which is larger) rather than SHERLOCK HOME.

    mkdir $SCRATCH/wcEcoli_out
cd wcEcoli
ln -s $SCRATCH/wcEcoli_out out

  2. Create a symbolic link to a shared sim data cache directory in $GROUP_SCRATCH that should contain a copy of the newest sim data object (it should be updated by the daily build):

    ln -s $GROUP_SCRATCH/wc_ecoli/cached cached