Skip to content

README.md

Latest commit

 

History

History
125 lines (86 loc) · 4.93 KB

README.md

File metadata and controls

125 lines (86 loc) · 4.93 KB

Docker

Build recipes for containers using Docker and Singularity or Apptainer.

Software versions

Please confer and update accordingly the software version tables in the respective containers files for each container.

Feedback

If you face any issues, or if you need additional software, please let us know by creating an issue.

Building notes

We use GitHub Actions to automatically build the containers. The build process is defined in the corresponding .github/workflows/docker_build*.yml files. In general, a created PR will trigger the build process and run the unit tests automatically. Resulting container files will be uploaded to the GitHub packages when new Git tags are pushed on the main branch.

Offline builds

It should be possible to build the Docker containers offline, using the provided Dockerfiles. Suggested steps for building the containers offline are as follows:

docker build --platform=linux/amd64 -t ghcr.io/comorment/<image> -f dockerfiles/<image>/Dockerfile .

To successfully build the r container, a GITHUB_PAT file with a valid GitHub Personal Access Token (PAT) is required, which can be created in the GitHub account settings and should have at least read:packages scope. This is needed to access a number of R package GitHub repositories.

docker build --secret id=github_pat,src=~/secrets/GITHUB_PAT --platform=linux/amd64 -t ghcr.io/comorment/r -f dockerfiles/r/Dockerfile .

Note about NREC machine (old)

We use NREC machine to develop and build containers. NREC machine has small local disk (~20 TB) and a larger external volume attached (~400 TB) If you use NREC machine, it's important to not store large data or install large software to your home folder which is located on a small disk, using /nrec/projects space instead:

Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda1                               20G  9.6G  9.7G  50% /
/dev/mapper/nrec_extvol-comorment      393G  346G   28G  93% /nrec/projects
/dev/mapper/nrec_extvol_2-comorment_2  935G  609G  279G  69% /nrec/space

Both docker and singularity were configured to avoid placing cached files into local file system. For docker this involves changing /etc/docker/daemon.json file by adding this:

{ 
    "data-root": "/nrec/projects/docker_root"
}

(as described https://tienbm90.medium.com/how-to-change-docker-root-data-directory-89a39be1a70b ; you may use docker info command to check the data-root)

For singularity, the configuration is described here https://sylabs.io/guides/3.6/user-guide/build_env.html and it was done for the root user by adding the following line into /etc/environment

export SINGULARITY_CACHEDIR="/nrec/projects/singularity_cache"

Common software, such as git-lfs, is installed to /nrec/projects/bin. Therefore it's reasonable for all users of the NREC comorment instance to add this folder to the path by changing ~/.bashrc and ~/.bash_profile.

export PATH="/nrec/projects/bin:$PATH"

A cloned version of comorment repositories is available here:

/nrec/projects/github/comorment/containers
/nrec/projects/github/comorment/reference

Feel free to change these folders and use git pull / git push. TBD: currently the folder is cloned as 'ofrei' user - I'm not sure if it will actually work to pull & push. But let's figure this out.

Testing container builds

Some basic checks for the functionality of the different container builds are provided in <containers>/tests/, implemented in Python. The tests can be executed using the Pytest testing framework.

To install Pytest in the current Python environment, issue:

pip install pytest  # --user optional

New virtual environment using conda:

conda create -n pytest python=3 pytest -y  # creates env "pytest"
conda activate pytest  # activates env "pytest"

Then, all checks can be executed by issuing:

cd <containers>
py.test -v tests  # with verbose output

Checks for individual containers (e.g., gwas.sif) can be executed by issuing:

py.test -v tests/test_<container-prefix>.py

Note that the proper container files (*.sif files) corresponding to the different test scripts must exist in <containers>/singularity/>, not only git LFS pointer files.

Git clone ignoring LFS

See stackoverflow.com/questions/42019529/how-to-clone-pull-a-git-repository-ignoring-lfs

GIT_LFS_SKIP_SMUDGE=1 git clone git@github.com:comorment/containers.git