Add singularity and slurm instructions

Author: Tom-Notch

.dockerignore

@@ -0,0 +1,8 @@
+# Applies to build only, ignore everything by default
+**
+
+# Keep only files needed for build
+!environment.yml
+!pyproject.toml
+!docker/
+!docker-compose.yml

.env

@@ -8,9 +8,12 @@ IMAGE_NAME=todo-image-name
 IMAGE_TAG=latest
 
 CONTAINER_NAME=${IMAGE_NAME}
-CONTAINER_HOME_FOLDER=/root
+IMAGE_USER=todo-image-user
+HOME_FOLDER=/home/${IMAGE_USER}
 CODE_FOLDER=${IMAGE_NAME}
 
 HOST_UID=$(id -u)
 HOST_GID=$(id -g)
 HOSTNAME=${HOSTNAME}
+
+BUILDER=multi-platform

README.md

@@ -1,13 +1,17 @@
-# New Docker Repository
+# New Cloud Computing Repository
 
-[![pre-commit](https://github.com/Tom-Notch/Docker-Repository-Template/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/Tom-Notch/Docker-Repository-Template/actions/workflows/pre-commit.yml)
+[![pre-commit](https://github.com/Tom-Notch/Cloud-Computing-Repository-Template/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/Tom-Notch/Cloud-Computing-Repository-Template/actions/workflows/pre-commit.yml)
 
 ## Dependencies
 
 - [Docker](https://docs.docker.com/get-docker/)
+- [Singularity/Apptainer](https://apptainer.org/)
 
 ## Usage Guidelines
 
+TLDR: Search for `todo` and update all occurrences to your desired name
+Docker and singularity is not a must unless you cannot install some dependencies locally on HPC shell environment due to permission issue
+
 ### Base Repo
 
 1. Change [LICENSE](LICENSE) if necessary
@@ -18,20 +22,59 @@
 
 ### Docker Config
 
-1. Modify **DOCKER_USER**, **IMAGE_NAME** in [.env](.env)
+1. Modify `DOCKER_USER`, `IMAGE_NAME`, `IMAGE_USER` in [.env](.env)
+
+   - [.env](env) will be loaded when you use docker compose for build/run/push/...
+   - `DOCKER_USER` refers to your docker hub account username
+   - `IMAGE_USER` refers to the default user inside the image, which is used to determine home folder
+
+1. Modify the service name from `default` to your service name in [docker-compose.yml](docker-compose.yml), add additional volume mounting options such as dataset directories
+
+1. Update [Dockerfile](docker/latest/Dockerfile) and [.dockerignore](.dockerignore)
+
+   - Existing dockerfile has screen & tmux config, oh-my-zsh, cmake, and other basic goodies
+   - Add any additional dependency installations at appropriate locations
+
+1. [build_docker_image.sh](scripts/build_docker_image.sh) to build and test the image locally in your machine's architecture
+
+   - Do this on a machine where you have docker permission, HPC clusters usually restrict docker access for security reasons
+   - The scripts uses buildx to build multi-arch image, you can disable this by removing redundant archs in [docker-compose.yml](docker-compose.yml)
+   - Building stage does not have GPU access, if some of your dependencies need GPU, build them inside a running container and commit to the final image
+
+1. To run and test a built image, use [run_docker_container.sh](scripts/run_docker_container.sh) or `docker compose up -d`
+
+   - The service by default will mount the whole repository onto `CODE_FOLDER` inside the container so any modification inside also takes effect outside, which is useful when you use vscode remote extension to develop inside a running container with docker context
+
+1. [push_docker_image.sh](scripts/push_docker_image.sh) to push the multi-arch image to docker hub
+
+   - You should have the docker hub repository set up before pushing
+
+### Singularity Config
+
+1. [pull_singularity_image.sh](scripts/pull_singularity_image.sh) to build the singularity image locally
+
+   - Singularity image can be built upon existing docker image
+
+1. [run_singularity_instance.sh](scripts/run_singularity_instance.sh) to test the image
+
+   - Add additional volume binding options to the script such as dataset directories, best practice is to define in [.env](.env) then export in [variables.sh](scripts/variables.sh) with `resolve_host_path` to turn relative path into absolute real path
+   - Singularity instances by default has less environment separation than docker containers unless you specify the additional options like the script
+
+### Job Config
 
-1. Modify the service name from **default** to your service name in [docker-compose.yml](docker-compose.yml)
+1. Modify job specifications under [jobs/](jobs/)
 
-1. Update [Dockerfile](docker/latest/Dockerfile)
+   - Each (HPC) Slurm environment has different partition definitions, which are often heterogeneous, you can query this by `sinfo` with some options
+   - All the jobs has `-l`(login) options in shebang so that any command working in your current shell environment should also run as a job
 
-1. [build.sh](scripts/build.sh) to build and test the image locally in your machine's architecture
+1. Submit job by `sbatch jobs/your-cluster/your-job.job` or `jobs/your-cluster/your-job.job`
 
-1. [push.sh](scripts/push.sh) to push the multi-arch image to the registry
+1. Recommend [turm](https://github.com/kabouzeid/turm) for job monitor, use `turn -u your-slurm-user` after installation
 
 ## Developer Quick Start
 
-- Run [scripts/dev-setup.sh](scripts/dev-setup.sh) to setup the development environment
+- Run [dev_setup.sh](scripts/dev_setup.sh) to setup the development environment
 
-## Note
+## Maintainer
 
-- This template currently only supports docker image for amd64 and arm64, if you want to support other architectures, please modify the [build.sh](scripts/build.sh) script and [docker-compose.yml](docker-compose.yml) accordingly
+- Mukai (Tom Notch) Yu: [mukaiy@andrew.cmu.edu](mailto:mukaiy@andrew.cmu.edu)

docker/latest/Dockerfile

@@ -1,19 +1,71 @@
 # Do not add --platform=linux/blabla since this is intended for multiplatform builds
-FROM tomnotch/bipvrobotics-base-image:latest
-ENV HOME_FOLDER=/root
+FROM nvidia/cuda:13.0.0-cudnn-devel-ubuntu24.04
+ENV HOME=$HOME_FOLDER
 WORKDIR $HOME_FOLDER/
 
 # Fix apt install stuck problem
 ENV DEBIAN_FRONTEND=noninteractive
 RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
 
+# Copy home folder config
+COPY --from=home-folder-config . $HOME_FOLDER/
+
+# Remove cuda source list
+# E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/Packages.gz File has unexpected size (853674 != 833365). Mirror sync in progress? [IP: 23.58.157.10 443]
+RUN rm /etc/apt/sources.list.d/cuda.list
+
 # update all obsolete packages to latest, install sudo, and cleanup
 RUN apt update -o Acquire::Check-Valid-Until=false -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true && \
     apt full-upgrade -y && \
     apt install -y sudo ca-certificates && \
     apt autoremove -y && \
     apt autoclean -y
 
+# fix local time problem
+RUN apt-get install -y --no-install-recommends tzdata && \
+    ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime && \
+    echo "America/New_York" > /etc/timezone && \
+    dpkg-reconfigure --frontend noninteractive tzdata
+
+# update all obsolete packages to latest, install sudo, and cleanup
+RUN apt update -o Acquire::Check-Valid-Until=false -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true && \
+    apt full-upgrade -y && \
+    apt install -y sudo ca-certificates && \
+    apt autoremove -y && \
+    apt autoclean -y
+
+    # Install some goodies
+RUN apt-get install -y --no-install-recommends \
+    build-essential \
+    clang-format \
+    curl \
+    dirmngr \
+    git \
+    gnupg \
+    htop \
+    less \
+    locate \
+    lsb-release \
+    nano \
+    ncdu \
+    net-tools \
+    perl \
+    screen \
+    software-properties-common \
+    tmux \
+    tmuxp \
+    tree \
+    unzip \
+    valgrind \
+    vim \
+    wget \
+    zsh
+
+# upgrade cmake to kitware official apt-get repo release version
+RUN wget https://apt-get.kitware.com/kitware-archive.sh -O- | sh -s && \
+    apt-get upgrade -y cmake && \
+    apt-get autoremove -y
+
 # # Add a new group and user
 # RUN addgroup --gid 1000 $USER && \
 #     adduser --uid 1000 --ingroup $USER --home $HOME_FOLDER --shell /bin/zsh --disabled-password --gecos "" $USER && \
@@ -25,44 +77,76 @@ RUN apt update -o Acquire::Check-Valid-Until=false -o Acquire::AllowInsecureRepo
 #     chmod 4755 /usr/local/bin/fixuid && \
 #     mkdir -p /etc/fixuid
 
-# # Switch to the new user
-# USER $USER:$USER
-
-#! Install OpenCV 4.2.0 with QUIRC support from source
-ENV OPENCV_VERSION=4.2.0
-RUN pip3 uninstall -y opencv && \
-    apt install -y --no-install-recommends libavcodec-dev libavformat-dev libswscale-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgtk-3-dev libpng-dev libjpeg-dev && \
-    git clone --depth 1 --recursive https://github.com/opencv/opencv.git $HOME_FOLDER/opencv -b $OPENCV_VERSION && \
-    #  follow the error information，replace all the “ipcp-unit-growth” with “ipa-cp-unit-growth” in 3rdparty/carotene/CMakeLists.txt and 3rdparty/carotene/hal/CMakeLists.txt
-    perl -pi -e 's/ipcp-unit-growth/ipa-cp-unit-growth/g' $HOME_FOLDER/opencv/3rdparty/carotene/CMakeLists.txt $HOME_FOLDER/opencv/3rdparty/carotene/hal/CMakeLists.txt && \
-    git clone --depth 1 --recursive https://github.com/opencv/opencv_contrib.git $HOME_FOLDER/opencv_contrib -b $OPENCV_VERSION && \
-    mkdir -p $HOME_FOLDER/opencv/build && \
-    cd $HOME_FOLDER/opencv/build && \
-    cmake \
-    -D CMAKE_CXX_STANDARD=20 \
-    -D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \
-    -D OPENCV_GENERATE_PKGCONFIG=ON \
-    -D BUILD_opencv_python3=ON \
-    -D OPENCV_PYTHON3_INSTALL_PATH=/usr/local/lib/python3.8/dist-packages \
-    # -D OPENCV_ENABLE_NONFREE=ON \
-    -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
-    # -D WITH_LAPACK=ON \
-    -D WITH_GTK=ON \
-    -D WITH_TBB=ON \
-    # -D WITH_QUIRC=ON \
-    -D WITH_GSTREAMER=ON \
-    -D WITH_V4L=ON \
-    # -D WITH_OPENGL=ON \
-    -D BUILD_TESTS=OFF \
-    -D BUILD_PERF_TESTS=OFF \
-    -D BUILD_EXAMPLES=OFF \
-    -D CMAKE_BUILD_TYPE=RELEASE \
-    -D CMAKE_INSTALL_PREFIX=/usr/local .. && \
-    make install -j$(($(nproc)-1)) && \
-    echo "export OpenCV_DIR=/usr/local/lib/cmake/opencv4/" >> ${HOME_FOLDER}/.zshrc && \
-    echo "export OpenCV_DIR=/usr/local/lib/cmake/opencv4/" >> ${HOME_FOLDER}/.bashrc && \
-    rm -rf $HOME_FOLDER/opencv && \
-    rm -rf $HOME_FOLDER/opencv_contrib
+# install zsh, Oh-My-Zsh, and plugins
+RUN sh -c "$(wget -O- https://github.com/deluan/zsh-in-docker/releases/latest/download/zsh-in-docker.sh)" -- \
+    -t https://github.com/romkatv/powerlevel10k \
+    -p git \
+    -p https://github.com/zsh-users/zsh-autosuggestions \
+    -p https://github.com/zsh-users/zsh-completions \
+    -p https://github.com/zsh-users/zsh-syntax-highlighting \
+    -a "[[ ! -f $HOME_FOLDER/.p10k.zsh ]] || source $HOME_FOLDER/.p10k.zsh" \
+    -a "POWERLEVEL9K_DISABLE_GITSTATUS=true" \
+    -a "bindkey -M emacs '^[[3;5~' kill-word" \
+    -a "bindkey '^H' backward-kill-word" \
+    -a "autoload -U compinit && compinit" \
+    -a "export PATH=~/.local/bin:$PATH"
+
+# change default shell for the $USER in the image building process for extra environment safety
+RUN chsh -s $(which zsh)
+
+# Use bash -lc for conda commands in build layers
+SHELL ["/bin/bash", "-lc"]
+
+# Install miniconda at /opt/conda
+RUN wget "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-$(uname -m).sh" -O ~/miniconda.sh && \
+    sh ~/miniconda.sh -b -p /opt/conda && \
+    rm ~/miniconda.sh
+
+# Add conda to PATH
+ENV PATH=$PATH:/opt/conda/bin
+
+# Init conda for zsh
+RUN conda init zsh
+
+#! Examples of actual dependency install:
+# #! Install OpenCV 4.2.0 with QUIRC support from source
+# ENV OPENCV_VERSION=4.2.0
+# RUN pip3 uninstall -y opencv && \
+#     apt install -y --no-install-recommends libavcodec-dev libavformat-dev libswscale-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgtk-3-dev libpng-dev libjpeg-dev && \
+#     git clone --depth 1 --recursive https://github.com/opencv/opencv.git $HOME_FOLDER/opencv -b $OPENCV_VERSION && \
+#     #  follow the error information，replace all the “ipcp-unit-growth” with “ipa-cp-unit-growth” in 3rdparty/carotene/CMakeLists.txt and 3rdparty/carotene/hal/CMakeLists.txt
+#     perl -pi -e 's/ipcp-unit-growth/ipa-cp-unit-growth/g' $HOME_FOLDER/opencv/3rdparty/carotene/CMakeLists.txt $HOME_FOLDER/opencv/3rdparty/carotene/hal/CMakeLists.txt && \
+#     git clone --depth 1 --recursive https://github.com/opencv/opencv_contrib.git $HOME_FOLDER/opencv_contrib -b $OPENCV_VERSION && \
+#     mkdir -p $HOME_FOLDER/opencv/build && \
+#     cd $HOME_FOLDER/opencv/build && \
+#     cmake \
+#     -D CMAKE_CXX_STANDARD=20 \
+#     -D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \
+#     -D OPENCV_GENERATE_PKGCONFIG=ON \
+#     -D BUILD_opencv_python3=ON \
+#     -D OPENCV_PYTHON3_INSTALL_PATH=/usr/local/lib/python3.8/dist-packages \
+#     # -D OPENCV_ENABLE_NONFREE=ON \
+#     -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
+#     # -D WITH_LAPACK=ON \
+#     -D WITH_GTK=ON \
+#     -D WITH_TBB=ON \
+#     # -D WITH_QUIRC=ON \
+#     -D WITH_GSTREAMER=ON \
+#     -D WITH_V4L=ON \
+#     # -D WITH_OPENGL=ON \
+#     -D BUILD_TESTS=OFF \
+#     -D BUILD_PERF_TESTS=OFF \
+#     -D BUILD_EXAMPLES=OFF \
+#     -D CMAKE_BUILD_TYPE=RELEASE \
+#     -D CMAKE_INSTALL_PREFIX=/usr/local .. && \
+#     make install -j$(($(nproc)-1)) && \
+#     echo "export OpenCV_DIR=/usr/local/lib/cmake/opencv4/" >> ${HOME_FOLDER}/.zshrc && \
+#     echo "export OpenCV_DIR=/usr/local/lib/cmake/opencv4/" >> ${HOME_FOLDER}/.bashrc && \
+#     rm -rf $HOME_FOLDER/opencv && \
+#     rm -rf $HOME_FOLDER/opencv_contrib
+
+# Add git safe directory
+RUN git config --global --add safe.directory "*"
 
 # end of sudo apt installs
 RUN apt full-upgrade -y && \
@@ -71,6 +155,12 @@ RUN apt full-upgrade -y && \
     apt clean -y && \
     rm -rf /var/lib/apt/lists/*
 
+# change owner of home folder
+RUN chown -R $IMAGE_USER:$IMAGE_USER $HOME_FOLDER
+
+# change user
+USER $IMAGE_USER
+
 # Set the default shell to zsh
 SHELL [ "/bin/zsh", "-c" ]
 

jobs/perceptron/gpu.job

@@ -0,0 +1,19 @@
+#!/bin/bash -l
+#SBATCH -N 1 # Number of nodes
+#SBATCH -p a100-gpu-shared # partition
+#SBATCH --ntasks-per-node=2 # number of tasks per node
+#SBATCH --gpus-per-task=1 # number of GPUs per task
+#SBATCH --cpus-per-task=16 # number of CPU cores per task
+#SBATCH --mem=100G # CPU RAM
+#SBATCH -t 48:00:00 # time
+#SBATCH -o todo_your_job_name_%j.out
+#SBATCH --job-name todo_your_job_name
+#SBATCH --mail-type END
+#SBATCH --mail-user todo-your-email@your-domain
+
+# echo commands to stdout
+set -x
+
+cd todo-your-code-directory || exit # this directory refers to the absolute path outside any docker/singularity container/instance
+conda activate todo-your-conda-env-name
+srun todo-your-code-entrypoint

jobs/psc/gpu-singularity.job

@@ -0,0 +1,20 @@
+#!/bin/bash -l
+#SBATCH -N 1 # Number of nodes
+#SBATCH -p GPU-shared # partition
+#SBATCH --gpus=h100-80:2 # gpu type and number, use --gres for GPU partition
+#SBATCH --ntasks-per-node=2 # number of tasks per node
+#SBATCH --cpus-per-task=12 # number of CPU cores per task
+#SBATCH --mem-per-gpu=100G # CPU RAM per GPU
+#SBATCH -t 48:00:00 # time
+#SBATCH -A todo-your-project-id
+#SBATCH -o todo_your_job_name_%j.out
+#SBATCH --job-name todo_your_job_name
+#SBATCH --mail-type END
+#SBATCH --mail-user todo-your-email@your-domain
+
+# echo commands to stdout
+set -x
+
+cd todo-your-code-directory || exit # this directory refers to the absolute path outside any docker/singularity container/instance
+scripts/run_singularity_instance.sh
+srun singularity exec instance://todo-your-container-name zsh -c "source ~/.zshrc && conda run -n todo-your-conda-env-name todo-your-code-entrypoint"

jobs/psc/gpu.job

@@ -0,0 +1,20 @@
+#!/bin/bash -l
+#SBATCH -N 1 # Number of nodes
+#SBATCH -p GPU-shared # partition
+#SBATCH --gpus=h100-80:2 # gpu type and number, use --gres for GPU partition
+#SBATCH --ntasks-per-node=2 # number of tasks per node
+#SBATCH --cpus-per-task=12 # number of CPU cores per task
+#SBATCH --mem-per-gpu=100G # CPU RAM per GPU
+#SBATCH -t 48:00:00 # time
+#SBATCH -A todo-your-project-id
+#SBATCH -o todo_your_job_name_%j.out
+#SBATCH --job-name todo_your_job_name
+#SBATCH --mail-type END
+#SBATCH --mail-user todo-your-email@your-domain
+
+# echo commands to stdout
+set -x
+
+cd todo-your-code-directory || exit # this directory refers to the absolute path outside any docker/singularity container/instance
+conda activate todo-your-conda-env-name
+srun todo-your-code-entrypoint