slurm support

[rjleveque]

There was a recent question on the claw-users mailing list about sample slurm scripts for running Clawpack or GeoClaw on cluster or HPC, and it would be nice to include some examples. @mandli pointed to this code: https://github.com/mandli/batch.

Personally I have used a script similar to this sample, if I want to do a single run using 6 OpenMP threads, for example:

#!/bin/bash

# JOB NAME
#SBATCH --job-name=run-geoclaw

# ALLOCATION DEFINITION
#SBATCH --account=tsunami
#SBATCH --partition=compute

# RESOURCES
#SBATCH --nodes=1  # total number of nodes allocated
#SBATCH --ntasks-per-node=6

# WALL TIME
#SBATCH --time=10:00:00

# MEMORY PER NODE
#SBATCH --mem=58G

# WORKING DIRECTORY ENTRYPOINT
# Specify the working directory for this job
#SBATCH --output=job-geoclaw.%j.out
#SBATCH --error=job-geoclaw.%j.err

# Run the commands to run your program here
# e.g. load modules, copy input.output files, run program, etc.

module load intel/oneAPI/2021.1.1

# Export all environment variables to the batch job session
#SBATCH --export=all

# Set or reset any environment variables here:
source $HOME/.bashrc   # needed for conda setup
export FC=ifort
export FFLAGS='-O2 -qopenmp'
export OMP_NUM_THREADS=6
export CLAW=/gscratch/tsunami/clawpack-v5.12.0
export PYTHONPATH=$CLAW:$PYTHONPATH
export EXE=$CLAW/xgeoclaw

date
pwd
conda activate /gscratch/tsunami/miniconda3/envs/geo5
make all              # makes scratch dir, data, output, plots
python plot_fgmax.py  # post-processing the fgmax output
date

Instead of doing a single run as in the example above, I often run a Python script along the lines of what was merged in v5.12.0 in this PR: #180. This is still work in progress and there may be better ways to do the multiprocessing, but at any rate if I set e.g. nprocs = 8 in that code to run 8 jobs in parallel and want to use 5 OpenMP threads for each, I would set

#SBATCH --ntasks-per-node=40
...
export OMP_NUM_THREADS=5
...
python run_cases_clawpack.py

[KYANJO]

Another example could be this one.

#!/bin/bash -l
#SBATCH -N 1                          # Request 1 node
#SBATCH --ntasks-per-node=1           # Run 1 task per node
#SBATCH -c 24                         # Request 24 CPU cores (OpenMP threads) per task
#SBATCH --time=20:00:00               # Set maximum runtime to 20 hours
#SBATCH --gres=gpu:1                  # Request 1 GPU (max of 1 on Borah)
#SBATCH -p gpu                        # Specify the GPU partition
#SBATCH --job-name=gclaw              # Name the job "gclaw"
#SBATCH --output=gclaw_00024.o%j      # Output file with job ID

# Load the SLURM module (if required by the system)
module load slurm

# Set the number of OpenMP threads based on SLURM allocation
# SLURM_CPUS_PER_TASK is set to the value of -c (24 in this case)
# If not set, default to 1 thread to avoid errors
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
  omp_threads=$SLURM_CPUS_PER_TASK
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads   # Export the number of threads for OpenMP

# Explanation for multiple threads:
# The -c 24 option requests 24 CPU cores per task, which are used as OpenMP threads.
# The OMP_NUM_THREADS environment variable tells OpenMP to use these 24 threads.
# Ensure your you have upto the number of threads you are requesting available

# Add Anaconda to PATH for Python environment
# Update the path to match your Anaconda or python installation
export PATH="/bsuhome/bkyanjo/anaconda3/bin:$PATH"

# Set PYTHONPATH to include Clawpack installation
# Adjust the path to point to your Clawpack directory
export PYTHONPATH=/bsuhome/bkyanjo/clawpack_src/clawpack-v5-9-0:$PYTHONPATH

# Run the application
make output