Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/building/hpc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ This section documents quick-start guides for a selection of supercomputers that
:maxdepth: 1

platforms/booster_jsc
platforms/jupiter_jsc
platforms/lumi_csc
platforms/maxwell_desy
platforms/perlmutter_nersc
Expand Down
75 changes: 75 additions & 0 deletions docs/source/building/platforms/jupiter_jsc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
Jupiter Booster @ JSC
=====================

This page only provides HiPACE++ specific instructions.
For more information please visit the `JSC documentation <https://apps.fz-juelich.de/jsc/hps/jupiter/index.html>`__.

Log in with ``<yourid>@login.jupiter.fz-juelich.de``. Note that you first need to log into JuDoor
to be added to a compute project, set up an SSH key and set up 2FA.

Running on GPU
--------------

Create a file ``profile.hipace`` and ``source`` it whenever you log in and want to work with HiPACE++:

.. code-block:: bash

# please set your project account
export proj=<your project id>
# required dependencies
module load CMake
module load GCC
module load OpenMPI
module load CUDA
module load HDF5
module load ccache # optional, accelerates recompilation
# optimize CUDA compilation for GH200
export AMREX_CUDA_ARCH=9.0

Install HiPACE++ (the first time, and whenever you want the latest version):

.. code-block:: bash

source profile.hipace
git clone https://github.com/Hi-PACE/hipace.git $HOME/src/hipace # only the first time
cd $HOME/src/hipace
rm -rf build
cmake -S . -B build -DHiPACE_COMPUTE=CUDA
cmake --build build -j 16

You can get familiar with the HiPACE++ input file format in our :doc:`../../run/get_started` section, to prepare an input file that suits your needs.
You can then create your directory in your ``$SCRATCH_<project id>``, where you can put your input file and adapt the following submission script:

.. code-block:: bash

#!/bin/bash -l
#SBATCH -A <your project id>
#SBATCH --partition=booster
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --time=00:05:00
#SBATCH --job-name=hipace
#SBATCH --output=hipace-%j-%N.txt
#SBATCH --error=hipace-%j-%N.err
export OMP_NUM_THREADS=1
module load GCC
module load OpenMPI
module load CUDA
module load HDF5
# fix issues with MPI
export UCX_CUDA_COPY_REG_WHOLE_ALLOC=on
export UCX_CUDA_COPY_MAX_REG_RATIO=0
export UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
export UCX_MEMTYPE_CACHE=n
srun -n 8 --cpu_bind=sockets $HOME/src/hipace/build/bin/hipace inputs

and use it to submit a simulation.

.. tip::
Parallel simulations can be largely accelerated by using GPU-aware MPI.
To utilize GPU-aware MPI, the input parameter ``comms_buffer.on_gpu = 1`` must be set.
Additionally ``comms_buffer.pre_register_memory = 1`` can help improve startup time.

Note that using GPU-aware MPI may require more GPU memory.
Loading