use with slurm?

[biddisco]

Is it possible to use tmpi with slurm, I tried a few quick experiments setting the mpirun env var to srun and launching jobs, but I didn't get it working (tmpi exits immediately). I don't usually use tmux, so didn't dig deep or experiment much, but if it was possible to have

tmpi 4 gdb --args /my/job/to/debug

expand out to do the equivalent of

srun -n 4 blah blah blah

and do the right thing, it'd be very useful indeed. On some machines I'm not able to use xterm, so can't run a simple

srun -n 4 xterm -e gdb --args /my/job/to/debug

which is one way of debugging under slurm.

[Azrael3000]

Hi John,

first of all nice talking to you again. I was working on SPH over a decade ago with that whole HDF5 stuff including those pesky ParaView plugins that you contributed to.

Regarding the issue at hand, I did some quick testing with the following changes

diff --git a/tmpi b/tmpi
index cb64351..e1620cd 100755
--- a/tmpi
+++ b/tmpi
@@ -10,9 +10,10 @@ reptyr_cmd=${REPTYR:-reptyr}
 
 mpich=$(${mpirun_cmd} --version |grep -i -e HYDRA -e Intel)
 openmpi=$(${mpirun_cmd} --version |grep -i "Open MPI")
+slurm=$(${mpirun_cmd} --version |grep -i "slurm")
 
-if [ -z "${mpich}" ] && [ -z "${openmpi}" ]; then
-    echo "Unknown MPI implementation. Only MPICH (and its derivatives) and OpenMPI are supported!"
+if [ -z "${mpich}" ] && [ -z "${openmpi}" ] && [ -z "${slurm}" ]; then
+    echo "Unknown MPI implementation. Only MPICH (and its derivatives), SLURM and OpenMPI are supported!"
     exit 1
 fi
 
@@ -75,9 +76,9 @@ if [[ ${1} == runmpi ]] ; then
     shift
     shift
 
-    mpi_rank=${OMPI_COMM_WORLD_RANK:-${PMI_RANK:--1}}
+    mpi_rank=${OMPI_COMM_WORLD_RANK:-${PMI_RANK:-${SLURM_PROCID:--1}}}
     if [[ ${mpi_rank} == -1 ]]; then
-        echo "Neither OMPI_COMM_WORLD_RANK nor PMI_RANK environment variables is not set by '${mpirun_cmd}'."
+        echo "Neither OMPI_COMM_WORLD_RANK, SLURM_PROCID nor PMI_RANK environment variables is not set by '${mpirun_cmd}'."
         exit 1
     fi
 
@@ -104,6 +105,8 @@ if [[ ${1} == runmpi ]] ; then
     else
         if [ -n "${mpich}" ]; then
             mpi_env="-e MPI -e HYD -e PMI"
+        elif [ -n "${slurm}" ]; then
+            mpi_env="-e SLURM -e SRUN"
         else
             mpi_env="-e MPI -e OPAL -e PMIX"
         fi
@@ -231,10 +234,10 @@ EOF
     trap "rm -f /tmp/tmpi_${USER}.lock" EXIT
 
     mpi_arg=""
-    if [ -n "${mpich}" ]; then
-        mpi_arg="" # used to be -pmi-port
-    else
+    if [ -n "${openmpi}" ]; then
         mpi_arg="--oversubscribe"
+    else
+        mpi_arg=""
     fi
 
     # re-execute ourself to spawn of the processes.

And then running tmpi with MPIRUN=slurm tmpi 4 gdb -x gdb_script.txt /to/exec

That seemed to spawn the window panes as expected with gdb running but all of them failed during MPI_Init. So some further investigation is required, but at least a start was made.

[biddisco]

Thanks for looking at this. I was away for holidays and didn't see your reply. I will try your patch and see if I can make any progress.

[Azrael3000]

Did you get any further with this John?