Incompatibilities with Slurm on Virgo

[bsobol]

Hi,

I've been experimenting with DDS (I run the up-to-date master branch of DDS) on Virgo and encountered two issues:

• When my custom batch script configuration contained #SBATCH --mem-per-cpu, I got the following error srun: fatal: cpus_per_task set by two different environment variables SLURM_CPUS_PER_TASK=2 != SLURM_TRES_PER_TASK=cpu:1
  This can be fixed by explicitly stating --cpus-per-task in srun invocation and I believe is connected with the change in slurm behavior described here: https://docs.icer.msu.edu/2023-05-04_LabNotebook_srun_threading_changes/
• The second one I don't understand, but the command passed to srun via bash -c was not being executed. It's somehow being fixed by dumping the script into a file and passing it to bash -c.

Here's the diff with changes I made in job.slurm.in file:

@@ -16,8 +16,10 @@
 # continue waiting for child processes by any means
 trap -- '' SIGINT SIGTERM
 
+echo 'trap  '"'"'kill $PID && wait'"'"'  SIGINT SIGTERM; eval JOB_WRK_DIR=%DDS_AGENT_ROOT_WRK_DIR%/${SLURM_JOB_NAME}_${SLURM_JOBID}_${SLURMD_NODENAME}; mkdir -p $JOB_WRK_DIR; cd $JOB_WRK_DIR; cp %DDS_SCOUT% $JOB_WRK_DIR/; ./DDSWorker.sh & PID=$!;  wait'  > srun_script.sh
+
 # execute DDS Scoullt
-srun --no-kill --kill-on-bad-exit=0 --output=slurm-%j-%N.out /usr/bin/env bash -c 'trap  '"'"'kill $PID && wait'"'"'  SIGINT SIGTERM; eval JOB_WRK_DIR=%DDS_AGENT_ROOT_WRK_DIR%/${SLURM_JOB_NAME}_${SLURM_JOBID}_${SLURMD_NODENAME}; mkdir -p $JOB_WRK_DIR; cd $JOB_WRK_DIR; cp %DDS_SCOUT% $JOB_WRK_DIR/; ./DDSWorker.sh & PID=$!;  wait' &
+srun --cpus-per-task $SLURM_CPUS_PER_TASK --no-kill --kill-on-bad-exit=0 --output=slurm-%j-%N.out /usr/bin/env bash 'srun_script.sh' &
 
 wait

BTW, is there an intended way to reserve >1 cpu per slot for multithreaded tasks?

Regards,
Bartosz

[AnarManafov]

Hey @bsobol ,

Reg. #SBATCH --mem-per-cpu.
Did you try to specify #SBATCH --cpus-per-task in the same script?
if yes, please, attach your script on the ticket. I will investigate.

Reg.

The second one I don't understand, but the command passed to srun via bash -c was not being executed. It's somehow being fixed by dumping the script into a file and passing it to bash -c.

I think, there is a confusion here.
You should not be providing any srun to the slurm config for DDS. This config should contain only additional instructions regarding resources, queues, etc.
The actual task(s) you want DDS to run should be provided via the topology file and activated via the topology commands.

BTW, is there an intended way to reserve >1 cpu per slot for multithreaded tasks?
Users should be able to change that via #SBATCH --cpus-per-task.
The current slurm plug-in for DDS was tuned for ALICE production, where they reserver the entire host for the DDS agent and each DDS task slot runs one user task with one thread.

If it helps to better understand what this slurm plug-in executes in the background, here is its template of the job script: https://github.com/FairRootGroup/DDS/blob/master/plugins/dds-submit-slurm/src/job.slurm.in

DDS is flexible, it doesn't matter how dds agent (i.e. worker script with the bynary payload) ends up on the worker node. If the current slurm plug-in can't do what you want you and I can always come up with another plug-in.

[bsobol]

@AnarManafov

#SBATCH --cpus-per-task is added by the DDS, and it corresponds to the number of slots provided to the dds-submit

I don't add additional srun to the slurm config. I meant the srun that's already in the job.slurm.in and runs a DDSWorker, please check my diff of this file above.

Here's the job.slurm that successfully executed. It was started with dds-submit -r slurm -n 8 --slots 9 -c slurm.cfg and it fails without the additional --cpus-per-task $SLURM_CPUS_PER_TASK added to srun.

#!/usr/bin/env bash

#SBATCH --nodes=8
#SBATCH --no-kill

#SBATCH --job-name=dds_agent_job
#SBATCH --chdir=/lustre/cbm/users/bsobol/.DDS/sessions/7b71f8da-b7e2-43e9-89f6-e2e5355a3c05/wrk/d977087e-b306-4e88-9f28-7560fd6becba/

#SBATCH --cpus-per-task=9

# slurm.cfg BEGIN
#SBATCH --account=cbm
#SBATCH --partition=main
#SBATCH --time=04:00:00
#SBATCH --mem-per-cpu=12GB
export SLURM_SINGULARITY_CONTAINER=/lustre/cbm/users/bsobol/dev_04_2025.sif
# slurm.cfg END

# ignore signals
# continue waiting for child processes by any means
trap -- '' SIGINT SIGTERM

echo 'trap  '"'"'kill $PID && wait'"'"'  SIGINT SIGTERM; eval JOB_WRK_DIR=/lustre/cbm/users/bsobol/.DDS/sessions/7b71f8da-b7e2-43e9-89f6-e2e5355a3c05/wrk/d977087e-b306-4e88-9f28-7560fd6becba//${SLURM_JOB_NAME}_${SLURM_JOBID}_${SLURMD_NODENAME}; mkdir -p $JOB_WRK_DIR; cd $JOB_WRK_DIR; cp /lustre/cbm/users/bsobol/.DDS/sessions/7b71f8da-b7e2-43e9-89f6-e2e5355a3c05/wrk/d977087e-b306-4e88-9f28-7560fd6becba/DDSWorker.sh $JOB_WRK_DIR/; ./DDSWorker.sh & PID=$!;  wait'  > srun_script.sh

# execute DDS Scoullt
srun --cpus-per-task $SLURM_CPUS_PER_TASK --no-kill --kill-on-bad-exit=0 --output=slurm-%j-%N.out /usr/bin/env bash 'srun_script.sh' &

wait

exit 0

[AnarManafov]

#SBATCH --cpus-per-task is added by the DDS, and it corresponds to the number of slots provided to the dds-submit

This is a default behaviour. If you need a different number, then use --enable-overbooking for dds-submit (dds-submit --help for more info) and specify the desired number of CPU via #SBATCH --cpus-per-task in your config script.

Let me know if that fixes the issue with the original slurm job script of DDS.

[bsobol]

--enable-overbooking allows me to control the number of CPUs requested, but it only changes how the #SBATCH --cpus-per-task directive appears in the final job script (DDS controlled vs manual setting), so the two issues from the first post still exist.

And to be clear, I achieved the behavior I needed (even better with --enable-overbooking), just wanted you to know that I faced those two problems.