NERSC profile to configure interaction with SLURM

[gipert]

It's possible to instruct Snakemake to interact with SLURM and execute groups of jobs in parallel on a computing node (so that few SLURM jobs are actually submitted). See relevant documentation:

• https://snakemake.readthedocs.io/en/stable/executing/grouping.html
• https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html

Note: this statement in the NERSC documentation is wrong: https://docs.nersc.gov/jobs/workflow/snakemake/#disadvantages-of-snakemake

I tried to write a Snakemake v7 profile for legend-simflow a while ago:

https://github.com/legend-exp/legend-simflow/blob/main/profiles/nersc-batch/config.yaml

but did not work properly, see

• Workflow not being executed in parallel on batch computing nodes (SLURM cluster grouped execution) snakemake/snakemake#2339
• sbatch call does not escape quotes in exec_job snakemake/snakemake-executor-plugin-slurm#29

We should try again with Snakemake v8.

See also: legend-exp/legend-simflow#8

[gipert]

Good news: all of this seems to work now! I'm testing it with legend-simflow