Add --slurm_extra_args

[Hoeze]

Is your feature request related to a problem? Please describe.
I would like to specify --exclude=<nodelist> to each sbatch call.
Unfortunately, I did not find a way to do that.

Describe the solution you'd like
Add --slurm_extra_args setting that gets appended to the sbatch arguments, e.g.:
--slurm_extra_args="--exclude=<nodelist>"

Describe alternatives you've considered
playing with sbatch environment variables, without success. Did not find any that would work for this use case

[cmeesters]

Hi,

thank you for this feature request. Have you tried it? Did it work? If not, which was the error message?

Also, note that the plugin already tries to exclude nodes where jobs failed without reason. Ordinarily, this is achieved on clusters with a node health check, which automatically disables faulty nodes. But knowing, that there is no such thing as "ordinarily" in the HPC world, the feature has been implemented.

[Hoeze]

Dear @cmeesters, thank you for your response.

Have you tried it? Did it work? If not, which was the error message?

the feature has been implemented.

I am not aware of such a feature being implemented yet. How would I test it?

Also, note that the plugin already tries to exclude nodes where jobs failed without reason

The jobs fail due to the specific CPU architecture on our two oldest nodes. I usually just skip these if necessary, but I did not find a way to skip them using the executor plugin.
Of course, I could have implemented custom resources, etc., but this would be overkill to exclude two nodes.

knowing, that there is no such thing as "ordinarily" in the HPC world

Yes, exactly 😄

[cmeesters]

How would I test it?

Well, slurm_extra is not a command line flag, but a resource parameter. Have a look in the docs. It is intended a "last resort". If you want this to be a default resource, try:

default_resources:
    slurm_extra: "--exclude=<nodelist>"
    # further parameters of your workflow profile go here
    # ...

If this is part of your workflow profile, you can use snakemake ... --workflow-profile <path to dir>.