How to use `--cluster-generic-cancel-cmd` on LSF

[keviny2]

My Snakefile looks like this,

rule test:
    output:
        "test.txt"
    resources:
        mem_mb=4000
    shell:
        "sleep 600 && touch {output}"

and my config.yaml file looks like,

executor: cluster-generic
cluster-generic-submit-cmd: 'bsub -R "rusage[mem={resources.mem_mb}GB]" -n 4'
cluster-generic-cancel-cmd: bkill

I fire up my workflow with snakemake --profile profile/ and then press ^C to terminate the process early. I get the following warning,

Job <36740856> is submitted to default queue <long>.: Illegal job ID.
1 out of 1 calls to --cluster-cancel failed. This is safe to ignore in most cases.

and the job isn't terminated. What am I missing?

I am using snakemake v8.11.3.

[peterch405]

@keviny2 did you find an answer? I'm wondering the same with SGE, cluster-generic-cancel-cmd: qdel throws the same error.

[keviny2]

no, i haven't found a solution. for now, i'm using bash commands to terminate pending jobs

[lutrarutra]

The issue is that the whole output of the submit command is assumed to be the job id, howerever, quite often the output is something like: Job <36740856> is submitted to default queue <long>. Which is given straight to the cancel command, and ofcourse this won't work.

Hence, the fix is to parse the job id from this output string. In my case, I forked the repo and modified couple lines in __init__.py:

if (match := re.match(r"Submitted batch job (\d+)", ext_jobid)):
    ext_jobid = match.group(1)
else:
    print(f"Error: Could not parse external job id from string '{ext_jobid}'")

You can look at example in my repo: https://github.com/lutrarutra/snakemake-executor-plugin-cemm-hpc