Missing documentation/ How to use

[JessicaTiebel]

Hi
I am using Snakemake for some months now, previously on a Slurm cluster where I used the slurm executer plugin.
Now I am workin on a LSF cluster and I wanted to adapt my workflow to the new cluster using this plugin. Unfourtunately I find it quite hard to use it, since there is little to no documentation how to use it.

How and where can I define the needed ressources?

Do I have to define the status, commit and cancel commands in the command line or can I also to that in a config file?

Is there anything else I need to specify?

[johanneskoester]

You can use a profile to specify those: https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles

[johanneskoester]

The documentation of this plugin can be found here: https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/cluster-generic.html
However, I agree that it is still sparse. Sorry, the transition to Snakemake 8 is quite big and still needs to mature a bit.

[keviny2]

Not sure if you resolved your issue @JessicaTiebel, but in case it helps anybody else, here's a simple example for LSF.

We can define resource requirements in the resource directive of a snakemake rule,

rule do_something:
    input:
        "my_input.txt"
    output:
        "my_output.txt",
    resources:
        mem_gb=96,
        walltime="12:00"
    shell:
        "touch {output}"

Then we can reference those resource specifications in the profile,

executor: cluster-generic
cluster-generic-submit-cmd: 'bsub -R "rusage[mem={resources.mem_gb}GB/host] span[hosts=1]" -W {resources.walltime}
jobs: 10

The syntax is somewhat similar to --cluster in snakemake<8.0 (see docs).

The plugin docs say you don't need to specify status, commit, and cancel commands, but you can specify them in the profile if needed.

[JessicaTiebel]

Hi,
Thank you very much for your answers. I also found the links that @johanneskoester supplied, but I couldn't really make sense of them. The example @keviny2 provided really helped a lot. It works now. I think I overcomplicated it in my first try.

[peterch405]

Could I maybe hijack this issue to try and get help migrating my profile from v7 to v8. I have posted a question on stackoverflow https://stackoverflow.com/questions/78426554/migrating-sge-profile-to-snakemake-8

Previously I was able to specify output and error logs in the cluster.yaml file as follows:

__default__:
    time: "05:00:00"
    mem: 20g
    cpus: 1
    nodes: 1
    output: "{out_dir}logs/cluster/{rule}.{wildcards.sample}.out"
    error: "{out_dir}logs/cluster/{rule}.{wildcards.sample}.err"

Since the cluster.yaml file is no longer supported my solution was:

executor: cluster-generic
cluster-generic-submit-cmd: qsub -V -cwd -l highp,h_data={resources.mem},h_rt={resources.time},nodes={resources.nodes} -pe shared {resources.cpus} -o {resources.output} -e {resources.error}

default-resources:
    mem: 20g
    time: "'05:00:00'"
    cpus: 1
    nodes: 1
    output: "'{out_dir}logs/cluster/{rule}.{wildcards.sample}.out'"
    error: "'{out_dir}logs/cluster/{rule}.{wildcards.sample}.err'"

But the out_dir, rule, and wildcards are not interpolated. Any suggestions how this could be achieved with the plugin?

[keviny2]

Would a solution be to redirect stdout and stderr in the shell directive in the Snakefile? For example,

rule my_rule:
    input:
        "some_input.txt"
    output:
        "some_output.txt",
    log:
        stdout="log_file.out",
        stderr="log_file.err",
    shell:
        """
        echo "Hello World" > {output} 1> {log.stdout} 2> {log.stderr}
        """

This is what I usually do and it's working for me so far.

[peterch405]

@keviny2 I haven't considered redirecting within each rule. Will require some rewrite, but is a good solution. Thank you!