Support for per-rule spot instances?

[SichongP]

Currently, if we want to use spot instances with Batch, we need to send jobs to a queue that is connected to an environment set up using spot instances. But in reality, it often makes sense to send some long-running jobs to on-demand instances while others to spot instances, to minimize chances of interuptions. As far as I can undertand, this is currently not possible?

I'm wondering what would be the best way to bring that support. Ideally, we want a directive spot: True in the rule definition, but this requires modifying Snakemake code.

Alternatively, we can add a command line argument --spot-rules that takes a comma separated string of rule names that should be sent to spot instances (and --spot-job-queue that specifies the job queue to use for those jobs). The argument to --spot-rules can be quite long and unmanageable but that's the kind of setting that makes sense in a snakemake profile config so maybe it's acceptable?

[radusuciu]

I've added support for something like this in my fork (https://github.com/radusuciu/snakemake-executor-plugin-aws-batch/) via a per rule queue resource option aws_batch_job_queue. Using that you can use a separate queue for any rule.

I'd be happy to contribute if there was interest.