`VALID_RESOURCES_MAPPING` is restrictive, and only applicable to Fargate compute environments.

[phmoferring]

The VALID_RESOURCES_MAPPING derived from the AWS documentation on ResourceRequirement is only required for Fargate compute environments and requires that job definitions be very specific about how much memory/threads to request. When you request an amount of memory that isn't a key to this dictionary, or a number of threads that doesn't match with the used memory value, things don't behave as expected, either selecting the minimum memory for the threads provided or erroring out respectively. (Happens in the _validate_resources method of the BatchJobBuilder class.)

It would be great if this handling was more flexible for EC2 compute environments, which can have any instance type and can be much more flexible than the Fargate related expectations. It's possible I get the chance to implement an improvement here, but I figured I'd document the problem in case others encounter the same.

[radusuciu]

I'm about to use this for a project (using EC2 instances) and seems like this is worth fixing.

Any thoughts on whether it'd be better to auto-detect the compute environment associated with a queue or if it can be a configuration item? The latter would be simpler to implement and would be more robust out of the box.

[phmoferring]

I'd say that because compute environments can have multiple instance types/families specified, it would be best if the code just used whatever values for memory/threads are provided by the job, with some handling for minimum and possibly maximum values. That way the onus is on the workflow/job itself to specify the ratio of memory:vcpu appropriate for the compute environment in question. If your workflow needs to be able to adjust to different compute environments, I'd use the workflow level config and dynamic resource functions to generalize the handling.

[radusuciu]

In that case, the validation could be as simple as:

# Basic sanity checks
if vcpu < 1:
    raise WorkflowError(f"vCPU value must be at least 1, got {vcpu}")
if mem < 1:
    raise WorkflowError(f"Memory value must be at least 1 MB, got {mem}")

return str(vcpu), str(mem)

[phmoferring]

I would probably recommend a much higher memory minimum, maybe 1024MiB, I doubt just the overhead of the docker image and snakemake starting the job in question would work with much less.

[SichongP]

Just to add to this thread, this is the cause of a very vaguely phrased error message that took me a few hours to track down the exact cause.

When a rule specifies a number of cpus that is not one of [1,2,4,8,16], an error is thrown:

Failed to submit AWS Batch job: min() iterable argument is empty

This also immediately terminates already submitted jobs and removes deployed workflow resources from remote path. If a job happens to start around this time, it may get another seeming unrelated error on the remote job log:

Failed to get expected local footprint (i.e. size) of s3://.../snakemake-workflow-sources.7db2e4a1dfb1775aa55f32c819bdc603e5afa93b8d15903a87451e118bed3cc4.tar.xz
ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

But that's because of now deleted workflow resources.

The error is caused by this line:

snakemake-executor-plugin-aws-batch/snakemake_executor_plugin_aws_batch/batch_job_builder.py

Line 54 in 2525ca0

min_mem = min([m for m, v in VALID_RESOURCES_MAPPING.items() if vcpu in v])