AWS batch and Elastic container registry

[SSmith-NUIG]

Hi,
I'm trying to get my Snakemake pipeline to use a custom docker image that I have stored in a private repo in the AWS ECR.

This docker image was created the usual Snakemake way with containerize, then tagged and pushed to the ECR following AWS documentation.

I can pull this container fine and it works interactively on my EC2 but I am unable to use it with Snakemake and AWS batch.

I have tried adding a global container at the top of the snakefile and also tried specifying the container in a rule, both don't work. Snakemake just keeps making the default container that it uses for creating a batch job and using that instead.

Does this plugin have any way to communicate with ECR?

[SSmith-NUIG]

I dont think this issue is limited to the ECR, I can't seem to use even publicly available docker images. Every job definition just creates and uses the default snakemake image that is made by aws batch.

I got this working by editing the following:

Plugin Patch: Support for image: in profile config.yaml

This enables users to specify a custom container image using the image: field in their profile config. With this patch, jobs submitted via the AWS Batch executor will use that image rather than defaulting to snakemake/snakemake.

✅ Changes made:

1. Add image to ExecutorSettings in __init__.py

@dataclass
class ExecutorSettings(ExecutorSettingsBase):
    ...
    image: Optional[str] = field(
        default=None,
        metadata={
            "help": "Docker image to use for AWS Batch job.",
            "env_var": False,
            "required": False,
            "snakemake_flag": "image"  # enables use of `aws-batch-image` in profile config
        },
    )

Update Executor.post_init() to prefer executor_settings.image

Replaced:

self.container_image = self.workflow.remote_execution_settings.container_image

With:

self.container_image = getattr(self.workflow.executor_settings, "image", None) \
                       or self.workflow.remote_execution_settings.container_image \
                       or "snakemake/snakemake:v9.1.4"

That's the snakemake version it kept defaulting to...

Set aws-batch-image: in .snakemake/config.yaml

e.g.:

executor: aws-batch
aws-batch-image:<acc_id>.dkr.ecr.<region>.amazonaws.com/image:latest

Though this could also just be done on the command line if you dont use profiles...

Now I just have to figure out how to use multiple container images in a single pipeline but for now this works.

[SichongP]

Snakemake is run on Batch in a container, which can be specified by --container-image command line argument. The container that should be used for each rule is separate and specified in workflow files

[SSmith-NUIG]

Yeah that's the issue I had though, batch was using the same default snakemake container for every rule. Even if I specified a container in the rule or at the top of the Snakefile it would still just use a default snakemake container.

[SichongP]

What was the command you used? Did you use --sdm or --use-singularity?

[SSmith-NUIG]

I tried both of those. If you have a way of getting it to work with the default plugin code I'd appreciate the help!

[phmoferring]

If you specify at the CLI --container-image, that image will be used as the top-level docker container when executed in batch. I don't think this corresponds to the same mechanism as specifying a container/containerized in the Snakefile. As I understand it, the --sdm argument is independent of this and is only relevant once a batch job is started and the snakemake command to run the specific rule in question is executed within the --container-image container. If you want to additionally specify job-level containers, your --container-image container must have apptainer installed and be using --sdm apptainer.

If you are using a private ECR image for job-level containerization you must pass --envvars DOCKER_USERNAME DOCKER_PASSWORD(these must be prepopulated in the top-level execution environment) to the initial snakemake command to send the top-level execution context's docker credentials through to AWS Batch -> docker -> apptainer, otherwise apptainer cannot authenticate and pull from your ECR even though the compute environment context may have the appropriate privileges to authenticate. There may be a cleaner way, but I have not found one thus far.