Hello Bottlerocket team:
We are using bottlerocket as OS for our AWS EKS v1.33.8 cluster.
After upgrading to bottlerocket v1.58.0, we started to see the following error reported by ArgoCD (job log of job downloading the mysql image):
state:
waiting:
message: >-
Back-off pulling image
"aws-access.deliveryzone.xxxxxxxx.com:443/blue/db-migration-mysql:1.0.1":
ErrImagePull: failed to pull and unpack image
"aws-access.deliveryzone.xxxxxxxx.com:443/blue/db-migration-mysql:1.0.1":
failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip
sha256:023a182c62a0ce5adf24030e7fee994ceaa333b22cdb5f1a0835501015edf3ed)
to overlayfs as "extract-251576486-TcNB
sha256:827b99c091a57f0fc1ad0a026084fa9335638ba911267b9764a05b991077a0da":
mount callback failed on
/var/lib/containerd/tmpmounts/containerd-mount318086458: permission
denied
reason: ImagePullBackOff
What I expected to happen:
Image to be downloaded and unpacked with all layers without any errors.
What actually happened:
When containerd tries to extract Docker image layers during docker pull in ArgoCD it fails with permission denied on the mount point.
How to reproduce the problem:
AWS EKS 1.33.8 cluster
Bottlerocket v1.58.0
download db-migration-mysql (multi-layered) image from Nexus repository
The error disappeared after rolling back to bottlerocket v1.57.0
Dockerfile (empty lines deleted):
FROM debian:bookworm-20250610-slim AS base
WORKDIR /app
RUN apt-get update
&& apt-get install -y --no-install-recommends apt-utils ca-certificates
&& rm -rf /var/lib/apt/lists/*
RUN update-ca-certificates
COPY tls-ca-chain/*.pem /tls-ca-chain/
FROM mysql:8.4.6 AS builder
WORKDIR /app
USER nonroot
COPY --from=base /tls-ca-chain /etc/ssl/certs/
install the bm-migrate binary
ADD bm-migrate.tar.gz bm-migrate
RUN mv bm-migrate/bm-migrate /usr/local/bin/bm-migrate
RUN touch .env
ADD migrations migrations
CMD ["bm-migrate"]
Hello Bottlerocket team:
We are using bottlerocket as OS for our AWS EKS v1.33.8 cluster.
After upgrading to bottlerocket v1.58.0, we started to see the following error reported by ArgoCD (job log of job downloading the mysql image):
What I expected to happen:
Image to be downloaded and unpacked with all layers without any errors.
What actually happened:
When containerd tries to extract Docker image layers during docker pull in ArgoCD it fails with permission denied on the mount point.
How to reproduce the problem:
AWS EKS 1.33.8 cluster
Bottlerocket v1.58.0
download db-migration-mysql (multi-layered) image from Nexus repository
The error disappeared after rolling back to bottlerocket v1.57.0
Dockerfile (empty lines deleted):
FROM debian:bookworm-20250610-slim AS base
WORKDIR /app
RUN apt-get update
&& apt-get install -y --no-install-recommends apt-utils ca-certificates
&& rm -rf /var/lib/apt/lists/*
RUN update-ca-certificates
COPY tls-ca-chain/*.pem /tls-ca-chain/
FROM mysql:8.4.6 AS builder
WORKDIR /app
USER nonroot
COPY --from=base /tls-ca-chain /etc/ssl/certs/
install the bm-migrate binary
ADD bm-migrate.tar.gz bm-migrate
RUN mv bm-migrate/bm-migrate /usr/local/bin/bm-migrate
RUN touch .env
ADD migrations migrations
CMD ["bm-migrate"]