Skip to content

Fix ECR BLOB_UPLOAD_INVALID by forcing new session on push retry#792

Open
am-saksham wants to merge 1 commit into
apple:mainfrom
am-saksham:main
Open

Fix ECR BLOB_UPLOAD_INVALID by forcing new session on push retry#792
am-saksham wants to merge 1 commit into
apple:mainfrom
am-saksham:main

Conversation

@am-saksham

Copy link
Copy Markdown

Fixes #790
Resolves apple/container#1895

Motivation

When pushing large, multi-architecture images to AWS ECR, network interruptions cause the push to fail with a 416 Range Not Satisfiable and BLOB_UPLOAD_INVALID error.

This occurs because the generic retry loop in RegistryClient.swift catches the failure, resets the data stream to byte 0, and blindly retries the exact same upload session URL. Because ECR expects the retry to resume from the last committed byte, it rejects the byte 0 payload.

Modifications

This PR decouples blob upload retries from the generic RegistryClient retry loop to allow for proper session regeneration.

  1. RegistryClient.swift: Added a retryOptionsOverride parameter to the generic request() signature. This allows specific operations to opt out of the default infinite retry loop while maintaining the client's standard behavior for auth and manifests.
  2. RegistryClient+Push.swift: Completely disabled the generic retry for blob POST and PUT requests by passing RetryOptions(maxRetries: 0, retryInterval: 0).
  3. RegistryClient+Push.swift: Wrapped the blob upload sequence in a dedicated, smart retry loop. If a PUT drops mid-upload, the loop now catches the error locally, requests a fresh session UUID via a new POST, and starts the byte 0 upload on that clean session.

Result

Interrupted pushes to AWS ECR now successfully recover by abandoning the stale session and initiating a fresh upload, eliminating the 416 error.

@sadysnaat

Copy link
Copy Markdown

@am-saksham thanks a lot for the quick PR.

I was thinking we should revisit the whole Distribution Spec implementation. We might want to implement resume upload with PATCH as well.

Let's see what everyone thinks about that.

@am-saksham

Copy link
Copy Markdown
Author

Thanks @sadysnaat! I completely agree that moving to the full Distribution Spec with chunked PATCH uploads and GET state recovery is the most robust long-term solution.

I put this POST/PUT session reset together as a lightweight stopgap to immediately unblock the ECR 416 errors without requiring a heavy rewrite of the upload pipeline.

Let me know if the team decides to merge this as a temporary bridge, or if you'd prefer to hold off for a complete PATCH refactor. Happy to help look into the PATCH implementation as well if the team decides to go that route!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants