Multipart upload signature error after updating the SDK

[HenryS1]

Describe the bug

Multipart upload stopped working for me after I updated the SDK from version 2.29.52 to 2.42.8.

I'm not completely sure that this is a bug, but it appears to me that the requests I'm making should be valid in the new SDK. If there is something wrong with my requests please tell me how I should remedy the problem.

Here is a minimal example in scala that works in 2.29.52 and doesn't work in 2.42.8.

            val bucket = "my-bucket"
            val key = "my-key"

            val createRequest = awsModel.CreateMultipartUploadRequest.builder()
              .bucket(bucket)
              .key(key)
              .build()

            val createResult = Await.result(s3Client.createMultipartUpload(createRequest).asScala, 10 seconds)
            
            val partRequest = awsModel.UploadPartRequest.builder()
              .bucket(bucket)
              .key(key)
              .uploadId(createResult.uploadId())
              .partNumber(1)
              .contentLength(5)
              .build()

            val completedPart = Await.result(s3Client.uploadPart(
              partRequest,
              AsyncRequestBody.fromByteBuffer(ByteString("Hello").asByteBuffer)
            ).asScala, 10 seconds)

            val completeRequest = awsModel.CompleteMultipartUploadRequest.builder()
              .bucket(bucket)
              .key(key)
              .uploadId(createResult.uploadId())
              .multipartUpload(
                awsModel.CompletedMultipartUpload.builder()
                  .parts(
                    awsModel.CompletedPart.builder()
                      .partNumber(1)
                      .eTag(completedPart.eTag)
                      .build()
                  )
                  .build()
              )
              .build()

            Await.result(s3Client.completeMultipartUpload(completeRequest).asScala, 10 seconds)

On the old SDK version the requests are all successful and I can see the object in S3. When I use the new SDK version I get the following error:

java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403, Request ID: 4BXHV234N122G6PP, Extended Request ID: j9jAU0fHyvZ+/KOVWSWAyxg0KDHIsw8rvQuLS7MJ/GEjSXu6iYQTNsWSl4p9x3ipnebpnr6zS6ei2hv2Fj+tccOgoD6yY0E4xaAJ6Zxz/Gg=) (SDK Attempt Count: 1)

After some debugging I found that the createMultipartUpload request succeeds and the uploadPart request fails.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The requests work and successfully create a new object in S3.

Current Behavior

I get this error:

java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403, Request ID: 4BXHV234N122G6PP, Extended Request ID: j9jAU0fHyvZ+/KOVWSWAyxg0KDHIsw8rvQuLS7MJ/GEjSXu6iYQTNsWSl4p9x3ipnebpnr6zS6ei2hv2Fj+tccOgoD6yY0E4xaAJ6Zxz/Gg=) (SDK Attempt Count: 1)

Reproduction Steps

Here is a minimal example in scala that works in 2.29.52 and doesn't work in 2.42.8.

            val bucket = "my-bucket"
            val key = "my-key"

            val createRequest = awsModel.CreateMultipartUploadRequest.builder()
              .bucket(bucket)
              .key(key)
              .build()

            val createResult = Await.result(s3Client.createMultipartUpload(createRequest).asScala, 10 seconds)
            
            val partRequest = awsModel.UploadPartRequest.builder()
              .bucket(bucket)
              .key(key)
              .uploadId(createResult.uploadId())
              .partNumber(1)
              .contentLength(5)
              .build()

            val completedPart = Await.result(s3Client.uploadPart(
              partRequest,
              AsyncRequestBody.fromByteBuffer(ByteString("Hello").asByteBuffer)
            ).asScala, 10 seconds)

            val completeRequest = awsModel.CompleteMultipartUploadRequest.builder()
              .bucket(bucket)
              .key(key)
              .uploadId(createResult.uploadId())
              .multipartUpload(
                awsModel.CompletedMultipartUpload.builder()
                  .parts(
                    awsModel.CompletedPart.builder()
                      .partNumber(1)
                      .eTag(completedPart.eTag)
                      .build()
                  )
                  .build()
              )
              .build()

            Await.result(s3Client.completeMultipartUpload(completeRequest).asScala, 10 seconds)

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.42.8

JDK version used

openjdk 21.0.5 2024-10-15 LTS

Operating System and version

Ubuntu 20.04.6 LTS

[debora-ito]

Hi @HenryS1

• Can you share a code sample of how you're creating s3Client?
• Are you able to generate the wirelogs? If you are, you can compare the request headers and signature of the UploadPart before and after. Instructions here
• Unrelated to the bug: is there any reason why you are not using the S3TransferManager?

If you decide to share the wirelogs here in the thread, please redact any sensitive information (like access keys).

I'll try to reproduce in the meantime.

[debora-ito]

Just realized in v2.30.0 the new s3 integrity protection was released, and although it shouldn't throw an error based in your repro code, it may be related. Can you try adding

requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)

when the newer SDK version is used, to see if the error goes away?
I'll continue to investigate the rootcause.

[HenryS1]

Hi @debora-ito

Thanks for your prompt response

Can you share a code sample of how you're creating s3Client

    S3AsyncClient.builder
      .credentialsProvider(DefaultCredentialsProvider.builder().build())
      .region(region)
      .httpClient(PekkoHttpClient.builder().withActorSystem(system).build())
      .build()

Adding requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED) when I construct the client makes the problem go away, although I see that that disables the integrity check so it's not a fix.

Regarding this

Unrelated to the bug: is there any reason why you are not using the S3TransferManager?

I'm uploading a stream of byte chunks and when I came across S3TransferManager before it seemed it was designed for files or directories of files, but I suppose I could use it with an AsyncRequestBody and a publisher. The multipart upload API is simple enough though and has worked fine for me so far so I'd like to stick with it for now.

Are you able to generate the wirelogs? If you are, you can compare the request headers and signature of the UploadPart before and after. Instructions here

I'll get onto that

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[HenryS1]

@debora-ito

I found that when I use the netty async client the upload works without disabling the integrity check so it must be something wrong with the pekko client. I'll create an issue in that repo.

Thanks for your help