Duplicate HeadRequests against S3 Express when using the LegacyMD5Plugin.

[ahmarsuhail]

Describe the bug

We're trying to upgrade S3A to SDK v2.32.23, see this PR. To restored third party support, we're adding the Md5 legacy plugin, like so:

   builder.addPlugin(LegacyMd5Plugin.create());

    // do not do request checksums as this causes third-party store problems.
    builder.requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED);

    // response checksum validation. Slow, even with CRC32 checksums.
    if (parameters.isChecksumValidationEnabled()) {
      builder.responseChecksumValidation(ResponseChecksumValidation.WHEN_SUPPORTED);
    }

However, this is causing one of our ITests to fail:

  @Test
  public void testSDKMetricsCostOfGetFileStatusOnFile() throws Throwable {
    describe("Performing getFileStatus() on a file");
    Path simpleFile = file(methodPath());
    // and repeat on the file looking at AWS wired up stats
    final S3AFileSystem fs = getFileSystem();
    LOG.info("Initiating GET request for {}", simpleFile);
    verifyMetrics(() -> fs.getFileStatus(simpleFile),
        with(STORE_IO_REQUEST, 1));
  }

This test checks that only a single HTTP request is made, but we're seeing 2 requests, with the first request always failing when run against a S3 express store.

Things to note:

• When making JUST a head request, this does not happen.
• But when replicating what this test case does, I am able to replicate it. The test case will first check if a test/ directory exists, if it doesn't exist, it'll create it, and then do a HEAD.

If I comment out the .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED), there are no retries and the test passes.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

There should be no retries. Currently the first HEAD always fails with

2025-09-18 11:19:42,219 [setup] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2025-09-18 11:19:42,219 [setup] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-0 >> "[\r][\n]"
2025-09-18 11:19:42,220 [setup] DEBUG http.wire (Wire.java:wire(87)) - http-outgoing-0 << "end of stream"
2025-09-18 11:19:42,220 [setup] DEBUG conn.DefaultManagedHttpClientConnection (LoggingManagedHttpClientConnection.java:close(79)) - http-outgoing-0: Close connection

Current Behavior

HEAD request fails with end of stream and is retried.

Reproduction Steps

Run the below code exactly to reproduce locally. Making just a HEAD request will succeed. Something about this sequence of operations causes the retry. Everything works as expected when requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED) is commented out.

public class TestClass {

    S3Client s3Client;

    public TestClass() {
        this.s3Client = S3Client.builder().region(Region.US_EAST_1)
                .addPlugin(LegacyMd5Plugin.create())
                .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
                .responseChecksumValidation(ResponseChecksumValidation.WHEN_SUPPORTED)
                .overrideConfiguration(o -> o.retryStrategy(b -> b.maxAttempts(1)))
                .build();
    }


    public void testS3Express(String bucket, String key) {
        s3Client.listObjectsV2(ListObjectsV2Request.builder()
                .bucket("s3-express-bucket")
                .maxKeys(2)
                .prefix("test/")
                .build());


        try {
            s3Client.headObject(HeadObjectRequest.builder().bucket("s3-express-bucket")
                    .key("test")
                    .build());
        } catch (Exception e) {
            System.out.println("Exception thrown: " + e.getMessage());
        }

        s3Client.putObject(PutObjectRequest
                .builder()
                .bucket("s3-express-bucket")
                .key("test/").build(), RequestBody.empty());

        s3Client.headObject(HeadObjectRequest.builder().bucket("s3-express-bucket")
                .key("<KEY>")
                .build());
    }

}

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.32.23

JDK version used

17

Operating System and version

Amazon linux

[bhoradc]

Hi @ahmarsuhail

Thanks for reporting the issue. I am able the reproduce the behavior you described above. I see that you have an internal ticket raised as well, we'll investigate and provide further update on it.

Regards,
Chaitanya

[joviegas]

@ahmarsuhail Thanks for the details.

We are actively working on this issue. The issue is specific to zero content-length PutObject to S3 Express buckets with checksums disabled.

In this specific case, the connection from the PutObject gets reused for the HeadObject call and it errors out since it's already closed by S3.

We are doing a deep dive on why we are getting "end of stream" after some time. (Note: if you add a 2-second delay between PutObject and HeadObject, the issue is not seen since the "end of stream" closes the connection before the HeadObject call.)

This issue is not seen with AwsCrtHttpClient since this HTTP client automatically closes the connection in these cases. We are investigating whether we need to handle this in ApacheSdkHttpClient.

We'll keep you updated on our progress.

Update : Communicated the issue with S3Express team , they are actively working to see why is stream getting closed for putObject for S3Express with EmptyRequestBody .

[joviegas]

@ahmarsuhail Also wanted to highlight that this issue surfaces because of:

.overrideConfiguration(o -> o.retryStrategy(b -> b.maxAttempts(1)))

Since it's set to 1, there will be no retry, hence the exception is thrown. In case of defaults, there will be a retry taking place and that will use a new connection for the retry, and the issue will not be seen.

What is the functional need to set maxAttempts to 1? As a workaround until it's fixed in S3 Express, can we set the maxAttempts to 2 as a workaround for this issue?

[ahmarsuhail]

Hey @joviegas, that .overrideConfiguration(o -> o.retryStrategy(b -> b.maxAttempts(1))) is only to show that the first HEAD fails in this reproduction code, we don't use that in S3A.

S3A will fallback to the SDK retry, and the request will succeed, but I'm not sure if we feel comfortable releasing something with an issue like this.

[steveloughran]

and the issue will not be seen.
s3a does retry, we saw this as we count the number of internal requests. Does add an extra HTTP request though, which adds time and money.

Do you think we should make the use of the legacy signing stuff optional? then for s3 express we just document "turn this off". or even automatically do it once we see it is S3 express?

[ahmarsuhail]

@steveloughran - we do only need it for third party stores, and those don't have S3 express anyway. So if we make it optional, and say if you're using third party, turn on legacy md5 checksumming, and otherwise don't add the plugin, we won't hit this issue.