S3AsyncClient CRT Client does not honor S3CrtProxyConfiguration - nonProxyHosts is not being read after Issue 6414 is closed

[UnirCs]

Describe the bug

This is an issue related to the closed issue #6414
PR for closing: #6511

Hi @alextwoods , @bhoradc , @debora-ito
Thanks for the time looking into the issue. I've seen the fix and try to get my code working with these changes:

• Uploading software.amazon.awssdk:s3 to 2.36.3
• Transitively, upload software.amazon.awssdk:crt-core2.36.3`
• Uploading software.amazon.awssdk.crt to 0.39.4

Being LocalStack deployed on my local machine:

System.setProperty("https.proxyHost", "proxy.example.com");
System.setProperty("https.proxyPort", "8080");
System.setProperty("http.nonProxyHosts", "127.0.0.1");
System.setProperty("aws.crt.log.level","Trace");
System.setProperty("aws.crt.log.destination","Stdout");

S3AsyncClient s3Async =
        S3AsyncClient.crtBuilder()
            .endpointOverride(localstack.getEndpoint())
            .credentialsProvider(
                StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(
                        localstack.getAccessKey(), localstack.getSecretKey())))
            .region(Region.of(localstack.getRegion()))
            .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
            .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
            .httpConfiguration(S3CrtHttpConfiguration.builder()
                .proxyConfiguration(AwsCloudStorageUtils.getProxyConfiguration())
                .build())
            .build();
S3TransferManager transferManager = S3TransferManager.builder()
        .s3Client(s3Async)
        .build();

ProxyConfig:

/**
   * Obtains the proxy configuration from system properties. If the https.proxyHost property is not
   * set, this method returns null indicating no proxy should be used.
   *
   * @return an S3CrtProxyConfiguration object if the proxy is configured, or null if no proxy is
   *     set.
   */
  public static S3CrtProxyConfiguration getProxyConfiguration() {
    String proxyHost = System.getProperty("https.proxyHost");

    // Si no hay host configurado, no usar proxy
    if (StringUtils.isEmpty(proxyHost)) {
      return null;
    }

    // Puerto por defecto es 8080 si no se especifica
    String proxyPortStr = System.getProperty("https.proxyPort");
    int proxyPort = 8080; // Puerto por defecto del SDK de AWS

    if (!StringUtils.isEmpty(proxyPortStr)) {
      try {
        proxyPort = Integer.parseInt(proxyPortStr.trim());
      } catch (NumberFormatException e) {
        // Si el puerto no es válido, usar el puerto por defecto
        proxyPort = 8080;
      }
    }

    return (S3CrtProxyConfiguration) S3CrtProxyConfiguration.builder()
        .host(proxyHost.trim())
        .port(proxyPort)
        .nonProxyHosts(getNonProxyHosts())
        .build();
  }

  /**
   * Retrieves and parses the http.nonProxyHosts system property. Converts wildcard patterns
   * (*.domain.com) to proper regex patterns (.*\.domain\.com). The property should contain hosts
   * separated by pipe (|) characters.
   * <p>
   * Example input: "169.254.169.254|169.254.170.2|*.amazonaws.com|*.corp" Example output:
   * {"169\.254\.169\.254", "169\.254\.170\.2", ".*\.amazonaws\.com", ".*\.corp"}
   *
   * @return a Set of String containing properly escaped regex patterns for non-proxy hosts
   */
  public static Set<String> getNonProxyHosts() {
    Set<String> result = new HashSet<>();

    String nonProxyHosts = System.getProperty("http.nonProxyHosts");
    if (StringUtils.isEmpty(nonProxyHosts)) {
      return result;
    }

    String[] hosts = nonProxyHosts.split("\\|");
    for (String host : hosts) {
      String trimmedHost = host.trim();
      result.add(trimmedHost);
    }

    return result;
  }

Then I store two files in a "directory" called test-files/ , with a non-Crt S3 Client. Then I try do download them with the Crt client above.
I see the same log in the execution

[INFO] [2025-10-29T14:33:41Z] [0000000300f67000] [http-connection] - (STATIC) Connecting to "127.0.0.1" through a tunnel via proxy "proxy.example.com"

Is this bug fixed really? Reading comments in PR I thought it was transparent for us. Is there any change that should be done from our side?
Full log included

BugAwsNonProxyHosts.txt

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Connect to my Localstack instance on 127.0.0.1 without any attempt to go through the invalid proxy "proxy.example.com"

Current Behavior

The connection is taking place through a tunnel via proxy:

BugAwsNonProxyHosts.txt

Reproduction Steps

Use the following values for these properties:

System.setProperty("https.proxyHost", "proxy.example.com");
System.setProperty("https.proxyPort", "8080");
System.setProperty("http.nonProxyHosts", "127.0.0.1");
System.setProperty("aws.crt.log.level","Trace");
System.setProperty("aws.crt.log.destination","Stdout");

1. Create a LocalStack sandbox for testing

@Container
  public static LocalStackContainer localstack =
      new LocalStackContainer(DockerImageName.parse("localstack/localstack:0.11.3"))
          .withServices(S3);

2. Create S3Client (not async), S3AsyncClient (Crt) and a TransferManager client based on the async s3 client:

        S3Client s3 = S3Client.builder()
            .endpointOverride(localstack.getEndpoint())
            .credentialsProvider(
                StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(
                        localstack.getAccessKey(), localstack.getSecretKey())))
            .region(Region.of(localstack.getRegion()))
            .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
            .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
            .build();

    S3AsyncClient s3Async =
        S3AsyncClient.crtBuilder()
            .endpointOverride(localstack.getEndpoint())
            .credentialsProvider(
                StaticCredentialsProvider.create(
                    AwsBasicCredentials.create(
                        localstack.getAccessKey(), localstack.getSecretKey())))
            .region(Region.of(localstack.getRegion()))
            .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
            .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
            .httpConfiguration(S3CrtHttpConfiguration.builder()
                .proxyConfiguration(AwsCloudStorageUtils.getProxyConfiguration())
                .build())
            .build();

    S3TransferManager transferManager = S3TransferManager.builder()
        .s3Client(s3Async)
        .build();

This is the configuration I am using:

public static S3CrtProxyConfiguration getProxyConfiguration() {
    String proxyHost = System.getProperty("https.proxyHost");

    if (StringUtils.isEmpty(proxyHost)) {
      return null;
    }

    String proxyPortStr = System.getProperty("https.proxyPort");
    int proxyPort = 8080;

    if (!StringUtils.isEmpty(proxyPortStr)) {
      try {
        proxyPort = Integer.parseInt(proxyPortStr.trim());
      } catch (NumberFormatException e) {
        proxyPort = 8080;
      }
    }

    return (S3CrtProxyConfiguration) S3CrtProxyConfiguration.builder()
        .host(proxyHost.trim())
        .port(proxyPort)
        .nonProxyHosts(getNonProxyHosts())
        .build();
  }


  public static Set<String> getNonProxyHosts() {
    Set<String> result = new HashSet<>();

    String nonProxyHosts = System.getProperty("http.nonProxyHosts");
    if (StringUtils.isEmpty(nonProxyHosts)) {
      return result;
    }

    String[] hosts = nonProxyHosts.split("\\|");
    for (String host : hosts) {
      String trimmedHost = host.trim();
      result.add(trimmedHost);
    }

    return result;
  }

3. Create buckets, upload files... using the S3Client (not async)
4. Try to download a directory (for example) with the AsyncS3Client via TransferManager

Possible Solution

No response

Additional Information/Context

Using 0.39.4 version of aws-crt: https://mvnrepository.com/artifact/software.amazon.awssdk.crt/aws-crt/0.39.4

AWS Java SDK version used

2.36.3

JDK version used

java version "21.0.1" 2023-10-17 LTS

Operating System and version

MacOS 26.0.1 (25A362)

[bhoradc]

Hi @UnirCs

Acknowledged, thanks for validating the fix. We are looking into it and will post further updates.

Regards,
Chaitanya

[alextwoods]

I believe the fix has not been released yet (Java SDK release yesterday was delayed) - this should be released in 2.37.0.

[bhoradc]

Hi @UnirCs

I have validated below code sample with Java SDK v2.37.0 and aws-crt v0.39.4 and the fix appears to be working.

Main.java

package org.example;

import org.apache.commons.lang3.StringUtils;
import org.testcontainers.containers.localstack.LocalStackContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.utility.DockerImageName;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
import software.amazon.awssdk.crt.Log;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.crt.S3CrtHttpConfiguration;
import software.amazon.awssdk.services.s3.crt.S3CrtProxyConfiguration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.DirectoryDownload;
import software.amazon.awssdk.transfer.s3.model.DownloadDirectoryRequest;

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashSet;
import java.util.Set;

import static org.testcontainers.containers.localstack.LocalStackContainer.Service.S3;

public class Main {
    
    @Container
    public static LocalStackContainer localstack =
            new LocalStackContainer(DockerImageName.parse("localstack/localstack:0.11.3"))
                    .withServices(S3);

    public static void main(String[] args) throws Exception {
        System.setProperty("https.proxyHost", "testingproxy.example.com");
        System.setProperty("https.proxyPort", "8080");
        System.setProperty("http.nonProxyHosts", "127.0.0.1");
        Log.initLoggingToStdout(Log.LogLevel.Trace);
        localstack.start();

        try {

            S3Client s3 = S3Client.builder()
                    .endpointOverride(localstack.getEndpoint())
                    .credentialsProvider(
                            StaticCredentialsProvider.create(
                                    AwsBasicCredentials.create(
                                            localstack.getAccessKey(), localstack.getSecretKey())))
                    .region(Region.of(localstack.getRegion()))
                    .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
                    .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
                    .build();

            S3AsyncClient s3Async =
                    S3AsyncClient.crtBuilder()
                            .endpointOverride(localstack.getEndpoint())
                            .credentialsProvider(
                                    StaticCredentialsProvider.create(
                                            AwsBasicCredentials.create(
                                                    localstack.getAccessKey(), localstack.getSecretKey())))
                            .region(Region.of(localstack.getRegion()))
                            .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
                            .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
                            .httpConfiguration(S3CrtHttpConfiguration.builder()
                                    .proxyConfiguration(getProxyConfigurationWithLocalStack())
                                    //.proxyConfiguration(getProxyConfiguration())
                                    .build())
                            .build();


            S3TransferManager transferManager = S3TransferManager.builder()
                    .s3Client(s3Async)
                    .build();

            String bucketName = "issue6414";
            s3.createBucket(CreateBucketRequest.builder().bucket(bucketName).build());

            s3.putObject(PutObjectRequest.builder()
                    .bucket(bucketName)
                    .key("test-dir/file1.txt")
                    .build(), software.amazon.awssdk.core.sync.RequestBody.fromString("Content 1"));

            s3.putObject(PutObjectRequest.builder()
                    .bucket(bucketName)
                    .key("test-dir/file2.txt")
                    .build(), software.amazon.awssdk.core.sync.RequestBody.fromString("Content 2"));

            Path downloadPath = Paths.get("downloaded-files");
            DirectoryDownload directoryDownload = transferManager.downloadDirectory(
                    DownloadDirectoryRequest.builder()
                            .destination(downloadPath)
                            .bucket(bucketName)
                            .listObjectsV2RequestTransformer(req -> req.prefix("test-dir/"))
                            .build());

            directoryDownload.completionFuture().join();
            System.out.println("Directory download completed successfully");

        } finally {
            localstack.stop();
        }
    }

    public static S3CrtProxyConfiguration getProxyConfigurationWithLocalStack() {
        String proxyHost = System.getProperty("https.proxyHost");
        System.out.println("Retrieved proxyHost System property: "+proxyHost);
        if (StringUtils.isEmpty(proxyHost)) {
            return null;
        }

        String proxyPortStr = System.getProperty("https.proxyPort");
        System.out.println("Retrieved proxyPort System property: "+proxyPortStr);
        int proxyPort = 8080;
        if (!StringUtils.isEmpty(proxyPortStr)) {
            try {
                proxyPort = Integer.parseInt(proxyPortStr.trim());
            } catch (NumberFormatException e) {
                proxyPort = 8080;
            }
        }

        return (S3CrtProxyConfiguration) S3CrtProxyConfiguration.builder()
                .host(proxyHost.trim())
                .port(proxyPort)
                .nonProxyHosts(getNonProxyHosts())
                .build();
    }

    public static S3CrtProxyConfiguration getProxyConfiguration() {
        String proxyHost = System.getProperty("https.proxyHost");

        if (StringUtils.isEmpty(proxyHost)) {
            return null;
        }

        String proxyPortStr = System.getProperty("https.proxyPort");
        int proxyPort = 8080;

        if (!StringUtils.isEmpty(proxyPortStr)) {
            try {
                proxyPort = Integer.parseInt(proxyPortStr.trim());
            } catch (NumberFormatException e) {
                proxyPort = 8080;
            }
        }

        return (S3CrtProxyConfiguration) S3CrtProxyConfiguration.builder()
                .host(proxyHost.trim())
                .port(proxyPort)
                .nonProxyHosts(getNonProxyHosts())
                //.nonProxyHosts(Set.of("localhost"))
                .build();
    }

    public static Set<String> getNonProxyHosts() {
        Set<String> result = new HashSet<>();

        String nonProxyHosts = System.getProperty("http.nonProxyHosts");
        System.out.println("pre-trimmed host is: "+nonProxyHosts);
        if (StringUtils.isEmpty(nonProxyHosts)) {
            return result;
        }

        String[] hosts = nonProxyHosts.split("\\|");
        for (String host : hosts) {
            String trimmedHost = host.trim();
            result.add(trimmedHost);
        }
        System.out.println("post-trimmed host is: "+result);
        return result;
    }
}

When you get a chance can you validate with the latest SDK version and let us know if the issue persists?

Regards,
Chaitanya

[UnirCs]

Hi @bhoradc , acknowledged . I test with Java SDK v2.37.0 and aws-crt v0.39.4 and now it is working!

Proof:

[INFO] [2025-10-31T08:53:38Z] [000000016f5cf000] [connection-manager] - id=0x1053c9b20: Requesting 1 new connections from http
[DEBUG] [2025-10-31T08:53:38Z] [000000016f5cf000] [http-connection] - Host "127.0.0.1" found in NO_PROXY, bypassing proxy
[TRACE] [2025-10-31T08:53:38Z] [000000016f5cf000] [http-connection] - static: attempting to initialize a new client channel to 127.0.0.1:55683
[TRACE] [2025-10-31T08:53:38Z] [000000016f5cf000] [channel-bootstrap] - id=0x600002566fb0: attempting to initialize a new client channel to 127.0.0.1:55683
[DEBUG] [2025-10-31T08:53:38Z] [000000016f5cf000] [channel-bootstrap] - id=0x600002566fb0: acquiring bootstrap reference
[DEBUG] [2025-10-31T08:53:38Z] [000000016f5cf000] [dns] - id=0x6000039b4630: Host resolution requested for 127.0.0.1
[DEBUG] [2025-10-31T08:53:38Z] [000000016f5cf000] [dns] - id=0x6000039b4630: cached entries found for 127.0.0.1 returning to caller.
[TRACE] [2025-10-31T08:53:38Z] [000000016f5cf000] [dns] - id=0x6000039b4630: vending address 127.0.0.1 for host 127.0.0.1 to caller
[TRACE] [2025-10-31T08:53:38Z] [000000016f5cf000] [channel-bootstrap] - id=0x600002566fb0: dns resolution completed. Kicking off connections on 1 addresses. First one back wins.

Thanks for the quick answer and the support!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.