Add a workaround for connection acquire issue (#6808)

* Add a workaround for connection acquire issue

There appears to be an issue in rare circumstances where a connection
acquire fails so that no connection is leased from the pool, and
validation of the connection subsequently fails, causing the underlying
HTTP client to throw.

This adds a workaround for this issue by rethrowing the exception
wrapped in an IOException to at least give the upper layer an
opportunity to retry the request.

* Add changelog

Author: dagnir

.changes/next-release/bugfix-Apache5HTTPClient-2866d44.json

@@ -0,0 +1,6 @@
+{
+    "type": "bugfix",
+    "category": "Apache 5 HTTP Client",
+    "contributor": "",
+    "description": "Fixed an issue in the Apache 5 HTTP client where requests could fail with `\"Endpoint not acquired / already released\"`. These failures are now converted to retryable I/O errors."
+}

http-clients/apache5-client/src/main/java/software/amazon/awssdk/http/apache5/Apache5HttpClient.java

@@ -291,6 +291,15 @@ private HttpExecuteResponse execute(HttpUriRequestBase apacheRequest, MetricColl
             HttpHost target = determineTarget(apacheRequest);
             ClassicHttpResponse httpResponse = httpClient.executeOpen(target, apacheRequest, localRequestContext);
             return createResponse(httpResponse, apacheRequest);
+        } catch (IllegalStateException e) {
+            // TODO: remove this when a permanent fix is made upstream
+            // This is a workaround for a race condition where a connection is not properly acquired when httpClient attempts
+            // to execute a request on a connection from the pool. For now, we rethrow this as an IOException so upper layers
+            // have a chance to retry if possible
+            if ("Endpoint not acquired / already released".equals(e.getMessage())) {
+                throw new IOException("Failed to execute HTTP request", e);
+            }
+            throw e;
         } finally {
             THREAD_LOCAL_REQUEST_METRIC_COLLECTOR.remove();
         }