Skip to content

Commit 1452e67

Browse files
jchrostek-ddclaude
andcommitted
fix(http): disable connection pooling to prevent stale connections in Lambda
## Problem After upgrading from extension v92 to v93, customers reported a sharp increase in "Max retries exceeded, returning request error" errors (SVLS-8672, GitHub issue #1092). ## Root Cause PR #1018 introduced HTTP client caching for performance improvements. However, the cached client maintains a connection pool that doesn't work well with Lambda's freeze/resume execution model: 1. Lambda executes, HTTP client created with connection pool 2. Extension flushes traces, connections remain open in pool 3. Lambda freezes (paused between invocations - seconds to minutes) 4. Lambda resumes, cached client reuses stale connections 5. TCP errors → "Max retries exceeded" In v92, a new HTTP client was created per-flush, so there were never stale connections to reuse. ## Solution Disable connection pooling by setting `pool_max_idle_per_host(0)`. This ensures each request gets a fresh connection, avoiding stale connection issues while still benefiting from client caching. This matches the pattern used in libdatadog's `new_client_periodic()` which explicitly disables pooling with the comment: "This client does not keep connections because otherwise we would get a pipe closed every second connection because of low keep alive in the agent." Fixes: SVLS-8672 Fixes: #1092 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 06947f1 commit 1452e67

1 file changed

Lines changed: 13 additions & 2 deletions

File tree

bottlecap/src/traces/http_client.rs

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,14 +173,25 @@ pub fn create_client(
173173
let proxy =
174174
hyper_http_proxy::Proxy::new(hyper_http_proxy::Intercept::Https, proxy.parse()?);
175175
let proxy_connector = hyper_http_proxy::ProxyConnector::from_proxy(connector, proxy)?;
176-
let client = http_common::client_builder().build(proxy_connector);
176+
// Disable connection pooling to avoid stale connections after Lambda freeze/resume.
177+
// In Lambda, the execution environment can be frozen for seconds to minutes between
178+
// invocations. Pooled connections become stale during this time, causing failures
179+
// when reused. Setting pool_max_idle_per_host(0) ensures each request gets a fresh
180+
// connection, matching the pattern used in libdatadog's new_client_periodic().
181+
let client = http_common::client_builder()
182+
.pool_max_idle_per_host(0)
183+
.build(proxy_connector);
177184
debug!(
178185
"HTTP_CLIENT | Proxy connector created with proxy: {:?}",
179186
proxy_https
180187
);
181188
Ok(client)
182189
} else {
183190
let proxy_connector = hyper_http_proxy::ProxyConnector::new(connector)?;
184-
Ok(http_common::client_builder().build(proxy_connector))
191+
// Disable connection pooling to avoid stale connections after Lambda freeze/resume.
192+
// See comment above for detailed explanation.
193+
Ok(http_common::client_builder()
194+
.pool_max_idle_per_host(0)
195+
.build(proxy_connector))
185196
}
186197
}

0 commit comments

Comments
 (0)