Skip to content

Commit d906255

Browse files
jchrostek-ddclaudeduncanista
authored
fix(http): disable connection pooling to prevent stale connections in Lambda (#1094)
## Summary Fixes a regression introduced in v93 where customers see a sharp increase in "Max retries exceeded, returning request error" errors after upgrading from v92. - Disables HTTP connection pooling for the trace/stats flusher by setting `pool_max_idle_per_host(0)` - Prevents stale connections from being reused after Lambda freeze/resume cycles ## Problem PR #1018 introduced HTTP client caching for performance improvements. However, the cached client maintains a connection pool that doesn't work well with Lambda's freeze/resume execution model: 1. Lambda executes, HTTP client created with connection pool 2. Extension flushes traces, connections remain open in pool 3. Lambda **freezes** (paused between invocations - can be seconds to minutes) 4. Lambda **resumes**, cached client reuses stale connections 5. TCP errors → "Max retries exceeded" In v92, a new HTTP client was created per-flush, so there were never stale connections to reuse. ## Solution Disable connection pooling by setting `pool_max_idle_per_host(0)`. This ensures each request gets a fresh connection, avoiding stale connection issues while still benefiting from client caching (TLS session reuse, configuration reuse, etc.). This matches the pattern used in libdatadog's `new_client_periodic()` which explicitly disables pooling with the comment: > "This client does not keep connections because otherwise we would get a pipe closed every second connection because of low keep alive in the agent." ## Related - Fixes [SVLS-8672](https://datadoghq.atlassian.net/browse/SVLS-8672) - Fixes #1092 - Regression introduced in #1018 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: jordan gonzález <30836115+duncanista@users.noreply.github.com>
1 parent bacfad2 commit d906255

File tree

1 file changed

+13
-2
lines changed

1 file changed

+13
-2
lines changed

bottlecap/src/traces/http_client.rs

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,14 +173,25 @@ pub fn create_client(
173173
let proxy =
174174
hyper_http_proxy::Proxy::new(hyper_http_proxy::Intercept::Https, proxy.parse()?);
175175
let proxy_connector = hyper_http_proxy::ProxyConnector::from_proxy(connector, proxy)?;
176-
let client = http_common::client_builder().build(proxy_connector);
176+
// Disable connection pooling to avoid stale connections after Lambda freeze/resume.
177+
// In Lambda, the execution environment can be frozen for seconds to minutes between
178+
// invocations. Pooled connections become stale during this time, causing failures
179+
// when reused. Setting pool_max_idle_per_host(0) ensures each request gets a fresh
180+
// connection, matching the pattern used in libdatadog's new_client_periodic().
181+
let client = http_common::client_builder()
182+
.pool_max_idle_per_host(0)
183+
.build(proxy_connector);
177184
debug!(
178185
"HTTP_CLIENT | Proxy connector created with proxy: {:?}",
179186
proxy_https
180187
);
181188
Ok(client)
182189
} else {
183190
let proxy_connector = hyper_http_proxy::ProxyConnector::new(connector)?;
184-
Ok(http_common::client_builder().build(proxy_connector))
191+
// Disable connection pooling to avoid stale connections after Lambda freeze/resume.
192+
// See comment above for detailed explanation.
193+
Ok(http_common::client_builder()
194+
.pool_max_idle_per_host(0)
195+
.build(proxy_connector))
185196
}
186197
}

0 commit comments

Comments
 (0)