feat(runtime): make connection pooling configurable

[dexhunter]

Issue Reference

• This PR fixes/closes [feat] Make Connection Pooling configurable #3072

Changes

Adds an explicit http.enable_connection_pooling option for regular HTTP-backed clients.

client<llm> MyClient {
  provider openai
  options {
    model "gpt-4o"
    http {
      enable_connection_pooling true
    }
  }
}

• Pooling remains disabled when the option is omitted or set to false, preserving the current fork-safe default.
• AWS clients reject the option because their SDK-managed transport cannot use this runtime setting.
• Documentation covers asyncio/thread/spawn use, the fork() caveat, and unsupported client types.

In a deterministic local HTTP/1.1 harness, ten sequential requests produced:

Configuration	New TCP connections
Default / false	10
true	1

This is a 90% reduction in new connections for repeated calls while leaving existing behavior unchanged by default.

Testing

• Unit tests added/updated
• Manual testing performed
• Tested with deterministic local HTTP/1.1 connection counting

Validation performed:

• cargo test -p internal-llm-client
• cargo test -p baml-runtime
• HTTP configuration validation fixtures, including invalid type and AWS rejection
• package formatting and Clippy checks
• pooling connection-count test repeated 20 times
• strict metric harness repeated three times, each reporting one new connection for ten requests

The metric harness also verifies response payloads, streaming, retries, cancellation, concurrency, forced connection close, recovery, and a fresh forked child process.

Screenshots

Not applicable.

PR Checklist

• I have read and followed the contributing guidelines
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where the behavior is not self-explanatory
• I have made corresponding changes to the documentation
• My changes generate no new warnings

Additional Notes

Applications using Python's fork() start method should create the BAML runtime in each child process instead of inheriting a pooled client from the parent. The new option is disabled by default for this reason.

Summary by CodeRabbit

• New Features

  • Added optional HTTP connection pooling for supported BAML clients via http.enable_connection_pooling.
  • Pooling is disabled by default and reduces repeated TCP/TLS handshakes when enabled.
  • Validation prevents unsupported pooling for fallback/round-robin and AWS Bedrock clients.

• Documentation

  • Added a new “Connection Pooling” reference page and included it in navigation.

• Bug Fixes / Validation

  • Improved diagnostics for invalid http configuration (e.g., wrong types and unsupported fields like total_timeout_ms).

• Tests

  • Added runtime tests to verify connection reuse behavior.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
promptfiddle	Skipped		Jul 10, 2026 11:22pm

[vercel]

@dexhunter is attempting to deploy a commit to the Boundary Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 7d7c37cb-5806-4aed-92f2-754a012ac2c9

📥 Commits

Reviewing files that changed from the base of the PR and between b6265f2 and 7c4a2fd.

📒 Files selected for processing (8)

• engine/baml-lib/baml/tests/validation_files/client/http_config_aws_pooling.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_invalid_pooling.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_regular_with_total.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_valid.baml
• engine/baml-lib/llm-client/src/clients/helpers.rs
• engine/baml-runtime/src/request/mod.rs
• fern/03-reference/baml/clients/connection-pooling.mdx
• fern/docs.yml

✅ Files skipped from review due to trivial changes (3)

• engine/baml-lib/baml/tests/validation_files/client/http_config_invalid_pooling.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_regular_with_total.baml
• fern/03-reference/baml/clients/connection-pooling.mdx

🚧 Files skipped from review as they are similar to previous changes (5)

• engine/baml-lib/baml/tests/validation_files/client/http_config_aws_pooling.baml
• fern/docs.yml
• engine/baml-lib/baml/tests/validation_files/client/http_config_valid.baml
• engine/baml-lib/llm-client/src/clients/helpers.rs
• engine/baml-runtime/src/request/mod.rs

---

📝 Walkthrough

Walkthrough

Adds opt-in HTTP connection pooling for supported BAML clients, provider-aware validation, runtime connection reuse tests, and documentation covering configuration and limitations.

Changes

HTTP Connection Pooling

Layer / File(s)	Summary
Pooling configuration and validation engine/baml-lib/llm-client/src/clients/helpers.rs, engine/baml-lib/baml/tests/validation_files/client/*	Adds enable_connection_pooling, validates its boolean type and provider support, improves HTTP-field diagnostics, and updates valid and invalid fixtures.
Runtime pooling behavior and tests engine/baml-runtime/src/request/mod.rs	Conditionally disables reqwest pooling and tests connection counts for enabled versus disabled pooling.
Connection pooling documentation fern/03-reference/baml/clients/connection-pooling.mdx, fern/docs.yml	Documents defaults, configuration, fork-related constraints, unsupported clients, and navigation.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BAMLClient
  participant create_http_client
  participant ReqwestTCPServer
  BAMLClient->>create_http_client: Configure enable_connection_pooling
  create_http_client->>ReqwestTCPServer: Send requests
  ReqwestTCPServer-->>create_http_client: Return HTTP responses
  create_http_client-->>BAMLClient: Reuse or recreate connections

Loading

Poem

A rabbit hops where fresh links spring,
Reusing paths for every ping.
A bool now guides the tunnel’s flow,
While tests count streams below.
Docs bloom softly, clear and bright—
Pooling dances through the night.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: configurable connection pooling.
Linked Issues check	✅ Passed	The changes add an opt-in HTTP pooling flag, keep it disabled by default, and document unsupported AWS/fork cases, matching #3072.
Out of Scope Changes check	✅ Passed	The added tests and docs support the pooling feature, and no unrelated code changes are apparent from the PR summary.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

fern/03-reference/baml/clients/connection-pooling.mdx (1)

34-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add links to related documentation pages.

The coding guidelines state that documentation should include well-designed links to guide readers to relevant pages. This page mentions fallback, round-robin, and AWS Bedrock clients but does not link to their respective documentation pages.

🔗 Suggested links

-Connection pooling is not available for fallback or round-robin clients. Set it
-on their leaf clients instead. It is also not available for AWS Bedrock clients,
-whose HTTP transport is managed by the AWS SDK.
+Connection pooling is not available for [fallback](/reference/baml/clients/strategy/fallback) or
+[round-robin](/reference/baml/clients/strategy/round-robin) clients. Set it on their leaf
+clients instead. It is also not available for [AWS Bedrock](/reference/baml/clients/providers/aws-bedrock)
+clients, whose HTTP transport is managed by the AWS SDK.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@fern/03-reference/baml/clients/connection-pooling.mdx` around lines 34 - 36,
Add inline links to the documentation pages for fallback clients, round-robin
clients, and AWS Bedrock clients in the connection-pooling limitation paragraph.
Preserve the existing wording while linking each referenced client type to its
corresponding documentation page.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@fern/03-reference/baml/clients/connection-pooling.mdx`:
- Around line 1-3: Add the required subtitle field to the frontmatter of the
“Connection Pooling” document, preserving the existing title and matching the
specified frontmatter format used by other fern/**/*.mdx files.

---

Nitpick comments:
In `@fern/03-reference/baml/clients/connection-pooling.mdx`:
- Around line 34-36: Add inline links to the documentation pages for fallback
clients, round-robin clients, and AWS Bedrock clients in the connection-pooling
limitation paragraph. Preserve the existing wording while linking each
referenced client type to its corresponding documentation page.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e4f337d3-f5ba-4fe0-853d-918adcd9d41b

📥 Commits

Reviewing files that changed from the base of the PR and between 8349784 and b6265f2.

📒 Files selected for processing (8)

• engine/baml-lib/baml/tests/validation_files/client/http_config_aws_pooling.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_invalid_pooling.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_regular_with_total.baml
• engine/baml-lib/baml/tests/validation_files/client/http_config_valid.baml
• engine/baml-lib/llm-client/src/clients/helpers.rs
• engine/baml-runtime/src/request/mod.rs
• fern/03-reference/baml/clients/connection-pooling.mdx
• fern/docs.yml

[aaronvg]

are you using AWS provider with baml?

[dexhunter]

No, I am not using AWS Bedrock for this path. The target is the regular HTTP-based providers. I kept AWS explicitly unsupported because its HTTP transport is owned by the AWS SDK, so exposing this reqwest pooling option there would be misleading. The motivation is #3072 plus the reproduced connection-count result: 10 requests use 10 connections by default and 1 connection when pooling is explicitly enabled. I have also rebased onto current canary and addressed the documentation review.