feat: add HTTP/2 load testing infrastructure

[yoon-park-rl]

Note: pair pr

Summary

A loadtest/ toolkit for benchmarking the SDK's request handling, plus raw-transport baselines. The SDK harness (loadtest.py) fires a burst of devboxes.create calls at a deliberately nonexistent blueprint (bp_nonexistent_loadtest_00000): every request fails fast server-side (HTTP 400, no devboxes created), isolating client + server request handling from provisioning.

Scripts (real-API)

Script	Transport under test
loadtest.py	The SDK (AsyncRunloop). USE_HTTP2=0 → HTTP/1.1; default → HTTP/2 (shared httpx pool). Always runs against the installed package in this checkout.
h2_test.py	Raw httpx HTTP/2, bypassing the SDK. Configurable connection count.
h2_single_conn.py	Raw httpx HTTP/2 on a single warmed connection (50-request burst).
raw_fetch_test.py	Raw httpx HTTP/1.1 keep-alive baseline.
alpn_check.py	Confirms the origin negotiates h2 via TLS ALPN.

Usage

# SDK: HTTP/2 (default) vs HTTP/1.1
source ~/env && REQUEST_COUNT=2000 uv run python loadtest/loadtest.py            # HTTP/2
source ~/env && REQUEST_COUNT=2000 USE_HTTP2=0 uv run python loadtest/loadtest.py # HTTP/1.1

# Raw httpx HTTP/2 vs HTTP/1.1 comparison
source ~/env && uv run python loadtest/h2_test.py
source ~/env && uv run python loadtest/raw_fetch_test.py

Test plan

• loadtest.py runs for both transports; all requests 400 (no devboxes created).
• HTTP/1.1 FD exhaustion warning fires correctly when limit is low.

Python vs TS comparison

Ran loadtest.py against the pair PR's loadtest.ts on the dev cluster (api.runloop.pro), 2000 requests each, all 400 by design (no devboxes created):

Mode	SDK	Throughput	Wall	p50	p99
HTTP/2	TS	2021 req/s	0.99s	752ms	961ms
HTTP/2	Python	287 req/s	6.96s	4592ms	6153ms
HTTP/1.1	TS	123 req/s	16.2s	6600ms	12171ms
HTTP/1.1	Python	99 req/s	20.2s	13739ms	19340ms

HTTP/1.1 is comparable (both bottlenecked on socket-per-request + the 4096 FD cap). On HTTP/2, Python is ~7× slower than TS against the same server. The raw h2_test.py probe isolates the cause: raw httpx-h2 plateaus at ~485–560 req/s and going from 10→50 connections barely moves it — so the ceiling is httpx/httpcore's pure-Python h2 state machine (CPU-bound under the GIL), not the SDK's max_connections=20 pool. TS's native node:http2 (C++) has no such ceiling. This is inherent to the transport stack, not a loadtest bug.

Also in this revision

• loadtest.py now captures a per-request error message (parity with the TS RequestResult) and prints a breakdown of the network_error bucket, so transport-level failures (timeouts, resets, pool exhaustion) are diagnosable.

---

🤖 Generated with Claude Code

Reflex session: https://reflex.runloop.ai/orgs/runloop/agents/agt_4Sg729tM9zLk7ayz20XNib

[jrvb-rl]

Thanks for putting this together, Yoon. I added one bit to bring the error reporting in line with the TS loadtest. I also had claude run a comparison and add the results to the PR description.

I'll land, so I can add sync SDK tests onto this.