feat(loadtest): add USE_SYNC to benchmark the blocking client

[jrvb-rl]

Summary

Adds USE_SYNC to loadtest/loadtest.py so the harness can benchmark the blocking Runloop client (driven by a ThreadPoolExecutor) in addition to the async AsyncRunloop client. Works with the existing USE_HTTP2 toggle, so the harness now covers async/sync × HTTP/1.1/HTTP/2.

The TS SDK has no synchronous client (Node has no blocking HTTP client), so USE_SYNC is Python-only; the TS harness stays async and appears as N/A in the sync rows below.

Sync concurrency model

The blocking client can't use asyncio.gather, so a burst is driven by a thread pool capped at workers = min(REQUEST_COUNT, CONCURRENCY).

CONCURRENCY defaults to 500. The blocking client is GIL-bound, so more worker threads add contention without raising throughput. Measured at 20 000 HTTP/1.1 requests against dev, a 500-worker pool beat a 5000-worker pool — 260 vs 199 req/s with ~10× lower p50 latency (1.6 s vs 16.9 s). Rationale is documented in the module docstring and README.

Test setup

• Target: dev cluster (api.runloop.pro), all requests hit the nonexistent blueprint → HTTP 400, no devboxes created.
• Host FD limit: 4096 (hard, unraisable without root) — relevant to the HTTP/1.1 @ 20 000 rows.
• ok = HTTP 400 (request reached the server); net_err = transport failure (surfaced via the new error breakdown).
• Throughput counts every request including fast failures, so read it alongside the ok/err split (notably for sync + HTTP/2).

Results — Python loadtest.py (async vs sync)

Sync rows use the shipped default CONCURRENCY=500 (worker count = min(REQUEST_COUNT, 500)).

HTTP/2

N	mode	throughput	p50	p99	ok / net_err
200	async	240.7 req/s	631 ms	730 ms	200 / 0
200	sync	428.5 req/s	273 ms	444 ms	7 / 193
2000	async	306.7 req/s	4343 ms	5732 ms	2000 / 0
2000	sync	413.1 req/s	977 ms	1887 ms	507 / 1493
20000	async	104.0 req/s	153449 ms	182824 ms	20000 / 0
20000	sync	404.7 req/s	823 ms	8448 ms	6968 / 13032

HTTP/1.1

N	mode	throughput	p50	p99	ok / net_err
200	async	166.5 req/s	832 ms	1095 ms	200 / 0
200	sync	99.7 req/s	894 ms	1504 ms	200 / 0
2000	async	103.1 req/s	13350 ms	18614 ms	2000 / 0
2000	sync	183.3 req/s	1906 ms	6561 ms	2000 / 0
20000	async	—	—	—	did not converge (FD-bound; killed >9 min)
20000	sync	260.1 req/s	1631 ms	5629 ms	20000 / 0

Results — TS loadtest.ts (async only)

N	HTTP/2	HTTP/1.1
200	575.9 req/s, p50 298 ms, 200/0	262.6 req/s, p50 684 ms, 200/0
2000	2079.6 req/s, p50 745 ms, 2000/0	123.8 req/s, p50 6513 ms, 2000/0
20000	2816.5 req/s, p50 4520 ms, 20000/0	149.9 req/s, p50 84586 ms, 15378 / 4622

Findings

1. TS native HTTP/2 scales up; Python httpx HTTP/2 scales down. TS throughput rises with load (576 → 2080 → 2817 req/s at 200/2000/20000). Python async HTTP/2 falls (241 → 307 → 104 req/s) and p50 balloons to 153 s at 20 000 — the pure-Python h2/httpcore stack is CPU/GIL-bound, not server-bound. At 20 000, TS HTTP/2 is ~27× Python's.
2. Python sync + HTTP/2 is unsafe under a thread burst. Many threads hitting the shared httpx HTTP/2 pool before its connection is established cause a thundering herd — 65–96 % of requests fail with Connection error at every scale (even 200 threads). The high "throughput" there is fast failures, not work done. Sync + HTTP/1.1 is unaffected (socket-per-request; nothing to race on), and async + HTTP/2 is fine (the event loop coordinates waiters on the pending connection). Recommendation: use the async client for HTTP/2, or HTTP/1.1 if you need the sync client concurrently.
3. The sync client is GIL-bound (~100–260 req/s). Throughput barely moves with worker count and fewer threads win — hence the 500 default.
4. HTTP/1.1 doesn't scale to 20 000 on a 4096-FD host. Async httpx (max_connections=None) fires 20 000 simultaneous connects and never converges; the sync thread-ramp self-throttles and completes; TS completes but sheds 4622 requests to FD exhaustion. This is the FD-exhaustion case the harness already warns about.

Also included

• Documented USE_SYNC / CONCURRENCY in loadtest/README.md.

---

🤖 Generated with Claude Code

Reflex session: https://reflex.runloop.ai/orgs/runloop/agents/agt_4Sg729tM9zLk7ayz20XNib

[reflex-loop]

✅ Reflex agent status: Completed

The agent completed its work.

This PR was created by Reflex.

View the agent run →

_{This comment updates in place as the agent works.}