Skip to content

Commit 8602fbd

Browse files
Reflexclaude
Reflex
and
claude
committed
test(loadtest): run against in-repo SDK and match HTTP/2-as-default
Update the SDK load harness for the new default transport: - `loadtest.ts` imports `../src/sdk.ts` (like the sibling loadtest scripts) so it always exercises the SDK code in this checkout, not a published build. - Set `http2` explicitly: HTTP/2 is now the default, so HTTP/1.1 must be opted into with `http2: false`. `USE_HTTP2` defaults to on; `USE_HTTP2=0` benchmarks HTTP/1.1. Relabel the HTTP/2 mode as native `node:http2`. - Drop the `@runloop/api-client` file dependency and `type: module` from `loadtest/package.json` (the latter would force the shared loadtest dir to ESM and break `push-to-loki.ts`); keep `undici` so the raw undici-vs-native comparison still runs — that comparison motivated the native H2 transport. - Document the real-API comparison scripts (SDK + raw undici/node:http2/https) in the loadtest README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent fb04677 commit 8602fbd

3 files changed

Lines changed: 68 additions & 41 deletions

File tree

‎loadtest/README.md‎

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,47 @@ duration. Not part of `npm test`; run on demand.
66

77
## Scripts
88

9-
| Script | Purpose |
10-
|---|---|
11-
| `h2-multiplex.ts` | Fire N concurrent requests, report throughput + p50/p95/p99. |
12-
| `h2-pool-growth.ts` | Ramp 1 → 50 → 200 → 50 → 1 concurrent. Observe pool size over time. |
13-
| `h2-leak.ts` | Soak at 50 r/s. Sample heap + FD count. Periodic GOAWAY injection. |
14-
| `h2-chaos.ts` | Server randomly drops sockets / RSTs / GOAWAYs / delays. Asserts ≥50 % GET success. |
15-
| `h2load.sh` | Wraps `nghttp2`'s `h2load` against a long-running test server. |
16-
| `sse-test.ts` | Pre-existing manual SSE round-trip. |
17-
| `push-to-loki.ts` | Runs all four timed tests and pushes results to Loki for Grafana. |
9+
| Script | Purpose |
10+
| ------------------- | ----------------------------------------------------------------------------------- |
11+
| `h2-multiplex.ts` | Fire N concurrent requests, report throughput + p50/p95/p99. |
12+
| `h2-pool-growth.ts` | Ramp 1 → 50 → 200 → 50 → 1 concurrent. Observe pool size over time. |
13+
| `h2-leak.ts` | Soak at 50 r/s. Sample heap + FD count. Periodic GOAWAY injection. |
14+
| `h2-chaos.ts` | Server randomly drops sockets / RSTs / GOAWAYs / delays. Asserts ≥50 % GET success. |
15+
| `h2load.sh` | Wraps `nghttp2`'s `h2load` against a long-running test server. |
16+
| `sse-test.ts` | Pre-existing manual SSE round-trip. |
17+
| `push-to-loki.ts` | Runs all four timed tests and pushes results to Loki for Grafana. |
18+
19+
## End-to-end transport comparison (real API)
20+
21+
Unlike the in-process scripts above, these hit a real Runloop API (`RUNLOOP_API_KEY`,
22+
optional `RUNLOOP_BASE_URL`). They send a burst of `devboxes.create` calls against a
23+
**deliberately nonexistent blueprint** (`bp_nonexistent_loadtest_00000`), so every
24+
request fails fast server-side (HTTP `400`) and **no devboxes are created** — isolating
25+
client + server _request handling_ from provisioning.
26+
27+
| Script | Transport under test |
28+
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
29+
| `loadtest.ts` | The **SDK** itself. `USE_HTTP2=0` → HTTP/1.1 (`node-fetch`); default / `USE_HTTP2=1` → HTTP/2 (native `node:http2`). Imports `../src/sdk.ts`, so it always runs against the SDK code in this checkout. |
30+
| `undici-test.ts`, `undici-single-conn.ts`, `undici-debug.ts` | Raw [undici](https://github.com/nodejs/undici) HTTP/2 pool, bypassing the SDK. |
31+
| `h2-test.ts`, `h2-single-conn.ts` | Raw `node:http2`, bypassing the SDK. |
32+
| `raw-fetch-test.ts` | Raw `node:https` (HTTP/1.1 keep-alive agent). |
33+
| `alpn-check.ts` | Confirms the origin negotiates `h2` via TLS ALPN. |
34+
35+
The raw-transport probes compare undici's HTTP/2 multiplexing against `node:http2`
36+
(and HTTP/1.1) directly — the comparison that motivated building the SDK's own
37+
`node:http2` transport. They're kept so that comparison stays reproducible.
38+
39+
```sh
40+
# SDK: HTTP/2 (default) vs HTTP/1.1, 2000-request burst
41+
source ~/env && REQUEST_COUNT=2000 npx tsx loadtest/loadtest.ts # HTTP/2
42+
source ~/env && REQUEST_COUNT=2000 USE_HTTP2=0 npx tsx loadtest/loadtest.ts # HTTP/1.1
43+
44+
# Raw undici vs node:http2 comparison (undici comes from loadtest/package.json)
45+
cd loadtest && npm install && source ~/env && npx tsx undici-test.ts
46+
```
47+
48+
HTTP/1.1 opens a socket per in-flight request; for large bursts raise the file-descriptor
49+
limit (`ulimit -n 65536`) or keep `REQUEST_COUNT` under it.
1850

1951
## Automated daily runs
2052

‎loadtest/loadtest.ts‎

Lines changed: 27 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
1-
import { Runloop } from "@runloop/api-client";
1+
// Import the in-repo SDK source directly (run via `npx tsx`) so the benchmark
2+
// always exercises the exact code in this checkout — not a published build.
3+
import { Runloop } from '../src/sdk.ts';
24

3-
const REQUEST_COUNT = parseInt(process.env.REQUEST_COUNT ?? "100000", 10);
5+
const REQUEST_COUNT = parseInt(process.env.REQUEST_COUNT ?? '100000', 10);
46
const RUNLOOP_BASE_URL = process.env.RUNLOOP_BASE_URL;
5-
const USE_HTTP2 = process.env.USE_HTTP2 === "1";
7+
// HTTP/2 is the SDK's default transport; set USE_HTTP2=0 to benchmark HTTP/1.1.
8+
const USE_HTTP2 = (process.env.USE_HTTP2 ?? '1') === '1';
69
const PROGRESS_INTERVAL_MS = 2000;
710

811
function buildClient(): Runloop {
@@ -13,9 +16,9 @@ function buildClient(): Runloop {
1316
if (RUNLOOP_BASE_URL) {
1417
opts.baseURL = RUNLOOP_BASE_URL;
1518
}
16-
if (USE_HTTP2) {
17-
(opts as any).http2 = true;
18-
}
19+
// Set explicitly: since HTTP/2 is the default, HTTP/1.1 must be opted into
20+
// with `http2: false` (omitting it would now select HTTP/2).
21+
(opts as any).http2 = USE_HTTP2;
1922
return new Runloop(opts);
2023
}
2124

@@ -26,26 +29,22 @@ interface RequestResult {
2629
error: string | null;
2730
}
2831

29-
async function sendRequest(
30-
client: Runloop,
31-
index: number,
32-
runId: string,
33-
): Promise<RequestResult> {
32+
async function sendRequest(client: Runloop, index: number, runId: string): Promise<RequestResult> {
3433
const start = performance.now();
3534
try {
3635
await client.devboxes.create({
37-
blueprint_id: "bp_nonexistent_loadtest_00000",
36+
blueprint_id: 'bp_nonexistent_loadtest_00000',
3837
name: `loadtest-${runId}-${index}`,
3938
environment_variables: {
40-
TEST_VAR_1: "value_one",
41-
TEST_VAR_2: "value_two",
39+
TEST_VAR_1: 'value_one',
40+
TEST_VAR_2: 'value_two',
4241
},
4342
metadata: {
4443
test_run: runId,
4544
index: String(index),
4645
},
4746
launch_parameters: {
48-
resource_size_request: "SMALL",
47+
resource_size_request: 'SMALL',
4948
keep_alive_time_seconds: 300,
5049
},
5150
});
@@ -75,26 +74,24 @@ function printMetrics(results: RequestResult[], wallClockMs: number): void {
7574

7675
const statusCounts = new Map<string, number>();
7776
for (const r of results) {
78-
const key = r.status != null ? String(r.status) : "network_error";
77+
const key = r.status != null ? String(r.status) : 'network_error';
7978
statusCounts.set(key, (statusCounts.get(key) ?? 0) + 1);
8079
}
8180

82-
console.log("\n=== Load Test Results ===");
81+
console.log('\n=== Load Test Results ===');
8382
console.log(`Requests: ${results.length}`);
8483
console.log(`Wall clock: ${(wallClockMs / 1000).toFixed(2)}s`);
85-
console.log(
86-
`Throughput: ${(results.length / (wallClockMs / 1000)).toFixed(1)} req/s`,
87-
);
88-
console.log("");
89-
console.log("Latency (ms):");
84+
console.log(`Throughput: ${(results.length / (wallClockMs / 1000)).toFixed(1)} req/s`);
85+
console.log('');
86+
console.log('Latency (ms):');
9087
console.log(` min: ${latencies[0].toFixed(1)}`);
9188
console.log(` p50: ${percentile(latencies, 50).toFixed(1)}`);
9289
console.log(` p90: ${percentile(latencies, 90).toFixed(1)}`);
9390
console.log(` p95: ${percentile(latencies, 95).toFixed(1)}`);
9491
console.log(` p99: ${percentile(latencies, 99).toFixed(1)}`);
9592
console.log(` max: ${latencies[latencies.length - 1].toFixed(1)}`);
96-
console.log("");
97-
console.log("Status codes:");
93+
console.log('');
94+
console.log('Status codes:');
9895
for (const [status, count] of [...statusCounts.entries()].sort()) {
9996
console.log(` ${status}: ${count}`);
10097
}
@@ -115,10 +112,10 @@ async function main(): Promise<void> {
115112

116113
console.log(`Starting load test: ${REQUEST_COUNT} concurrent requests`);
117114
console.log(`Run ID: ${runId}`);
118-
console.log(`HTTP mode: ${USE_HTTP2 ? "HTTP/2 (undici)" : "HTTP/1.1 (node-fetch)"}`);
119-
console.log(`Base URL: ${RUNLOOP_BASE_URL ?? "(SDK default)"}`);
115+
console.log(`HTTP mode: ${USE_HTTP2 ? 'HTTP/2 (SDK native node:http2)' : 'HTTP/1.1 (node-fetch)'}`);
116+
console.log(`Base URL: ${RUNLOOP_BASE_URL ?? '(SDK default)'}`);
120117
console.log(`File descriptor limit: ${fdLimit}`);
121-
console.log("");
118+
console.log('');
122119

123120
let completed = 0;
124121
const progressTimer = setInterval(() => {
@@ -146,14 +143,14 @@ async function main(): Promise<void> {
146143

147144
async function checkFileDescriptorLimit(): Promise<number> {
148145
try {
149-
const { execSync } = await import("child_process");
150-
return parseInt(execSync("ulimit -n", { encoding: "utf-8" }).trim(), 10);
146+
const { execSync } = await import('child_process');
147+
return parseInt(execSync('ulimit -n', { encoding: 'utf-8' }).trim(), 10);
151148
} catch {
152149
return -1;
153150
}
154151
}
155152

156153
main().catch((err) => {
157-
console.error("Fatal error:", err);
154+
console.error('Fatal error:', err);
158155
process.exit(1);
159156
});

‎loadtest/package.json‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,10 @@
11
{
22
"name": "runloop-loadtest",
33
"private": true,
4-
"type": "module",
54
"scripts": {
65
"test": "tsx loadtest.ts"
76
},
87
"dependencies": {
9-
"@runloop/api-client": "file:..",
108
"tsx": "^4.0.0",
119
"undici": "^7.26.0"
1210
},

0 commit comments

Comments
 (0)