You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR Commands — comment on this PR to trigger (requires collaborator approval):
Command
Description
/benchmark -f <framework>
Run all benchmark tests
/benchmark -f <framework> -t <test>
Run a specific test
/benchmark -f <framework> --save
Run and save results (updates leaderboard on merge)
Always specify -f <framework>. Results are automatically compared against the current leaderboard.
Run benchmarks locally
You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.
25% of clients started
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
48.
finished in 5.03s, 76413.00 req/s, 1.84GB/s
requests: 382065 total, 390257 started, 382065 done, 382065 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 382142 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 9.22GB (9901658301) total, 14.12MB (14806403) headers (space savings 76.48%), 9.20GB (9876005760) data
min max mean sd +/- sd
time for request: 258us 555.65ms 104.81ms 72.06ms 70.40%
time for connect: 2.45ms 94.82ms 21.58ms 17.07ms 88.28%
time to 1st byte: 5.72ms 248.11ms 63.61ms 44.38ms 75.39%
req/s : 193.77 470.18 298.45 50.74 73.05%
[info] CPU 5777.7% | Mem 508MiB
=== Best: 75972 req/s (CPU: 5777.7%, Mem: 508MiB) ===
[info] saved results/static-h2/256/mq-bridge.json
httparena-bench-mq-bridge
httparena-bench-mq-bridge
==============================================
=== mq-bridge / static-h2 / 1024c (tool=h2load) ===
==============================================
[info] waiting for server...
[info] server ready
[run 1/3]
starting benchmark...
7Main benchmark duration is started for thread #58.
.
.Main benchmark duration is started for thread #41.
.
progress: Main benchmark duration is started for thread #37.
50% of clients started
progress: Warm-up phase is over for thread #.75
16Warm-up phase is over for thread #.Warm-up phase is over for thread #Main benchmark duration is started for thread #3.
60.
.
22.
.
12.
progress: Warm-up phase is over for thread #20.
87% of clients started
5.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
. Stopping all clients.
. Stopping all clients.Stopped all clients for thread #63
34
55
4. Stopping all clients.
59Stopped all clients for thread #
27
finished in 5.06s, 64038.20 req/s, 1.55GB/s
requests: 320191 total, 352959 started, 320191 done, 320191 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 320258 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 7.75GB (8321877399) total, 11.92MB (12496913) headers (space savings 76.31%), 7.73GB (8300234877) data
min max mean sd +/- sd
time for request: 308us 1.29s 460.24ms 164.50ms 73.44%
time for connect: 3.30ms 710.90ms 267.39ms 182.27ms 60.45%
time to 1st byte: 24.35ms 1.38s 514.87ms 323.08ms 59.28%
req/s : 44.59 93.77 62.52 7.34 68.95%
[info] CPU 4941.6% | Mem 2.2GiB
[run 2/3]
starting benchmark...
26.
.Warm-up phase is over for thread #Warm-up phase is over for thread #17.47
.
.Warm-up phase is over for thread #
.52Main benchmark duration is started for thread #61.
.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
43Stopped all clients for thread #0
. Stopping all clients.
24
22
finished in 5.05s, 73236.60 req/s, 1.77GB/s
requests: 366183 total, 398951 started, 366183 done, 366183 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 366276 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8.85GB (9505936519) total, 13.63MB (14288881) headers (space savings 76.32%), 8.83GB (9481201808) data
min max mean sd +/- sd
time for request: 341us 1.28s 418.58ms 135.84ms 73.12%
time for connect: 3.86ms 261.61ms 78.81ms 51.51ms 66.99%
time to 1st byte: 17.23ms 806.85ms 251.56ms 157.85ms 61.04%
req/s : 52.40 95.59 71.50 6.10 70.51%
[info] CPU 5929.7% | Mem 2.2GiB
[run 3/3]
starting benchmark...
% of clients startedMain benchmark duration is started for thread #8.
.
progress: Main benchmark duration is started for thread #375Warm-up phase is over for thread #% of clients started
36.
progress: 81Warm-up phase is over for thread #% of clients started41
.
37.
% of clients startedMain benchmark duration is started for thread #20.
24.Main benchmark duration is started for thread #
61.Warm-up phase is over for thread #.53
.Warm-up phase is over for thread #Main benchmark duration is started for thread #50.24
.
0.
.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
14.
63. Stopping all clients.
26
0Stopped all clients for thread #50
34Stopped all clients for thread #57. Stopping all clients.Stopped all clients for thread #
7
finished in 5.07s, 73663.20 req/s, 1.78GB/s
requests: 368316 total, 401084 started, 368316 done, 368316 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 368464 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8.91GB (9565130948) total, 13.72MB (14387741) headers (space savings 76.30%), 8.89GB (9540234431) data
min max mean sd +/- sd
time for request: 240us 1.11s 417.73ms 133.51ms 72.28%
time for connect: 3.09ms 243.70ms 75.64ms 56.94ms 68.75%
time to 1st byte: 7.26ms 762.64ms 245.69ms 162.70ms 62.01%
req/s : 55.18 95.19 71.92 5.97 69.73%
[info] CPU 5880.8% | Mem 2.3GiB
=== Best: 72675 req/s (CPU: 5880.8%, Mem: 2.3GiB) ===
[info] saved results/static-h2/1024/mq-bridge.json
httparena-bench-mq-bridge
httparena-bench-mq-bridge
[info] skip: mq-bridge does not subscribe to baseline-h2c
[info] skip: mq-bridge does not subscribe to json-h2c
[info] skip: mq-bridge does not subscribe to baseline-h3
[info] skip: mq-bridge does not subscribe to static-h3
[info] skip: mq-bridge does not subscribe to gateway-64
[info] skip: mq-bridge does not subscribe to gateway-h3
[info] skip: mq-bridge does not subscribe to production-stack
[info] skip: mq-bridge does not subscribe to unary-grpc
[info] skip: mq-bridge does not subscribe to unary-grpc-tls
[info] skip: mq-bridge does not subscribe to stream-grpc
[info] skip: mq-bridge does not subscribe to stream-grpc-tls
[info] skip: mq-bridge does not subscribe to echo-ws
[info] skip: mq-bridge does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-16-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-4-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/async-db-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-h2-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-h2-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
[info] restoring loopback MTU to 65536
.Main benchmark duration is started for thread #56.
.
42.
.
21..
finished in 5.05s, 529006.80 req/s, 1.76GB/s
requests: 2645034 total, 2677802 started, 2645034 done, 2645034 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2645733 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8.80GB (9444893691) total, 89.96MB (94330226) headers (space savings 76.19%), 8.66GB (9302890242) data
min max mean sd +/- sd
time for request: 54us 187.43ms 60.26ms 30.69ms 68.78%
time for connect: 19us 1.05ms 386us 216us 63.87%
time to 1st byte: 1.20ms 136.21ms 21.55ms 23.97ms 85.74%
req/s : 444.51 621.56 516.53 26.18 69.43%
[info] CPU 3200.5% | Mem 367MiB
[run 3/3]
starting benchmark...
63Main benchmark duration is started for thread #20.
.
22.
22Main benchmark duration is started for thread #53.
.
38Main benchmark duration is started for thread #10.Warm-up phase is over for thread #.
.
.
3231.
.Main benchmark duration is started for thread #32.
.
15.
Application protocol: h2c
63. Stopping all clients.
29. Stopping all clients.
38Stopped all clients for thread #22
. Stopping all clients.
20. Stopping all clients.
16. Stopping all clients.
finished in 5.05s, 534857.80 req/s, 1.78GB/s
requests: 2674289 total, 2707057 started, 2674289 done, 2674289 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2674807 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8.89GB (9549528014) total, 91.07MB (95493340) headers (space savings 76.15%), 8.76GB (9405836490) data
min max mean sd +/- sd
time for request: 53us 173.71ms 59.67ms 30.80ms 67.72%
time for connect: 24us 988us 377us 210us 64.36%
time to 1st byte: 1.12ms 121.77ms 23.90ms 21.76ms 75.59%
req/s : 455.91 637.81 522.25 28.65 67.29%
[info] CPU 3249.2% | Mem 397MiB
=== Best: 529664 req/s (CPU: 3249.2%, Mem: 397MiB) ===
[info] saved results/json-h2c/1024/mq-bridge-h2c.json
httparena-bench-mq-bridge-h2c
httparena-bench-mq-bridge-h2c
==============================================
=== mq-bridge-h2c / json-h2c / 4096c (tool=h2load) ===
==============================================
[info] waiting for server...
[info] server ready
[run 1/3]
starting benchmark...
52.Warm-up phase is over for thread #14.
37Warm-up phase is over for thread #.53Warm-up phase is over for thread #Warm-up phase is over for thread #26.23.
.
54
.
Application protocol: h2c
5. Stopping all clients.
4
. Stopping all clients.
24. Stopping all clients.
finished in 5.11s, 432137.80 req/s, 1.43GB/s
requests: 2160689 total, 2291761 started, 2160689 done, 2160689 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2162409 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 7.17GB (7697106344) total, 73.96MB (77554515) headers (space savings 76.04%), 7.06GB (7580418640) data
min max mean sd +/- sd
time for request: 44us 1.24s 300.68ms 235.71ms 52.96%
time for connect: 15us 6.70ms 2.39ms 1.78ms 59.01%
time to 1st byte: 3.65ms 965.77ms 345.84ms 247.75ms 65.43%
req/s : 62.75 172.12 105.46 14.96 68.55%
[info] CPU 2868.0% | Mem 989MiB
[run 2/3]
starting benchmark...
.30Warm-up phase is over for thread #41.
.
.
.
..
3960..
45Main benchmark duration is started for thread #2.
.
1Main benchmark duration is started for thread #16.
.
57.Main benchmark duration is started for thread #11.
Application protocol: h2c
44.
52.
28.
40
42. Stopping all clients.
38. Stopping all clients.
finished in 5.09s, 497664.00 req/s, 1.65GB/s
requests: 2488320 total, 2619392 started, 2488320 done, 2488320 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2489949 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8.26GB (8867325040) total, 85.07MB (89202157) headers (space savings 76.07%), 8.13GB (8733093182) data
min max mean sd +/- sd
time for request: 56us 992.92ms 245.99ms 187.62ms 56.78%
time for connect: 17us 7.35ms 3.03ms 2.26ms 53.47%
time to 1st byte: 3.40ms 718.65ms 40.76ms 56.66ms 92.43%
req/s : 71.99 196.45 121.44 17.20 68.80%
[info] CPU 3185.7% | Mem 1.1GiB
[run 3/3]
starting benchmark...
48.
4.
.Main benchmark duration is started for thread #12.
.Main benchmark duration is started for thread #25.
39.
33.
.Warm-up phase is over for thread #56.
.Main benchmark duration is started for thread #50.
8.
Application protocol: h2c
54Main benchmark duration is over for thread #. Stopping all clients.
10Main benchmark duration is over for thread #30. Stopping all clients.
. Stopping all clients.
33. Stopping all clients.
. Stopping all clients.
26Main benchmark duration is over for thread #. Stopping all clients.
61Stopped all clients for thread #Stopped all clients for thread #33
26
finished in 5.11s, 489734.20 req/s, 1.63GB/s
requests: 2448671 total, 2579743 started, 2448671 done, 2448671 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2450344 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8.13GB (8725253704) total, 83.72MB (87789482) headers (space savings 76.07%), 8.00GB (8593147798) data
min max mean sd +/- sd
time for request: 52us 942.03ms 250.82ms 187.53ms 56.13%
time for connect: 16us 7.35ms 2.57ms 1.96ms 57.96%
time to 1st byte: 4.56ms 765.97ms 39.04ms 38.55ms 91.26%
req/s : 76.15 190.90 119.50 16.23 69.82%
[info] CPU 3220.3% | Mem 1.1GiB
=== Best: 489184 req/s (CPU: 3185.7%, Mem: 1.1GiB) ===
[info] saved results/json-h2c/4096/mq-bridge-h2c.json
httparena-bench-mq-bridge-h2c
httparena-bench-mq-bridge-h2c
[info] skip: mq-bridge-h2c does not subscribe to baseline-h3
[info] skip: mq-bridge-h2c does not subscribe to static-h3
[info] skip: mq-bridge-h2c does not subscribe to gateway-64
[info] skip: mq-bridge-h2c does not subscribe to gateway-h3
[info] skip: mq-bridge-h2c does not subscribe to production-stack
[info] skip: mq-bridge-h2c does not subscribe to unary-grpc
[info] skip: mq-bridge-h2c does not subscribe to unary-grpc-tls
[info] skip: mq-bridge-h2c does not subscribe to stream-grpc
[info] skip: mq-bridge-h2c does not subscribe to stream-grpc-tls
[info] skip: mq-bridge-h2c does not subscribe to echo-ws
[info] skip: mq-bridge-h2c does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2c-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2c-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-h2c-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-h2c-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-h2c-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adding mq-bridge to benchmark:
PR Commands — comment on this PR to trigger (requires collaborator approval):
/benchmark -f <framework>/benchmark -f <framework> -t <test>/benchmark -f <framework> --saveAlways specify
-f <framework>. Results are automatically compared against the current leaderboard.Run benchmarks locally
You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.
Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.