Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

HTTP framework benchmark platform.

26 test profiles. 64-core dedicated hardware. Same conditions for every framework.
27 test profiles. 64-core dedicated hardware. Same conditions for every framework.

[View Leaderboard](https://www.http-arena.com/) | [Documentation](https://www.http-arena.com/docs/) | [Add a Framework](https://www.http-arena.com/docs/add-framework/)

Expand Down Expand Up @@ -33,6 +33,7 @@ Always specify `-f <framework>`. Results are automatically compared against the
| Connection | `baseline`, `pipelined`, `limited-conn` | Mixed GET/POST with query parsing (512/4K conns), 16× batched pipelining, short-lived connections that close after 10 requests |
| Workload | `json`, `json-comp`, `json-tls`, `upload`, `static` | JSON serialization, gzip/brotli compression, HTTP/1.1 over TLS, 20 MB body ingestion, 20-file static asset serving |
| Database | `async-db`, `crud` | Async Postgres sequential scan; realistic REST API with cached reads, list, upsert, update, and optional Redis cache |
| Templates | `fortunes` * | DB query + HTML template render (TechEmpower-style Fortunes). Reference-only — measures template-engine throughput, not part of the composite score |
| Multi-endpoint | `api-4`, `api-16` | Mixed baseline + JSON + async-db at CPU-budget cliffs (4 and 16 CPUs) |
| H/2 | `baseline-h2`, `static-h2`, `baseline-h2c`, `json-h2c` | Baseline + static over TLS with h2 stream multiplexing; baseline + JSON over cleartext h2 (prior-knowledge, port 8082) |
| H/3 | `baseline-h3`, `static-h3` | Baseline and static over QUIC with TLS 1.3 |
Expand Down
49 changes: 49 additions & 0 deletions data/pgdb-seed.sql
Original file line number Diff line number Diff line change
Expand Up @@ -100043,3 +100043,52 @@ INSERT INTO users VALUES
(17, 'Bob Martinez', 'bob@bench.local', 'free'),
(8, 'Carla Nguyen', 'carla@bench.local', 'pro'),
(99, 'Dmitri Volkov', 'dmitri@bench.local', 'enterprise');

-- ── fortunes table ─────────────────────────────────────────────────────────
-- Backs the /fortunes profile (template-engine benchmark). 12 fixed rows;
-- the handler appends a 13th row in-memory at request time and sorts by
-- message before rendering. Row 11 contains a raw <script> tag — the
-- validator asserts it is HTML-escaped in the rendered output, which is
-- the load-bearing check for "the engine actually escapes user content".
-- Dataset matches the TechEmpower Fortunes spec for cross-benchmark
-- comparability; see docs/test-profiles/h1/isolated/fortunes/.

CREATE TABLE fortune (
id INTEGER PRIMARY KEY,
message TEXT NOT NULL
);

INSERT INTO fortune (id, message) VALUES
(1, 'fortune: No such file or directory'),
(2, 'A computer scientist is someone who fixes things that aren''t broken.'),
(3, 'After enough decimal places, nobody gives a damn.'),
(4, 'A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1'),
(5, 'A computer program does what you tell it to do, not what you want it to do.'),
(6, 'Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen'),
(7, 'Any program that runs right is obsolete.'),
(8, 'A list is only as strong as its weakest link. — Donald Knuth'),
(9, 'Feature: A bug with seniority.'),
(10, 'Computers make very fast, very accurate mistakes.'),
(11, '<script>alert("This should not be displayed in a browser alert box.");</script>'),
(12, 'フレームワークのベンチマーク');

-- Synthetic rows 13..200 — bulk content so per-request render time is
-- proportional to a real engine workload instead of being dwarfed by the
-- PG round-trip. With only the 12 TE-canonical rows the profile measures
-- the database driver more than the template engine; adding 188 rows
-- means render cost scales linearly while the query cost stays fixed
-- (still one round-trip, still single-page table read). Each row contains
-- &, ', " and an em-dash so escape work runs on every cell, not just on
-- row 11. Diverges from TE Fortunes' 12-row spec by design — see
-- docs/test-profiles/h1/isolated/fortunes/.
INSERT INTO fortune (id, message)
SELECT g, CASE g % 7
WHEN 0 THEN 'Adage #' || g || ': "Premature optimization is the root of all evil." — D. Knuth'
WHEN 1 THEN 'Adage #' || g || ': There are 10 kinds of people: those who understand binary & those who don''t.'
WHEN 2 THEN 'Adage #' || g || ': Walking on water & developing from a spec are easy if both are frozen — E. Berard'
WHEN 3 THEN 'Adage #' || g || ': In theory there is no difference between theory & practice. In practice there is.'
WHEN 4 THEN 'Adage #' || g || ': "It''s not a bug, it''s an undocumented feature." — anonymous'
WHEN 5 THEN 'Adage #' || g || ': The cheapest, fastest & most reliable components are those that aren''t there.'
ELSE 'Adage #' || g || ': Programs must be written for people to read & only incidentally for machines to execute.'
END
FROM generate_series(13, 200) AS g;
2 changes: 2 additions & 0 deletions frameworks/aspnet-minimal/Models.cs
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,5 @@ sealed class CrudWriteResponse
public int Price { get; set; }
public int Quantity { get; set; }
}

public sealed record Fortune(int Id, string Message);
16 changes: 16 additions & 0 deletions frameworks/aspnet-minimal/Pages/Fortunes.cshtml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
@page "/fortunes"
@model FortunesModel
@{ Layout = null; }
<!DOCTYPE html>
<html>
<head><title>Fortunes</title></head>
<body>
<table>
<tr><th>id</th><th>message</th></tr>
@foreach (var f in Model.Fortunes)
{
<tr><td>@f.Id</td><td>@f.Message</td></tr>
}
</table>
</body>
</html>
31 changes: 31 additions & 0 deletions frameworks/aspnet-minimal/Pages/Fortunes.cshtml.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;

public sealed class FortunesModel : PageModel
{
public List<Fortune> Fortunes { get; private set; } = [];

public async Task<IActionResult> OnGetAsync()
{
if (AppData.PgDataSource is null)
return new StatusCodeResult(500);

var list = new List<Fortune>(201);
await using (var cmd = AppData.PgDataSource.CreateCommand("SELECT id, message FROM fortune"))
await using (var reader = await cmd.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
list.Add(new Fortune(reader.GetInt32(0), reader.GetString(1)));
}
}

// Runtime-injected row defeats whole-page memoization: the rendered
// HTML must vary per request, even though the seeded rows don't.
list.Add(new Fortune(0, "Additional fortune added at request time."));
list.Sort(static (a, b) => string.CompareOrdinal(a.Message, b.Message));

Fortunes = list;
return Page();
}
}
7 changes: 7 additions & 0 deletions frameworks/aspnet-minimal/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
var builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.Services.AddMemoryCache();
builder.Services.AddRazorPages();

var certPath = Environment.GetEnvironmentVariable("TLS_CERT") ?? "/certs/server.crt";
var keyPath = Environment.GetEnvironmentVariable("TLS_KEY") ?? "/certs/server.key";
Expand Down Expand Up @@ -84,6 +85,12 @@
app.MapPost("/crud/items", Handlers.CrudCreate);
app.MapPut("/crud/items/{id:int}", Handlers.CrudUpdate);

// /fortunes is served by the Razor page at Pages/Fortunes.cshtml
// (route "/fortunes" declared via the page's @page directive). MapRazorPages
// wires up the MVC/Razor pipeline so the page model can render Razor markup
// — the standard ASP.NET production path for HTML responses.
app.MapRazorPages();

app.MapStaticAssets();

app.Run();
1 change: 1 addition & 0 deletions frameworks/aspnet-minimal/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
"static",
"async-db",
"crud",
"fortunes",
"baseline-h2",
"static-h2",
"baseline-h2c",
Expand Down
3 changes: 3 additions & 0 deletions requests/fortunes-get.raw
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
GET /fortunes HTTP/1.1
Host: localhost:8080

2 changes: 1 addition & 1 deletion scripts/benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ fi

# Start the postgres sidecar if any subscribed test needs it.
need_pg=false
for t in async-db crud api-4 api-16 gateway-64 gateway-h3 production-stack; do
for t in async-db crud api-4 api-16 gateway-64 gateway-h3 production-stack fortunes; do
if framework_subscribes_to "$t"; then need_pg=true; break; fi
done
$need_pg && postgres_start
Expand Down
2 changes: 1 addition & 1 deletion scripts/lib/framework.sh
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ framework_start() {

# Profiles that exercise the database get DATABASE_URL + per-profile conn cap.
case "$endpoint" in
async-db|crud|api-4|api-16)
async-db|crud|api-4|api-16|fortunes)
args+=(-e "DATABASE_URL=$DATABASE_URL" -e "DATABASE_MAX_CONN=256")
;;
esac
Expand Down
2 changes: 2 additions & 0 deletions scripts/lib/profiles.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ declare -A PROFILES=(
[static]="1|200|0-31,64-95|1024,4096,6800|static"
[async-db]="1|0|0-31,64-95|1024|async-db"
[crud]="1|200|1-31,65-95|4096|crud"
[fortunes]="1|0|0-31,64-95|1024|fortunes"
[baseline-h2]="1|0|0-31,64-95|256,1024|h2"
[static-h2]="1|0|0-31,64-95|256,1024|static-h2"
[baseline-h2c]="1|0|0-31,64-95|256,1024,4096|h2c"
Expand All @@ -43,6 +44,7 @@ PROFILE_ORDER=(
json json-comp json-tls
upload api-4 api-16
static async-db crud
fortunes
baseline-h2 static-h2
baseline-h2c json-h2c
baseline-h3 static-h3
Expand Down
7 changes: 7 additions & 0 deletions scripts/lib/tools/gcannon.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@ gcannon_build_args() {
--raw "$REQUESTS_DIR/async-db-5.raw,$REQUESTS_DIR/async-db-10.raw,$REQUESTS_DIR/async-db-20.raw,$REQUESTS_DIR/async-db-35.raw,$REQUESTS_DIR/async-db-50.raw"
-c "$conns" -t "$THREADS" -d 10s -p "$pipeline" -r 25)
;;
fortunes)
# Single endpoint, fixed 12-row seed + 1 runtime-injected row.
# No --raw rotation: every request is the same GET; the per-request
# variance lives inside the handler (sort + render).
args=("http://localhost:$PORT/fortunes"
-c "$conns" -t "$THREADS" -d "$duration" -p "$pipeline")
;;
json)
args=("http://localhost:$PORT"
--raw "$REQUESTS_DIR/json-1.raw,$REQUESTS_DIR/json-5.raw,$REQUESTS_DIR/json-10.raw,$REQUESTS_DIR/json-15.raw,$REQUESTS_DIR/json-25.raw,$REQUESTS_DIR/json-40.raw,$REQUESTS_DIR/json-50.raw"
Expand Down
71 changes: 69 additions & 2 deletions scripts/validate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ fi
HARD_NOFILE=$(ulimit -Hn 2>/dev/null || echo 1048576)
# Docker --ulimit nofile rejects "unlimited"; fall back to a large numeric cap
[[ "$HARD_NOFILE" =~ ^[0-9]+$ ]] || HARD_NOFILE=1048576
if has_test "async-db" || has_test "crud" || has_test "api-4" || has_test "api-16" || has_test "gateway-64" || has_test "gateway-h3" || has_test "production-stack"; then
if has_test "async-db" || has_test "crud" || has_test "api-4" || has_test "api-16" || has_test "gateway-64" || has_test "gateway-h3" || has_test "production-stack" || has_test "fortunes"; then
docker_args=(-d --name "$CONTAINER_NAME" --network host --security-opt seccomp=unconfined
--ulimit memlock=-1:-1 --ulimit nofile="$HARD_NOFILE:$HARD_NOFILE")
else
Expand Down Expand Up @@ -138,7 +138,7 @@ if [ "$ENGINE" = "io_uring" ]; then
fi

# Start Postgres sidecar if async-db is needed
if has_test "async-db" || has_test "crud" || has_test "api-4" || has_test "api-16" || has_test "gateway-64" || has_test "gateway-h3" || has_test "production-stack"; then
if has_test "async-db" || has_test "crud" || has_test "api-4" || has_test "api-16" || has_test "gateway-64" || has_test "gateway-h3" || has_test "production-stack" || has_test "fortunes"; then
echo "[postgres] Starting Postgres sidecar for validation..."
docker rm -f "$PG_CONTAINER" 2>/dev/null || true
docker run -d --name "$PG_CONTAINER" --network host \
Expand Down Expand Up @@ -1025,6 +1025,73 @@ print(f'{count} {has_rating} {has_tags} {has_active_bool}')
fi
fi

# ───── Fortunes (GET /fortunes) — template-engine benchmark ─────
#
# Feature-based validation, not byte-exact. Engines disagree on whitespace
# and attribute formatting; what matters is that the rendered HTML actually
# loops the DB rows, includes the runtime-injected row, escapes user
# content, and is sized like a real page (not stripped to win the bench).

if has_test "fortunes"; then
FORTUNES_DOCS="$DOCS_BASE/h1/isolated/fortunes/validation"
echo "[test] fortunes endpoint"

body=$(curl -s --max-time 30 "http://localhost:$PORT/fortunes" || true)

check_header "GET /fortunes Content-Type" "Content-Type" "text/html" "$FORTUNES_DOCS" \
"http://localhost:$PORT/fortunes"

if echo "$body" | grep -qi '<!doctype html>'; then
echo " PASS [GET /fortunes <!DOCTYPE html>]"
PASS=$((PASS + 1))
else
fail_with_link "[GET /fortunes <!DOCTYPE html>]: missing — layout/partial likely not rendered" "$FORTUNES_DOCS"
fi

# Each row in the rendered table must produce a <tr>. 201 data rows are
# required (200 seeded + 1 runtime-injected); a header row is allowed
# but not required, so the band is 201–210 to absorb implementation-
# specific extras (footer rows, etc.).
tr_count=$(echo "$body" | grep -oi '<tr' | wc -l)
if [ "$tr_count" -ge 201 ] && [ "$tr_count" -le 210 ]; then
echo " PASS [GET /fortunes <tr> count=$tr_count]"
PASS=$((PASS + 1))
else
fail_with_link "[GET /fortunes <tr> count]: expected 201–210, got $tr_count" "$FORTUNES_DOCS"
fi

# Runtime-injected row text — proves the handler appended id=0 in memory
# rather than caching a pre-rendered page from the DB rows alone.
if echo "$body" | grep -qF 'Additional fortune added at request time.'; then
echo " PASS [GET /fortunes runtime-injected row]"
PASS=$((PASS + 1))
else
fail_with_link "[GET /fortunes runtime-injected row]: missing 'Additional fortune added at request time.'" "$FORTUNES_DOCS"
fi

# XSS escape — load-bearing check. Row 11 contains a raw <script> tag
# in the DB; the rendered output must encode it as &lt;script&gt; and
# must NOT contain the raw <script>alert sequence anywhere.
if echo "$body" | grep -qF '&lt;script&gt;' && ! echo "$body" | grep -qF '<script>alert'; then
echo " PASS [GET /fortunes XSS escape]"
PASS=$((PASS + 1))
else
fail_with_link "[GET /fortunes XSS escape]: <script> in row 11 not properly HTML-escaped" "$FORTUNES_DOCS"
fi

# Size sanity — catches stripped pages and empty bodies. A 201-row
# table plus a layout typically lands between 18 KB and 40 KB; the band
# is generous to absorb whitespace and per-engine formatting, but
# rejects empty / fragment / pathologically-large outputs.
size=${#body}
if [ "$size" -ge 18432 ] && [ "$size" -le 65536 ]; then
echo " PASS [GET /fortunes body size=${size}B]"
PASS=$((PASS + 1))
else
fail_with_link "[GET /fortunes body size]: expected 18432–65536 bytes, got $size" "$FORTUNES_DOCS"
fi
fi

# ───── CRUD (list + read + create + update /crud/items) ─────

if has_test "crud"; then
Expand Down
7 changes: 6 additions & 1 deletion site/content/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ html.dark .test-card-endpoint { color: #64748b; }
</style>

<div class="tests-section">
<h2>26 Test Profiles Across H/1.1, H/2, H/3, gRPC and WebSocket</h2>
<h2>27 Test Profiles Across H/1.1, H/2, H/3, gRPC and WebSocket</h2>
<p class="tests-sub">Every framework is tested under diverse, realistic workloads — from raw throughput to JSON processing, gRPC unary calls, and WebSocket echo.</p>

<div class="tests-proto">
Expand Down Expand Up @@ -124,6 +124,11 @@ html.dark .test-card-endpoint { color: #64748b; }
<div class="test-card-desc">Realistic REST API with paginated list, cached reads, create, and update against Postgres.</div>
<div class="test-card-endpoint">GET/POST/PUT /crud/items</div>
</a>
<a class="test-card" href="docs/test-profiles/h1/isolated/fortunes">
<div class="test-card-title">Fortunes (Templates) *</div>
<div class="test-card-desc">DB query + HTML template rendering with auto-escape. Reference-only — measures template-engine throughput.</div>
<div class="test-card-endpoint">GET /fortunes</div>
</a>
</div>
</div>

Expand Down
1 change: 1 addition & 0 deletions site/content/docs/scoring/composite-score.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ Not all profiles count toward the composite score. Profiles marked as **scored**
| Static | Yes | 20 static files served over HTTP/1.1 |
| Async DB | Yes | Async Postgres query with connection pooling |
| CRUD | Yes | Realistic REST API against Postgres: cached reads (75%), updates (15%), list (5%), upsert create (5%). Cache-aside with 200ms TTL (in-process or Redis sidecar) |
| Fortunes | No (*) | DB query + HTML template render. Reference-only — engine-comparison test, not part of the composite ranking |
| TCP Frag | No (*) | Baseline with MTU 69 — TCP fragmentation stress |

### H/1.1 Workload
Expand Down
1 change: 1 addition & 0 deletions site/content/docs/test-profiles/h1/isolated/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ Single-endpoint benchmarks that measure framework performance on one task at a t
{{< card link="static" title="Static Files" subtitle="Serves 20 static files — CSS, JS, HTML, fonts, images — over HTTP/1.1." icon="photograph" >}}
{{< card link="pipelined" title="Pipelined (16x)" subtitle="16 requests sent back-to-back per connection, testing raw I/O and pipeline batching." icon="fast-forward" >}}
{{< card link="crud" title="CRUD (REST API)" subtitle="Realistic REST API with paginated list, cached reads, create, and update against Postgres." icon="database" >}}
{{< card link="fortunes" title="Fortunes (Templates) *" subtitle="DB query + HTML template rendering with auto-escape. Reference-only — measures template-engine throughput, not part of the composite score." icon="document-text" >}}
{{< /cards >}}
10 changes: 10 additions & 0 deletions site/content/docs/test-profiles/h1/isolated/fortunes/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
title: Fortunes (Templates)
---

Database-backed HTML rendering benchmark — measures template-engine throughput end-to-end: Postgres read, runtime row injection, sort, escape-aware HTML rendering. Reference-only profile (not part of the composite score).

{{< cards >}}
{{< card link="implementation" title="Implementation Guidelines" subtitle="Endpoint specification, expected output, and type-specific rules." icon="code" >}}
{{< card link="validation" title="Validation" subtitle="All checks executed by the validation script for this test profile." icon="check-circle" >}}
{{< /cards >}}
Loading