You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: site/content/docs/add-framework/_index.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,4 +10,5 @@ Adding a framework to HttpArena takes a few steps: create a Dockerfile, add meta
10
10
{{< card link="test-profiles" title="Test Profiles" subtitle="All HTTP endpoints your framework must implement, organized by test profile." icon="code" >}}
11
11
{{< card link="meta-json" title="meta.json" subtitle="Framework metadata — display name, language, type, and which tests to participate in." icon="document-text" >}}
12
12
{{< card link="testing" title="Testing & Submitting" subtitle="How to validate your implementation locally and submit a pull request." icon="check-circle" >}}
13
+
{{< card link="ci" title="CI & Runner" subtitle="GitHub Actions workflows and the self-hosted benchmark runner." icon="cog" >}}
HttpArena uses four GitHub Actions workflows to automate validation, benchmarking, and deployment.
8
+
9
+
### Validate Framework
10
+
11
+
**Trigger:** Automatically on every PR that modifies files under `frameworks/` or `scripts/validate.sh`.
12
+
13
+
Detects which frameworks were changed in the PR and runs `./scripts/validate.sh` against each one. If validation fails, the PR check fails — you must fix the issues before merging.
14
+
15
+
### Benchmark
16
+
17
+
**Trigger:** Automatically when a push to `main` modifies files under `frameworks/`, or manually via workflow dispatch.
18
+
19
+
When triggered automatically, it detects which frameworks changed in the latest commit and benchmarks only those. When triggered manually, you can specify:
20
+
-**Framework** — a specific framework name, or leave empty to benchmark all changed frameworks
21
+
-**Profile** — a specific test profile (e.g. `baseline`, `baseline-h2`), or leave empty to run all profiles
22
+
23
+
Results are committed and pushed to `main` automatically by the HttpArena Bot.
24
+
25
+
### Benchmark PR
26
+
27
+
**Trigger:** Manual only (workflow dispatch). Requires a PR number and framework name.
28
+
29
+
Checks out the PR branch, runs the benchmark, and posts the results as a comment on the PR. This lets maintainers benchmark a new framework submission before merging, so contributors can see how their implementation performs on the hosted runner. An optional profile parameter lets you run a specific test instead of the full suite.
30
+
31
+
### Deploy Site
32
+
33
+
**Trigger:** Automatically when a push to `main` modifies files under `site/`, or manually via workflow dispatch.
34
+
35
+
Builds the Hugo site and deploys it to GitHub Pages. This runs on GitHub-hosted Ubuntu runners (not the self-hosted runner).
36
+
37
+
## Hosted runner
38
+
39
+
The Validate, Benchmark, and Benchmark PR workflows run on a **self-hosted runner** — a dedicated bare-metal machine configured for reproducible, low-noise benchmarking. This ensures all frameworks are tested on identical hardware under controlled conditions, with CPU governors locked, background services minimized, and no resource contention from other CI jobs.
40
+
41
+
Only the Deploy Site workflow uses GitHub-hosted runners, since it only builds static HTML and doesn't need controlled hardware.
Copy file name to clipboardExpand all lines: site/content/docs/add-framework/testing.md
-38Lines changed: 0 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -44,41 +44,3 @@ The PR should include:
44
44
-`Dockerfile`
45
45
-`meta.json`
46
46
- Source files for your server implementation
47
-
48
-
## GitHub Actions
49
-
50
-
HttpArena uses four GitHub Actions workflows to automate validation, benchmarking, and deployment.
51
-
52
-
### Validate Framework
53
-
54
-
**Trigger:** Automatically on every PR that modifies files under `frameworks/` or `scripts/validate.sh`.
55
-
56
-
Detects which frameworks were changed in the PR and runs `./scripts/validate.sh` against each one. If validation fails, the PR check fails — you must fix the issues before merging.
57
-
58
-
### Benchmark
59
-
60
-
**Trigger:** Automatically when a push to `main` modifies files under `frameworks/`, or manually via workflow dispatch.
61
-
62
-
When triggered automatically, it detects which frameworks changed in the latest commit and benchmarks only those. When triggered manually, you can specify:
63
-
-**Framework** — a specific framework name, or leave empty to benchmark all changed frameworks
64
-
-**Profile** — a specific test profile (e.g. `baseline`, `baseline-h2`), or leave empty to run all profiles
65
-
66
-
Results are committed and pushed to `main` automatically by the HttpArena Bot.
67
-
68
-
### Benchmark PR
69
-
70
-
**Trigger:** Manual only (workflow dispatch). Requires a PR number and framework name.
71
-
72
-
Checks out the PR branch, runs the benchmark, and posts the results as a comment on the PR. This lets maintainers benchmark a new framework submission before merging, so contributors can see how their implementation performs on the hosted runner. An optional profile parameter lets you run a specific test instead of the full suite.
73
-
74
-
### Deploy Site
75
-
76
-
**Trigger:** Automatically when a push to `main` modifies files under `site/`, or manually via workflow dispatch.
77
-
78
-
Builds the Hugo site and deploys it to GitHub Pages. This runs on GitHub-hosted Ubuntu runners (not the self-hosted runner).
79
-
80
-
## Hosted runner
81
-
82
-
The Validate, Benchmark, and Benchmark PR workflows run on a **self-hosted runner** — a dedicated bare-metal machine configured for reproducible, low-noise benchmarking. This ensures all frameworks are tested on identical hardware under controlled conditions, with CPU governors locked, background services minimized, and no resource contention from other CI jobs.
83
-
84
-
Only the Deploy Site workflow uses GitHub-hosted runners, since it only builds static HTML and doesn't need controlled hardware.
Copy file name to clipboardExpand all lines: site/content/docs/scoring/composite-score.md
+6-1Lines changed: 6 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -115,7 +115,12 @@ Framework A still leads because its raw throughput advantage outweighs B's CPU e
115
115
116
116
## Engine-level implementations
117
117
118
-
Frameworks with type `engine` are excluded from the composite ranking and from the normalization pool. They can still be compared in individual test profiles. Only `framework` entries — those using standard, production-grade HTTP libraries — are scored here.
118
+
Engines and frameworks are scored **separately** — each type has its own composite ranking and normalization pool. The scored profiles differ by type:
119
+
120
+
-**Frameworks** are scored on all H1 profiles (Baseline, Pipelined, Short-lived, JSON, Upload, Compression, Noisy, Mixed) plus all H2/H3 profiles.
121
+
-**Engines** are scored on a reduced H1 set (Baseline, Pipelined, Short-lived only) plus all H2/H3 profiles, since most engines don't implement the heavier endpoints (JSON, DB, upload, compression).
122
+
123
+
The Type filter on the composite leaderboard switches between the two rankings.
0 commit comments