Skip to content

Commit ef75559

Browse files
committed
update readme
1 parent 31e29ef commit ef75559

4 files changed

Lines changed: 48 additions & 39 deletions

File tree

site/content/docs/add-framework/_index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,5 @@ Adding a framework to HttpArena takes a few steps: create a Dockerfile, add meta
1010
{{< card link="test-profiles" title="Test Profiles" subtitle="All HTTP endpoints your framework must implement, organized by test profile." icon="code" >}}
1111
{{< card link="meta-json" title="meta.json" subtitle="Framework metadata — display name, language, type, and which tests to participate in." icon="document-text" >}}
1212
{{< card link="testing" title="Testing & Submitting" subtitle="How to validate your implementation locally and submit a pull request." icon="check-circle" >}}
13+
{{< card link="ci" title="CI & Runner" subtitle="GitHub Actions workflows and the self-hosted benchmark runner." icon="cog" >}}
1314
{{< /cards >}}
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
title: CI & Runner
3+
---
4+
5+
## GitHub Actions
6+
7+
HttpArena uses four GitHub Actions workflows to automate validation, benchmarking, and deployment.
8+
9+
### Validate Framework
10+
11+
**Trigger:** Automatically on every PR that modifies files under `frameworks/` or `scripts/validate.sh`.
12+
13+
Detects which frameworks were changed in the PR and runs `./scripts/validate.sh` against each one. If validation fails, the PR check fails — you must fix the issues before merging.
14+
15+
### Benchmark
16+
17+
**Trigger:** Automatically when a push to `main` modifies files under `frameworks/`, or manually via workflow dispatch.
18+
19+
When triggered automatically, it detects which frameworks changed in the latest commit and benchmarks only those. When triggered manually, you can specify:
20+
- **Framework** — a specific framework name, or leave empty to benchmark all changed frameworks
21+
- **Profile** — a specific test profile (e.g. `baseline`, `baseline-h2`), or leave empty to run all profiles
22+
23+
Results are committed and pushed to `main` automatically by the HttpArena Bot.
24+
25+
### Benchmark PR
26+
27+
**Trigger:** Manual only (workflow dispatch). Requires a PR number and framework name.
28+
29+
Checks out the PR branch, runs the benchmark, and posts the results as a comment on the PR. This lets maintainers benchmark a new framework submission before merging, so contributors can see how their implementation performs on the hosted runner. An optional profile parameter lets you run a specific test instead of the full suite.
30+
31+
### Deploy Site
32+
33+
**Trigger:** Automatically when a push to `main` modifies files under `site/`, or manually via workflow dispatch.
34+
35+
Builds the Hugo site and deploys it to GitHub Pages. This runs on GitHub-hosted Ubuntu runners (not the self-hosted runner).
36+
37+
## Hosted runner
38+
39+
The Validate, Benchmark, and Benchmark PR workflows run on a **self-hosted runner** — a dedicated bare-metal machine configured for reproducible, low-noise benchmarking. This ensures all frameworks are tested on identical hardware under controlled conditions, with CPU governors locked, background services minimized, and no resource contention from other CI jobs.
40+
41+
Only the Deploy Site workflow uses GitHub-hosted runners, since it only builds static HTML and doesn't need controlled hardware.

site/content/docs/add-framework/testing.md

Lines changed: 0 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -44,41 +44,3 @@ The PR should include:
4444
- `Dockerfile`
4545
- `meta.json`
4646
- Source files for your server implementation
47-
48-
## GitHub Actions
49-
50-
HttpArena uses four GitHub Actions workflows to automate validation, benchmarking, and deployment.
51-
52-
### Validate Framework
53-
54-
**Trigger:** Automatically on every PR that modifies files under `frameworks/` or `scripts/validate.sh`.
55-
56-
Detects which frameworks were changed in the PR and runs `./scripts/validate.sh` against each one. If validation fails, the PR check fails — you must fix the issues before merging.
57-
58-
### Benchmark
59-
60-
**Trigger:** Automatically when a push to `main` modifies files under `frameworks/`, or manually via workflow dispatch.
61-
62-
When triggered automatically, it detects which frameworks changed in the latest commit and benchmarks only those. When triggered manually, you can specify:
63-
- **Framework** — a specific framework name, or leave empty to benchmark all changed frameworks
64-
- **Profile** — a specific test profile (e.g. `baseline`, `baseline-h2`), or leave empty to run all profiles
65-
66-
Results are committed and pushed to `main` automatically by the HttpArena Bot.
67-
68-
### Benchmark PR
69-
70-
**Trigger:** Manual only (workflow dispatch). Requires a PR number and framework name.
71-
72-
Checks out the PR branch, runs the benchmark, and posts the results as a comment on the PR. This lets maintainers benchmark a new framework submission before merging, so contributors can see how their implementation performs on the hosted runner. An optional profile parameter lets you run a specific test instead of the full suite.
73-
74-
### Deploy Site
75-
76-
**Trigger:** Automatically when a push to `main` modifies files under `site/`, or manually via workflow dispatch.
77-
78-
Builds the Hugo site and deploys it to GitHub Pages. This runs on GitHub-hosted Ubuntu runners (not the self-hosted runner).
79-
80-
## Hosted runner
81-
82-
The Validate, Benchmark, and Benchmark PR workflows run on a **self-hosted runner** — a dedicated bare-metal machine configured for reproducible, low-noise benchmarking. This ensures all frameworks are tested on identical hardware under controlled conditions, with CPU governors locked, background services minimized, and no resource contention from other CI jobs.
83-
84-
Only the Deploy Site workflow uses GitHub-hosted runners, since it only builds static HTML and doesn't need controlled hardware.

site/content/docs/scoring/composite-score.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,12 @@ Framework A still leads because its raw throughput advantage outweighs B's CPU e
115115

116116
## Engine-level implementations
117117

118-
Frameworks with type `engine` are excluded from the composite ranking and from the normalization pool. They can still be compared in individual test profiles. Only `framework` entries — those using standard, production-grade HTTP libraries — are scored here.
118+
Engines and frameworks are scored **separately** — each type has its own composite ranking and normalization pool. The scored profiles differ by type:
119+
120+
- **Frameworks** are scored on all H1 profiles (Baseline, Pipelined, Short-lived, JSON, Upload, Compression, Noisy, Mixed) plus all H2/H3 profiles.
121+
- **Engines** are scored on a reduced H1 set (Baseline, Pipelined, Short-lived only) plus all H2/H3 profiles, since most engines don't implement the heavier endpoints (JSON, DB, upload, compression).
122+
123+
The Type filter on the composite leaderboard switches between the two rankings.
119124

120125
## Why this approach
121126

0 commit comments

Comments
 (0)