Skip to content

Commit 960fd7b

Browse files
authored
Merge pull request #586 from MDA2AV/add-libreactorng
Add libreactorng engine
2 parents 979ea8e + c288f55 commit 960fd7b

18 files changed

Lines changed: 365 additions & 1 deletion

frameworks/libreactorng/Dockerfile

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
FROM ubuntu:24.04 AS build
2+
3+
ENV DEBIAN_FRONTEND=noninteractive
4+
RUN apt-get update && apt-get install -y --no-install-recommends \
5+
build-essential autoconf automake libtool libltdl-dev pkg-config \
6+
git ca-certificates \
7+
liburing-dev libssl-dev \
8+
&& rm -rf /var/lib/apt/lists/*
9+
10+
# Build libreactorng from source. The library is small (~200 KiB) and has
11+
# zero third-party runtime deps, so we build it fresh every image rather
12+
# than relying on a distro package (none ships it).
13+
WORKDIR /tmp/libreactorng
14+
RUN git clone --depth 1 https://github.com/fredrikwidlund/libreactorng.git . \
15+
&& ./autogen.sh \
16+
&& ./configure CFLAGS="-O3 -march=native" \
17+
&& make -j"$(nproc)" \
18+
&& make install \
19+
&& ldconfig
20+
21+
# Build the bench server against the installed libreactor.
22+
WORKDIR /app
23+
COPY src/server.c .
24+
RUN gcc -O3 -march=native -Wall -Wextra -o server server.c \
25+
$(pkg-config --libs --cflags libreactor)
26+
27+
FROM ubuntu:24.04
28+
RUN apt-get update && apt-get install -y --no-install-recommends \
29+
liburing2 libssl3 libltdl7 ca-certificates \
30+
&& rm -rf /var/lib/apt/lists/*
31+
COPY --from=build /usr/local/lib/libreactor* /usr/local/lib/
32+
COPY --from=build /app/server /server
33+
RUN ldconfig
34+
EXPOSE 8080
35+
CMD ["/server"]

frameworks/libreactorng/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# libreactorng
2+
3+
[libreactorng](https://github.com/fredrikwidlund/libreactorng) — Fredrik Widlund's io_uring-native event framework, the successor to the epoll-based libreactor that held top placements on TechEmpower plaintext/JSON for years.
4+
5+
## Stack
6+
7+
- **Language:** C
8+
- **Engine:** io_uring (Linux)
9+
- **Dependencies:** libreactor (built from source), liburing, libssl
10+
- **Build:** `ubuntu:24.04``ubuntu:24.04`
11+
12+
## Endpoints
13+
14+
| Endpoint | Method | Description |
15+
|----------|--------|-------------|
16+
| `/pipeline` | GET | Returns `ok` (text/plain) |
17+
| `/baseline11` | GET | Sum of integer query args |
18+
| `/baseline11` | POST | Sum of query args + integer body |
19+
| `/baseline2` | GET | Same as baseline11 GET (parity with H/2 profile) |
20+
21+
## Notes
22+
23+
- Single `on_request` callback dispatches on `session->request.target`; libreactor parses method / target / body for us.
24+
- One reactor per logical CPU in the container's affinity mask, forked up front. Each worker creates its own `SO_REUSEPORT` socket so the kernel distributes incoming accepts.
25+
- Requires `--security-opt seccomp=unconfined` (default Docker seccomp blocks several io_uring ops). The harness adds this automatically for frameworks declaring `"engine": "io_uring"` in `meta.json`.
26+
- Response bodies computed on the stack are safe — `http_write_response` copies through `stream_allocate` before returning control to the event loop.
27+
28+
## Known limitation
29+
30+
libreactor's HTTP server keeps connections open unconditionally — it ignores the `Connection` request header and the only teardown API (`server_disconnect``stream_close`) is abortive. That makes the TCP-fragmentation validation checks in `scripts/validate.sh` (which send `Connection: close` and then `recv` until EOF) time out waiting for a close that never comes. Plain `curl`-driven checks are fine because curl uses `Content-Length`. Fixing this cleanly needs a write-completion hook in libreactor's `stream_t`, which isn't exposed in the public API.

frameworks/libreactorng/meta.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"display_name": "libreactorng",
3+
"language": "C",
4+
"type": "engine",
5+
"engine": "io_uring",
6+
"description": "libreactorng — Fredrik Widlund's io_uring-native event framework, the successor to the long-running epoll-based libreactor. Built directly on Linux io_uring syscalls with zero third-party runtime deps. Minimal server dispatches /pipeline, /baseline11, /baseline2 via the built-in HTTP parser; one reactor process per logical CPU via SO_REUSEPORT.",
7+
"repo": "https://github.com/fredrikwidlund/libreactorng",
8+
"enabled": true,
9+
"tests": [
10+
"baseline",
11+
"pipelined",
12+
"limited-conn"
13+
],
14+
"maintainers": []
15+
}
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
/* HttpArena minimal bench server on top of libreactorng.
2+
*
3+
* Uses libreactor's built-in HTTP parser (session->request.{method,target,body})
4+
* so this file is just dispatch + integer arithmetic.
5+
*
6+
* Multi-process: one libreactor per logical CPU in the container's affinity
7+
* mask, each listening on its own SO_REUSEPORT socket so the kernel balances
8+
* accepted connections across workers.
9+
*/
10+
#define _GNU_SOURCE
11+
#include <stdio.h>
12+
#include <stdlib.h>
13+
#include <string.h>
14+
#include <stdint.h>
15+
#include <signal.h>
16+
#include <unistd.h>
17+
#include <sched.h>
18+
#include <sys/socket.h>
19+
#include <netinet/in.h>
20+
#include <netinet/tcp.h>
21+
22+
#include <reactor.h>
23+
24+
/* Parse a leading signed integer. Skips whitespace, stops at the first
25+
* non-digit. Matches the contract of nginx/h2o reference implementations. */
26+
static int64_t parse_int(const char *p, const char *end)
27+
{
28+
while (p < end && (*p == ' ' || *p == '\t' || *p == '\r' || *p == '\n')) p++;
29+
int neg = 0;
30+
if (p < end && *p == '-') { neg = 1; p++; }
31+
int64_t n = 0;
32+
while (p < end && *p >= '0' && *p <= '9') {
33+
n = n * 10 + (*p - '0');
34+
p++;
35+
}
36+
return neg ? -n : n;
37+
}
38+
39+
/* Sum integer values across "k1=v1&k2=v2..." — ignores keys, non-integer
40+
* values silently skip. */
41+
static int64_t sum_query(const char *p, size_t len)
42+
{
43+
const char *end = p + len;
44+
int64_t sum = 0;
45+
while (p < end) {
46+
const char *eq = memchr(p, '=', end - p);
47+
if (!eq) break;
48+
const char *v = eq + 1;
49+
const char *amp = memchr(v, '&', end - v);
50+
const char *ve = amp ? amp : end;
51+
sum += parse_int(v, ve);
52+
p = amp ? amp + 1 : end;
53+
}
54+
return sum;
55+
}
56+
57+
static void on_request(reactor_event_t *event)
58+
{
59+
server_session_t *s = (server_session_t *) event->data;
60+
string_t method = s->request.method;
61+
string_t target = s->request.target;
62+
63+
/* Split target at the first '?' to get path + query string. */
64+
const char *t = (const char *) data_base(target);
65+
size_t t_len = data_size(target);
66+
const char *q = memchr(t, '?', t_len);
67+
size_t path_len = q ? (size_t)(q - t) : t_len;
68+
const char *qs = q ? q + 1 : NULL;
69+
size_t qs_len = q ? (t_len - path_len - 1) : 0;
70+
71+
/* Connection: close is NOT honored — libreactor keeps the session open
72+
* after the response, and the only teardown primitive it exposes
73+
* (server_disconnect → stream_close) is abortive: it closes before the
74+
* queued response bytes reach the socket. That causes the TCP
75+
* fragmentation validation checks to time out reading for an EOF that
76+
* never comes. Fixing it cleanly would need a write-completion hook in
77+
* libreactor's stream, which isn't in the public API. Known limitation. */
78+
79+
if (path_len == 9 && memcmp(t, "/pipeline", 9) == 0) {
80+
server_plain(s, data_string("ok"), NULL, 0);
81+
return;
82+
}
83+
84+
if (path_len == 11 && memcmp(t, "/baseline11", 11) == 0) {
85+
int64_t sum = qs ? sum_query(qs, qs_len) : 0;
86+
if (string_equal(method, string("POST"))) {
87+
const char *b = (const char *) data_base(s->request.body);
88+
size_t b_len = data_size(s->request.body);
89+
if (b_len > 0) sum += parse_int(b, b + b_len);
90+
}
91+
/* Stack buffer is safe: http_write_response copies via stream_allocate
92+
* before this handler returns to the event loop. */
93+
char buf[32];
94+
int n = snprintf(buf, sizeof(buf), "%lld", (long long) sum);
95+
server_plain(s, data(buf, n), NULL, 0);
96+
return;
97+
}
98+
99+
if (path_len == 10 && memcmp(t, "/baseline2", 10) == 0) {
100+
int64_t sum = qs ? sum_query(qs, qs_len) : 0;
101+
char buf[32];
102+
int n = snprintf(buf, sizeof(buf), "%lld", (long long) sum);
103+
server_plain(s, data(buf, n), NULL, 0);
104+
return;
105+
}
106+
107+
server_respond(s, string("404 Not Found"), string("text/plain"),
108+
data_string("Not Found"), NULL, 0);
109+
}
110+
111+
static int make_reuseport_socket(int port)
112+
{
113+
int s = socket(AF_INET, SOCK_STREAM, 0);
114+
if (s < 0) { perror("socket"); exit(1); }
115+
int on = 1;
116+
setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
117+
setsockopt(s, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on));
118+
setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on));
119+
struct sockaddr_in addr = {
120+
.sin_family = AF_INET,
121+
.sin_port = htons(port),
122+
.sin_addr.s_addr = htonl(INADDR_ANY),
123+
};
124+
if (bind(s, (struct sockaddr *) &addr, sizeof(addr)) < 0) {
125+
perror("bind"); exit(1);
126+
}
127+
if (listen(s, 4096) < 0) { perror("listen"); exit(1); }
128+
return s;
129+
}
130+
131+
int main(void)
132+
{
133+
signal(SIGPIPE, SIG_IGN);
134+
135+
/* Respect Docker --cpuset-cpus via the affinity mask. sysconf() would
136+
* report the host CPU count which isn't what we want inside a pinned
137+
* container. */
138+
cpu_set_t cs;
139+
int workers = 1;
140+
if (sched_getaffinity(0, sizeof(cs), &cs) == 0) workers = CPU_COUNT(&cs);
141+
if (workers < 1) workers = 1;
142+
143+
for (int i = 1; i < workers; i++) {
144+
pid_t pid = fork();
145+
if (pid < 0) { perror("fork"); break; }
146+
if (pid == 0) break;
147+
}
148+
149+
int fd = make_reuseport_socket(8080);
150+
151+
server_t server;
152+
reactor_construct();
153+
server_construct(&server, on_request, NULL);
154+
server_open_socket(&server, fd);
155+
reactor_loop();
156+
server_destruct(&server);
157+
reactor_destruct();
158+
return 0;
159+
}

site/data/baseline-4096.json

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -508,6 +508,26 @@
508508
"status_4xx": 0,
509509
"status_5xx": 0
510510
},
511+
{
512+
"framework": "libreactorng",
513+
"language": "C",
514+
"rps": 4086741,
515+
"avg_latency": "1.00ms",
516+
"p99_latency": "2.91ms",
517+
"cpu": "6398.5%",
518+
"memory": "113MiB",
519+
"connections": 4096,
520+
"threads": 64,
521+
"duration": "5s",
522+
"pipeline": 1,
523+
"bandwidth": "444.12MB/s",
524+
"input_bw": "315.69MB/s",
525+
"reconnects": 0,
526+
"status_2xx": 20433705,
527+
"status_3xx": 0,
528+
"status_4xx": 0,
529+
"status_5xx": 0
530+
},
511531
{
512532
"framework": "mark",
513533
"language": "PHP",

site/data/baseline-512.json

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -508,6 +508,26 @@
508508
"status_4xx": 0,
509509
"status_5xx": 0
510510
},
511+
{
512+
"framework": "libreactorng",
513+
"language": "C",
514+
"rps": 3749681,
515+
"avg_latency": "136us",
516+
"p99_latency": "375us",
517+
"cpu": "6372.7%",
518+
"memory": "71MiB",
519+
"connections": 512,
520+
"threads": 64,
521+
"duration": "5s",
522+
"pipeline": 1,
523+
"bandwidth": "407.50MB/s",
524+
"input_bw": "289.65MB/s",
525+
"reconnects": 0,
526+
"status_2xx": 18748406,
527+
"status_3xx": 0,
528+
"status_4xx": 0,
529+
"status_5xx": 0
530+
},
511531
{
512532
"framework": "mark",
513533
"language": "PHP",

site/data/current.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
"docker": "29.3.0",
1111
"docker_runtime": "runc",
1212
"governor": "performance",
13-
"commit": "f4baef2e",
13+
"commit": "94a3aad8",
1414
"tcp": {
1515
"lo_mtu": "1500",
1616
"congestion": "cubic",

site/data/frameworks.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,13 @@
255255
"type": "tuned",
256256
"engine": "netty"
257257
},
258+
"libreactorng": {
259+
"dir": "libreactorng",
260+
"description": "libreactorng \u2014 Fredrik Widlund's io_uring-native event framework, the successor to the long-running epoll-based libreactor. Built directly on Linux io_uring syscalls with zero third-party runtime deps. Minimal server dispatches /pipeline, /baseline11, /baseline2 via the built-in HTTP parser; one reactor process per logical CPU via SO_REUSEPORT.",
261+
"repo": "https://github.com/fredrikwidlund/libreactorng",
262+
"type": "engine",
263+
"engine": "io_uring"
264+
},
258265
"mark": {
259266
"dir": "mark",
260267
"description": "Mark is a high performance API micro framework based on workerman, helps you quickly write APIs with PHP.",

site/data/limited-conn-4096.json

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -508,6 +508,26 @@
508508
"status_4xx": 0,
509509
"status_5xx": 0
510510
},
511+
{
512+
"framework": "libreactorng",
513+
"language": "C",
514+
"rps": 2486388,
515+
"avg_latency": "1.51ms",
516+
"p99_latency": "4.18ms",
517+
"cpu": "5524.0%",
518+
"memory": "165MiB",
519+
"connections": 4096,
520+
"threads": 64,
521+
"duration": "5s",
522+
"pipeline": 1,
523+
"bandwidth": "270.21MB/s",
524+
"input_bw": "192.07MB/s",
525+
"reconnects": 1242393,
526+
"status_2xx": 12431941,
527+
"status_3xx": 0,
528+
"status_4xx": 0,
529+
"status_5xx": 0
530+
},
511531
{
512532
"framework": "mark",
513533
"language": "PHP",

site/data/limited-conn-512.json

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -508,6 +508,26 @@
508508
"status_4xx": 0,
509509
"status_5xx": 0
510510
},
511+
{
512+
"framework": "libreactorng",
513+
"language": "C",
514+
"rps": 2331986,
515+
"avg_latency": "204us",
516+
"p99_latency": "1.15ms",
517+
"cpu": "5259.4%",
518+
"memory": "91MiB",
519+
"connections": 512,
520+
"threads": 64,
521+
"duration": "5s",
522+
"pipeline": 1,
523+
"bandwidth": "253.42MB/s",
524+
"input_bw": "180.14MB/s",
525+
"reconnects": 1165994,
526+
"status_2xx": 11659931,
527+
"status_3xx": 0,
528+
"status_4xx": 0,
529+
"status_5xx": 0
530+
},
511531
{
512532
"framework": "mark",
513533
"language": "PHP",

0 commit comments

Comments
 (0)