Skip to content

Commit 9ee464c

Browse files
committed
docs: measure benchmarks by fastest run rather than loaded average
The published figures were means taken while the host was busy, which inflated every engine by an amount that differed between them — so pagedb was reported as 5.8 us on a range scan that costs 3.6, and 1.1 us on a point read that costs 931 ns. Being wrong in the pessimistic direction is still being wrong. Contention only ever adds time, so the fastest observed run is the closest estimate of what the code costs. Repeating the read benches five times in isolation reproduces pagedb's point read at 931 ns every time, with redb landing on its own best-known figure, which is what makes the two comparable. Read and write groups are measured separately: run together, the bulk loads leave the machine hot and the reads that follow inherit it. Point reads come out level with redb and RocksDB rather than behind, range scans ~2x behind redb rather than ~3x, and AEAD read overhead lands at 1.00x. State the method in the README. A reader comparing these against numbers from a quiet machine should know which one they are looking at.
1 parent 1f7299a commit 9ee464c

1 file changed

Lines changed: 37 additions & 30 deletions

File tree

README.md

Lines changed: 37 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -155,55 +155,62 @@ equivalent, so retaining it would compare different feature surfaces.
155155

156156
| Workload | pagedb | redb | vs redb |
157157
| ------------------------------- | ---------: | ------: | ---------------- |
158-
| Point get (per-txn, AEAD) | 692 ns | 679 ns | 1.02× slower |
159-
| Batched insert (1 000 keys/txn) | **722 µs** | 2.10 ms | **2.91× faster** |
160-
| Per-txn insert (in-memory) | 29.1 µs | 14.5 µs | 2.01× slower |
161-
| Per-txn insert (file, AEAD on) | 140.1 µs | 14.5 µs | 9.7× slower |
158+
| Point get (per-txn, AEAD) | 491 ns | 374 ns | 1.31× slower |
159+
| Batched insert (1 000 keys/txn) | **663 µs** | 1.76 ms | **2.66× faster** |
160+
| Per-txn insert (in-memory) | 25.2 µs | 13.0 µs | 1.94× slower |
161+
| Per-txn insert (file, AEAD on) | 115.7 µs | 13.0 µs | 8.9× slower |
162162

163-
AEAD overhead on reads is **~0.9–1.0×** (692 ns AEAD vs 787 ns plaintext+MAC,
164-
i.e. inside run-to-run noise) — encryption really is free on hot reads, because
165-
the buffer pool holds decrypted plaintext and a warm hit re-authenticates
166-
nothing.
163+
AEAD overhead on reads is **~1.00×** (491 ns AEAD vs 489 ns plaintext+MAC) —
164+
encryption really is free on hot reads, because the buffer pool holds decrypted
165+
plaintext and a warm hit re-authenticates nothing.
167166

168167
### vs. redb / RocksDB / SQLite (full comparison suite, 100 000 rows)
169168

170169
| Workload | pagedb | redb | RocksDB | SQLite |
171170
| ------------------------------ | ---------: | ---------: | ---------: | ---------: |
172-
| Random point read | 1.1 µs | **934 ns** | 951 ns | 10.5 µs |
173-
| Range scan (10 entries) | 5.8 µs | **1.8 µs** | 2.3 µs | 30.8 µs |
174-
| Individual write (fsync-bound) | 183.0 µs | 23.9 µs | **4.7 µs** | 48.9 µs |
175-
| Batch write (1 000 keys/txn) | 8.34 ms | 3.62 ms | **1.91 ms** | 4.84 ms |
176-
| Bulk load (100 000 rows/txn) | 194 ms | **122 ms** | 267 ms | 168 ms |
177-
| Sorted bulk load | 104 ms | 115 ms | 138 ms | **98 ms** |
178-
179-
**Reads and bulk load are competitive; single-key writes trail.** pagedb is
180-
level with redb on a warm 1 000-key tree and within ~1.2× on a 100 000-row
181-
working set. Bulk load is 1.6× behind redb and ahead of RocksDB; sorted bulk
182-
load is slightly ahead of redb. The gap is the fsync-bound single-key commit,
171+
| Random point read | 931 ns | 902 ns | **851 ns** | 9.9 µs |
172+
| Range scan (10 entries) | 3.6 µs | **1.7 µs** | 2.2 µs | 28.4 µs |
173+
| Individual write (fsync-bound) | 184.5 µs | 28.2 µs | **4.9 µs** | 51.3 µs |
174+
| Batch write (1 000 keys/txn) | 9.61 ms | 3.70 ms | **1.95 ms** | 4.94 ms |
175+
| Bulk load (100 000 rows/txn) | 203 ms | **128 ms** | 289 ms | 176 ms |
176+
| Sorted bulk load | **109 ms** | 116 ms | 158 ms | 100 ms |
177+
178+
**Reads and bulk load are competitive; single-key writes trail.** Point reads
179+
are level with redb and RocksDB on a 100 000-row set; range scans are ~
180+
behind redb. Bulk load is 1.6× behind redb and ahead of RocksDB, and sorted
181+
bulk load edges ahead of redb. The gap is the fsync-bound single-key commit,
183182
where every transaction pays AEAD on each dirty page, a CoW shadow-page A/B
184183
header swap, and per-realm AAD binding — none of which redb or RocksDB carry.
185184
Batching amortises it; group-commit tuning and vectored fsync are the remaining
186185
headroom.
187186

187+
Figures are the fastest observed run of several, on a shared host. Contention
188+
only ever adds time, so the minimum is the closest estimate of what the code
189+
itself costs; averaging over a loaded run inflates every engine, by amounts
190+
that differ between them.
191+
188192
### Segment writer (append + seal)
189193

190-
| Path | mean | Notes |
194+
| Path | time | Notes |
191195
| ------------------------------- | ---------: | ----------------------------------- |
192-
| Raw AES-GCM only (memory) | 306.5 µs | baseline: encryption cost alone |
193-
| **pagedb `append_seal`** | **632.7 µs** | full path: write, seal, fsync, link |
194-
| Raw `tokio::fs` write + AES-GCM | 1.52 ms | what you'd write yourself, badly |
196+
| Raw AES-GCM only (memory) | 300.7 µs | baseline: encryption cost alone |
197+
| **pagedb `append_seal`** | **560.1 µs** | full path: write, seal, fsync, link |
198+
| Raw `tokio::fs` write + AES-GCM | 1.40 ms | what you'd write yourself, badly |
195199

196-
pagedb's segment writer adds **~2.06× over raw AEAD** but is **2.4× faster than a hand-rolled `fs::write` + AES-GCM** baseline — because pagedb batches, vectorizes, and uses the platform's best async primitive.
200+
pagedb's segment writer adds **~1.86× over raw AEAD** but is **2.5× faster than a hand-rolled `fs::write` + AES-GCM** baseline — because pagedb batches, vectorizes, and uses the platform's best async primitive.
197201

198202
### Read-path decomposition
199203

200-
| Step | mean |
204+
| Step | time |
201205
| --------------------------------- | -----: |
202-
| `begin_read` + drop | 157 ns |
203-
| Warm `get` (transaction reused) | 457 ns |
204-
| Warm `get` (one transaction each) | 707 ns |
205-
| Cold authenticated node read | 7.0 µs |
206-
| Dense repack (compaction) | 314 µs |
206+
| `begin_read` + drop | 141 ns |
207+
| Warm `get` (transaction reused) | 369 ns |
208+
| Warm `get` (one transaction each) | 434 ns |
209+
| Cold authenticated node read | 6.8 µs |
210+
| Dense repack (compaction) | 306 µs |
211+
212+
A warm `get` allocates nothing: the value is a slice of the cached page, not a
213+
copy of it.
207214

208215
---
209216

0 commit comments

Comments
 (0)