Skip to content

Commit 1acd6e5

Browse files
committed
Enhance benchmark documentation with detailed performance insights
- Updated the `README.md` to include comprehensive benchmark results for various cache policies, focusing on hit rates, scan resistance, adaptation speed, throughput, and memory overhead. - Added new sections to summarize key insights from the benchmarks, aiding users in understanding the performance characteristics of different caching strategies. - Improved clarity and organization of benchmark data, ensuring it aligns with Rustdoc standards for better usability and accessibility. - Provided a policy selection guide to assist users in choosing the best cache policy based on specific use cases and workload patterns.
1 parent c72351e commit 1acd6e5

1 file changed

Lines changed: 92 additions & 21 deletions

File tree

benches/README.md

Lines changed: 92 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -124,27 +124,98 @@ Human-readable console reports (no criterion overhead, instant results):
124124

125125
## Latest Results
126126

127-
### Micro-ops (ns/op)
128-
129-
| Cache | get_hit | insert_evict |
130-
|-------|---------|--------------|
131-
| LRU | TBD | TBD |
132-
| LRU-K | TBD | TBD |
133-
| LFU | TBD | TBD |
134-
| Clock | TBD | TBD |
135-
| S3-FIFO | TBD | TBD |
136-
| 2Q | TBD | TBD |
137-
138-
### Workload Throughput (Melem/s, 200k ops)
139-
140-
| Cache | uniform | hotset | scan | zipfian |
141-
|-------|---------|--------|------|---------|
142-
| LRU | TBD | TBD | TBD | TBD |
143-
| LRU-K | TBD | TBD | TBD | TBD |
144-
| LFU | TBD | TBD | TBD | TBD |
145-
| Clock | TBD | TBD | TBD | TBD |
146-
| S3-FIFO | TBD | TBD | TBD | TBD |
147-
| 2Q | TBD | TBD | TBD | TBD |
127+
*Generated with `cargo bench --bench reports -- all` (capacity=4096, universe=16384, ops=200000)*
128+
129+
### Hit Rate Comparison
130+
131+
| Policy | uniform | hotset | scan | zipfian | scrambled | latest | scan_resist | flash_crowd |
132+
|--------|---------|--------|------|---------|-----------|--------|-------------|-------------|
133+
| LRU | 24.62% | 90.65% | 0.00% | 80.17% | 90.22% | **35.45%** | 16.75% | 88.50% |
134+
| LRU-K | 24.61% | 90.64% | 0.00% | 82.57% | **91.13%** | 25.91% | 20.28% | 89.40% |
135+
| LFU | 24.61% | 90.64% | 0.00% | 82.57% | **91.13%** | 25.82% | 20.28% | 89.40% |
136+
| Heap-LFU | 24.67% | 90.67% | **22.52%** | 74.81% | 90.03% | 26.51% | 23.42% | 85.32% |
137+
| Clock | 24.66% | 90.65% | 0.00% | 80.75% | 90.45% | 34.96% | 17.63% | 88.74% |
138+
| S3-FIFO | 24.77% | 90.63% | 0.00% | **82.84%** | 91.07% | 23.45% | **24.00%** | **89.49%** |
139+
| 2Q | 24.78% | 90.63% | 0.00% | 82.37% | 90.71% | 31.55% | 16.04% | 89.36% |
140+
141+
**Key insights**:
142+
- **uniform**: All policies equal (~24.7%) - random access reaches theoretical limit
143+
- **scan**: Heap-LFU alone survives (22.5%); all others collapse to 0%
144+
- **zipfian**: S3-FIFO wins (82.8%) - frequency-aware policies outperform recency-only
145+
- **latest**: LRU wins (35.5%) - recency-biased workload is LRU's sweet spot
146+
- **scan_resistance**: S3-FIFO leads (24.0%) - handles mixed scan + point queries best
147+
148+
### Scan Resistance
149+
150+
| Policy | Baseline | During Scan | Recovery | Score |
151+
|--------|----------|-------------|----------|-------|
152+
| LRU | 79.65% | 7.03% | 68.54% | 0.86 |
153+
| LRU-K | 79.66% | 7.69% | 78.54% | **0.99** |
154+
| LFU | 79.66% | 7.69% | 78.54% | **0.99** |
155+
| Heap-LFU | 79.21% | 21.89% | 75.79% | 0.96 |
156+
| S3-FIFO | 79.66% | 7.69% | 78.82% | **0.99** |
157+
| 2Q | 79.66% | 7.69% | 78.54% | **0.99** |
158+
| Clock | 79.66% | 6.90% | 68.54% | 0.86 |
159+
160+
**Score** = recovery / baseline (1.0 = full recovery after scan pollution)
161+
162+
- **Winners (0.99)**: LRU-K, LFU, S3-FIFO, 2Q - recover almost fully after scans
163+
- **Losers (0.86)**: LRU, Clock - scans permanently degrade performance by ~11%
164+
165+
### Adaptation Speed
166+
167+
| Policy | Ops to 50% | Ops to 80% | Stable HR |
168+
|--------|------------|------------|-----------|
169+
| LRU | 3,072 | 5,120 | **49.32%** |
170+
| LRU-K | **1,024** | **2,048** | 9.08% |
171+
| LFU | **1,024** | **2,048** | 9.08% |
172+
| Heap-LFU | 2,048 | 2,048 | 9.86% |
173+
| S3-FIFO | 8,192 | 11,264 | 44.34% |
174+
| 2Q | 3,072 | 11,264 | 33.50% |
175+
| Clock | 3,072 | 6,144 | **49.32%** |
176+
177+
- **Fastest warmup**: LRU-K reaches 80% of stable in only 2,048 ops
178+
- **Slowest warmup**: S3-FIFO takes 11,264 ops (frequency tracking needs history)
179+
- **Trade-off**: S3-FIFO has better long-term hit rates but slower warmup
180+
181+
### Throughput (ops/sec)
182+
183+
*Zipfian 1.0 workload, p99 latency in parentheses*
184+
185+
| Policy | uniform | zipfian | hotset | scan | loop_small |
186+
|--------|---------|---------|--------|------|------------|
187+
| LRU | 20.6M (125ns) | 16.2M (125ns) | 7.3M (375ns) | 4.6M (334ns) | 24.6M (42ns) |
188+
| S3-FIFO | 19.9M (167ns) | 15.0M (167ns) | 24.3M (125ns) | 23.3M (84ns) | 26.2M (42ns) |
189+
| Clock | **32.2M (84ns)** | **16.9M (84ns)** | **26.1M (83ns)** | **37.3M (42ns)** | 25.8M (42ns) |
190+
191+
- **Clock is fastest** with lowest tail latency across all workloads
192+
- **S3-FIFO** trades ~12% throughput for better hit rates
193+
- **LRU** struggles on scan workload (constant eviction overhead)
194+
195+
### Memory Overhead
196+
197+
*Shallow struct size (heap allocations not included)*
198+
199+
| Policy | Struct Size |
200+
|--------|-------------|
201+
| LRU | 56 bytes |
202+
| Clock | 72 bytes |
203+
| 2Q | 96 bytes |
204+
| LRU-K | 104 bytes |
205+
| S3-FIFO | 224 bytes |
206+
207+
S3-FIFO has 4x the metadata overhead of LRU due to its three-queue structure.
208+
209+
### Policy Selection Guide
210+
211+
| Use Case | Best Policy | Why |
212+
|----------|-------------|-----|
213+
| General purpose | **S3-FIFO** | Best hit rate on realistic workloads |
214+
| Low latency critical | **Clock** | Fastest ops, lowest p99 |
215+
| Scan-heavy / DB buffers | **S3-FIFO** or **2Q** | Excellent scan resistance |
216+
| Fast warmup needed | **LRU** | Quickest to steady state |
217+
| Memory constrained | **LRU** | Smallest metadata footprint |
218+
| Mixed read/write | **LRU-K** | Good balance across patterns |
148219

149220
## Adding a New Policy
150221

0 commit comments

Comments
 (0)