Summary
Current benchmarks focus on mean execution time against reference libraries with synthetic data. While this is a solid baseline, it misses several dimensions that matter in production: tail latency, memory usage, concurrency, data realism, and scalability curves.
This is a tracking issue for a set of improvements to make zerodep benchmarks more representative of real-world workloads.
Current strengths
- Apple-to-apple comparison against 30+ reference libraries
- S/M/L data size tiers across most modules
- Network modules (httpclient, websocket, httpserver) use real local server fixtures
- readability uses real CNN/BBC/Guardian HTML pages
Gaps vs production reality
| Gap |
Impact |
Description |
| Mean-only reporting |
Low |
Report shows only mean; min/max/stddev already collected but not displayed |
| No memory metrics |
Medium |
Peak RSS matters in containerized deployments; parsers (yaml, soup, multipart, protobuf) are most affected |
| No concurrency benchmarks |
High |
httpclient tested async but single-request only; no asyncio.gather or thread-pool contention |
| Synthetic data dominance |
Medium |
Most modules use generated data; real configs, dirty HTML, edge-case payloads are absent |
| No cold start measurement |
Low |
First-call overhead (import, regex compile, table init) hidden by warmup rounds |
| Fixed S/M/L, no scale curve |
Medium |
Three data points can't reveal complexity inflection points |
Sub-issues
Summary
Current benchmarks focus on mean execution time against reference libraries with synthetic data. While this is a solid baseline, it misses several dimensions that matter in production: tail latency, memory usage, concurrency, data realism, and scalability curves.
This is a tracking issue for a set of improvements to make zerodep benchmarks more representative of real-world workloads.
Current strengths
Gaps vs production reality
asyncio.gatheror thread-pool contentionSub-issues
P1P1P1P2P2P3