Skip to content

Improve benchmark realism for production scenarios #80

@Oaklight

Description

@Oaklight

Summary

Current benchmarks focus on mean execution time against reference libraries with synthetic data. While this is a solid baseline, it misses several dimensions that matter in production: tail latency, memory usage, concurrency, data realism, and scalability curves.

This is a tracking issue for a set of improvements to make zerodep benchmarks more representative of real-world workloads.

Current strengths

  • Apple-to-apple comparison against 30+ reference libraries
  • S/M/L data size tiers across most modules
  • Network modules (httpclient, websocket, httpserver) use real local server fixtures
  • readability uses real CNN/BBC/Guardian HTML pages

Gaps vs production reality

Gap Impact Description
Mean-only reporting Low Report shows only mean; min/max/stddev already collected but not displayed
No memory metrics Medium Peak RSS matters in containerized deployments; parsers (yaml, soup, multipart, protobuf) are most affected
No concurrency benchmarks High httpclient tested async but single-request only; no asyncio.gather or thread-pool contention
Synthetic data dominance Medium Most modules use generated data; real configs, dirty HTML, edge-case payloads are absent
No cold start measurement Low First-call overhead (import, regex compile, table init) hidden by warmup rounds
Fixed S/M/L, no scale curve Medium Three data points can't reveal complexity inflection points

Sub-issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions