feat: Phase 3 — --track-stats, per-command HDR, sustained steady-state mode#6
Open
paulorsousa wants to merge 7 commits into
Open
feat: Phase 3 — --track-stats, per-command HDR, sustained steady-state mode#6paulorsousa wants to merge 7 commits into
paulorsousa wants to merge 7 commits into
Conversation
Companion to rusty-sidekiq's new `track_stats` ProcessorConfig field —
makes Ruby Sidekiq's default `Sidekiq[:track_stats] = true` writes
selectable from the bench tool CLI:
--track-stats # presence flag, off by default
On the wire, per processed job, the flag adds:
HSET <identity>:work <tid> <work_json> # before dispatch
HDEL <identity>:work <tid> # after dispatch
INCR stat:processed # on success
INCR stat:processed:<date> # on success
The Phase 3 benchmark passes this flag to match production
Sidekiq's wire shape; the Phase 2 sweeps keep omitting it for direct
comparability with prior runs. Off-by-default also keeps tests and ad-hoc
runs identical to historical output.
Submodule pin bumped to sidekiq-rs paulo/perf-probe (58a2cb6).
Tests:
- track_stats_default_is_off
- track_stats_flag_is_presence_based
Captures the new `brpop_latency_tx` signal from rusty-sidekiq's
`Processor::fetch`, builds a parallel HDR histogram, and surfaces the
percentiles in three places:
* Trial line: a second indented `BRPOP p50/p99/p99.9/max` row prints
just below the existing job-level line when samples were recorded.
* `TrialResult.brpop_latency: LatencyStats` for in-process consumers.
* JSON output: `brpop_latency_us` alongside `latency_us` per trial.
Empty histograms (e.g. processor never started, all jobs failed before
the first dequeue) are detected via `total_count > 0` and suppressed
from the trial-line output. Channel/collector shutdown mirrors the
existing job-latency path — drop the main sentinel after cancellation,
let the fetcher's clones drop on processor exit, then await the
collector for the final histogram.
Submodule pin bumped to sidekiq-rs paulo/perf-probe (20d2907).
Adds an opt-in workload shape for Phase 3 of the Sidekiq bench:
producer + consumer run concurrently for a fixed wall-clock window
instead of the existing pre-fill-then-drain pattern.
CLI surface:
--duration-secs N # opt in; --jobs ignored in this mode
--target-queue-depth N # soft cap on in-flight (produced − completed)
Producer (new `producer::stream_enqueue`):
- LPUSHes one job at a time on a dedicated MultiplexedConnection,
timing each call and pushing the µs into an unbounded mpsc channel.
- Soft-caps in-flight; when at the cap, sleeps 100 µs instead of
busy-spinning.
- Stops cleanly on a CancellationToken signal at duration window end.
Consumer: unchanged rusty-sidekiq Processor. BRPOP HDR (added in the
prior commit) gives the corresponding pop-side measurement so both
sides of the round-trip are observable.
Window measurement: snapshot completed/errors/elapsed the instant the
duration timer fires, BEFORE shutting down producer + consumer. The
~5 s graceful-drain at trial end would otherwise dilute jobs/s with
drain work that didn't happen under sustained-load conditions.
Verified via local smoke: stat:processed counter matches reported
total to within one job (the race between snapshot and final
INCR on the last completing job).
Header line + memory-warning guard updated so steady-state runs no
longer print a misleading `jobs=N` / "estimated peak memory" line.
Tests: `duration_secs_default_is_none_burst_mode`,
`duration_secs_opts_into_steady_state`.
Phase 3 Experiment 3 (latency-vs-fill) needs the bench tool to run on top of a 25 / 100 / 240 GB pre-filled LIST without wiping it. The companion `scripts/sidekiq-prefill-list-backlog.py` does the fill; this flag lets the steady-state trial start without DELing the queue first. Only meaningful with `--duration-secs`. Ignored in burst-then-drain mode (which always pre-fills its own backlog via the bulk pipeline, so a pre-existing one would just collide with --jobs accounting). Note for interpretation: with --no-clear and a deep pre-existing backlog, BRPOP returns the oldest pre-fill job from the tail first. Producer LPUSH-pushed jobs go to the head and are dequeued LIFO behind the entire pre-fill. The job-level e2e histogram will report whatever the dequeued job's `enqueued_at` says — usually the pre-fill time, so e2e looks artificially huge. Per-command HDR (LPUSH + BRPOP) is the meaningful measurement in this mode.
Phase 3 pre-flight (2026-06-02 on RS 25 GB at us-east-1 intra-AZ) exposed a silent-drop bug: the e2e histogram's 60-second upper bound discards every sample over a minute, and `--no-clear` runs against a pre-existing backlog routinely dequeue jobs many minutes (or hours) old. Result: `latency_us.total_count == 0` even after a full 30 s trial that processed 1.4M jobs cleanly. HDR's behavior at out-of-range is silent (record() returns Err that we don't propagate). At 60 s the design was fine for Phases 1-2 (producer and consumer in the same trial); at 1 hour the design covers Phase 3 Experiment 3 without forcing us to also report a "drop count" metric. Memory cost: HDR allocates O(precision * log(range)) buckets — going from 60 M to 3.6 G grows the bucket count by log2(60) ≈ 6 extra indexes, low single-digit KB per histogram. Negligible. Centralized as `metrics::HIST_UPPER_BOUND_US` and a shared `empty_histogram()` helper; all four prior call sites switched over.
Phase 3 pre-flight (2026-06-02) exposed a producer bottleneck: the original `stream_enqueue` ran sequential LPUSH-await-LPUSH on a single MultiplexedConnection, capping at ~290 µs RTT ≈ 3.4K jobs/s on a peered RS endpoint. At 200 workers the consumers starved waiting for new jobs — BRPOP p50 = 58 ms turned out to be queue-empty wait time, not real Redis processing time. At 2000 workers it would be worse. Fix: fan out `parallelism` (default 64) concurrent producer tasks, each owning a clone of the same MultiplexedConnection. The redis crate's multiplexer pipelines requests through the single socket, so N tasks scale producer throughput nearly linearly to ~3.4K × N jobs/s. The shared atomic counter (idx + total_pushed) gives each task a unique job index without coordination cost. CLI surface: `--producer-parallelism N` (default 64). All other flags unchanged. Default 64 sustains ~200K jobs/s comfortably and gives 2000-worker trials enough push headroom. Connection is now passed by value (Connection: Clone) instead of by &mut ref, since each task needs its own owned handle for &mut query_async.
paulorsousa
force-pushed
the
paulo/track-stats
branch
from
June 25, 2026 13:46
8d6116f to
199b9db
Compare
…hape)
The steady-state producer historically did a one-shot SADD over all
queues upfront, then bare LPUSHes per push. Ruby Sidekiq's client
(`Sidekiq::Client.push`) instead pipelines BOTH commands per push:
SADD queues <queue>
LPUSH queue:<queue> <job_json>
So our producer was emitting 1 cmd/job while a real Sidekiq client
emits 2 cmds/job. At 64K push/s sustained that's 64K SADDs/sec of
producer-side Redis work we were missing — small CPU-wise (SADD-on-
hot-member is one of Redis's cheapest commands) but methodologically
asymmetric vs the consumer side, which already goes through
rusty-sidekiq's Processor and emits faithful BRPOP + heartbeat traffic.
New flag `--producer-mode={sidekiq,raw}` (default `sidekiq`):
- `sidekiq`: per-push SADD+LPUSH pipelined in a single round trip,
matching the Ruby client wire shape. The recorded LPUSH HDR samples
the full pair's RTT, which is what a real Sidekiq client sees.
- `raw`: bare LPUSH per push (Phase 1/2 baseline). Use when comparing
Phase 3 trials against pre-Phase-3 sweeps where the producer ran
in the bare-LPUSH shape.
Phase 3 Apollo-faithful results should re-run with the default
`sidekiq` mode; the original Experiment 1 sweep (committed in
2cb2004) was effectively `--producer-mode=raw`. Expect a ~3-5%
throughput drop at saturated cells once corrected.
The upfront one-shot SADD still happens in both modes — primes the
queues set so the per-push SADD is always a no-op fast path on a hot
member from the very first push.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four additions to the bench tool that drive Phase 3 of the Sidekiq plan (240 GB single-shard scenario). Depends on redis-performance/sidekiq-rs#4 landing first — the submodule pin in this PR points at that branch.
--track-stats— presence flag. Off by default (Phase 2 reproductions keep their lean wire shape). When set, forwards to rusty-sidekiq's newProcessorConfig.track_stats, which makes every processed job emit the four Redis commands Ruby Sidekiq'sSidekiq[:track_stats] = true(its default) generates.Per-command HDR percentiles for BRPOP and LPUSH, alongside the existing job-level (enqueue → perform) histogram. BRPOP samples are collected from rusty-sidekiq's new
brpop_latency_tx; LPUSH samples are collected from the new steady-state producer. Surfaced in the trial-line output, the summary table, and the JSON (brpop_latency_us/lpush_latency_us/latency_us).Sustained steady-state mode — opt-in via
--duration-secs N. Producer + consumer run concurrently for a fixed wall-clock window. Producer LPUSHes one job at a time on a dedicatedMultiplexedConnection, soft-caps in-flight at--target-queue-depth. Required for latency-stability ask — the existing burst-then-drain shape doesn't measure sustained behavior. Window measurement snapshotscompleted/errors/elapsedthe instant the duration timer fires, before the ~5 s graceful-drain at trial end, so jobs/s reflects only the sustained window.--no-clear— opt-in skip of the per-trialDEL queue:<name>. Required by Experiment 3 (latency-vs-fill at 25/100/240 GB) so a separately-staged backlog survives trial start. Only meaningful with--duration-secs; ignored in burst-then-drain mode.CLI surface added
--track-statsProcessorConfig.track_stats--duration-secs N--jobsis ignored;--warmup-jobsis skipped--target-queue-depth Nproduced − completed--no-clearImplementation notes
producer::stream_enqueuefor the steady-state producer; the existingbulk_enqueue(pipelined pre-fill) is unchanged.Test plan
🤖 Generated with Claude Code