Skip to content

Commit 5f78ddb

Browse files
HanSur94HanSur94
authored
test(benchmarks): isolated microbenchmarks for core hot-path kernels (#213)
Adds five focused, deterministic micro-benchmarks isolating core hot-path kernels (MinMax/LTTB downsampling, binary_search, violation_cull, FastSenseDataStore range query, delimited CSV parse), each guarded by a machine-independent scaling gate. New files only; no library/production code changed. Includes benchmarks/.reports/coverage.md tracking the ranked surface and remaining gaps.
1 parent 3dddde2 commit 5f78ddb

6 files changed

Lines changed: 966 additions & 0 deletions

File tree

‎benchmarks/.reports/coverage.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Benchmark coverage notes
2+
3+
Tracks what `/bench-evolve` has added and which performance-critical paths
4+
still lack isolated benchmark coverage. Newest entries first.
5+
6+
## Performance-critical surface (ranked) and coverage status
7+
8+
| Path | Why it matters | Coverage |
9+
|------|----------------|----------|
10+
| **Downsampling kernels** (`minmax_downsample` / `lttb_downsample` → `minmax_core_mex` / `lttb_core_mex`) | Runs on every render + every zoom/pan, over the full dataset (≤50M pts). The library's core value. | ✅ `bench_downsample_kernels.m` (isolated, both methods) — *added 2026-06-24*. Also exercised indirectly in `benchmark.m` / `benchmark_zoom.m` / `benchmark_features.m` (render-mixed). |
11+
| **`binary_search`** (`binary_search_mex`) | Range-window lookup on raw full-N sorted arrays; on the resolve path for every zoom/pan + every tag range query (`FastSense.m`, `FastSenseToolbar.m`, `SensorTag.m`). | ✅ `bench_binary_search.m` (isolated, log-scaling gate) — *added 2026-06-24*. |
12+
| **Violation marker path** (`violation_cull` → `violation_cull_mex`; constant + step-function branches) | Fused detect+cull on every threshold render/zoom for thresholds with `ShowViolations` (incl. time-varying step thresholds). | ✅ `bench_violation_cull.m` (isolated, both branches, linear-scaling gate) — *added 2026-06-24*. |
13+
| **Disk range-query** (`FastSenseDataStore.getRange`, `resolve_disk_mex`) | Out-of-core read on every zoom/pan of a disk-backed line. The large-data story's hot read path. | ✅ `bench_datastore_range.m` (fixed-window query, indexed-read gate) — *added 2026-06-24*. Store create/slice still only exploratory (`benchmark_datastore.m` / `profile_datastore.m`). |
14+
| **CSV ingestion** (`dispatchDelimitedParse_` → `delimited_parse_mex`, fallback `readRawDelimited_`) | Front door for raw sensor data into the Tag pipeline; MEX is ~10–40× the textscan fallback. Slow parse = slow load for big logs. | ✅ `bench_delimited_parse.m` (isolated, row-scaling gate) — *added 2026-06-24*. |
15+
| **Pyramid build** (`FastSense.buildPyramidLevel`) | Multi-level pre-downsample cache built at render for large lines (powers O(1) zoom). Full-N at render. | ◐ Partial — it is essentially `minmax_downsample` per level (already gated by `bench_downsample_kernels.m`) + chunked disk reads; only *memory*-benchmarked end-to-end (`benchmark_memory.m`). Low marginal value to isolate; private method. |
16+
| **`to_step_function_mex`** | SIMD step-function conversion — a compiled, deployed, correctness-tested kernel (`TestToStepFunctionMex`). | ⏸️ **DEFERRED** — no confirmed production caller. `MonitorTag.recompute_` emits a binary vector (no step conversion); `StateTag.getXY` is pass-through; only the test suite calls it. The `dispatchDelimitedParse_` comment citing it is stale. **Investigate whether it's still wired into any render path (or is vestigial) before benchmarking.** |
17+
| **Tag layer** (SensorTag/MonitorTag/CompositeTag getXY, resolve, append) | Live-tick recompute path. | ✅ `bench_sensortag_getxy`, `bench_monitortag_tick`, `bench_monitortag_append`, `bench_compositetag_merge`, `bench_consumer_migration_tick`, `bench_tag_pipeline_1k`. |
18+
| **Dashboard refresh / load** | Live dashboard refresh rate. | ✅ `bench_dashboard`, `bench_dashboard_live`, `bench_dashboard_load`. |
19+
| **Full render vs plot(), zoom/pan, memory, features** | End-to-end render comparison. | ✅ `benchmark.m`, `benchmark_zoom.m`, `benchmark_memory.m`, `benchmark_features.m`. |
20+
21+
## Change log
22+
23+
### 2026-06-24 — `bench_downsample_kernels.m`
24+
- **Gap closed:** isolated downsampling-kernel microbenchmark. Previously the
25+
only coverage was a single `minmax_downsample(x,y,1000)` call buried inside
26+
the render-heavy `benchmark.m`; **LTTB had zero coverage anywhere**.
27+
- **What it does:** times `minmax_downsample` and `lttb_downsample` as pure
28+
computation (no figure/render) across a 10K→10M size sweep, same ~2000-pt
29+
output budget for both, reporting per-call ms + throughput (Mpts/s).
30+
- **Gate:** machine-independent — fits the empirical log-log scaling exponent
31+
over the large-N portion and asserts it stays ≤ 1.3 (catches super-linear
32+
creep regardless of host speed).
33+
- **Reaches the private wrappers** by `cd`-ing into `libs/FastSense/private`
34+
(current folder is always searched, even when named `private`) — works in
35+
both MATLAB and Octave, unlike the `addpath(.../private)` trick that
36+
`benchmark.m` uses (Octave-only; MATLAB rejects private dirs on the path).
37+
- **First run (MATLAB R2025b, MEX active):** MinMax ~764 Mpts/s @ 10M,
38+
LTTB ~349 Mpts/s @ 10M; scaling exponents 0.88 / 0.87 → PASS.
39+
40+
### 2026-06-24 — `bench_binary_search.m`
41+
- **Gap closed:** isolated range-lookup microbenchmark. `binary_search` is the
42+
most broadly-used uncovered kernel — the resolve/zoom window lookup in
43+
`FastSense.m` (4060/4103/4178/4460), timestamp lookup (1683), toolbar
44+
click/range, and tag range resolve (`SensorTag.m:152`), on raw full-N sorted
45+
arrays, every zoom/pan. Re-prioritised **above** the violation marker path
46+
this run: `violation_cull` runs on already-downsampled display data
47+
(small-N, per-frame), whereas `binary_search` hits the raw full-N array.
48+
- **What it does:** times 20k scalar `'left'`/`'right'` lookups across a
49+
10K→50M sweep, reporting per-query µs + Mqueries/s.
50+
- **Gate:** machine-independent — fits the per-query log-log exponent over the
51+
large-N portion and asserts it stays ≤ 0.6, catching the catastrophic
52+
O(log N)→O(N) (linear-scan) regression regardless of host speed.
53+
- **MEX detection caveat (baked into the bench):** `binary_search_mex` lives in
54+
`libs/FastSense/private` and is visible to `binary_search.m` (its parent) but
55+
NOT from `benchmarks/`. A plain `exist('binary_search_mex','file')` in the
56+
bench misreports as fallback; the bench instead checks the built binary for
57+
the current platform on disk (`['binary_search_mex.' mexext]`).
58+
- **First run (MATLAB R2025b, MEX active):** ~0.95 µs/query @ 10K → ~1.8 µs @ 50M;
59+
exponent 0.09 (firmly logarithmic), growth 1.9× over the sweep → PASS.
60+
61+
### 2026-06-24 — `bench_violation_cull.m`
62+
- **Gap closed:** isolated threshold-marker microbenchmark. `violation_cull` is
63+
the fused detect+cull kernel called per (threshold x line) on every
64+
render/zoom (`FastSense.m:1368/1371`, `4468/4471`); only
65+
`bench_event_marker_regression.m` touched a neighbouring path before.
66+
- **What it does:** times both threshold branches as pure computation — a
67+
constant threshold (thX=0 sentinel) and a 5-knot step-function threshold —
68+
across a 1K→1M input sweep, reporting per-call ms + throughput. Annotated
69+
that production input is the displayed/downsampled data (~few thousand pts,
70+
the low end); upper sizes verify linear scaling.
71+
- **Gate:** machine-independent — log-log scaling exponent over N >= 1e4 must
72+
stay <= 1.3 (catches super-linear creep in detect+cull).
73+
- **Reaches the private wrapper** via the `cd`-into-`libs/FastSense/private`
74+
trick (see [[benchmarking-private-mex-kernels]]).
75+
- **First run (MATLAB R2025b, MEX active):** constant ~288 Mpts/s @ 1M, step
76+
~261 Mpts/s @ 1M; at the realistic ~1K size both are sub-10 µs. Scaling
77+
exponents 0.93 / 0.92 → PASS.
78+
79+
### 2026-06-24 — `bench_datastore_range.m`
80+
- **Gap closed:** focused, deterministic gate for the disk-backed range-query
81+
path (`FastSenseDataStore.getRange`), which every zoom/pan on a disk-backed
82+
line hits. Previously only exploratory scripts existed (`benchmark_datastore.m`
83+
is a .mat-vs-SQLite sweep and Linux-only — shells out to `free`;
84+
`profile_datastore.m` is a profiler script). No figure needed.
85+
- **What it does:** builds a chunked store at each size, fires fixed-size view
86+
windows (width scaled so each query returns ~10k pts regardless of N), times
87+
`getRange`, and reports create time + per-query ms + queries/s.
88+
- **Gate:** machine-independent — the indexed store must read only the window,
89+
so per-query time must stay ~constant as the dataset grows; asserts the
90+
query-time-vs-total-N exponent <= 0.5 (a full-scan regression → ~1.0).
91+
- **Robustness:** warms up a throwaway store first (absorbs one-time SQLite/MEX
92+
init), and always `cleanup()`s each store (try/catch + post-loop) so temp DBs
93+
never leak even if the gate trips.
94+
- **First run (MATLAB R2025b, mksqlite active):** query time flat at ~0.16 ms
95+
across 100K→5M (50× more data), exponent −0.11, exactly 10002 pts/query → PASS.
96+
97+
### Pivot note this run
98+
Intended target was `to_step_function_mex`, but a fresh survey found it has **no
99+
confirmed production caller** (see table) — benchmarking it would violate the
100+
"path that matters" rule. Deferred it (flagged for investigation) and pivoted to
101+
the disk range-query gate instead.
102+
103+
### 2026-06-24 — `bench_delimited_parse.m`
104+
- **Gap closed:** isolated CSV-ingestion microbenchmark. `delimited_parse_mex`
105+
(via `dispatchDelimitedParse_`) is the parse front door for the Tag pipeline,
106+
documented at ~10–40× the textscan fallback, with zero coverage
107+
(BatchTagPipeline / delimited ingestion was entirely unbenchmarked).
108+
- **What it does:** generates deterministic 4-column CSVs of growing row count,
109+
times `dispatchDelimitedParse_` (file generation excluded), reports parse ms +
110+
rows/s + MB/s. Always deletes its temp files (per-iter + onCleanup backstop).
111+
- **Gate:** machine-independent — log-log row-scaling exponent over rows ≥ 1e4
112+
must stay ≤ 1.3 (catches super-linear parse creep, e.g. O(rows²) realloc).
113+
- **Reaches the private wrapper** via `cd`-into-`libs/SensorThreshold/private`
114+
(see [[benchmarking-private-mex-kernels]]).
115+
- **First run (MATLAB R2025b, MEX active):** ~5.7 M rows/s (~205 MB/s) at 100K–500K
116+
rows; exponent 0.98 (essentially linear) → PASS.
117+
118+
### Pivot notes this run
119+
Two earmarked targets were rejected on fresh survey:
120+
- **Pyramid build** — `buildPyramidLevel` is just `minmax_downsample` per level
121+
(already gated) + chunked reads; private; low marginal value. Downgraded to
122+
◐ Partial in the table, not benchmarked.
123+
- **DerivedTag.recompute_** — thin dispatch around a user-supplied `ComputeFn`
124+
(`[X,Y] = ComputeFn(Parents)`), so a microbench would mostly measure the test
125+
closure, not a FastSense kernel. Deferred unless paired with a built-in
126+
compute/alignment path worth isolating.
127+
128+
### Next gap for the following iteration
129+
Survey fresh, but leading candidates (higher-level paths now that the core MEX
130+
kernels are covered):
131+
- **EventStore persistence scaling** — `EventStore.save` (atomic temp-rename
132+
write) / `load` as event count grows; relevant for long-running live
133+
dashboards. Confirm it isn't already covered by `bench_event_marker_regression`
134+
/ `bench_dashboard_*` (those attach stores but may not stress save/load at scale).
135+
- **LiveEventPipeline per-tick processing** (`processMonitorTag_`) on the live
136+
refresh path — confirm it isn't already covered by `bench_monitortag_tick`.
137+
- Still open: the `to_step_function_mex` wiring question (filed as a background task).

‎benchmarks/bench_binary_search.m‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
function bench_binary_search()
2+
%BENCH_BINARY_SEARCH Isolated microbenchmark of the range-lookup hot path.
3+
%
4+
% binary_search is the gateway to every range query in FastSense. On each
5+
% zoom/pan and render it locates the visible index window in a raw, sorted,
6+
% full-length X array — FastSense.m (resolve/zoom window, timestamp lookup),
7+
% FastSenseToolbar.m (click-to-point, range select) and SensorTag.m (tag
8+
% range resolve) all call it, against arrays up to tens of millions of
9+
% points. It is MEX-accelerated (binary_search_mex) with a pure-MATLAB
10+
% fallback, yet has no benchmark anywhere.
11+
%
12+
% The cost of any single call is tiny (O(log N) comparisons), so absolute
13+
% throughput is not the point. The point is the GATE: binary search must
14+
% stay logarithmic. If the MEX silently stops loading, or a change turns
15+
% the search into a linear scan, large-data zoom/pan responsiveness
16+
% collapses — and nothing else in the suite would catch it. This benchmark
17+
% times many scalar lookups (both 'left' and 'right') across a wide size
18+
% sweep and asserts the per-query time scales sub-linearly with N.
19+
%
20+
% Per-query time grows only weakly with N (a mix of ~log2(N) comparisons
21+
% and cache-miss penalty as the array spills out of cache), so the
22+
% empirical log-log exponent stays well below the linear-scan exponent of
23+
% ~1.0. The gate (exponent <= 0.6) cleanly separates the two regimes and
24+
% is machine-independent.
25+
%
26+
% Warmup dissolves first-call/JIT overhead; each measurement loops over a
27+
% fixed query batch so per-call dispatch stays representative of production
28+
% (binary_search is always called scalar); median of nRuns defuses spikes.
29+
%
30+
% Run:
31+
% octave --no-gui --eval "install(); bench_binary_search();"
32+
%
33+
% Exits 0 with "PASS: ..." on success; raises assert() (non-zero exit) if
34+
% either direction's per-query scaling exponent exceeds the gate.
35+
%
36+
% See also binary_search, binary_search_mex, bench_downsample_kernels.
37+
38+
here = fileparts(mfilename('fullpath'));
39+
addpath(fullfile(here, '..'));
40+
install();
41+
% binary_search lives in libs/FastSense/ (not a private/ folder), so
42+
% install() puts it on the path and it is directly callable here.
43+
44+
sizes = [1e4, 1e5, 1e6, 1e7, 5e7];
45+
labels = {'10K', '100K', '1M', '10M', '50M'};
46+
47+
nQueries = 20000; % scalar lookups timed per (size, direction, run)
48+
nRuns = 5; % median of nRuns
49+
50+
% Deterministic seed — works in both MATLAB and Octave
51+
if exist('rng', 'file') == 2
52+
rng(0);
53+
else
54+
rand('state', 0); %#ok<RAND>
55+
end
56+
57+
% binary_search_mex lives in libs/FastSense/private. It is visible to
58+
% binary_search.m (its parent folder) and is what the wrapper actually
59+
% dispatches to — but it is NOT visible from this benchmark's context,
60+
% so a plain exist('binary_search_mex','file') here would misreport as a
61+
% fallback. Detect the built binary for THIS platform on disk instead.
62+
mexPath = fullfile(here, '..', 'libs', 'FastSense', 'private', ...
63+
['binary_search_mex.' mexext]);
64+
useMex = (exist(mexPath, 'file') ~= 0);
65+
66+
nSizes = numel(sizes);
67+
tLeft = zeros(1, nSizes); % per-query seconds, 'left'
68+
tRight = zeros(1, nSizes); % per-query seconds, 'right'
69+
70+
fprintf('\n=== binary_search range-lookup microbenchmark ===\n');
71+
fprintf(' binary_search_mex: %s\n', tf_(useMex));
72+
fprintf(' %d scalar lookups per measurement, median of %d runs\n', nQueries, nRuns);
73+
fprintf(' %s\n', repmat('-', 1, 74));
74+
fprintf(' %-6s | %-14s %-12s | %-14s %-12s\n', ...
75+
'N', 'left (us/q)', 'left Mq/s', 'right (us/q)', 'right Mq/s');
76+
fprintf(' %s\n', repmat('-', 1, 74));
77+
78+
for c = 1:nSizes
79+
n = sizes(c);
80+
x = linspace(0, 100, n); % sorted ascending (binary_search contract)
81+
vals = 100 * rand(1, nQueries); % query targets within range (not timed)
82+
83+
tLeft(c) = timeSearch_(x, vals, 'left', nRuns);
84+
tRight(c) = timeSearch_(x, vals, 'right', nRuns);
85+
86+
fprintf(' %-6s | %12.4f %10.2f | %12.4f %10.2f\n', ...
87+
labels{c}, ...
88+
tLeft(c) * 1e6, 1 / tLeft(c) / 1e6, ...
89+
tRight(c) * 1e6, 1 / tRight(c) / 1e6);
90+
91+
clear x vals;
92+
end
93+
fprintf(' %s\n', repmat('-', 1, 74));
94+
95+
% ---- Scaling gate: per-query time must stay sub-linear in N ----
96+
% Fit over N >= 1e5 (small N is dominated by fixed call/dispatch overhead
97+
% and would flatten the slope). O(log N) + cache effects keep the exponent
98+
% well under 1.0; a linear-scan regression drives it toward 1.0.
99+
fitMask = sizes >= 1e5;
100+
slopeLeft = scalingExponent_(sizes(fitMask), tLeft(fitMask));
101+
slopeRight = scalingExponent_(sizes(fitMask), tRight(fitMask));
102+
growthLeft = tLeft(end) / max(tLeft(1), eps);
103+
104+
gate = 0.6;
105+
fprintf(' Per-query scaling exponent (large-N fit, linear-scan ~1.0):\n');
106+
fprintf(' left : %.2f (gate: <= %.1f)\n', slopeLeft, gate);
107+
fprintf(' right : %.2f (gate: <= %.1f)\n', slopeRight, gate);
108+
fprintf(' per-query growth 10K->50M (left): %.1fx\n', growthLeft);
109+
fprintf(' %s\n', repmat('-', 1, 74));
110+
111+
assert(slopeLeft <= gate, ...
112+
sprintf(['FAIL: binary_search ''left'' per-query exponent %.2f exceeds %.1f — ' ...
113+
'search is no longer logarithmic (linear-scan regression?).'], slopeLeft, gate));
114+
assert(slopeRight <= gate, ...
115+
sprintf(['FAIL: binary_search ''right'' per-query exponent %.2f exceeds %.1f — ' ...
116+
'search is no longer logarithmic (linear-scan regression?).'], slopeRight, gate));
117+
fprintf(' PASS: lookups stay sub-linear (gate: exponent <= %.1f).\n\n', gate);
118+
end
119+
120+
function t = timeSearch_(x, vals, dir, nRuns)
121+
%TIMESEARCH_ Median-of-nRuns per-query time of binary_search over a batch.
122+
% Warms up first, then times nQueries back-to-back scalar lookups per
123+
% run and returns the median run divided by nQueries.
124+
nq = numel(vals);
125+
binary_search(x, vals(1), dir); %#ok<*NASGU> % warmup
126+
binary_search(x, vals(end), dir);
127+
runTimes = zeros(1, nRuns);
128+
for r = 1:nRuns
129+
t0 = tic;
130+
for q = 1:nq
131+
binary_search(x, vals(q), dir);
132+
end
133+
runTimes(r) = toc(t0);
134+
end
135+
t = median(runTimes) / nq;
136+
end
137+
138+
function slope = scalingExponent_(ns, times)
139+
%SCALINGEXPONENT_ Log-log slope of per-query time vs N (the growth exponent).
140+
% slope -> 0 indicates flat/logarithmic scaling; -> 1 indicates linear.
141+
times = max(times, eps);
142+
p = polyfit(log10(ns(:)), log10(times(:)), 1);
143+
slope = p(1);
144+
end
145+
146+
function s = tf_(b)
147+
if b
148+
s = 'active';
149+
else
150+
s = 'fallback (pure MATLAB)';
151+
end
152+
end

0 commit comments

Comments
 (0)