You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
handle_infinite_request_xorq (xorq_loading.py:421) serves each scroll window by executing the session's expression through window_to_parquet(filtered_expr, start, end, sort_col, ascending). For scan-backed expressions, limit/offset pushdown makes this near-free. For aggregate-backed expressions the entire aggregation re-runs for every 300-row window, and a column sort re-runs the aggregation plus a full sort. _expr_count is cached per expression (#889); the window execution is not.
Evidence
2026-06-10 parking_ticket_analysis replay (0.14.17): scan-backed entries page in 5–12ms; aggregate-backed entries (habitual_speeder_stats: group_by over 131.7M rows → 4.5M; tickets_per_day: 3 aggregations + 2 joins) take 1.35–1.75s per window, every window.
Scrolling an aggregate-backed table costs one full aggregation per viewport, and sorting costs aggregation + sort per window — ~1.4s added to every scroll/sort interaction on the parking catalog, scaling with input size, on entries whose final results are small (5K–4.5M rows).
Suggested fix
Serve windows from a materialized result when the source is chained. #916 already classifies exactly these op trees (filter / aggregate / join / distinct / sort) to decide stat-source materialization; reusing that materialized table for row serving — or accepting an embedder-provided result parquet (buckaroo-data/tallyman#31 sketches the protocol) — would make paging O(window) instead of O(aggregation).
Context
Identified during the 2026-06-10 parking_ticket_analysis replay and re-measured by tallyman's perf suite. Present on current main (handle_infinite_request_xorq → window_to_parquet); distinct from #899 (repeat /load_expr cost) and untouched by #901/#913/#916, which address the load/stat side.
Problem
handle_infinite_request_xorq(xorq_loading.py:421) serves each scroll window by executing the session's expression throughwindow_to_parquet(filtered_expr, start, end, sort_col, ascending). For scan-backed expressions, limit/offset pushdown makes this near-free. For aggregate-backed expressions the entire aggregation re-runs for every 300-row window, and a column sort re-runs the aggregation plus a full sort._expr_countis cached per expression (#889); the window execution is not.Evidence
Impact
Scrolling an aggregate-backed table costs one full aggregation per viewport, and sorting costs aggregation + sort per window — ~1.4s added to every scroll/sort interaction on the parking catalog, scaling with input size, on entries whose final results are small (5K–4.5M rows).
Suggested fix
Serve windows from a materialized result when the source is chained. #916 already classifies exactly these op trees (filter / aggregate / join / distinct / sort) to decide stat-source materialization; reusing that materialized table for row serving — or accepting an embedder-provided result parquet (buckaroo-data/tallyman#31 sketches the protocol) — would make paging O(window) instead of O(aggregation).
Context
Identified during the 2026-06-10 parking_ticket_analysis replay and re-measured by tallyman's perf suite. Present on current main (
handle_infinite_request_xorq→window_to_parquet); distinct from #899 (repeat /load_expr cost) and untouched by #901/#913/#916, which address the load/stat side.