Skip to content

row paging on an aggregate-backed xorq session re-executes the full aggregation per 300-row window #923

Description

@paddymul

Problem

handle_infinite_request_xorq (xorq_loading.py:421) serves each scroll window by executing the session's expression through window_to_parquet(filtered_expr, start, end, sort_col, ascending). For scan-backed expressions, limit/offset pushdown makes this near-free. For aggregate-backed expressions the entire aggregation re-runs for every 300-row window, and a column sort re-runs the aggregation plus a full sort. _expr_count is cached per expression (#889); the window execution is not.

Evidence

  • 2026-06-10 parking_ticket_analysis replay (0.14.17): scan-backed entries page in 5–12ms; aggregate-backed entries (habitual_speeder_stats: group_by over 131.7M rows → 4.5M; tickets_per_day: 3 aggregations + 2 joins) take 1.35–1.75s per window, every window.
  • tallyman perf regression suite (perf(suite): report-only perf regression suite + CI workflow tallyman#36; synthetic 4M rows → 1.4M-group count/min/max aggregate, 4-core CI runner): first page 0.26s vs 0.05s for the scan-backed control over the same data.

Impact

Scrolling an aggregate-backed table costs one full aggregation per viewport, and sorting costs aggregation + sort per window — ~1.4s added to every scroll/sort interaction on the parking catalog, scaling with input size, on entries whose final results are small (5K–4.5M rows).

Suggested fix

Serve windows from a materialized result when the source is chained. #916 already classifies exactly these op trees (filter / aggregate / join / distinct / sort) to decide stat-source materialization; reusing that materialized table for row serving — or accepting an embedder-provided result parquet (buckaroo-data/tallyman#31 sketches the protocol) — would make paging O(window) instead of O(aggregation).

Context

Identified during the 2026-06-10 parking_ticket_analysis replay and re-measured by tallyman's perf suite. Present on current main (handle_infinite_request_xorqwindow_to_parquet); distinct from #899 (repeat /load_expr cost) and untouched by #901/#913/#916, which address the load/stat side.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions