perf(gfql): gate a low-cardinality pure count(*) onto value_counts, and keep q1 off it

CHANGELOG.md

@@ -14,6 +14,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL secondary node property indexes (`create_index('node_prop', column=...)` / `g.gfql_index_node_props([...])`)**: a seed predicate on a NON-key column — `MATCH (m {id: 42})` where the graph's node id binding is some other column — previously cost a full node scan, because the registry only indexed the node-id binding and the CSR adjacencies. A property index is the same pay-as-you-go sidecar as the existing kinds: sorted distinct values over node **row positions** (CSR, so duplicate values are indexable), never reorders `.nodes`, fingerprint-validated so a `.nodes()` rebind is treated as absent (safe miss, never a wrong answer), engine-polymorphic (numpy host / cupy on-device), and policy-gated (`off`/`use`/`auto`/`force`). The seeded fixed-hop planner picks the **most selective** indexed scalar predicate in the seed filter using a free CSR-offset estimate, gathers those candidates, and applies every remaining predicate to them — so results are identical whether the index is present, absent, stale, or cost-gated out. `show_indexes()` lists property indexes; `drop_index('node_prop', column=...)` drops one. Only integer columns are indexable today (float NaN ordering, strings on cupy, and nulls all decline to the scan); widening that is additive. **Perf (dgx-spark, official LDBC SNB SF1, 3.18M nodes, warm median, value-identical 19-row result):** interactive-short IS7 `71.6 ms -> 19.5 ms` (**3.7x**), with a one-time `112 ms` build — the seed lookup itself goes from a `51.2 ms` scan to `0.096 ms`.
 
 ### Performance
+- **A single-key pure `count(*)` with provably LOW group cardinality skips polars' partitioned group-by**: inside the fused single-hop grouped-aggregate lane, `group_by(maintain_order=True).agg(pl.len())` carries a FLAT ~2 ms coordination cost that exists only at low group cardinality — measured on dgx-spark (polars 1.35.2, 20 threads, interleaved, 90 samples/arm/cell, 214 cells, ZERO value mismatches), int keys at 20,000 rows go 32 groups `2.054 ms` → 48 groups `0.411` → 64 groups `0.291`. `value_counts` has no such cost, so for a pure `count(*)` it is the same value for a fraction of the time. It is NOT a drop-in: `value_counts` scales WORSE with input rows and at 1,000,000 rows loses even at 2 groups (`4.137 → 8.591 ms`), and applied ungated the identical formulation makes the matched graph-benchmark **q1** cell (~20,000 groups over ~200,000 rows) **2.7 ms slower at 20k and 8.6 ms slower at 100k** — q1 is a cell that currently wins, so an ungated swap trades one cell's loss for another's regression. The formulation is therefore chosen only behind two STATIC, O(1), **upper** bounds: group cardinality ≤ the HEIGHT of the alias node frame supplying the group key (every group value is a property value of some row of that one frame, so distinct values cannot exceed its height), and aggregate input rows ≤ the height of the already-filtered EDGE frame (the semi-joins only remove rows; the property inner-join can multiply them, so the row bound is only claimed once exactly one alias carries properties and its node ids are unique — a check that runs on a frame already known to be ≤ 32 rows). Both bounds over-estimate, and over-estimating is the safe direction: a loose bound can only DECLINE a shape the fast formulation would have served, never route a high-cardinality aggregate into it. The thresholds — **32 groups and 100,000 rows** — were fixed from the crossover curve BEFORE the formulation was validated on any query, because a threshold chosen after seeing the verdicts is unfalsifiable; 48 groups already fails at 0.96× and 150,000 rows at 0.80× on string keys. Strictly additive: every decline falls through to the untouched `group_by`, so the blast radius is a decline away from zero. It DECLINES a non-single-key or non-pure-`count(*)` aggregate (including `count(<property>)`, which counts non-null values rather than rows), a group key not supplied by exactly one alias, a second alias also contributing property columns, a group-key alias frame that is too tall / missing its node-id column / carrying duplicate node ids, and an edge frame over the row bound. **Measured** on dgx-spark under the exclusive perf lock, matched graph-benchmark q1–q9 lane on the canonical query text (`gb_queries.py`, md5 `6e7ae268a5a41742587fcb87854b6e27`), 24 position-balanced slots per scale (12 per arm), Kuzu 0.11.3 re-run in-session, per-slot medians: **q4 at 20k `4.96 → 3.73 ms` (−1.23 ms, −24.7%), the two arms' slot ranges NOT overlapping**, which takes the board's last 20k loss from `1.65×` to `1.25×` of same-session Kuzu. Under the board's overlap rule that scores a TIE rather than a loss, but the overlap is **0.015 ms** and the median still favours Kuzu, so it is reported as *a loss narrowed to near-parity, not parity*. Engagement is exactly one cell: on the real board data the gate is consulted for q1/q2/q3/q4 and **admits only q4 at 20k** — q4 at 100k declines because its 7,117-row City frame carries only 3 distinct countries and an O(1) height bound cannot see the 3, and q1/q2 decline on BOTH bounds at both scales. q1, q2, q3 and q8 are arm-vs-arm ties with overlapping ranges, and the pandas arm — which never enters this polars-only lane — ties on all nine cells at both scales, a built-in null control. Value identity is the gate on the number: one canonical value per query across every slot, both arms, both engines and both scales, matching Kuzu on every cell.
 - **The polars single-hop GROUPED AGGREGATE builds ONE lazy plan instead of ~7 eager collects**: `MATCH (a {..})-[{..}]->(b {..}) [WHERE ..] RETURN <alias>.<prop> AS k, <agg> AS v ORDER BY .. [LIMIT n]` lowers to a fast path that semi-joins the edge frame against both node domains, inner-joins the projected properties on, groups, sorts and limits. Every one of those ops was issued EAGERLY — each its own `lazy().collect(_eager=True)` — so each intermediate materialized in full, the `select([src, dst])` projection could not be pushed into the semi-joins, and the `head()` could not reach back into the plan at all. That path serves three of the nine matched graph-benchmark cells (q1, q3, q4), and 74–96% of each of those queries' wall time sat inside those collects. The same op sequence is now expressed as a single lazy plan collected once — the algebra is character-identical (same `.unique()` id frames, same semi-joins, same un-deduplicated property lookups, same `group_by(maintain_order=True).agg(..)`, same per-key `nulls_last` sort, same `head`), so the value is identical, row ORDER included. The lane is strictly additive: the eager code is untouched and is the fallback on every decline. It DECLINES — never answering differently, only forgoing the speedup — for a non-eager-polars input frame, a property column missing from its alias' node frame (the eager twin discovers that MID-CHAIN and declines the whole fast path, so the guard is hoisted ahead of plan construction rather than left to be discovered after a plan already exists), source and destination bound to the same edge column, a projected column colliding with an endpoint column or the internal lookup key, an untranslatable aggregate, and — the correctness crux — **a result row order that ORDER BY does not fully determine**. Without every group key in the sort, the eager twin's order falls back to `maintain_order=True` group first-appearance order over an EAGER join output, which a lazy plan is free to change by re-ordering or re-siding joins; measured with an ungated variant of the same plan over 4 graph sizes × 4 seeds × 4 order-undetermined shapes, 47 of 64 comparisons diverged from the eager twin, and under `LIMIT` the divergence is a different ROW SET rather than a different row order. Not a GPU change: like the eager code and the fused two-star lane, it collects on CPU polars for both `polars` and `polars-gpu`. **Measured** on dgx-spark, matched graph-benchmark q1–q9 lane, one perf lock per experiment, master tree vs PR tree position-balanced `M P P M P M M P`, per-slot medians (never best-of), rows and canonical values compared on every cell, and replicated end to end in a second independent run: **`engine='polars'` q1 `13.31 → 8.96 ms` (−32.7%), q3 `8.54 → 5.53 ms` (−35.3%), q4 `6.99 → 5.05 ms` (−27.8%) at 20k**, per-slot ranges non-overlapping on all three in both runs; at 100k q1 `42.59 → 31.45 ms` (−26.2%) and q4 `12.19 → 10.43 ms` (−14.4%), with q3 `−9.1%…−13.0%` (non-overlapping in one run, overlapping in the other). Against same-session embedded Kuzu at 20k that widens q1 from a 1.12× win to **1.66×**, moves q3 from a **1.37× LOSS to a TIE** (Kuzu 6.29 ms; the two slot ranges overlap, so it is a tie and not a win), and narrows q4 from a 2.10× loss to **1.51× — still a loss**. The cells this lane is not called for are unchanged: q5 and q8 are ties at both scales in both runs, and q8 stays a win over Kuzu. The pandas arm — untouched by this change — reproduces the reference board to +0.8%…+9.4% and same-session Kuzu reproduces it to −2.2%…+1.5% on the cells at issue, which is what shows the harness matches it. Value identity is the gate throughout: a differential over 19 shapes × 10 graphs, compared ROW-ORDER and COLUMN-ORDER sensitively against both the eager code and the pandas oracle, found zero divergences from the eager code; pinned tests cover multiplicity on BOTH arms of the hop (duplicate node rows, parallel edges, self-loops), null placement on group keys and on aggregate values, empty matches, dangling endpoints, non-numeric ids, degenerate column bindings, and every decline. One PRE-EXISTING divergence is disclosed rather than quietly changed: the polars property lookup is not deduplicated by node id while the pandas one is, so a node table carrying the same id twice multiplies matched rows on polars only — the fused lane reproduces the eager polars answer exactly, and a test pins both sides.
 - **Native polars chain combine is proportional to the traversal result, not to the graph**: two graph-sized terms sat inside a combine whose answer is a handful of rows, and both are gone. (1) `_combine_edges` ran the prev/next endpoint gates for EVERY step, including the node steps whose edge frame is `g._edges.clear()` — zero rows. The eager combine skipped those, but the collect-once rewrite lazified the step frames and `.lazy()` erases the height, so the skip silently went dead. The cost lands on the side that is NOT empty: for the first step the gate's key side is the whole node table, and polars builds the hash table on that side before discovering the probe side has no rows (isolated: 6.99 ms for one such join at N=2M, and a chain pays one per node step). The pre-lazy row count is now recorded when the step frame is still eager and an empty step is dropped from the id union — it can contribute no ids, so the result is unchanged by construction. The skip keys on KNOWN-empty only; a frame that arrives already lazy reports no height and is planned normally. (2) The output node rows were materialized in TWO passes over the node table — one for the ids the steps kept, one more for the surviving edges' endpoints the first pass missed — then concatenated. The output node set is the UNION of those two id sides, so the ids are unioned first and the node table is read ONCE. The row-level `unique(subset=[node])` is preserved verbatim: those rows feed `how="left"` alias joins where a node table carrying the same id twice would multiply rows. Measured on synthetic LDBC-IS5-shaped graphs (one-row answer, polars-engine resident indexes), varying one dimension at a time: **at fixed E=2M, N=250k → 4M went 10.17 → 45.48 ms before and 7.60 → 17.17 ms after (2.65× at 4M, and the node-count slope is 3.7× flatter)**; the edge-count slope is unchanged, as expected for a node-side fix, with the constant ~6 ms lower. Parity: identical full frames (all columns, row order included) across 280 shape × graph combinations and 400 duplicate-node-id combinations, plus the 1003-case polars chain differential suite. Pinned by tests that assert the boundary rather than a wall clock: the node universe must not appear in the edge plan at all, and the node table must be read at most once per query.
 - **GFQL polars chain stops deduplicating semi-join key sides**: the native polars executor applied `.unique()` to every frame it fed into a `how="semi"` join. A semi-join emits a left row iff at least one matching right row exists, so duplicate keys can neither change which rows come back nor multiply them the way an inner join would — the deduplication was a full hash pass over the key column bought for no observable effect. On an unfiltered hop the key side **is** the node table, so this put **O(N) work inside a query whose answer is O(degree)**: the seeded single-hop plan built two such key frames per hop, each costing ~53 ms at 3.18M nodes — more than the rest of the query combined. The `.unique()` is now dropped everywhere the frame is provably a semi key side only (the `_semi` helper, the alias hop-window and next-edge endpoint gates, the two-hop fast path's endpoint gate, the `start_nodes` gate, the single-hop planner's id frames, and the index layer's `select_by_ids` polars branch — where the cuDF and pandas branches already used `isin` with no dedup, so the three engines now agree). It is deliberately **kept** on the alias frame that feeds a `how="left"` join, where duplicates genuinely would multiply rows, and the eager multi-hop loop is untouched (its frames also flow into concat/anti-join bookkeeping, a separate argument). Measured on a 3.18M-node / 14M-edge polars graph (LDBC SNB SF1-shaped), a seeded typed hop goes **127.4 → 57.1 ms (2.23×)** with identical row counts; at 1.75M edges **102.8 → 31.4 ms (3.28×)**, confirming the removed cost scales with node count, not edge count. Parity held across 380 differential comparisons covering duplicate node keys, null ids, dangling edges, duplicate `start_nodes`, and 11 traversal shapes; the gfql and chain suites show identical failure sets before and after. Pinned by tests that assert the boundary rather than the speed: duplicates reaching a semi key side must not change results or multiply rows, a dangling endpoint must still be excluded (the gate is load-bearing, not vacuous), and duplicate `start_nodes` must be inert.

bin/test-polars.sh

@@ -38,6 +38,7 @@ POLARS_TEST_FILES=(
     graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py
     graphistry/tests/compute/gfql/test_residual_polars_native.py
     graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_fused_polars.py
+    graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_lowcard_count.py
     # module-level `importorskip("polars")` files that previously ran in no lane at all
     graphistry/tests/compute/gfql/test_engine_polars_narrow_combine.py
     graphistry/tests/compute/gfql/test_engine_polars_semi_key_dedup.py