Merge master into audit/gpu-engine-params-1788-1790

Resolves both conflicts and corrects two defects the conflict exposed.

bin/test-polars.sh -- take master's version unchanged. This branch adds
graphistry/tests/compute/gfql/test_rows_table_named_middle.py and
test_rewrite_param_discard.py to POLARS_TEST_FILES, but #1805 (49db91c,
this branch's own merge-base) already added both. Keeping the branch side
would list each file twice, so the polars lane would run both modules twice
-- and that lane's py3.12 cell is the coverage cell, currently 442s against
a 600s cap. Master's version also carries #1814's xdist plumbing, which the
branch side predates.

CHANGELOG.md -- the entry claimed "neither file was in bin/test-polars.sh,
so test_rows_table_named_middle.py ... ran in no CI lane at all". That is
false at this branch's base: both files are listed there at lines 43 and 50.
Rewritten to the residual that is true -- the suites were parametrized over
pandas + polars only, so cuDF and polars-gpu never exercised the table
guards; and the module-level pytest.importorskip("polars") skipped the whole
file in test-gfql-core, so its pandas cases ran only in the polars lane.
The measured claims (45 passed / 3 xfailed / 0 skipped on GB10; 24 mutation
failures, 6 per engine) are unchanged.

Also types engine_skip_reason, per review: the engine -> required-modules
map is hoisted to a module-level Mapping[str, Tuple[str, ...]] constant
rather than rebuilt inline per call, the signature is
(engine: str, smoke: Callable[[], object]) -> Optional[str], and
gpu_environment_reason's "str | None" string annotation is normalized to
Optional[str] to match (this file is imported under py3.8 lanes).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx

Author: lmeyerov

CHANGELOG.md

@@ -1,10 +1,20 @@
 #!/usr/bin/env python3
 """FAIR GFQL-vs-Ladybug: run Ladybug's benchmark ops as GFQL Cypher MATCH...RETURN
-(the row pipeline), NOT dataframe shortcuts. Compares against LadybugDB's PUBLISHED
-5M/20M numbers (their figures, their hardware — a cross-machine comparison; treat
-ratios as indicative). Ladybug best (5M/20M, ms): full_scan 3789, range 7.5,
-point 0.3, count 3.3, out_degree100 59.8, scan_rel_props 15722,
-scan_rel_rowid 14562. Kuzu count 46.
+(the row pipeline), NOT dataframe shortcuts. Op shapes follow
+https://github.com/LadybugDB/kuzu-ladybug-benchmark.
+
+The `LADYBUG` reference column below is a MEASUREMENT, not a literal: LadybugDB 0.18.1,
+embedded, on the same host and in the same session as the GFQL arm, on the same generated
+5M-node / 20M-edge graph, warm medians (2 warmups + 5 timed runs, median across two
+position-balanced slots), timed at Ladybug's FASTEST result-producing scope of the four
+measured (zero-copy Arrow / polars / pandas / row iterator) -- Arrow for every scan,
+the row iterator for the single-row point lookup -- because GFQL returns a materialized
+columnar frame and choosing a slower scope for the competitor is not a comparison. Result values were digest-identical
+across the two engines on every op. Runner, raw JSON and full provenance:
+pyg-bench `benchmarks/ladybug_scan/` and `results/ladybug-scan-20260728/`.
+
+Numbers here are from dgx-spark (GB10); on another box, re-run the pyg-bench lane rather
+than reusing these.
 
 FAIRNESS: each engine is benchmarked on a graph built in ITS OWN NATIVE frame type
 (pandas/polars/cuDF), built ONCE outside the timing loop. An earlier version built the
@@ -49,8 +59,9 @@ def build(engine):
     "scan_rel_props": "MATCH (a)-[o]->(b) RETURN a.id, b.id, o.since",
     "scan_rel_rowid": "MATCH (a)-[r]->(b) RETURN a.id, b.id",
 }
-LADYBUG = {"full_scan": 3789, "range": 7.5, "point": 0.3, "count": 3.3,
-           "scan_rel_props": 15722, "scan_rel_rowid": 14562}
+# LadybugDB 0.18.1, dgx-spark, 2026-07-28, Arrow materialization (see module docstring).
+LADYBUG = {"full_scan": 364.28, "range": 7.57, "point": 0.30, "count": 3.08,
+           "scan_rel_props": 1925.61, "scan_rel_rowid": 1449.13}
 
 def _size(res):
     df = getattr(res, "_nodes", None)

benchmarks/gfql/bench_ladybug_cypher.py

@@ -5,6 +5,8 @@ set -ex
 # - Extra args are passed through to the pytest phase
 # - Set POLARS_COV=1 to collect coverage over graphistry/compute; the coverage
 #   data file location is taken from $COVERAGE_FILE (as the CI py3.12 lane sets it)
+# - Set POLARS_XDIST to pick the pytest-xdist worker spec (default `auto`); `0` forces
+#   the serial path, which is what an A/B of this lane compares against
 # - Non-zero exit code on fail
 
 # Assume [polars,test] installed
@@ -24,11 +26,13 @@ POLARS_TEST_FILES=(
     graphistry/tests/compute/gfql/test_engine_polars_chain.py
     graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py
     graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py
+    # engine-parametrized (pandas/polars/cudf/polars-gpu); the pandas params also run in
+    # test-gfql-core, but only this lane has polars installed
+    graphistry/tests/compute/gfql/test_varlen_bounded_engine_parity_1787.py
     graphistry/tests/compute/gfql/test_engine_polars_with_match_reentry.py
-    # #1788 / #1790 regression suites: engine-parametrized over pandas/cuDF/polars/polars-gpu,
-    # so their POLARS params only ever run here (test-gfql-core installs no polars)
-    graphistry/tests/compute/gfql/test_rows_table_named_middle.py
-    graphistry/tests/compute/gfql/test_rewrite_param_discard.py
+    # engine-parametrized: its pandas params run in test-gfql-core, but the polars params
+    # only ever run here (the file has no module-level importorskip, so nothing else flags it)
+    graphistry/tests/compute/gfql/test_exec_context_scoping.py
     graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py
     graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py
     graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py
@@ -39,6 +43,7 @@ POLARS_TEST_FILES=(
     graphistry/tests/compute/gfql/test_polars_rows_entity_groupby.py
     graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py
     graphistry/tests/compute/gfql/test_residual_polars_native.py
+    graphistry/tests/compute/gfql/cypher/test_grouped_aggregate_fused_polars.py
     # module-level `importorskip("polars")` files that previously ran in no lane at all
     graphistry/tests/compute/gfql/test_engine_polars_narrow_combine.py
     graphistry/tests/compute/gfql/test_engine_polars_semi_key_dedup.py
@@ -61,15 +66,40 @@ POLARS_TEST_FILES=(
     graphistry/tests/test_engine_frame_helpers.py
 )
 
+# PARALLELISM. The py3.12 cell of this lane is the coverage cell and has repeatedly run out
+# of its CI budget; xdist is the lever that does not require a workflow edit (pytest-xdist is
+# already in the [test] extra, and test-gfql-core already runs `-n auto` under --cov, so
+# coverage+xdist is an established combination in this repo).
+#   * worker spec `auto` = os.cpu_count(): 4 on a GitHub-hosted ubuntu-latest runner, and it
+#     scales DOWN on a 2-vCPU runner where a fixed `-n 4` could be slower than serial.
+#   * --maxprocesses caps the count so a 24-core dev box does not fan out 24 polars processes
+#     that then oversubscribe polars' own thread pool.
+#   * --dist load (xdist's default) balances per test. `loadfile` was measured too: it is
+#     bounded by the single largest module and only reaches 1.4x where `load` reaches 3.2x.
+#     No test in this lane depends on execution order or on cross-test module state, and the
+#     pass/skip node-id sets were compared serial-vs-parallel and are identical; POLARS_XDIST_DIST
+#     is the escape hatch if a future order-dependent test needs `loadfile`/`loadscope`.
+XDIST_ARGS=()
+if [ "${POLARS_XDIST:-auto}" != "0" ]; then
+    XDIST_ARGS=(
+        -n "${POLARS_XDIST:-auto}"
+        --maxprocesses "${POLARS_XDIST_MAX:-4}"
+        --dist "${POLARS_XDIST_DIST:-load}"
+    )
+fi
+
 COV_ARGS=()
 if [ -n "${POLARS_COV:-}" ]; then
     COV_ARGS=(--cov=graphistry --cov-report=)
 fi
 
-python -B -m pytest -vv "${COV_ARGS[@]}" "${POLARS_TEST_FILES[@]}" "$@"
+python -B -m pytest -vv "${XDIST_ARGS[@]}" "${COV_ARGS[@]}" "${POLARS_TEST_FILES[@]}" "$@"
 
 # cypher-lowering polars-parametrized cases (round ties, lower/upper, =~, numeric fns);
-# appended into the same coverage data file when POLARS_COV=1 (CI audit reads it)
+# appended into the same coverage data file when POLARS_COV=1 (CI audit reads it).
+# Left SERIAL on purpose: it is one module and ~8s of the lane, so worker startup would eat
+# the gain. Appending into the data file the xdist phase produced is verified — the merged
+# result is line-for-line identical to running the whole script in one go.
 COV_APPEND_ARGS=()
 if [ -n "${POLARS_COV:-}" ]; then
     COV_APPEND_ARGS=(--cov=graphistry --cov-report= --cov-append)

bin/test-polars.sh

@@ -171,22 +171,22 @@ guess. The measured pairs behind every claim live in :doc:`performance` (and
        Kuzu as the store; GFQL for bulk read analytics.
    * - **LadybugDB**
      - Actively-maintained **Kuzu fork** (Kuzu is archived); embedded C++, strongly-typed
-       Cypher, opt-in ART *or* hash indexing, zero-copy Arrow/CSR scans, and **out-of-core
-       billion-scale** (query a 1.8B-edge graph in <8 GB RAM).
-     - Against **LadybugDB's published numbers** for their own 5M-node / 20M-edge suite
-       (their figures, their hardware; GFQL measured separately on the same Cypher row
-       pipeline, each engine on its **native** frames — a cross-machine comparison, so
-       read it as indicative): GFQL **wins the scan-shaped ops** — full node scans and
-       relationship property/rowid scans — by large margins. **Point lookups** are close
-       in absolute terms but favor Ladybug's B-tree/hash **index seek** over a columnar
-       scan; a resident GFQL node-id index (tracked in issue #1676) should close that.
-       Ladybug wins the two ops backed by persistent structure: point lookups and a
+       Cypher, opt-in ART *or* hash indexing, zero-copy Arrow/CSR scans, and an on-disk
+       mode its docs describe as processing larger-than-memory workloads.
+     - **Measured on one host, in one session, on the same 5M-node / 20M-edge graph** —
+       LadybugDB 0.18.1 embedded against GFQL ``engine='polars'``, result values
+       digest-identical on every op, Ladybug timed at its fastest (zero-copy Arrow)
+       materialization: GFQL wins the **node-scan** shapes — a full node scan **6.2×**
+       and a 1,001-row range scan **1.5×**. **Point lookups** are close in absolute terms
+       but favor Ladybug's hash/ART **index seek** over a columnar scan; a resident GFQL
+       node-id index (tracked in issue #1676) should close that. Ladybug also holds a
        cached relationship ``COUNT(*)`` (an O(1) count vs GFQL's O(E) endpoint-validated
        scan — a dataframe has no referential integrity). GFQL's angle is
        dataframe-native, in-process, and GPU-accelerated with no separate store to
-       load/index. Details and reproducer in the :doc:`performance` methodology notes.
-     - **Complement:** Ladybug is a durable embedded store with an out-of-core mode
-       (billion-scale in <8 GB RAM); GFQL is a query engine over your dataframes. GFQL's
+       load/index. Numbers, provenance and reproducer in the :doc:`performance`
+       methodology notes.
+     - **Complement:** Ladybug is a durable embedded store with an on-disk,
+       larger-than-memory mode; GFQL is a query engine over your dataframes. GFQL's
        *default* is in-memory, but it is **not limited to it** — Polars streaming
        (``GFQL_POLARS_CPU_STREAMING=1``, disk-spill) and the cudf-polars streaming executor
        (``GFQL_POLARS_GPU_EXECUTOR=streaming``) are larger-than-memory paths

docs/source/gfql/engines.rst

@@ -136,9 +136,10 @@ gathers (PR #1770). Neo4j still wins recent-replies — reported as-is.
 OLAP multi-join
 ~~~~~~~~~~~~~~~
 
-On the graph-benchmark OLAP multi-join queries at 100k-node scale with
-``engine='polars'``: **q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200×)**; q9 is
-**14.2×**.
+On the graph-benchmark OLAP multi-join queries at 100k-node scale (100,000 persons /
+2.4M ``FOLLOWS``) with ``engine='polars'``, against embedded Kuzu 0.11.3 on the same host
+and in the same session: GFQL wins q8, and **q9 runs in 66.6 ms vs 84.1 ms (1.26×)**. Both
+sides run the same canonical query text and return identical values.
 
 When not to use GFQL
 ~~~~~~~~~~~~~~~~~~~~
@@ -226,14 +227,20 @@ Methodology (prior sweep)
   ``benchmarks/gfql/pandas_vs_polars.py``, and ``benchmarks/gfql/index_vs_kuzu_prepared.py``
   (vs kuzu). Numbers on this page are rendered from saved runs; the page does not re-run
   them.
-- **LadybugDB comparison** (referenced qualitatively in :doc:`engines`): the Ladybug
-  figures are **their published results on their hardware**; the GFQL side ran on the host
-  above via ``benchmarks/gfql/bench_ladybug_cypher.py`` (5M/20M synthetic per their suite
-  shape, native frames per engine, warm medians) — a cross-machine comparison, disclosed
-  as such. GFQL won the scan-shaped ops by large margins (full node scan ~65×,
-  relationship property/rowid scans ~3.5–3.7×); Ladybug won the two ops backed by
-  persistent structure — point lookups (index seek vs columnar scan) and a cached
-  relationship ``COUNT(*)``.
+- **LadybugDB comparison** (referenced qualitatively in :doc:`engines`): **both sides
+  measured on the host above, in one session, on the same generated 5M-node / 20M-edge
+  graph** — LadybugDB **0.18.1** embedded in a host venv against GFQL ``engine='polars'``
+  in the container above. Op shapes are those of
+  `LadybugDB/kuzu-ladybug-benchmark <https://github.com/LadybugDB/kuzu-ladybug-benchmark>`_
+  via ``benchmarks/gfql/bench_ladybug_cypher.py``; warm medians (2 warmups + 5 timed runs),
+  slots interleaved L G G L, and **every op's result values are digest-identical across the
+  two engines**. Ladybug is timed at its **fastest** result-producing scope of the four measured — for
+  both cells below that is zero-copy Arrow, not its Python row iterator — because GFQL
+  returns a materialized columnar frame.
+  GFQL wins the node-scan shapes: **full node scan 59.0 ms vs 364.3 ms (6.2×)** and the
+  1,001-row **range scan 5.1 ms vs 7.6 ms (1.5×)**. Point lookups stay with Ladybug's
+  index seek over a columnar scan (a resident GFQL node-id index is tracked in issue
+  #1676), as does a cached relationship ``COUNT(*)``.
 
 There is **no universal winner**: ``polars`` typically takes over from ~10K edges up
 (``pandas`` still wins trivial sub-millisecond operations), and the right GPU

docs/source/gfql/performance.rst

@@ -48,6 +48,29 @@ def clear_row_exec_context(g: "Plottable") -> "Plottable":
     ``WITH`` query carries the seed, and a follow-up query on that result -- a different
     graph entirely -- is answered against it (the same #1786 defect, one hop removed).
     Returns ``g`` unchanged when there is nothing to strip, so the common path is free.
+
+    WHY ``None`` AND NOT "RESTORE WHAT ``g`` CARRIED ON ENTRY". ``attach`` above INHERITS on
+    the way in (a ``None`` argument keeps what ``g`` already carries), so the asymmetry is
+    real and the save/restore question is a fair one. Three reasons it is settled the other
+    way, each pinned by ``tests/compute/gfql/test_exec_context_scoping.py``:
+
+    * This function is PURE -- it returns ``g.bind()`` and never writes through to the object
+      it was handed, so an outer scope that set the field still has it afterwards. There is no
+      caller state to save. That is what separates this from #1786, which WAS an in-place
+      write onto the caller's own graph; restoring is the fix for a mutation, and there is no
+      mutation here.
+    * The only channel restore would change is the RETURN VALUE, and putting the seed back
+      there IS the second half of #1786. Measured: hand-restoring the seed onto a result
+      changes the answer of the next query run on that result (7 -> 2 -> 1 rows).
+    * No execution frame ever inherits a context it did not set. The cross-segment ``WITH``
+      seed travels as the explicit ``start_nodes`` PARAMETER (``chain_impl(..., start_nodes=)``,
+      ``_compiled_query_reentry_state``), never through this field, so the field's lifetime is
+      exactly ONE boundary-call run. Instrumenting ``attach`` over ``tests/compute`` recorded
+      3907 calls and ZERO inheriting ones. (53 of them DID enter on a graph already carrying a
+      seed -- nested boundary frames -- but each was handed the IDENTICAL ``start_nodes``
+      parameter, so the value is still owned by the frame that sets it.) The test above re-runs
+      that as an assertion, so a future path that starts relying on inheritance reopens this
+      decision loudly.
     """
     if g._gfql_start_nodes is None and g._gfql_rows_base_graph is None:
         return g

graphistry/compute/gfql/exec_context.py

@@ -1312,6 +1312,41 @@ def _names(lf: pl.LazyFrame) -> List[str]:
                     )
                     if _resolved_min != 1:
                         return None
+            # #1787, same root-cause family as the unbounded shapes #1781 declined: pandas'
+            # step_pairs come from the var-length `edge_op.execute` hop, whose hop-window
+            # pruning -- and, when seeded, its per-seed BFS -- changes an edge multiplicity
+            # this raw-edge rebuild cannot reproduce. Declining is a DELIBERATE divergence
+            # from master, which served these: parity-or-NIE means a loud error, never a
+            # different number. Shrink the gate again once the multiplicity is reconstructible.
+            # WHICH shapes, why each boundary sits where it does, and the counts that prove
+            # each one are executable rather than prose -- every claim that used to be written
+            # out here is now a named test in:
+            #   graphistry/tests/compute/gfql/test_varlen_bounded_engine_parity_1787.py
+            #
+            # Keyed on an EXPLICIT window, NOT on `sem.is_multihop`: `-[*1..1]-` resolves to
+            # min == max == 1, is therefore not multihop, and pandas still routes it here.
+            if op.min_hops is not None or op.max_hops is not None:
+                _vl_max = op.max_hops if op.max_hops is not None else op.hops
+                _vl_min = op.min_hops if op.min_hops is not None else (
+                    op.hops if op.hops is not None else 1
+                )
+                # a seed is anything that starts the segment from less than the whole node
+                # set: a filtered start alias, or a re-entry / `WITH` seed frame
+                _prev_op = ops[idx - 1] if idx >= 1 else None
+                _seeded_start = start_nodes is not None or (
+                    isinstance(_prev_op, ASTNode) and bool(_prev_op.filter_dict)
+                )
+                if _vl_max is not None:
+                    if op.direction == "undirected":
+                        if _vl_max == 1:  # the degenerate window `-[*1..1]-` / `-[*1]-`
+                            return None
+                        if _seeded_start or idx > 1:  # doubled-pair expansion over-counts
+                            return None
+                    # `max_reached_hop` (compute/hop.py) is a dedup-by-node BFS eccentricity,
+                    # not a longest-walk length, so pandas prunes where this rebuild expands
+                    # a different edge multiset
+                    elif _vl_min >= 3 or (_vl_min >= 2 and _seeded_start):
+                        return None
             if op.direction not in ("forward", "reverse", "undirected"):
                 return None
             if any(