fix(gfql): index AUTO gate is eager-polars-only; nodeless graphs materialize-first

Review findings (plans/review-pr-1767/review.md): B1 edges-only polars
graph crashed in the NODE_ID build (materialize under AUTO synthesizes
pandas; polars gathers ran on a pandas frame) — gfql_index_all now
materializes + engine-aligns BEFORE any build so all indexes land valid,
and create_index re-aligns for direct calls. M1 LazyFrame frames under
AUTO crashed — the gate now requires EAGER polars DataFrames on ALL
present frames (LazyFrame/mixed keep the legacy pandas path). Spy test
asserts the index SERVED (non-None), not merely was called. AUTO-query
trade-off documented in CHANGELOG (deferred to #1743 by policy hold).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

Author: lmeyerov

CHANGELOG.md

@@ -18,7 +18,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine <gfql/engines>` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator).
 
 ### Fixed
-- **GFQL `gfql_index_all()`/`create_index(engine='auto')` no longer swaps polars frames to pandas — the resident index finally serves `engine='polars'` hops**: `resolve_engine(AUTO)` maps polars input frames to PANDAS (the legacy input-format policy), so the AUTO index build coerced-and-REPLACED the user's polars frames with pandas copies. Every later `hop(engine='polars')` then paid a full-frame pandas→polars re-conversion per call — an O(E) tax (~220ms at 4M edges, ~1.7s at 32M on the C3 ladder) — while the pandas-engine index could never serve it (`get_valid` requires engine match + frame identity, and the per-call conversion broke both). The index layer is already engine-polymorphic (numpy sidecar arrays + polars row-gather), so the index API now resolves AUTO over resident polars frames to POLARS and indexes them in place: frames preserved (identity intact), polars-engine index resident and valid. Measured on dgx (warm median of 30, parity-checked vs the pandas indexed oracle every rung): polars seeded 1-hop 221→0.80ms at 4M edges (276×), and the ladder goes from linear (16.6→1751ms over 0.25M→32M) to sub-ms at every rung (0.17/0.35/0.80/0.83ms at 0.25M/1M/4M/16M). Explicit engines are honored unchanged (`gfql_index_all(engine='pandas')` still coerces). Pinned in `test_index.py::TestIndexAutoPreservesPolarsFrames` (frames+index engine preserved, index-engagement spy with pandas-oracle parity, explicit-engine coercion contract).
+- **GFQL `gfql_index_all()`/`create_index(engine='auto')` no longer swaps polars frames to pandas — the resident index finally serves `engine='polars'` hops**: `resolve_engine(AUTO)` maps polars input frames to PANDAS (the legacy input-format policy), so the AUTO index build coerced-and-REPLACED the user's polars frames with pandas copies. Every later `hop(engine='polars')` then paid a full-frame pandas→polars re-conversion per call — an O(E) tax (~220ms at 4M edges, ~1.7s at 32M on the C3 ladder) — while the pandas-engine index could never serve it (`get_valid` requires engine match + frame identity, and the per-call conversion broke both). The index layer is already engine-polymorphic (numpy sidecar arrays + polars row-gather), so the index API now resolves AUTO over resident polars frames to POLARS and indexes them in place: frames preserved (identity intact), polars-engine index resident and valid. Measured on dgx (warm median of 30, parity-checked vs the pandas indexed oracle every rung): polars seeded 1-hop 221→0.80ms at 4M edges (276×), and the ladder goes from linear (16.6→1751ms over 0.25M→32M) to sub-ms at every rung (0.17/0.35/0.80/0.83ms at 0.25M/1M/4M/16M). Explicit engines are honored unchanged (`gfql_index_all(engine='pandas')` still coerces); LazyFrame or mixed-engine frames keep the legacy pandas path (eager-polars-only gate); edges-only polars graphs materialize + engine-align BEFORE index builds so all three indexes land valid. Known trade-off, deliberate: `engine='auto'` QUERIES on a polars graph still resolve pandas (the #1743 routing policy is unchanged), so they re-coerce per call and scan — the accelerated surface is explicit `engine='polars'`; AUTO-first users can `gfql_index_all(engine='pandas')` to keep the old behavior. Pinned in `test_index.py::TestIndexAutoPreservesPolarsFrames` (frames+index engine preserved, index-engagement spy with pandas-oracle parity, explicit-engine coercion contract).
 - **GFQL polars engine natively runs `UNWIND` of a carried `collect()` list column**: `... WITH collect(x) AS xs UNWIND xs AS y RETURN ...` (exploding a list-valued column produced by `collect()`, the row-pipeline analogue of the IC6 unwind shape) previously declined on polars — `unwind_polars` only accepted a scalar literal list. Now explodes a `List`-dtype column reference (`with_columns` copy → `filter(list.len() > 0)` → `explode`), matching the pandas oracle exactly: empty-list/null cells → 0 rows, nulls *within* a list survive, source column retained; the pre-filter also makes it independent of the polars-2.0 `empty_as_null` default. Non-list columns, name collisions, unknown identifiers, and nested-list literals still decline (honest NIE, no pandas bridge). Differential fuzz vs pandas over ~3500 collect→UNWIND→{RETURN, grouped, filtered, ORDER BY, aggregate, nested-unwind} queries: 0 disagreements. (The `MATCH ... WITH collect(..) UNWIND .. MATCH` form — IC6 with a trailing re-traversal — is compile-rewritten into WITH→MATCH reentry, a separate path.) Tests in `test_engine_polars_row_pipeline.py`.
 - **GFQL polars engine natively runs the LDBC IC11/IC6 undirected variable-length bindings table (`-[*1..k]-`), unblocking the cross-alias `WHERE NOT(person=friend)` clause**: a bounded UNDIRECTED variable-length MATCH that materializes a bindings row table (`rows(binding_ops=...)`, emitted whenever a downstream clause — e.g. a cross-alias same-path `WHERE NOT a = b` / `a <> b` — references two node aliases on the path) previously declined on polars (`binding_rows_polars` hard-NIE'd every undirected multihop), while pandas handled it. This was the single residual blocking the official LDBC IC11/IC6 queries on polars (the cross-alias WHERE lowering itself was already native — directed var-length + `WHERE NOT a = b` already matched pandas). `binding_rows_polars` now supports undirected var-length with **`min_hops == 1`** via a doubled-pair join with immediate-backtrack avoidance, an exact port of the pandas oracle `_gfql_multihop_binding_rows` (`avoid_immediate_backtrack=True`): a `__prev__` marker (seeded null, dtype-matched to the id column) drops immediate backtracks each hop (Kleene mask: null prev kept), and the step-pair set reproduces pandas' edge multiplicity exactly — each non-loop edge contributes each directed orientation TWICE (so a length-1 pair appears x2, length-2 x4) while self-loops contribute `(u,u)` x2 only (not double-counted). The multiplicity rule was derived by instrumenting pandas' `step_pairs` (which flow from the var-length `edge_op.execute` hop + `orient_edges`), NOT by reading code alone. Scoped to `min_hops == 1` because that is where the raw-edge reconstruction provably matches pandas: every edge is trivially a length-1 path so the var-length hop's backward pruning removes nothing; `min_hops == 0` (zero-hop, undoubled) and `min_hops >= 2` (backward-pruned / long-walk divergence) still **decline with an honest `NotImplementedError`** rather than risk silent-wrong multiplicities. Differential fuzz vs the pandas oracle over ~2500 random graphs (self-loops, parallel + antiparallel edges, `*1..2`…`*1..5`, all WHERE/RETURN variants): 0 disagreements; the out-of-scope windows decline (never diverge). Tests in `test_engine_polars_binding_rows.py` (parity + multiplicity/backtrack/self-loop/string-id pins + `*0..2`/`*2..3`/`*2..2` decline pins).
 - **GFQL polars engine natively runs whole-entity `count(DISTINCT n)` / identity aggregation (#1709)**: `g.gfql("MATCH (a {..}) RETURN count(DISTINCT a) AS c")` and its grouped/filtered/traversal variants (`(a)-[]->(b) RETURN count(DISTINCT b)`) previously declined on polars — the Cypher aggregation lowers `count(DISTINCT b)` to the `__gfql_node_id__` identity sentinel, and the polars `lower_expr` `Identifier` branch resolved only the prefixed `alias.__gfql_node_id__` form, not the bare sentinel, so it NIE'd. Fix threads the graph node-id column through a `_NODE_ID` contextvar (published alongside the schema in `_lower_with_schema`, wired through the four callers) so the bare sentinel resolves to the id column **only if present**, else declines (never invents/mis-resolves a column). Differential fuzz vs the pandas oracle: 600 + 300 cases, 0 disagreements. Deliberately still honest-NIE (verified declines, not silent-wrong): whole-entity `collect(b)` (list-of-entities repr not ported, cf #1650) and multi-alias binding-table identity aggregation (bare id absent on connected-pattern binding tables — adjacent to #1273). Tests in `test_engine_polars_row_pipeline.py`.

graphistry/compute/gfql/index/api.py

@@ -123,11 +123,15 @@ def _resolve_index_engine(engine: EngineAbstractType, g: Plottable) -> Engine:
     caller said AUTO and the resident frames are polars, index them in place.
     Explicit engines are honored unchanged.
     """
+    def _is_eager_polars(df: object) -> bool:
+        # EAGER frames only: LazyFrame passes the module check but no index/coercion
+        # path can gather rows from it — those graphs keep the legacy pandas path.
+        return df is not None and 'polars' in str(type(df).__module__) and type(df).__name__ == 'DataFrame'
     eng = resolve_engine(engine, g)
     abstract = EngineAbstract(engine) if isinstance(engine, str) else engine
     if abstract == EngineAbstract.AUTO and eng == Engine.PANDAS:
-        df = g._edges if g._edges is not None else g._nodes
-        if df is not None and 'polars' in str(type(df).__module__):
+        frames = [f for f in (g._edges, g._nodes) if f is not None]
+        if frames and all(_is_eager_polars(f) for f in frames):
             return Engine.POLARS
     return eng
 
@@ -193,6 +197,11 @@ def create_index(
 
     if kind == NODE_ID:
         g2 = g.materialize_nodes() if g._nodes is None else g
+        if g._nodes is None:
+            # materialize_nodes synthesizes under its own AUTO rules (pandas for polars
+            # edges), so re-align the frames with the engine this index builds in —
+            # otherwise build_node_id_index runs polars gathers on a pandas frame.
+            g2 = _coerce_input_formats(g2, eng)
         node_col = g2._node
         assert node_col is not None and g2._nodes is not None
         _check_column(column, node_col, kind)
@@ -277,6 +286,12 @@ def gfql_index_all(g: Plottable,
     per-id semantics), so this convenience SKIPS it rather than raising — seeded
     traversal stays correct via the un-indexed node materialization path. (Explicit
     ``create_index(NODE_ID)`` still raises, since the caller asked for it specifically.)"""
+    if g._nodes is None:
+        # Materialize + engine-align BEFORE any index builds: materialize_nodes
+        # synthesizes (and can rebind edges) under its own AUTO rules, which would
+        # invalidate adjacency indexes built first (identity fingerprint).
+        from graphistry.compute.ComputeMixin import _coerce_input_formats
+        g = _coerce_input_formats(g.materialize_nodes(), _resolve_index_engine(engine, g))
     g = gfql_index_edges(g, "both", engine=engine)
     try:
         g = create_index(g, NODE_ID, engine=engine)

graphistry/tests/compute/gfql/index/test_index.py

@@ -884,10 +884,14 @@ def test_auto_polars_hop_engages_index(self, monkeypatch):
         g = graphistry.nodes(ndf, "id").edges(edf, "src", "dst").gfql_index_all()
         hits = []
         orig = index_api.index_seeded_hop
-        monkeypatch.setattr(index_api, "index_seeded_hop",
-                            lambda *a, **k: (hits.append(1), orig(*a, **k))[1])
+
+        def spy(*a, **k):
+            out = orig(*a, **k)
+            hits.append(out is not None)  # None = scan fallback; only a real serve counts
+            return out
+        monkeypatch.setattr(index_api, "index_seeded_hop", spy)
         r = g.hop(nodes=pl.DataFrame({"id": [7]}), hops=1, direction="forward", engine="polars")
-        assert hits, "resident polars index did not serve the polars hop"
+        assert any(hits), "resident polars index did not SERVE the polars hop (call alone is not service)"
         # parity vs the pandas indexed oracle on the same data
         gp = graphistry.nodes(ndf.to_pandas(), "id").edges(edf.to_pandas(), "src", "dst").gfql_index_all()
         rp = gp.hop(nodes=pd.DataFrame({"id": [7]}), hops=1, direction="forward", engine="pandas")
@@ -899,3 +903,24 @@ def test_explicit_engine_still_coerces(self):
         assert isinstance(gi._edges, pd.DataFrame)
         from graphistry.compute.gfql.index import show_indexes
         assert set(show_indexes(gi)["engine"]) == {"pandas"}
+
+
+    def test_nodeless_polars_graph_indexes(self):
+        # B1 pin: edges-only polars graph — materialize_nodes synthesizes under its
+        # own AUTO rules; the NODE_ID build must re-align engines, not crash.
+        pl = pytest.importorskip("polars")
+        g = graphistry.edges(pl.DataFrame({"src": [0, 1, 2], "dst": [1, 2, 3]}), "src", "dst")
+        gi = g.gfql_index_all()
+        from graphistry.compute.gfql.index import show_indexes
+        idx = show_indexes(gi)
+        assert len(idx) >= 2 and idx["valid"].all()
+
+    def test_lazyframe_auto_keeps_legacy_pandas_path(self):
+        # M1 pin: LazyFrame frames under AUTO must coerce to pandas like master
+        # (the eager-polars-only gate), never crash in a polars index build.
+        pl = pytest.importorskip("polars")
+        g = graphistry.nodes(pl.LazyFrame({"id": [0, 1, 2]}), "id").edges(
+            pl.LazyFrame({"src": [0, 1], "dst": [1, 2]}), "src", "dst")
+        gi = g.gfql_index_all()
+        from graphistry.compute.gfql.index import show_indexes
+        assert set(show_indexes(gi)["engine"]) == {"pandas"}