perf(gfql): route engine=auto to native polars for polars-frame graphs

CHANGELOG.md

@@ -32,6 +32,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **Seeded typed-hop property projection keeps its fast path through DISTINCT / ORDER BY / SKIP / LIMIT**: those trailing row ops are plain frame operations, so the fast path now delegates them to the canonical chain instead of declining the whole query.
 
 ### Changed
+- **`engine='auto'` (the default) now routes to the native polars engine when both bound frames are polars (#1743)**: AUTO on an all-polars-frame graph runs the native polars engine, falling back to the legacy AUTO path when the native engine declines a shape (`NotImplementedError`). Frames out follow the serving engine: polars for shapes the native engine serves, pandas for declined ones. Explicit `engine=` selections and policy-bearing calls are unchanged (the native executor does not emit the `postload`/`postchain` hooks, so policy-carrying queries stay on the generic path), AUTO never selects `polars-gpu`, and cuDF-frame graphs are untouched.
 - **The widened residual translator gets its cost back: the lowering is memoized, and the fused lane stops emitting a NaN mask it can prove is dead**. Widening `_residual_polars_expr` to `row_pipeline.lower_single_alias_predicate` (previous entry) turned a regex match into a full parse-and-lower on a path where every board residual was *already* native, and it regressed the matched graph-benchmark q1-q9 lane at both scales — q7 by 1.24x (20k) and 1.16x (100k) with disjoint per-slot ranges, q5 by 1.13x/1.09x also disjoint, q6 slower within overlap; the three cells that moved are exactly the three carrying residuals (`engine='polars'`, RUNS=101 WARMUP=10, 15 position-balanced slots, arms differing only by the pygraphistry tree, base `ed3904515` vs PR `7e9bd662d`). **Two measured mechanisms, two scoped fixes.** (1) *Parse tax*: `lower_single_alias_predicate` re-parsed and re-lowered on every call, at 3.3 us/call before the widening and 162 us/call after it — the cost is not the parse so much as the schema-width `LazyFrame` probe `_expr_output_dtype` runs per operand, and the predicate strings are a tiny fixed set per query. It is now memoized behind a bounded LRU (512 entries, so a long-lived process cannot grow it without limit), at 2.3 us/call. The key is `(expr, alias, columns_nan_free, ((column, dtype-repr), ...))`: names AND dtypes AND their order, because the same predicate over the same column names lowers differently when a dtype changes, and `str(dtype)` rather than the dtype object because polars' `DataType.__eq__` equates a dtype class with its parameterized instances and would HIT across dtypes that lower differently. Parser AVAILABILITY is deliberately NOT in the key — it is process state, not an argument — so it is probed ahead of the cache (~0.2 us), and no entry can outlive a parser that has gone away. (2) *A costlier expression*, which is why the loss grew with scale: the general lowering wraps every float comparison in `& col.is_nan().not()` so NaN compares IEEE/pandas/Cypher-style rather than polars-style (NaN = largest), and that mask took `p.age >= 23` from 0.23 ms to 0.52 ms at 100k. On the connected-join fused lane the mask is provably dead **for a bare column read**: every frame there is a filtered/joined projection of the graph's own `_nodes`, which `_coerce_input_formats` already ran through `nan_clean._pl_nan_to_null` on the way into `gfql()`, and selection/filtering/joining cannot introduce a NaN a column did not hold — the same justification the pre-widening narrow translator gave, and the same one `predicates.filter_expr_by_dict_polars` already relies on. So the lane opts in via a new `lowering_context.COLUMNS_NAN_FREE` contextvar, threaded by an explicit `lower_single_alias_predicate(..., columns_nan_free=True)` that nothing else sets. **The default is guard-ON**, so the general row-table lowering — whose frames have NOT been through gfql ingest — is untouched and any future caller is safe without knowing this exists. **Suppression is scoped to COLUMN REFERENCES, never to computed operands**: `n.a/n.b` is NaN at 0.0/0.0 and `sqrt(n.x)` is NaN at x<0 even on a perfectly clean column, so in-query float math manufactures NaN no ingest can have removed and keeps the mask on both lanes; a float LITERAL keeps its own (constant-false) term too, since only column reads are in scope. That restores exactly the guard-free set of the pre-widening narrow matcher. **Result**: q7 returns to parity with base — TIE at 1.02x at both scales, from 1.24x/1.16x — and q5 (0.95x/0.99x) and q6 (0.98x/1.01x) return to TIE as well, with no other cell moved and every cell's VALUE identical across all three arms at both scales. Tests pin the scoping in both directions: the fused lane emits a guard-free expression, the general row-table lowering still emits the mask, a computed operand keeps it and is answered IEEE-style over a real in-query 0.0/0.0, a natively-built polars frame carrying a genuine NaN is shown normalized by ingest before the lane ever sees it, and the memo is shown to agree with the uncached lowering on every shape while a dtype change alone produces a different expression.
 - **The native polars residual translator covers the whole single-alias predicate vocabulary the connected-join WHERE renderer can emit, because it now delegates to the row evaluator's own lowering instead of re-deriving it**: `_residual_polars_expr` (`graphistry/compute/gfql_fast_paths.py`) recognized exactly two hand-written regex shapes — `(<toLower|lower|toUpper|upper>(a.col) = 'lit')` and `(a.col <op> literal)` for `= >= <= > <` — and returned `None` for everything else, and a single `None` drops the entire fused single-collect connected-join plan and re-evaluates that alias's residual through a `where_rows` chain. It now parses the residual with the row-expression parser, rewrites `alias.col` to the bare column its own frame actually carries, and lowers it with `lower_expr` through the new `row_pipeline.lower_single_alias_predicate`. **Parity is by construction, not by re-derivation.** The fallback being replaced is `where_rows_polars`, which is that same parser plus that same `lower_expr` under the same `_SCHEMA` dtypes followed by `table.filter(...)`; `alias.col ↔ col` is a bijection over the one frame involved; so the fast lane builds the expression the fallback would build and applies it the way the fallback would apply it. The accept set, the decline set and the answers are the evaluator's, and the dtype, temporal-literal, int-literal-division and float-NaN guards are inherited rather than restated, so the fast lane cannot drift from the evaluator when those guards change. Newly covered — each pinned by a differential against the forced chain fallback on the same frame AND by the expected rows, so a differential both sides get wrong cannot pass as parity: `!=` and `<>` (including the sharp edge, 3-valued NULL: a NULL operand yields NULL and `filter` DROPS the row, exactly as for `=`, where a pandas-shaped object-dtype `!=` would have kept it), `IS NULL`, `IS NOT NULL`, `IN [literals]`, `OR`, `NOT`, nested boolean combinations, `CASE WHEN`, arithmetic, and the row lowering's function whitelist (`substring`, `size`, …). Three former declines are retired as unnecessary rather than preserved: the **escaped-literal** decline (`'it\u0027s'`, `'C:\u005Cx'`) is gone because the literal is now unescaped by the evaluator's own parser instead of compared as raw regex-captured text, and is replaced by a non-vacuous differential over rows that really contain a quote and a backslash; the purely **layout** rejections (missing outer parens, reversed operand order, a function on the column side) are gone because nothing is matched by text shape any more; and `=`/`!=`/`IS NULL`/`IN` on a **Categorical** column translate now, since those are not `.str` kernels and the evaluator answers them (`toLower`/`toUpper` on Categorical still declines, because `.str` on Categorical raises in polars only). Every remaining decline is a shape the fallback declines too, so the row op's designed parity-or-error `NotImplementedError` still surfaces instead of a raw polars `ComputeError`: a property access on another alias, a bare identifier (including the whole-entity identity sentinel, which resolves only through the row table's `_NODE_ID`), an absent column, a numeric literal against a String column and the converse, a case function on a non-String column, and any node type outside the row lowering's own whitelist (map/subscript/slice/quantifier/comprehension). **`STARTS WITH` / `ENDS WITH` / `CONTAINS` / `=~` are deliberately NOT covered, and the reason is reachability, not difficulty**: `_pushdown_connected_join_where_filters` cannot render those to a row filter, so on the polars engine the whole comma-pattern query is rejected upstream with `GFQLValidationError` and no such residual ever reaches this translator — pinned by a test that spies the translator and asserts it is never called. Teaching it those shapes would be unreachable code; the real gap is in the connected-join WHERE renderer, where pandas answers those queries today and polars does not. **The justification is the vocabulary the Cypher front end already accepts, not a number** — and the number, once measured, was negative: the board's ten residual translations were all ALREADY native under the narrow matcher, so the widening converted no decline there and only added cost, regressing q7 by 1.24x/1.16x (20k/100k, non-overlapping per-slot ranges). Both mechanisms and the fix are the next entry; the widened COVERAGE is unaffected by it.
 - **`is_polars_df` is a type predicate, and the engine-frame vocabulary has a canonical home**: `graphistry/compute/typing.py` now exports `PolarsFrame` (the `pl.DataFrame | pl.LazyFrame` union), a `PolarsT` TypeVar and `PolarsSeriesT`, TYPE_CHECKING-only so polars stays an optional dependency and no runtime import or version floor is added; `polars/dtypes.py`, which had defined the first two, re-exports them so there is ONE definition. `DataFrameT` is deliberately left pinned to `pd.DataFrame` — widening it into a cross-engine union fans out across every checked module, and the fix for a polars-only helper is to name polars, not to blur the pandas alias. Against that vocabulary `Engine.is_polars_df` is now declared `(df: object) -> TypeGuard["PolarsFrame"]` instead of `(df: Any) -> bool`, so the polars branch of an engine-dispatching helper is actually CHECKED against the polars API rather than silently accepted. `TypeGuard`, not `TypeIs` (PEP 742): `TypeIs` also narrows the negative branch, but to do that soundly it requires the narrowed type to be consistent with the declared input, and a polars frame is not a subtype of the pandas type these call sites declare — that is the difference from `dtypes.is_lazy`, whose input is already a `PolarsFrame`. A companion `is_polars_series` (the same import-light module check) narrows the SERIES call sites to `pl.Series`, which the frame guard would have mistyped. Consequences: the six per-branch `# type: ignore` comments in `Engine.py`'s dispatch block were written against an older stub set and named codes (`attr-defined`, `no-any-return`) that no longer applied — they were dead, suppressing nothing, and are now scoped to the one thing that genuinely cannot typecheck, the polars return value flowing out of a `DataFrameT`/`SeriesT` signature; `_pl_nan_to_null` takes the `PolarsFrame` union instead of the constrained `PolarsT` TypeVar, which a union argument cannot bind (an `@overload` set would keep the per-flavour precision but is unusable here — on the polars-less type-lint lane every signature collapses to `Any -> Any`); and two rebindings that mixed a polars result into a pandas-typed local (`gfql/row/frame_ops.py`, `gfql_unified.py`) return or branch directly instead. Typing only: no runtime behaviour, no public API, no new dependency. Measured on a stub-visible mypy lane with polars importable, repo-wide errors go 163 -> 157 (11 -> 9 files); the lane CI actually runs today is unaffected and still reports no issues.

graphistry/compute/gfql_unified.py

@@ -1852,6 +1852,37 @@ def gfql(self: Plottable,
     :returns: Resulting Plottable
     :rtype: Plottable
     """
+    # engine inference: resolve_engine(AUTO) maps polars frames to PANDAS (polars predates
+    # Engine.POLARS there), silently bridging polars-frame graphs onto the generic pandas path
+    # and handing pandas frames back. Measured on the matched graph-benchmark q1-q9 lane through
+    # this exact surface (dgx-spark, perf lock, position-balanced over 8 slots): 2.45-11.9x at 20k
+    # and 4.8-37.2x at 100k, values identical. Route AUTO to the native polars engine instead;
+    # an honest NIE (unsupported shape) falls back to the legacy AUTO path -- allowed here
+    # because the user pinned no engine -- at a flat ~0.23 ms, the decline being raised at
+    # planning before any data is touched.
+    #
+    # ``policy is None`` is REQUIRED, not conservatism: the native polars executor does not go
+    # through ``chain_impl``, so it never emits the ``postload``/``postchain`` hooks that path
+    # emits. Routing a policy-carrying query there would silently stop enforcing a DENYING
+    # ``postload`` policy (measured: deny-on-postload blocks under the generic path and does not
+    # under the native one) -- a governance hook that stops firing is worse than a slow query.
+    # The NIE fallback compounds it: re-running the query would fire ``preload``/``precompile``/
+    # ``postcompile`` twice for one user call. Explicit ``engine='polars'`` is unchanged and still
+    # carries the pre-existing hook gap; this guard only refuses to make that gap the default.
+    if (
+        (engine == EngineAbstract.AUTO or engine == EngineAbstract.AUTO.value)
+        and policy is None
+        and is_polars_df(self._edges) and (self._nodes is None or is_polars_df(self._nodes))
+    ):
+        try:
+            return gfql(
+                self, query, engine=Engine.POLARS.value, output=output, policy=policy,
+                where=where, language=language, params=params, validate=validate,
+                shortest_path_backend=shortest_path_backend,
+            )
+        except NotImplementedError:
+            logger.debug('AUTO polars-native attempt declined; falling back to generic path')
+
     context = ExecutionContext()
 
     if policy and context.policy_depth >= 1:

graphistry/tests/compute/gfql/cypher/test_lowering.py

@@ -15998,7 +15998,7 @@ def test_connected_join_polars_nullable_columns_match_master(predicate: str, exp
     query = f"MATCH (p)-[]->(x), (p)-[]->(y) WHERE {predicate} RETURN count(p) AS n"
 
     out = g.gfql(query)._nodes
-    records = out.to_dict(orient="records") if hasattr(out, "to_dict") else out.to_dicts()
+    records = out.to_dicts() if hasattr(out, "to_dicts") else out.to_dict(orient="records")
     assert records == expected
 
 
@@ -16028,7 +16028,7 @@ def test_arrow_table_nodes_match_master_results() -> None:
     def run(predicate: str) -> Any:
         query = f"MATCH (p)-[]->(x), (p)-[]->(y) WHERE {predicate} RETURN count(p) AS n"
         out = g.gfql(query)._nodes
-        return out.to_dict(orient="records") if hasattr(out, "to_dict") else out.to_dicts()
+        return out.to_dicts() if hasattr(out, "to_dicts") else out.to_dict(orient="records")
 
     assert run("p.name = 5") == []
     assert run("p.name = 'ann'") == [{"n": 4}]
@@ -16085,7 +16085,7 @@ def test_connected_join_polars_decimal_matches_master(predicate: str, expected:
     query = f"MATCH (p)-->(a), (p)-->(b) WHERE {predicate} RETURN count(p) AS n"
 
     result = g.gfql(query)._nodes
-    records = result.to_dict(orient="records") if hasattr(result, "to_dict") else result.to_dicts()
+    records = result.to_dicts() if hasattr(result, "to_dicts") else result.to_dict(orient="records")
     assert records == expected
 
 
@@ -16100,7 +16100,7 @@ def test_connected_join_polars_backed_graph_matches_pandas_results() -> None:
 
     g_polars = graphistry.nodes(pl.from_pandas(nodes), "id").edges(pl.from_pandas(edges), "s", "d")
     result = g_polars.gfql(query)._nodes
-    records = result.to_dict(orient="records") if hasattr(result, "to_dict") else result.to_dicts()
+    records = result.to_dicts() if hasattr(result, "to_dicts") else result.to_dict(orient="records")
     assert records == []
 
 

graphistry/tests/compute/gfql/index/test_index.py

@@ -1399,3 +1399,65 @@ def test_engine_mismatch_reason_reports_all_undirected_indexes(
         "resident edge_out_adj, edge_in_adj index "
         f"engine={resident_engine}, requested engine={requested_engine} -> scan"
     ), reason
+
+
+# ---------------------------------------------------------------------------
+# 1743: engine=auto on polars-frame graphs routes to the native polars engine,
+# so a polars-engine index built explicitly is actually consulted through
+# g.gfql(...) with NO engine argument. Regression pin for the #1767 cliff:
+# before the AUTO route, resolve_engine(AUTO) mapped polars frames to pandas,
+# so the resident polars index was engine-mismatched and every query scanned.
+# ---------------------------------------------------------------------------
+
+def _polars_indexed_graph():
+    pl = pytest.importorskip("polars")
+    nodes = pd.DataFrame({
+        "id": np.arange(1, 13, dtype=np.int64),
+        "public": np.arange(100, 112, dtype=np.int64),
+    })
+    edges = pd.DataFrame({
+        "src": [1, 1, 1, 2, 2, 3, 4, 5, 6, 8],
+        "dst": [2, 2, 3, 4, 5, 5, 6, 6, 7, 1],
+        "type": ["A", "A", "A", "B", "B", "B", "C", "C", "D", "REV"],
+    })
+    g = graphistry.nodes(pl.from_pandas(nodes), "id").edges(
+        pl.from_pandas(edges), "src", "dst"
+    )
+    return g.gfql_index_all(engine="polars")
+
+
+def test_auto_engine_gfql_serves_polars_index_1767_cliff():
+    """#1767 cliff pin: polars frames + explicit polars index + gfql with NO engine
+    argument must serve path=index on engine=polars (AUTO routes native, so the
+    resident index is no longer engine-mismatched into a silent scan)."""
+    from graphistry.compute.gfql.index import index_trace
+
+    gi = _polars_indexed_graph()
+    q = ("MATCH (source {public:100})-[:A]->(destination) "
+         "RETURN destination.public AS destination ORDER BY destination")
+    with index_trace() as steps:
+        out = gi.gfql(q)  # no engine argument: default AUTO, default index policy
+    assert "polars" in type(out._nodes).__module__, "AUTO must answer in polars frames"
+    served = [s for s in steps if s.get("served") is True]
+    assert served, ("no index-served step; AUTO fell off the polars route", steps)
+    for s in served:
+        assert s.get("path") == "index", steps
+        assert s.get("engine") == "polars", steps
+    assert out._nodes["destination"].to_list() == [101, 102]
+
+
+def test_auto_engine_hop_residual_still_scans_1767():
+    """Documented residual (#1743 scope boundary, executable): direct g.hop() with no
+    engine on the same polars-frame graph still resolves AUTO to pandas, so the
+    resident polars index declines with an engine mismatch and the hop scans."""
+    from graphistry.compute.gfql.index import index_trace
+
+    gi = _polars_indexed_graph()
+    seeds = pd.DataFrame({"id": [1]})
+    with index_trace() as steps:
+        out = gi.hop(nodes=seeds, hops=1, direction="forward")  # no engine argument
+    assert "pandas" in type(out._nodes).__module__, "hop AUTO still bridges to pandas"
+    assert not any(s.get("path") == "index" for s in steps), steps
+    mismatch = [s for s in steps if s.get("decision_code") == "engine_mismatch"]
+    assert mismatch, ("expected an engine_mismatch decline", steps)
+    assert "requested engine=pandas" in mismatch[-1]["decision_reason"], steps