Skip to content

Commit 83db420

Browse files
lmeyerovclaude
andcommitted
fix(gfql): IS5 projection slice — pivot dtype parity, empty edges frame, engine-mismatch decline
Review-skill findings (plans/review-pr-1766/review.md): B1 int/bool property dtypes diverged from the full pandas path (rows-pivot upcasts non-id int/float->float64, bool->object; id keeps dtype). The lean tail now applies the same casts; unverified dtype classes (datetimes, extension dtypes, categoricals) decline to the full path. M1 res._edges was None where the full path yields an empty edges frame; now the helper edges' zero-row head. M2 requested-vs-actual engine mismatch (e.g. polars frames + engine= 'pandas') declines — the full path converts to the requested engine. M3 CHANGELOG corrected to the measured 116.3->38.3ms (was 'low single-digit ms'). Pinned in TestSeededProjectionDtypeAndEdgesParity (dtype parity pandas+ polars, datetime decline, edges empty-not-None, both mismatch directions), all engagement-asserted. 14-case fast-vs-full probe matrix: exact parity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
1 parent 440c60d commit 83db420

3 files changed

Lines changed: 116 additions & 2 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
99
<!-- Do Not Erase This Section - Used for tracking unreleased changes -->
1010

1111
### Performance
12-
- **GFQL seeded fast path covers single-alias property RETURNs (LDBC IS5 shape) (#1755)**: `MATCH (m {id})-[:T]->(p) RETURN p.a AS x, p.b` — which lowers to `rows(source=p)+select(items)` rather than a whole-row projection — now takes the seeded typed-hop fast path: the deduped destination rows are renamed/selected directly (value-identical to the rows-pivot + select pipeline; row order may differ). Conservative declines keep full-path semantics for everything else: cross-alias refs (`RETURN m.x, p.y`), mixed whole-row+property, DISTINCT/ORDER BY/LIMIT (extra lowered ops), expr items, and properties absent from the node frame (the full path's null/error semantics must apply). Differential fast-vs-full sweep across all of the above on pandas + polars: exact parity, engagement pinned both ways. Measured on LDBC SNB SF1 (dgx): `message-creator` (IS5) drops from 116 ms to low single-digit ms on polars.
12+
- **GFQL seeded fast path covers single-alias property RETURNs (LDBC IS5 shape) (#1755)**: `MATCH (m {id})-[:T]->(p) RETURN p.a AS x, p.b` — which lowers to `rows(source=p)+select(items)` rather than a whole-row projection — now takes the seeded typed-hop fast path: the deduped destination rows are renamed/selected directly (value-identical to the rows-pivot + select pipeline; row order may differ). Dtype parity with the full pandas path is exact: non-id properties ride the rows-pivot there, which upcasts int/float to float64 and bool to object, so the lean tail applies the same casts (the id property keeps its dtype); dtype classes not verified against the pivot (datetimes, pandas extension dtypes, categoricals) decline. `res._edges` is the same empty edges frame the full path yields (never None). Conservative declines keep full-path semantics for everything else: cross-alias refs (`RETURN m.x, p.y`), mixed whole-row+property, DISTINCT/ORDER BY/LIMIT (extra lowered ops), expr items, properties absent from the node frame (the full path's null/error semantics must apply), and requested-vs-actual engine mismatches (the full path converts to the requested engine). Differential fast-vs-full sweep across all of the above on pandas + polars: exact parity incl. dtypes and edges, engagement pinned both ways. Measured on LDBC SNB SF1 (dgx, harness row-validated): `message-creator` (IS5) 116.3→38.3 ms on polars (the residue is the seed scan, targeted separately by resident-index coverage).
1313
- **GFQL seeded typed-hop fast path (#1755)**: a seeded typed 1-hop — native chain `[n({id}), e_forward(), n({type})]` or Cypher `MATCH (m {id})-[:T]->(p) RETURN p` — previously paid the full two-pass chain machinery (~20-40ms: whole-frame combines, full-column type filters, rows-pivot projection). A new seed-first fast path recognizes exactly this shape and reduces the graph to the seed's 1-hop neighborhood before any of that work: native chain 32.6→0.93ms (35×), Cypher RETURN 39.2→1.9ms (20×) on pandas, with cuDF covered via the shared DataFrame API (30.8→4.7ms). Value-identical to the full path by construction (same rows/columns/dtypes; row order and index may differ) — the helpers return `None` and fall through for anything outside the exact shape or carrying full-path side-channels (multi-hop, variable-length, undirected, predicate filters, reverse patterns, missing bindings, list-`labels` columns, policy hooks, same-path WHERE, OPTIONAL MATCH null rows, and WITH..MATCH carried seeds all decline). Null ids/endpoints never link (membership sets are null-dropped, matching the full pipeline's joins). Verified with an independent oracle (results checked against the creator set hand-computed from the raw frames, not merely fast-vs-slow agreement) plus a differential fast-vs-full sweep across shapes × engines in `test_seeded_typed_hop_fastpath.py`.
1414
- **GFQL seeded typed-hop fast path on polars/polars-gpu (#1755)**: extends the seeded typed-hop Cypher fast path to the polars engines via native polars filters (`_seeded_typed_return_dst_polars`), so a seeded typed 1-hop RETURN also lands in single-digit ms on polars (13.7→3.4ms, 4×) and polars-gpu (24.6→2.5ms, 10×). Dispatch is on the ACTUAL frame type (`is_polars_df`), not the requested engine, because WITH..MATCH reentry can request polars while handing pandas-materialized frames. Same decline contract as the pandas path (undirected/multi-hop/varlen/predicates fall through; value-identical for the covered shape), verified by the per-engine differential sweep and oracle tests in `test_seeded_typed_hop_fastpath.py`.
1515
- **GFQL connected-join simple residuals filter natively on polars (#1729/#1755)**: on the polars engines, a connected-join node residual of the #1729 scalar shapes — case-insensitive equality ``(tolower(a.col) = tolower('lit'))`` and scalar comparisons ``(a.col <op> literal)`` for ``= >= <= > <`` — previously dispatched a full sub-``chain()`` per alias (the polars row pipeline has no native ``where_rows``), costing several ms per residual on OLAP group-aggregate queries. `_residual_polars_expr` now translates these shapes to native ``pl.Expr`` filters applied directly to the alias frame (graph-benchmark q5 10.2→6.6ms, q6 10.7→8.9ms on dgx). Null semantics match the ``where_rows`` evaluator (null comparisons drop the row); any unrecognized shape, alias mismatch, or absent column falls back to the existing chain path — and if *any* expr in a residual group fails to translate, the whole group falls back (never a partial mix). Byte-parity verified across an 8-case adversarial differential (nulls, unicode/casefold, numeric ranges, mixed residuals) plus the full suite.

graphistry/compute/gfql_fast_paths.py

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1957,6 +1957,11 @@ def _execute_seeded_typed_hop_fast_path(
19571957
is_polars = is_polars_df(nodes_frame)
19581958
if is_polars != is_polars_df(base_graph._edges):
19591959
return None # mixed-engine node/edge frames: decline, full path decides
1960+
if select_items is not None and (requested_engine in (Engine.POLARS, Engine.POLARS_GPU)) != is_polars:
1961+
# Requested-vs-actual engine mismatch (e.g. polars frames + engine='pandas'):
1962+
# the full path CONVERTS the result to the requested engine, so the lean
1963+
# projection would leak actual-engine frames. Decline; the full path decides.
1964+
return None
19601965
helper = _seeded_typed_return_dst_polars if is_polars else _seeded_typed_return_dst_pandas_cudf
19611966
dst_res = helper(base_graph, n0, n2, e1, src, dst, node, direction)
19621967
if dst_res is None:
@@ -1971,11 +1976,36 @@ def _execute_seeded_typed_hop_fast_path(
19711976
import polars as pl
19721977
out_frame = p_rows.select([pl.col(prop).alias(out) for out, prop in select_items])
19731978
else:
1979+
# dtype parity with the full path: non-id properties ride the rows-pivot,
1980+
# which upcasts int/float -> float64 and bool -> object; the node-id
1981+
# property passes through the pivot key and keeps its dtype. Any dtype
1982+
# class not verified against the pivot (datetimes, pandas extension
1983+
# dtypes like nullable Int64/StringDtype, categoricals) declines to the
1984+
# full path rather than risk a silent dtype divergence.
1985+
import numpy as np
1986+
casts: Dict[str, str] = {}
1987+
for out_name, prop in select_items:
1988+
if prop == node:
1989+
continue
1990+
d = p_rows[prop].dtype
1991+
if not isinstance(d, np.dtype):
1992+
return None
1993+
if d == np.dtype(bool):
1994+
casts[out_name] = "object"
1995+
elif d.kind in "iuf":
1996+
casts[out_name] = "float64"
1997+
elif d.kind != "O":
1998+
return None
19741999
out_frame = p_rows[[prop for _, prop in select_items]].copy()
19752000
out_frame.columns = [out for out, _ in select_items]
2001+
for out_name, target in casts.items():
2002+
out_frame[out_name] = out_frame[out_name].astype(target)
19762003
out = base_graph.bind()
19772004
out._nodes = out_frame
1978-
out._edges = None
2005+
# full-path parity: a property RETURN yields an EMPTY edges frame with the
2006+
# edge schema (never None — res._edges must stay usable). The helper's
2007+
# edges are the matched hop edges, so take their zero-row head.
2008+
out._edges = _edges.head(0)
19792009
return out
19802010
assert projection is not None # narrowed by the gate above
19812011
# Lean projection: p_rows already IS the RETURN-alias (destination) node set.

graphistry/tests/compute/gfql/test_seeded_typed_hop_fastpath.py

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -688,3 +688,87 @@ def test_out_of_shape_declines_with_parity(self, q, label):
688688
import pytest as _pt
689689
with _pt.raises(Exception):
690690
_run_diff(g, "pandas", q, fast=False)
691+
692+
693+
class TestSeededProjectionDtypeAndEdgesParity:
694+
"""Review pins (plans/review-pr-1766/review.md): B1 int-property dtype parity
695+
(the full pandas path's rows-pivot upcasts non-id int/float props to float64
696+
and bool to object; the id prop keeps its dtype), M1 res._edges is an EMPTY
697+
edges frame (never None), M2 requested-vs-actual engine mismatch declines."""
698+
699+
Q = ("MATCH (m:Message {id:10})-[{type:'HAS_CREATOR'}]->(p:Person) "
700+
"RETURN p.id AS pid, p.age AS a, p.flag AS f")
701+
702+
def _typed_graph(self):
703+
ndf = pd.DataFrame({
704+
"id": [0, 1, 2, 10, 11],
705+
"type": ["Person"] * 3 + ["Message"] * 2,
706+
"age": [30, 40, 50, 0, 0], # int64: pivot upcasts to float64
707+
"flag": [True, False, True, False, False], # bool: pivot upcasts to object
708+
"ts": pd.to_datetime(["2020-01-01"] * 5), # datetime: fast path declines
709+
})
710+
edf = pd.DataFrame({"src": [10, 10, 11], "dst": [0, 1, 2],
711+
"type": ["HAS_CREATOR"] * 3})
712+
return graphistry.nodes(ndf, "id").edges(edf, "src", "dst")
713+
714+
def _pl_graph(self):
715+
pl = pytest.importorskip("polars")
716+
g = self._typed_graph()
717+
return graphistry.nodes(pl.from_pandas(g._nodes), "id").edges(
718+
pl.from_pandas(g._edges), "src", "dst")
719+
720+
def _fast_and_full(self, g, engine, q, expect_engage=True):
721+
hits = {"n": 0}
722+
real = gfql_unified._execute_seeded_typed_hop_fast_path
723+
724+
def spy(*a, **k):
725+
r = real(*a, **k)
726+
hits["n"] += r is not None
727+
return r
728+
gfql_unified._execute_seeded_typed_hop_fast_path = spy
729+
try:
730+
fast = g.gfql(q, engine=engine)
731+
finally:
732+
gfql_unified._execute_seeded_typed_hop_fast_path = real
733+
full = _run_diff(g, engine, q, fast=False)
734+
assert bool(hits["n"]) == expect_engage, f"engaged={hits['n']} expected={expect_engage}"
735+
return fast, full
736+
737+
def test_pandas_int_bool_dtype_parity(self):
738+
fast, full = self._fast_and_full(self._typed_graph(), "pandas", self.Q)
739+
pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full))
740+
dt = dict(zip(fast._nodes.columns, map(str, fast._nodes.dtypes)))
741+
assert dt == {"pid": "int64", "a": "float64", "f": "object"}
742+
743+
def test_polars_int_bool_dtype_parity(self):
744+
pytest.importorskip("polars")
745+
fast, full = self._fast_and_full(self._pl_graph(), "polars", self.Q)
746+
assert dict(fast._nodes.schema) == dict(full._nodes.schema)
747+
pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full))
748+
749+
def test_pandas_datetime_property_declines(self):
750+
q = self.Q.replace("p.flag AS f", "p.ts AS t")
751+
fast, full = self._fast_and_full(self._typed_graph(), "pandas", q, expect_engage=False)
752+
pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full))
753+
754+
@pytest.mark.parametrize("engine", ["pandas", "polars"])
755+
def test_edges_empty_frame_not_none(self, engine):
756+
g = self._typed_graph() if engine == "pandas" else self._pl_graph()
757+
if engine == "polars":
758+
pytest.importorskip("polars")
759+
fast, full = self._fast_and_full(g, engine, self.Q)
760+
assert fast._edges is not None
761+
assert fast._edges.shape[0] == 0
762+
assert sorted(map(str, fast._edges.columns)) == sorted(map(str, full._edges.columns))
763+
assert full._edges.shape[0] == 0
764+
765+
def test_engine_mismatch_declines(self):
766+
pytest.importorskip("polars")
767+
# polars frames + engine='pandas': full converts to pandas; fast must decline
768+
fast, full = self._fast_and_full(self._pl_graph(), "pandas", self.Q, expect_engage=False)
769+
assert type(fast._nodes).__module__.startswith("pandas")
770+
pd.testing.assert_frame_equal(_canon_nodes(fast), _canon_nodes(full))
771+
# pandas frames + engine='polars' (reentry direction): also declines
772+
fast2, full2 = self._fast_and_full(self._typed_graph(), "polars", self.Q, expect_engage=False)
773+
assert type(fast2._nodes).__module__ == type(full2._nodes).__module__
774+
pd.testing.assert_frame_equal(_canon_nodes(fast2), _canon_nodes(full2))

0 commit comments

Comments
 (0)