Skip to content

Commit f19f2cc

Browse files
lmeyerovclaude
andcommitted
feat(gfql): native polars undirected var-length bindings (-[*1..k]-) for IC11/IC6 cross-alias WHERE
binding_rows_polars now materializes the bounded UNDIRECTED variable-length bindings table for min_hops==1 (the LDBC IC11/IC6 -[*1..k]- shape), unblocking the cross-alias same-path `WHERE NOT a = b` / `a <> b` clause on polars (the WHERE lowering itself was already native; the residual was the undirected `rows(binding_ops=...)` materialization declining). Exact port of the pandas oracle `_gfql_multihop_binding_rows` (avoid_immediate_backtrack=True): a `__prev__` marker (seeded null, dtype-matched to the id column) drops immediate backtracks each hop, and the step-pair set reproduces pandas' edge multiplicity (each non-loop edge each directed orientation x2; self-loops (u,u) x2 only). Scoped to min_hops==1 where the raw-edge reconstruction provably matches pandas; min_hops 0 and >=2 decline with an honest NotImplementedError (never silent-wrong). Differential fuzz vs pandas over ~2500 random graphs (self-loops, parallel + antiparallel edges, *1..2..*1..5, all WHERE/RETURN variants): 0 disagreements; out-of-scope windows decline. Tests: parity + multiplicity/backtrack/self-loop/ string-id pins + *0..2/*2..3/*2..2 decline pins. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
1 parent e676fb0 commit f19f2cc

3 files changed

Lines changed: 173 additions & 22 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
1313

1414
### Fixed
1515
- **GFQL polars engine natively runs `UNWIND` of a carried `collect()` list column**: `... WITH collect(x) AS xs UNWIND xs AS y RETURN ...` (exploding a list-valued column produced by `collect()`, the row-pipeline analogue of the IC6 unwind shape) previously declined on polars — `unwind_polars` only accepted a scalar literal list. Now explodes a `List`-dtype column reference (`with_columns` copy → `filter(list.len() > 0)` → `explode`), matching the pandas oracle exactly: empty-list/null cells → 0 rows, nulls *within* a list survive, source column retained; the pre-filter also makes it independent of the polars-2.0 `empty_as_null` default. Non-list columns, name collisions, unknown identifiers, and nested-list literals still decline (honest NIE, no pandas bridge). Differential fuzz vs pandas over ~3500 collect→UNWIND→{RETURN, grouped, filtered, ORDER BY, aggregate, nested-unwind} queries: 0 disagreements. (The `MATCH ... WITH collect(..) UNWIND .. MATCH` form — IC6 with a trailing re-traversal — is compile-rewritten into WITH→MATCH reentry, a separate path.) Tests in `test_engine_polars_row_pipeline.py`.
16+
- **GFQL polars engine natively runs the LDBC IC11/IC6 undirected variable-length bindings table (`-[*1..k]-`), unblocking the cross-alias `WHERE NOT(person=friend)` clause**: a bounded UNDIRECTED variable-length MATCH that materializes a bindings row table (`rows(binding_ops=...)`, emitted whenever a downstream clause — e.g. a cross-alias same-path `WHERE NOT a = b` / `a <> b` — references two node aliases on the path) previously declined on polars (`binding_rows_polars` hard-NIE'd every undirected multihop), while pandas handled it. This was the single residual blocking the official LDBC IC11/IC6 queries on polars (the cross-alias WHERE lowering itself was already native — directed var-length + `WHERE NOT a = b` already matched pandas). `binding_rows_polars` now supports undirected var-length with **`min_hops == 1`** via a doubled-pair join with immediate-backtrack avoidance, an exact port of the pandas oracle `_gfql_multihop_binding_rows` (`avoid_immediate_backtrack=True`): a `__prev__` marker (seeded null, dtype-matched to the id column) drops immediate backtracks each hop (Kleene mask: null prev kept), and the step-pair set reproduces pandas' edge multiplicity exactly — each non-loop edge contributes each directed orientation TWICE (so a length-1 pair appears x2, length-2 x4) while self-loops contribute `(u,u)` x2 only (not double-counted). The multiplicity rule was derived by instrumenting pandas' `step_pairs` (which flow from the var-length `edge_op.execute` hop + `orient_edges`), NOT by reading code alone. Scoped to `min_hops == 1` because that is where the raw-edge reconstruction provably matches pandas: every edge is trivially a length-1 path so the var-length hop's backward pruning removes nothing; `min_hops == 0` (zero-hop, undoubled) and `min_hops >= 2` (backward-pruned / long-walk divergence) still **decline with an honest `NotImplementedError`** rather than risk silent-wrong multiplicities. Differential fuzz vs the pandas oracle over ~2500 random graphs (self-loops, parallel + antiparallel edges, `*1..2`…`*1..5`, all WHERE/RETURN variants): 0 disagreements; the out-of-scope windows decline (never diverge). Tests in `test_engine_polars_binding_rows.py` (parity + multiplicity/backtrack/self-loop/string-id pins + `*0..2`/`*2..3`/`*2..2` decline pins).
1617
- **GFQL polars engine natively runs multi-source (node-cartesian) MATCH (#1273)**: comma-separated disconnected aliases — `g.gfql("MATCH (a {..}), (b {..}) RETURN a.id, b.id")` — previously declined on polars (the `rows(binding_ops)` `node_cartesian` branch hard-NIE'd) while pandas handled them. New `_cartesian_node_bindings_polars` mirrors the pandas oracle `_gfql_cartesian_node_bindings_row_table`: each alias is independently filtered, projected into the per-alias lookup schema (bare `alias` id, `alias.id`, `alias.<prop>`, and pandas' leaked `alias.alias=True` flag incl. property-shadowing), then left-major cross-joined for row-order parity. Scalar/aliased/`count(*)` projections over the cartesian match pandas exactly (verified end-to-end); whole-entity `RETURN a, b` stays an honest NIE (separate projection surface). Two parity-safe declines mirror shapes where pandas itself errors (proven on master, so both engines fail identically, never diverge): an anonymous node op, and ≥4 named aliases (pandas' bare-id merge residue collides). Differential fuzz vs pandas over ~10k lowered cases: 0 disagreements. Tests in `test_engine_polars_binding_rows.py`.
1718
- **GFQL polars chain — variable-length node aliases are hop-distance gated; the chain is now silent-wrong-free (#1741)**: a node named after a variable-length edge (`g.gfql("MATCH (a)-[*1..2]-(b) RETURN b")`) previously carried its alias regardless of hop distance, so an undirected walk that backtracked into the seed wrongly returned it (pandas correctly excludes it — trail semantics). The polars chain now auto-injects the #1741 hop-distance label (name resolved against the user's node columns via the `reserved_columns` registry, so a user column named like the internal one is never clobbered) and applies pandas' alias `[min_hop, max_hop]` window in `_apply_node_names`. The one shape the gate can't yet cover — a node alias after a **forward/reverse `min_hops>1`** edge, whose labels need pandas' layered backward walk (not ported) — now **declines with an honest `NotImplementedError` (#1748)** instead of returning nodes outside the window; the decline is precise (differential vs pandas: 30/30 named shapes NIE, 30/30 unnamed run and match, 0 over-decline). 4-engine A/B on the stacked build: pandas/cudf 144/144, polars/polars-gpu 112 agree / **0 disagree** / 32 honest-NIE — zero silent-wrong shapes remain on the polars chain. Adapts the mechanism from the retired #1742 (declined undirected varlen+alias, now natively gated). Tests: `TestVarlenAliasHopGate` (pandas-parity across directions, the `*2..3` decline + unnamed-still-runs pins, a column-collision test).
1819
- **GFQL native polars `label_node_hops` on the plain BFS — correct, direction-dependent hop labels (#1741 groundwork)**: the polars eager hop now emits pandas-parity node hop-distance labels instead of declining `label_node_hops`. The labeling rule is DIRECTION-DEPENDENT (the subtle part, derived by differential fuzz vs the pandas oracle, not by reading pandas alone): forward/reverse label EVERY destination of a hop first-wins (matching pandas `hop.py` `new_node_ids`), so a seed re-entered at hop 1 IS labeled; undirected labels destinations MINUS everything already visited and pre-seeds the seen-set with the seeds, so a seed re-reached by a backtracking walk stays NULL (its shortest-path distance). Two correctness fixes over the first cut: (1) the undirected seed pre-seed must NOT depend on `return_as_wave_front` — a seed filtered out of the frontier by `source_node_match` or suppressed in wave-front mode was being re-labeled (A/B vs pandas over direction × hops × `return_as_wave_front` × `source_node_match` × `destination_node_match` × 10 graphs: was 456/24, now 480/480); (2) a requested `label_node_hops` name that collides with an existing column is redirected to `<name>_1` like pandas' `resolve_label_col`, not the polars left-join auto-suffix `<name>_right`/`DuplicateError`. `label_edge_hops` and `min_hops>1` labels stay honest NIEs. Amplified `test_engine_polars_hop.py` (`TestHopLabelsDifferential` over the seed-filter / wave-front / multi-seed axes, an explicit direction-asymmetry contrast pin, and a collision test).

graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py

Lines changed: 83 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1057,17 +1057,31 @@ def _names(lf: pl.LazyFrame) -> List[str]:
10571057
sem = EdgeSemantics.from_edge(op)
10581058
if sem.is_multihop:
10591059
# Bounded directed var-length (`-[*1..k]->`, graph-bench q3) is
1060-
# supported via iterative pair joins; everything else declines:
1061-
# unbounded (`[*]`, needs fixed-point + termination error),
1062-
# undirected multihop (immediate-backtrack avoidance not ported),
1063-
# and aliased var-length edges (pandas rejects those outright).
1060+
# supported via iterative pair joins. Bounded UNDIRECTED var-length
1061+
# with min_hops == 1 (`-[*1..k]-`, the LDBC IC11/IC6 shape) is now
1062+
# also supported via a doubled-pair join with immediate-backtrack
1063+
# avoidance (see the execution branch below). Everything else declines:
1064+
# unbounded (`[*]`, needs fixed-point + termination error), aliased
1065+
# var-length edges (pandas rejects those outright), and undirected
1066+
# var-length with min_hops != 1 (`-[*0..k]-` / `-[*2..k]-`): pandas'
1067+
# step_pairs come from the var-length `edge_op.execute` hop, whose
1068+
# backward hop-window pruning / zero-hop handling changes the edge
1069+
# multiplicity in a way this raw-edge reconstruction only reproduces
1070+
# for min_hops == 1 (every edge is trivially a length-1 path, so no
1071+
# pruning occurs) — fuzz-verified vs the pandas oracle. Decline the
1072+
# rest honestly rather than risk silent-wrong multiplicities.
10641073
if (
1065-
op.direction == "undirected"
1066-
or bool(op.to_fixed_point)
1074+
bool(op.to_fixed_point)
10671075
or (op.max_hops is None and op.hops is None)
10681076
or isinstance(op._name, str)
10691077
):
10701078
return None
1079+
if op.direction == "undirected":
1080+
_resolved_min = op.min_hops if op.min_hops is not None else (
1081+
op.hops if op.hops is not None else 1
1082+
)
1083+
if _resolved_min != 1:
1084+
return None
10711085
if op.direction not in ("forward", "reverse", "undirected"):
10721086
return None
10731087
if any(
@@ -1174,23 +1188,71 @@ def _names(lf: pl.LazyFrame) -> List[str]:
11741188
return None
11751189
min_hops = int(min_hops_value)
11761190
max_hops = int(max_hops_value)
1177-
pairs = oriented.select(["__from__", "__to__"])
11781191
state_cols = _names(state)
1179-
reachable = [state] if min_hops == 0 else []
1180-
current = state
1181-
# Lazy: build all max_hops iterations (no eager .height early-break —
1182-
# empty intermediates lazily join to empty, so the result is
1183-
# identical; the pandas break is an optimization, not semantics).
1184-
for _hop in range(1, max_hops + 1):
1185-
current = (
1186-
current.join(pairs, left_on="__current__", right_on="__from__", how="inner")
1187-
.drop("__current__")
1188-
.rename({"__to__": "__current__"})
1189-
.select(state_cols)
1192+
if sem.is_undirected:
1193+
# Bounded UNDIRECTED var-length, min_hops == 1 (gated above): the
1194+
# LDBC IC11/IC6 `-[*1..k]-` shape. Mirror the pandas oracle
1195+
# (`_gfql_multihop_binding_rows`, avoid_immediate_backtrack=True)
1196+
# EXACTLY, including its edge multiplicity: pandas' `step_pairs`
1197+
# come from the undirected var-length hop + `orient_edges`, which
1198+
# emits each NON-loop edge as (u,v)x2 AND (v,u)x2, and each
1199+
# SELF-loop as (u,u)x2 (loops are not double-counted). Reconstruct
1200+
# that here: `exec_rows` = both directions of non-loops + one row
1201+
# per self-loop; the final `pairs` doubles `exec_rows`
1202+
# (fuzz-verified vs pandas over random graphs incl. self-loops,
1203+
# parallel + antiparallel edges). A `__prev__` column (seeded null)
1204+
# carries the just-left node so each hop can drop immediate
1205+
# backtracks (`__to__ == __prev__`), matching pandas' Kleene mask
1206+
# (null prev -> kept).
1207+
normal = edges_f.filter(pl.col(src) != pl.col(dst))
1208+
loops = edges_f.filter(pl.col(src) == pl.col(dst))
1209+
fwd = normal.select([pl.col(src).alias("__from__"), pl.col(dst).alias("__to__")])
1210+
rev = normal.select([pl.col(dst).alias("__from__"), pl.col(src).alias("__to__")])
1211+
loop = loops.select([pl.col(src).alias("__from__"), pl.col(dst).alias("__to__")])
1212+
exec_rows = pl.concat([fwd, rev, loop], how="vertical")
1213+
pairs = pl.concat([exec_rows, exec_rows], how="vertical")
1214+
prev_col = "__prev__"
1215+
reachable = [state.select(state_cols)] if min_hops == 0 else []
1216+
# Seed the backtrack marker with the SAME dtype as __current__ so a
1217+
# non-Int64 node id (e.g. string ids) compares/concats cleanly.
1218+
current = state.with_columns(
1219+
pl.lit(None).cast(state.collect_schema()["__current__"]).alias(prev_col)
11901220
)
1191-
if _hop >= min_hops:
1192-
reachable.append(current)
1193-
state = pl.concat(reachable, how="vertical") if reachable else state.limit(0)
1221+
for _hop in range(1, max_hops + 1):
1222+
joined = current.join(
1223+
pairs, left_on="__current__", right_on="__from__", how="inner"
1224+
)
1225+
joined = joined.filter(
1226+
pl.col(prev_col).is_null() | (pl.col("__to__") != pl.col(prev_col))
1227+
)
1228+
# new prev = the node we are leaving (old __current__); new
1229+
# __current__ = __to__. Set prev BEFORE dropping __current__.
1230+
joined = (
1231+
joined.with_columns(pl.col("__current__").alias(prev_col))
1232+
.drop("__current__")
1233+
.rename({"__to__": "__current__"})
1234+
)
1235+
current = joined.select(state_cols + [prev_col])
1236+
if _hop >= min_hops:
1237+
reachable.append(current.select(state_cols))
1238+
state = pl.concat(reachable, how="vertical") if reachable else state.limit(0)
1239+
else:
1240+
pairs = oriented.select(["__from__", "__to__"])
1241+
reachable = [state] if min_hops == 0 else []
1242+
current = state
1243+
# Lazy: build all max_hops iterations (no eager .height early-break —
1244+
# empty intermediates lazily join to empty, so the result is
1245+
# identical; the pandas break is an optimization, not semantics).
1246+
for _hop in range(1, max_hops + 1):
1247+
current = (
1248+
current.join(pairs, left_on="__current__", right_on="__from__", how="inner")
1249+
.drop("__current__")
1250+
.rename({"__to__": "__current__"})
1251+
.select(state_cols)
1252+
)
1253+
if _hop >= min_hops:
1254+
reachable.append(current)
1255+
state = pl.concat(reachable, how="vertical") if reachable else state.limit(0)
11941256
else:
11951257
state = (
11961258
state.join(oriented, left_on="__current__", right_on="__from__", how="inner")

0 commit comments

Comments
 (0)