Commit 3f2128e
* feat(gfql): native polars unbounded directed var-length binding rows (LDBC IS6, #1709)
The Cypher multi-alias bindings table (`rows(binding_ops=...)`) lowered natively
for fixed-length and BOUNDED variable-length segments, but an unbounded
fixed-point segment (`-[*]->` / `-[*0..]->`) declined with
"polars engine does not yet natively support cypher row op 'rows'". That was the
last shape blocking engine='polars' on LDBC SNB interactive-short-6, the one
interactive-short query polars could not answer.
Unbounded DIRECTED fixed point now lowers natively. Termination is
data-dependent, so instead of pandas' expand-paths-until-empty (which
materializes every partial path at every hop) the lowering runs a dedup-by-node
frontier walk to find the exhaustion depth D, then reuses the SAME lazy bounded
pair-join loop the `-[*1..k]->` arm uses with max_hops=D. Deduping by endpoint
changes no walk's EXISTENCE, only its multiplicity, which the bounded loop then
reproduces in full -- so the emitted rows are pandas-identical while the
exponential path expansion happens once, lazily. A cycle reachable from the seed
means infinitely many paths: that raises the same E108 "require terminating
variable-length segments" error pandas raises, detected by a pigeonhole bound on
the distinct nodes rather than after the blow-up.
Also fixes a latent silent-wrong found by the differential fuzz: the polars
builder rebuilt bindings from the CHAIN OUTPUT while pandas rebuilds from the
pre-chain base graph. The traversal prunes to what IT considers matched, which is
not the bindings builder's match set -- a zero-hop var-length segment binds a seed
with no matching outgoing edge, and the traversal drops that node. So
`-[*0..k]->` could silently return fewer rows than pandas. The builder now sources
its node/edge tables from the same base graph pandas uses (and that the indexed
builder was already handed).
Honest declines unchanged in spirit (NIE, never a silent answer): undirected
unbounded, aliased var-length relationships, and unbounded segments without
to_fixed_point (pandas truncates at a bound this lowering cannot reconstruct).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1
* review: share the bounded var-length expansion; type the polars path bag lazily
- extract `_directed_varlen_reachable_polars` so the unbounded fixed-point arm and
the bounded `-[*1..k]->` arm run literally the same loop instead of two copies
kept in sync by a comment.
- pin the generic bindings builder's path bag as a LazyFrame. It always was one;
mypy inferred eager because `filter_by_dict_polars` declared the eager type.
- make `filter_by_dict_polars` frame-polymorphic via a constrained TypeVar
(DataFrame | LazyFrame) rather than declaring one flavour and being called with
both -- the eager viz lane and the lazy chain lane take the same `.filter(expr)`
path, and the TypeVar keeps the caller's flavour on the way out. Removes the
mypy noise this file was carrying, so the new code lands at delta 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1
* review: keep the localized ignores matching the new error code
Widening filter_by_dict_polars to a constrained TypeVar changes the diagnostic at
its two engine-neutral DataFrameT call sites from arg-type to type-var, so the
existing localized ignore no longer applied. Retarget it and add the twin in
gfql_fast_paths. Both sites are already gated on a polars engine; the cast is what
the surrounding code has always asserted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1
* refactor(gfql): move the var-length row specializations out of row_pipeline.py
Pure code move, no behaviour change. row_pipeline.py 1638 -> 1528 lines; the
unbounded/variable-length arms live in polars/varlen_rows.py with a docstring
explaining the exhaustion-depth walk and why deduping by endpoint is sound.
Motivation is rebase surface: #1757, #1714 and the seeded-chain work all touch
row_pipeline.py, and ~270 lines of var-length specialization sitting in the core
flow made every one of those a conflict.
RowPipelineMixin stays imported FUNCTION-LOCALLY exactly as at the original site —
hoisting it to module scope reintroduces an import cycle (row.pipeline -> lazy
engine -> back), so the move preserves it and says why.
2244 polars tests pass, ruff clean.
* review(#1781): decline `-[*k..]->` for k>=2, and fix the coverage baseline
Adversarial review of this branch found a silent-wrong regression and a red CI.
BLOCKER — `-[*k..]->` with k >= 2 returned WRONG COUNTS, no error.
The unbounded-directed arm of the `rows` gate had no `min_hops` guard. Cypher
`-[*k..]->` lowers to `{hops: null, min_hops: k, to_fixed_point: true,
direction: forward}`, so every k >= 2 was SERVED. Pandas' `step_pairs` come
from the var-length `edge_op.execute` hop, which empties when its
`max_reached_hop < min_hops` and otherwise drops edges labelled below
min_hops; `max_reached_hop` is a dedup-by-node BFS eccentricity, not a
longest-walk length, so the raw-edge reconstruction here expands a different
edge multiset. On a 7-node acyclic graph, both engines answering:
MATCH (a)-[*3..]->(b) RETURN count(*) pandas 0 polars 30
MATCH (a {id: 0})-[*2..]->(b) RETURN count(*) pandas 24 polars 41
85 of 1200 acyclic fuzz cases diverged, with zero declines. Master declines all
of them. The second gate this PR's description relies on (`_is_native_multihop`
in `_chain_traversal_polars`) never runs here: `RETURN count(*)` lowers to a
pure-CALL chain, so the `rows` gate is the only gate. The existing decline test
missed it because it pins `to_fixed_point=False` — a shape Cypher never emits.
Now the unbounded arm requires resolved `min_hops <= 1`, mirroring the
undirected arm; 0 and 1 (`-[*]->`, `-[*0..]->`, the IS6 walk) stay served.
BLOCKER — CI was red and the coverage gate never ran. `varlen_rows.py` was
missing from `coverage_baselines/ci-polars-py3.12.json`, whose own note requires
a complete list, so `test-polars (3.12)` exited 3 and `changed-line-coverage`
(which needs it) was SKIPPED — this branch's changed lines were never gated.
Also fixed:
* the cycle path raised a different exception CLASS and `.code` per engine
(pandas GFQLTypeError/E303 via execute_call's wrapper, polars a raw
GFQLValidationError/E108 because the native kernel runs before that
wrapper). Repo control flow keys on `.code`. Now wrapped identically.
* cycle detection was bounded by the GLOBAL node count with an eager collect
per hop, so a two-node cycle reachable from one seed cost O(graph) collects
(measured linear in global N) before raising. Now bounded by the REACHABLE
set: a walk of h edges visits h+1 nodes, all within `seen`, so h >= |seen|
IS a repeat. Exact in both directions, and the O(E) `node_cap` scan it
replaces is gone entirely (`pairs_df.height == 0` is the same test).
* the "hoisting reintroduces an import cycle" note was FALSE — disproven by
hoisting. The sibling import moves to the import block and the mid-file
`# noqa: E402` goes; `RowPipelineMixin` stays function-local, matching the
engine convention. Unused typing imports and a stale line count removed.
* `to_fixed_point` WITH an explicit bound was newly served and undocumented.
It is parity-correct (pandas ignores the flag once max_hops is set) but was
pinned by nothing; now tested and in the changelog.
Tests, all mutation-checked (reverting each fix fails the matching test):
* gate-level, k in 0..3, either side of the boundary
* end-to-end through Cypher on the repro graph — the one that actually
catches the blocker, since both engines answer and only values differ
* the cycle test now asserts class and `.code` are ENGINE-INDEPENDENT rather
than pinning polars' own code (the old form passed while they disagreed)
* a long acyclic walk must not be mistaken for a cycle, and a small reachable
cycle behind a large unreachable remainder must still raise — the two
failure modes of the new bound
Verified in-container on dgx-spark: 86 passed in the binding-rows file, 276
with the row-pipeline file, and `graphistry/tests/compute` with `--gpus all`
gives an IDENTICAL 9-failure set to this branch's base (pre-existing dask/cuDF
coercion), 6821 passed vs the base's 6804. mypy differential: 166 errors on
both trees, so these changes add none. ruff clean under the repo config.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
* review(#1781): decline to_fixed_point with an explicit bound
The re-review of the previous commit found that the gate rewrite which closed
the unbounded `-[*k..]->` hole left a narrower one open, and that the test I
added to pin it could not have caught it.
Master declined on `bool(op.to_fixed_point)` ALONE. Restructuring the gate to
key on the RESOLVED MAX incidentally let `to_fixed_point=True` combined with an
explicit bound fall through to the bounded arm — where it hits the same
reconstruction gap as the unbounded case, silently, for min_hops >= 3:
fwd min=3 max=4 tfp=True pandas 0 polars 30 (master: NIE)
fwd min=3 max=5 tfp=True pandas 0 polars 30
fwd min=4 max=5 tfp=True pandas 0 polars 6
fwd hops=3 tfp=True pandas 0 polars 24
reverse spellings: identical
Not reachable from Cypher — `cypher/parser.py` sets to_fixed_point False for
`*k` and `*i..k`, and only leaves it True for `*` and `*k..`, which always have
max_hops None. So this is the AST / `rows(binding_ops=...)` wire surface only.
Hard to reach is not correct, and the previous commit additionally claimed in
the CHANGELOG that the shape "matches pandas", which is false.
Declining it restores master's behaviour exactly, so it cannot regress anything
that previously worked.
WHY THE OLD TEST WAS BLIND, since the same mistake is easy to repeat: it
compared flagged-vs-unflagged WITHIN each engine and its parametrization
stopped at min_hops=2. A within-engine comparison never consults the pandas
oracle, so it cannot see a divergence where BOTH engines answer; and min_hops
<= 2 happens to agree on that fixture. It now asserts the DECLINE, runs to
min_hops=4, and is joined by a value test on the divergence graph that compares
ROW COUNTS against the oracle (engine-neutral: a bare `rows()` call emits
engine-specific scaffolding columns on every shape, pre-existing and unrelated).
Mutation-checked: removing the guard fails 9 tests, including the value test —
the property the previous version lacked.
Verified in-container on dgx-spark: binding-rows file 90 passed;
`graphistry/tests/compute` with `--gpus all` gives 6825 passed and a failure set
byte-identical to this branch's base (the 9 pre-existing dask/cuDF coercion
failures).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 674443e commit 3f2128e
10 files changed
Lines changed: 697 additions & 85 deletions
File tree
- graphistry
- compute
- gfql
- index
- lazy/engine/polars
- tests/compute/gfql
- coverage_baselines
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
| |||
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
| 30 | + | |
| 31 | + | |
28 | 32 | | |
29 | 33 | | |
30 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
152 | | - | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
153 | 155 | | |
154 | 156 | | |
155 | 157 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
508 | 508 | | |
509 | 509 | | |
510 | 510 | | |
| 511 | + | |
511 | 512 | | |
512 | | - | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
513 | 532 | | |
514 | 533 | | |
515 | 534 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
24 | 30 | | |
25 | 31 | | |
26 | 32 | | |
| |||
282 | 288 | | |
283 | 289 | | |
284 | 290 | | |
285 | | - | |
| 291 | + | |
286 | 292 | | |
287 | 293 | | |
288 | 294 | | |
| |||
307 | 313 | | |
308 | 314 | | |
309 | 315 | | |
310 | | - | |
| 316 | + | |
311 | 317 | | |
312 | 318 | | |
313 | 319 | | |
314 | 320 | | |
315 | 321 | | |
316 | 322 | | |
317 | 323 | | |
318 | | - | |
| 324 | + | |
319 | 325 | | |
320 | 326 | | |
321 | 327 | | |
| |||
0 commit comments