Skip to content

Commit 77e84b0

Browse files
lmeyerovclaude
andcommitted
perf(gfql/polars): reuse the synthetic edge id as the stable edge order
The native polars chain executor attaches a synthetic edge id (`EID`) via `with_row_index` when the graph has no edge binding, then — a few hundred lines later — materializes a SECOND `with_row_index` column (`EORD`) over that same frame to restore eager edge order for the fused combine. Both are 0..n-1 over the same frame in the same order, because the second runs on the frame the first produced, so the second is pure duplication of a graph-sized materialization. Reuse EID as EORD when it was attached. Both columns were already dropped before returning, so the single drop now covers it (guarded on `EID != EORD` for the case where the graph brought its own edge binding and EORD is still a separate column). Measured on a 3.18M-node / 14M-edge polars graph with polars-engine indexes, a seeded typed pattern: 148.22 -> 135.43 ms (-8.6%). `with_row_index` was ~34 ms/query across the eager calls before this. Parity: 108 polars-path cases (9 traversal shapes x 12 random seeds, engine='polars' with polars-engine indexes) hashed over column ORDER, dtypes, row count and full content -- 0 differ, 0 raised. This is a new sweep: the existing 650-case differential runs engine=auto, which resolves to pandas and never exercises this executor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VARwnAqmvczhz5EmP7TYyZ
1 parent 46d4821 commit 77e84b0

1 file changed

Lines changed: 13 additions & 3 deletions

File tree

  • graphistry/compute/gfql/lazy/engine/polars

graphistry/compute/gfql/lazy/engine/polars/chain.py

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -928,8 +928,17 @@ def _plain_edge(op):
928928
from graphistry.compute.util import generate_safe_column_name
929929
from graphistry.compute.gfql.lazy import collect_all
930930
NORD = generate_safe_column_name("__gfql_norder__", g._nodes, prefix="__gfql_", suffix="__")
931-
EORD = generate_safe_column_name("__gfql_eorder__", g._edges, prefix="__gfql_", suffix="__")
932-
g_lz = _LazyShim(g._nodes.with_row_index(NORD).lazy(), g._edges.with_row_index(EORD).lazy(),
931+
if added_edge_index:
932+
# EID was attached above as `with_row_index` over THIS frame in THIS order, so it
933+
# already IS the stable edge order. Materializing a second identical 0..n-1 column
934+
# over a graph-sized edge frame is pure duplication (measured: `with_row_index`
935+
# cost ~34ms/query across the eager calls on a 14M-edge graph). Reuse EID.
936+
EORD = EID
937+
edges_lz = g._edges.lazy()
938+
else:
939+
EORD = generate_safe_column_name("__gfql_eorder__", g._edges, prefix="__gfql_", suffix="__")
940+
edges_lz = g._edges.with_row_index(EORD).lazy()
941+
g_lz = _LazyShim(g._nodes.with_row_index(NORD).lazy(), edges_lz,
933942
node_col, src, dst, g._edge)
934943
steps_lz = [(op, _LazyShim.step(p)) for op, p in steps]
935944
edge_steps_lz = [(op, _LazyShim.step(p)) for op, p in edge_steps]
@@ -947,7 +956,8 @@ def _plain_edge(op):
947956

948957
final_nodes = final_nodes.sort(NORD).drop(NORD)
949958
final_edges = final_edges.sort(EORD).drop(EORD)
950-
if added_edge_index:
959+
if added_edge_index and EID != EORD:
960+
# When EORD IS EID the drop above already removed it (same column, one drop).
951961
final_edges = final_edges.drop(EID)
952962
final_edges, final_nodes = collect_all([final_edges, final_nodes])
953963
final_edges = _restore_edge_dtypes(final_edges, src, dst, _endpoint_restore)

0 commit comments

Comments
 (0)