Skip to content

Commit 46d4821

Browse files
lmeyerovclaude
andcommitted
perf(gfql): shrink the right side of a left-merge when the left is empty
`_lean_prefilter_right` shrinks `right` to the keys present in `left` before a how='left' merge. It declined to do so when `left` was EMPTY -- the one case where shrinking is both maximally profitable and trivially correct: no left key can match anything, and a left merge keeps only the right rows that DO match, so the result is empty whatever `right` holds. The cost of declining is graph-sized. Instrumenting a single-node named query on a 3.18M-node / 14M-edge graph showed the dominant work was: safe_merge(left=0x1, right=14000000x4, how='left') i.e. a zero-row intermediate joined against every edge in the graph. Returning a zero-row slice of `right` keeps the identical columns and dtypes -- so the merge still produces the same schema -- without materializing the frame. Measured on that graph: a named seed-only query goes 112.64 -> 17.29 ms (6.5x), same result shape. Patterns whose intermediates are non-empty are unaffected. Parity: 650 differential comparisons (pandas + polars x 13 shapes x 25 seeds, dense graph, null-carrying columns) -- 0 divergences. Index + lean-combine suites: no new failures (38 pre-existing GPU-image failures unchanged, 184 passed). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VARwnAqmvczhz5EmP7TYyZ
1 parent b5b18e2 commit 46d4821

1 file changed

Lines changed: 11 additions & 1 deletion

File tree

graphistry/compute/chain_lean_combine.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,16 @@ def _lean_prefilter_right(left: DataFrameT, right: DataFrameT, key: str, engine:
102102
right_len = len(right)
103103
except Exception:
104104
return right
105-
if left_len == 0 or left_len * _LEAN_SHRINK_RATIO > right_len:
105+
if left_len == 0:
106+
# No left key can match anything, and a how='left' merge keeps only the right
107+
# rows that DO match, so the result is empty whatever `right` holds. Hand back
108+
# a zero-row slice -- same columns and dtypes, so the merge still produces the
109+
# identical schema -- instead of letting the join materialize the whole frame.
110+
# This is the case where shrinking is both maximally profitable and trivially
111+
# correct, and it was the one case the gate below declined: an empty
112+
# intermediate against a graph-sized side (measured: a 0-row left joined
113+
# against 14M edges dominated a 112ms single-node query).
114+
return right[0:0]
115+
if left_len * _LEAN_SHRINK_RATIO > right_len:
106116
return right
107117
return right[right[key].isin(left[key])]

0 commit comments

Comments
 (0)