Skip to content

Commit b5b18e2

Browse files
lmeyerovclaude
andcommitted
docs(changelog): indexed edge_match candidate-row evaluation; tighten typing
CHANGELOG entry for the O(E)->O(degree) edge_match fix, plus review follow-ups on the implementation itself: - type the filter's state properly (Dict[str, SeriesT] / List[Tuple[str, Any]] and SeriesT on the gather) instead of bare `dict`/`list`/`Any`, per the repo's engine-agnostic typing convention. - correct an overclaim in the docstring: "never examines more elements than the eager mask" is false for a fixed-point UNDIRECTED walk, where the out- and in-indices are filtered separately and the total approaches 2E against the eager form's E. Stated accurately now, with the reason it is still a constant factor on an already-O(E) traversal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VARwnAqmvczhz5EmP7TYyZ
1 parent 6cb3843 commit b5b18e2

2 files changed

Lines changed: 18 additions & 8 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
1212
- **GFQL secondary node property indexes (`create_index('node_prop', column=...)` / `g.gfql_index_node_props([...])`)**: a seed predicate on a NON-key column — `MATCH (m {id: 42})` where the graph's node id binding is some other column — previously cost a full node scan, because the registry only indexed the node-id binding and the CSR adjacencies. A property index is the same pay-as-you-go sidecar as the existing kinds: sorted distinct values over node **row positions** (CSR, so duplicate values are indexable), never reorders `.nodes`, fingerprint-validated so a `.nodes()` rebind is treated as absent (safe miss, never a wrong answer), engine-polymorphic (numpy host / cupy on-device), and policy-gated (`off`/`use`/`auto`/`force`). The seeded fixed-hop planner picks the **most selective** indexed scalar predicate in the seed filter using a free CSR-offset estimate, gathers those candidates, and applies every remaining predicate to them — so results are identical whether the index is present, absent, stale, or cost-gated out. `show_indexes()` lists property indexes; `drop_index('node_prop', column=...)` drops one. Only integer columns are indexable today (float NaN ordering, strings on cupy, and nulls all decline to the scan); widening that is additive. **Perf (dgx-spark, official LDBC SNB SF1, 3.18M nodes, warm median, value-identical 19-row result):** interactive-short IS7 `71.6 ms -> 19.5 ms` (**3.7x**), with a one-time `112 ms` build — the seed lookup itself goes from a `51.2 ms` scan to `0.096 ms`.
1313

1414
### Performance
15+
- **GFQL indexed typed-edge traversals stop scanning the whole edge frame (#1658)**: the index path built the simple-equality `edge_match` mask over ALL `E` edges — one `(series == val)` over the full column — and then read that mask only at `rows[edge_keep[rows]]`, the handful of positions the CSR adjacency lookup had already returned. That put an **O(E) predicate scan inside an O(degree) traversal**, which is why an indexed seeded typed hop scaled with the graph while the underlying `g.hop()` stayed flat. The predicate is now evaluated on the gathered candidate rows instead: `_build_edge_row_filter` does the schema-level validation up front (the dtype-mismatch decline that keeps error parity with the scan is O(1) and unchanged) and `_EdgeMatchRowFilter.mask_for(rows)` applies `col == val` to just those rows, using each frame's native `==` exactly as before (so cuDF string columns stay on the cuDF layer). This never examines more elements in total than the eager mask did — it is strictly less work, not a tradeoff — and an evaluation failure mid-traversal abandons the indexed path entirely so the scan fallback stays parity-safe. Measured on a 3.18M-node / 14M-edge polars graph (LDBC SNB SF1-shaped), a seeded typed-edge query with alias markers goes **309.9 → 57.4 ms (5.4×)**, value-identical, with the full-column compare (previously 248 ms, 81% of the query) gone from the profile. Parity held across 650 differential comparisons — pandas + polars × 13 traversal shapes × 25 random seeds on a dense graph with null-carrying string and numeric columns — comparing columns, dtypes, row counts and full content, including the decline paths (membership `edge_match` still routes to the scan; a dtype mismatch still raises the same `GFQLSchemaError`). Pinned by two structural regression tests that assert the *shape* rather than a wall-clock number: the predicate must see only candidate rows, and growing the graph 8× at fixed degree must not grow the number of rows it examines.
1516
- **GFQL indexed fixed-hop bindings now dispatch BEFORE the canonical traversal (pandas / cuDF / native Polars)**: the resident-index fixed-hop path (`rows(binding_ops=...)`, the Cypher multi-alias `MATCH … RETURN` shape) was consulted only *inside* row materialization — after the engine had already run the full canonical traversal — so its compact path bag was computed on top of the graph-sized work it exists to avoid. Both the pandas/cuDF chain boundary and the native Polars chain now ask the same shared, structural gate first and, only when it can serve the WHOLE middle exactly, skip the canonical traversal and hand the compact state to the unchanged row materializer. Engagement stays operator/index/dtype/cost based — no query, schema, or hop-count recognition — and every unsupported, seeded, prefiltered, policy-bearing, shortest-path, or cost-gated shape still declines to canonical execution with identical results. On a standard LDBC SNB SF1 interactive-short profile this removed the redundant two-hop typed-mask and 16-merge buckets and cut the profiled call ~11.9× (1263 ms → 106 ms), exact 19-row oracle preserved.
1617
- **Indexed seed lookup now covers a unique node id PLUS extra scalar constraints**: `{node_id: v, other: w}` previously fell back to a full node-table scan even though the unique node-id index can gather at most one row; it now gathers that one row and applies the remaining constraints to it. Missing ids and non-matching extra constraints return the same empty result as before, and duplicate-id graphs (which cannot build the index) are unaffected.
1718
- **Seeded typed-hop property projection keeps its fast path through DISTINCT / ORDER BY / SKIP / LIMIT**: those trailing row ops are plain frame operations, so the fast path now delegates them to the canonical chain instead of declining the whole query.

graphistry/compute/gfql/index/traverse.py

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,12 @@
1414
"""
1515
from __future__ import annotations
1616

17-
from typing import Any, List, Optional, Tuple, cast
17+
from typing import Any, Dict, List, Optional, Tuple, cast
1818

1919
from typing_extensions import TypeGuard
2020

2121
from graphistry.Engine import Engine
22-
from graphistry.compute.typing import DataFrameT
22+
from graphistry.compute.typing import DataFrameT, SeriesT
2323
from graphistry.Plottable import Plottable
2424
from .engine_arrays import (
2525
array_namespace, col_to_array, ids_to_array, take_rows, select_by_ids,
@@ -82,8 +82,12 @@ class _EdgeMatchRowFilter:
8282
therefore put an O(E) predicate scan inside an O(degree) traversal, which is what
8383
made the indexed path scale with the graph instead of with the answer. Evaluating
8484
``col == val`` on the gathered candidate rows makes the predicate proportional to
85-
the edges the traversal actually visits, and never examines more elements in total
86-
than the eager mask did.
85+
the edges the traversal actually visits. Each edge row is returned by a given index
86+
at most once (frontiers are set-differenced against ``visited``), so a seeded hop
87+
examines O(edges traversed) elements; the worst case is a fixed-point undirected
88+
walk that reaches the whole graph, where the out- and in-indices are filtered
89+
separately and the total approaches 2E against the eager form's E — a constant
90+
factor on a query that was already O(E) in its traversal alone.
8791
8892
Column values are compared with each frame's native ``==`` (so cudf string columns
8993
stay on the cudf layer rather than becoming a cupy string compare), matching the
@@ -92,7 +96,12 @@ class _EdgeMatchRowFilter:
9296

9397
__slots__ = ("_series", "_items", "_engine")
9498

95-
def __init__(self, series: dict, items: list, engine: Engine) -> None:
99+
def __init__(
100+
self,
101+
series: Dict[str, SeriesT],
102+
items: List[Tuple[str, Any]],
103+
engine: Engine,
104+
) -> None:
96105
self._series = series
97106
self._items = items
98107
self._engine = engine
@@ -123,7 +132,7 @@ def mask_for(self, rows: ArrayLike) -> Optional[ArrayLike]:
123132
return None
124133

125134

126-
def _gather_series(series: Any, rows: ArrayLike, engine: Engine) -> Any:
135+
def _gather_series(series: SeriesT, rows: ArrayLike, engine: Engine) -> SeriesT:
127136
"""Positionally gather ``rows`` out of a single column. O(len(rows))."""
128137
if engine in (Engine.POLARS, Engine.POLARS_GPU):
129138
import numpy as np
@@ -149,8 +158,8 @@ def _build_edge_row_filter(
149158
_is_numeric_dtype_safe, _is_string_dtype_safe,
150159
)
151160
n_edges = int(edges.shape[0])
152-
series: dict = {}
153-
items: list = []
161+
series: Dict[str, SeriesT] = {}
162+
items: List[Tuple[str, Any]] = []
154163
for col, val in edge_match.items():
155164
if col not in edges.columns:
156165
return None

0 commit comments

Comments
 (0)