garden(gfql/index): team-polish — dead code, IF EXISTS, docstrings, bench honesty

Review-wave findings on the index PR:
- Delete dead engine_arrays.to_backend/cp_to_numpy (never called) + unused
  get_registry import (explain.py); remove index_ddl_smoke.py 'if False'
  scaffolding + unused variable.
- DropIndex.missing_ok was serialized but never honored. Now real IF EXISTS
  semantics: 'DROP GFQL INDEX [IF EXISTS] <name|FOR kind>' — plain DROP of a
  missing index raises (SQL-style, matching the existing custom-name test),
  IF EXISTS / missing_ok=true is a no-op; wire default flipped to false
  (field is new in this PR — no consumers). +1 test covering DDL both forms,
  wire JSON both ways, and resident-drop.
- Replace internal review-wave codes (F1/B1/B2/B3/I4/I5/I6/P0-1) in
  production comments with plain prose or in-repo test-name pointers
  (test-file section labels stay — they title the regression tests).
- ComputeMixin: real docstrings on create_index/drop_index/show_indexes/
  gfql_index_edges/gfql_index_all/gfql_explain (were one-liner-pointer or
  missing); gfql() docstring now documents index_policy (was invisible to
  help() despite being the flagship knob).
- index_bulk_olap_bench.py: polars-gpu cell used raise_on_fail=False — the
  exact silent-CPU-fallback pattern the CHANGELOG bans; now GPU-or-error.
- Label index_smoke/index_ddl_smoke as container-runnable mirrors of the
  canonical pytest suite; add benchmarks/gfql/README.md catalog for the 7
  index_* harnesses (which pair is canonical for published claims).

Validated: index suite 44 pass CPU-lane (cudf/polars-gpu lanes = local
no-GPU artifact, dgx-verified separately); ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: lmeyerov

benchmarks/gfql/README.md

@@ -382,3 +382,30 @@ So the honest current comparison is:
 - Neo4j is workable for the smaller Twitter analog, but already materially slower than both Graphistry CPU and GPU on the exact same shape.
 - On the selected GPlus benchmark shape, Neo4j is already dramatically slower than Graphistry CPU (`83.61s`) and Graphistry GPU (`3.41s`) before teardown/cleanup is even done.
 - Raw notes: `plans/gfql-gpu-pagerank-benchmark/results/neo4j_summary.md`
+
+## GFQL physical-index harnesses (`index_*.py`)
+
+Catalog for the seeded-traversal adjacency-index work (PR #1658; see
+`docs/source/gfql/index_adjacency.rst` for the user guide + published numbers):
+
+- `index_smoke.py` / `index_ddl_smoke.py` — correctness smokes: differential
+  parity (index path == scan path) and DDL/wire/`index_policy`/`gfql_explain`
+  round-trips. Container-runnable MIRRORS of the canonical pytest suite
+  (`graphistry/tests/compute/gfql/index/test_index.py`).
+- `index_perf.py` — microbenchmark: seeded 1-hop/2-hop index-vs-scan across
+  engines and frontier sizes; the cost-gate calibration numbers come from here.
+- `index_takeover_bench.py` + `index_vs_kuzu_prepared.py` — **canonical
+  competitor pair** behind the published "9–36x vs Kuzu" seeded-traversal
+  claims: guarded timings (path-taken assertions + result==scan oracle),
+  prepared-statement fairness on the Kuzu side.
+- `index_vs_dbs.py` — broader 3-way sweep (GFQL / Kuzu / Neo4j) over the same
+  seeded shapes; superset of the pair above, heavier setup (dockerized Neo4j).
+- `index_bulk_olap_bench.py` — bulk group-by/degree OLAP shapes across the 4
+  engines (answers "does the index help bulk scans?" — it does not; scans win).
+- `index_largegraph_bench.py` — scale probe on large real graphs
+  (flat-in-N behavior of the seeded path up to ~80M edges).
+
+Methodology for all: warm medians, one engine per process for large runs,
+NO-CHEATING guards (a timing is void unless the intended path was taken AND
+the result matches the scan oracle). Run on an idle box; GPU rows need
+`--gpus all` and RAPIDS.

benchmarks/gfql/index_bulk_olap_bench.py

@@ -114,7 +114,9 @@ def degall(edf, engine, reps, warm):
         import polars as pl
         df = pl.from_pandas(edf)
         if engine == "polars-gpu":
-            eng = pl.GPUEngine(executor="in-memory", raise_on_fail=False)
+            # raise_on_fail=True: GPU-or-error — a silent CPU fallback would report
+            # CPU time as a polars-gpu result (banned; see the polars-gpu CHANGELOG fix)
+            eng = pl.GPUEngine(executor="in-memory", raise_on_fail=True)
             fn = lambda: df.lazy().group_by("src").len().collect(engine=eng)
         else:
             fn = lambda: df.group_by("src").len()

benchmarks/gfql/index_ddl_smoke.py

@@ -1,5 +1,9 @@
 #!/usr/bin/env python3
-"""Smoke test: Cypher DDL + JSON wire + index_policy auto/force + gfql_explain."""
+"""Smoke test: Cypher DDL + JSON wire + index_policy auto/force + gfql_explain.
+
+Container-runnable MIRROR of graphistry/tests/compute/gfql/index/test_index.py
+(the pytest suite is canonical); kept for quick in-container smoke without pytest.
+"""
 from __future__ import annotations
 
 import sys
@@ -63,9 +67,7 @@ def main():
 
     # --- index_policy auto/force build-on-demand + explain ---
     gp = make_graph()  # fresh, no resident indexes
-    exp_use = gp.gfql_explain([], index_policy="use") if False else None  # explain needs a real query
-    # explain with a seeded 1-hop chain expressed in cypher
-    q = "MATCH (a)-[e]->(b) RETURN b"  # not seeded -> not coverable; use python chain instead
+    # explain with a seeded 1-hop chain (explain needs a real, seeded query)
     from graphistry.compute.ast import n, e_forward
     seeded_chain = [n({"id": 0}), e_forward(hops=1)]
     rep_off = gp.gfql_explain(seeded_chain, index_policy="off")

benchmarks/gfql/index_smoke.py

@@ -1,6 +1,9 @@
 #!/usr/bin/env python3
 """Differential parity smoke test for GFQL physical indexes.
 
+Container-runnable MIRROR of graphistry/tests/compute/gfql/index/test_index.py
+(the pytest suite is canonical); kept for quick in-container smoke without pytest.
+
 Oracle: the index fast path MUST return the same subgraph (node-id set + edge
 multiset) as the scan/join path, across engines / directions / hops / wavefront.
 A fast wrong answer is not a win.

graphistry/compute/ComputeMixin.py

@@ -514,23 +514,42 @@ def hop(self, *args, **kwargs):
 
     # ---- GFQL physical indexes (pay-as-you-go seeded-traversal acceleration) ----
     def create_index(self, kind, *, column=None, name=None, engine='auto'):
+        """Build a GFQL physical index for O(degree) seeded traversal.
+
+        :param kind: 'edge_out_adj' (forward hops), 'edge_in_adj' (reverse hops), or 'node_id' (node lookup)
+        :param column: column to index (defaults to the binding for the kind, e.g. the edge source column)
+        :param name: optional custom index name (defaults to 'kind:column')
+        :param engine: 'auto' | 'pandas' | 'cudf' | 'polars' — array backend for the index
+        :returns: new Plottable with the index resident (original is untouched)
+
+        **Example**
+            ::
+
+                g2 = g.create_index('edge_out_adj')
+                g2.gfql([n({'id': 0}), e_forward(hops=2)], index_policy='use')
+
+        See :doc:`gfql/index_adjacency` for policies, cost gating, and benchmarks.
+        """
         from graphistry.compute.gfql.index import create_index as _ci
         return _ci(self, kind, column=column, name=name, engine=engine)
-    create_index.__doc__ = "Build a GFQL physical index (edge_out_adj|edge_in_adj|node_id). See graphistry/compute/gfql/index/api.py."
 
     def drop_index(self, kind=None):
+        """Drop one resident GFQL index (by kind) or all (kind=None). Idempotent; returns a new Plottable."""
         from graphistry.compute.gfql.index import drop_index as _di
         return _di(self, kind)
 
     def show_indexes(self):
+        """Return a pandas DataFrame describing resident GFQL indexes (name, kind, column, valid). Empty if none; ``valid=False`` marks a stale index after a frame rebind."""
         from graphistry.compute.gfql.index import show_indexes as _si
         return _si(self)
 
     def gfql_index_edges(self, direction='both', engine='auto'):
+        """Convenience: build the edge adjacency index(es) — 'forward', 'reverse', or 'both'. Returns a new Plottable."""
         from graphistry.compute.gfql.index import gfql_index_edges as _gie
         return _gie(self, direction, engine=engine)
 
     def gfql_index_all(self, engine='auto'):
+        """Convenience: build all GFQL physical indexes (both edge adjacencies + node_id). Returns a new Plottable."""
         from graphistry.compute.gfql.index import gfql_index_all as _gia
         return _gia(self, engine=engine)
 
@@ -581,9 +600,19 @@ def gfql(self, *args, **kwargs):
             g = _copy.copy(self)
             g._gfql_index_policy = policy
         return gfql_base(g, *args, **kwargs)
-    gfql.__doc__ = gfql_base.__doc__
+    gfql.__doc__ = (gfql_base.__doc__ or "") + """
+
+        **GFQL physical indexes**
+
+        :param index_policy: 'off' (never use indexes), 'use' (use resident,
+            cost-gated; default planner behavior), 'auto' (build on demand), or
+            'force' (always probe the index). Also accepts index DDL strings
+            (``CREATE GFQL INDEX ...``) / wire ops as the query — routed to the
+            index registry. See :meth:`create_index` and :doc:`gfql/index_adjacency`.
+    """
 
     def gfql_explain(self, query, *, index_policy='use', engine='auto'):
+        """Explain how the GFQL planner would run ``query``: per-hop index-vs-scan choice, cost-gate numbers, and resident-index validity. Read-only (no execution). Returns a report object; print it for a human-readable plan."""
         from graphistry.compute.gfql.index.explain import gfql_explain as _ge
         return _ge(self, query, index_policy=index_policy, engine=engine)
 

graphistry/compute/gfql/index/api.py

@@ -25,7 +25,7 @@
 # full scan is cheaper than per-seed index probes, so `use` falls back to scan and
 # never loses to the un-indexed path. Engine-aware because vectorized-scan engines
 # (polars/cudf/GPU) scan far faster than pandas, so their crossover is much smaller
-# (F1). Measured N=1e5 deg8: pandas ~0.5, polars ~0.02. GPU values provisional
+# (see test_index_cost_gate_engine_aware). Measured N=1e5 deg8: pandas ~0.5, polars ~0.02. GPU values provisional
 # (dgx-gated) — conservatively grouped with polars.
 _COST_GATE_FRAC = {Engine.PANDAS: 0.5}
 _COST_GATE_FRAC_DEFAULT = 0.02
@@ -332,7 +332,7 @@ def maybe_index_hop(
 
     # Honor max_hops: the scan resolves the hop count as ``max_hops or hops``
     # (compute/hop.py); the index must run the SAME number of accumulating hops.
-    # (B1: max_hops was passed through *rest and silently ignored — the index ran
+    # (regression: max_hops was passed through *rest and silently ignored — the index ran
     # `hops` (default 1) while the scan ran max_hops → wrong answer.)
     _max_hops = rest.get("max_hops")
     eff_hops = _max_hops if _max_hops is not None else hops

graphistry/compute/gfql/index/build.py

@@ -78,7 +78,7 @@ def build_node_id_index(
     row position per unique key (``row_positions[group_offsets[:-1]]``, length U,
     aligned with ``unique_keys``) and (b) REFUSE (return None) when ids aren't unique:
     a unique-key CSR can't reproduce the scan's "all rows per id" semantics, so the
-    caller falls back to the correct ``select_by_ids`` isin path. (B2: a non-unique
+    caller falls back to the correct ``select_by_ids`` isin path. (Regression guard: a non-unique
     node-id index dropped reached nodes / emitted unrelated rows.)"""
     xp, backend = array_namespace(engine)
     keys = col_to_array(nodes, node_col, engine)

graphistry/compute/gfql/index/cypher_ddl.py

@@ -1,8 +1,8 @@
 """Targeted recognizer for GFQL index DDL Cypher statements.
 
     CREATE GFQL INDEX [<name>] FOR <kind> [ON <column>]
-    DROP   GFQL INDEX <name>
-    DROP   GFQL INDEX FOR <kind> [ON <column>]
+    DROP   GFQL INDEX [IF EXISTS] <name>
+    DROP   GFQL INDEX [IF EXISTS] FOR <kind> [ON <column>]
     SHOW   GFQL INDEXES
 
 The mandatory ``GFQL`` token disambiguates from standard property ``CREATE INDEX``
@@ -25,12 +25,12 @@
     re.IGNORECASE,
 )
 _DROP_FOR_RE = re.compile(
-    r"^\s*DROP\s+GFQL\s+INDEX\s+FOR\s+" + _KIND
+    r"^\s*DROP\s+GFQL\s+INDEX\s+(?P<ifexists>IF\s+EXISTS\s+)?FOR\s+" + _KIND
     + r"(?:\s+ON\s+(?P<col>[A-Za-z_]\w*))?\s*;?\s*$",
     re.IGNORECASE,
 )
 _DROP_NAME_RE = re.compile(
-    r"^\s*DROP\s+GFQL\s+INDEX\s+(?P<name>[A-Za-z_][\w:]*)\s*;?\s*$",
+    r"^\s*DROP\s+GFQL\s+INDEX\s+(?P<ifexists>IF\s+EXISTS\s+)?(?P<name>[A-Za-z_][\w:]*)\s*;?\s*$",
     re.IGNORECASE,
 )
 _SHOW_RE = re.compile(r"^\s*SHOW\s+GFQL\s+INDEXES\s*;?\s*$", re.IGNORECASE)
@@ -54,10 +54,11 @@ def parse_index_ddl(query: str) -> Optional[Any]:
                            name=m.group("name"))
     m = _DROP_FOR_RE.match(query)
     if m:
-        return DropIndex(kind=m.group("kind").lower(), column=m.group("col"))
+        return DropIndex(kind=m.group("kind").lower(), column=m.group("col"),
+                         missing_ok=bool(m.group("ifexists")))
     m = _DROP_NAME_RE.match(query)
     if m:
-        return DropIndex(name=m.group("name"))
+        return DropIndex(name=m.group("name"), missing_ok=bool(m.group("ifexists")))
     if looks_like_index_ddl(query):
         raise ValueError(
             f"Malformed GFQL INDEX DDL: {query!r}. Expected e.g. "

graphistry/compute/gfql/index/engine_arrays.py

@@ -51,27 +51,6 @@ def ids_to_array(ids: Any, col: str, engine: Engine) -> Any:
     return col_to_array(ids, col, engine)
 
 
-def to_backend(arr: Any, backend: str) -> Any:
-    """Move a host/device array to the index backend (numpy<->cupy)."""
-    import numpy as np
-
-    if backend == "cupy":
-        import cupy as cp  # type: ignore
-
-        return cp.asarray(arr)
-    if "cupy" in type(arr).__module__:
-        return cp_to_numpy(arr)
-    return np.asarray(arr)
-
-
-def cp_to_numpy(arr: Any) -> Any:
-    try:
-        return arr.get()
-    except Exception:
-        import numpy as np
-
-        return np.asarray(arr)
-
 
 def take_rows(df: Any, positions: Any, engine: Engine) -> Any:
     """Positionally gather rows of ``df`` by an integer array ``positions``.

graphistry/compute/gfql/index/explain.py

@@ -3,14 +3,14 @@
 Seeded queries are cheap once indexed, so explain *executes* the query under a
 decision trace and reports the real per-hop index-vs-scan path (plus resident
 indexes). This makes "it silently scanned" detectable and assertable rather than
-a mystery — the top human-factors need from the design review (P0-1).
+a mystery — the top human-factors need from the design review.
 """
 from __future__ import annotations
 
 from typing import Any, Dict, cast
 
 from graphistry.Engine import resolve_engine
-from .api import index_trace, get_registry, show_indexes
+from .api import index_trace, show_indexes
 
 
 def gfql_explain(g: Any, query: Any, *, index_policy: str = "use", engine: str = "auto") -> Dict[str, Any]: