Merge pull request #1840 from graphistry/feat/gfql-range-chain-parity

test(gfql): lock range-chain index decline

Author: lmeyerov

CHANGELOG.md

@@ -9,6 +9,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 <!-- Do Not Erase This Section - Used for tracking unreleased changes -->
 
 ### Added
+- **Structured GFQL index-planner diagnostics**: `gfql_explain()` and the planner trace now expose a stable `decision_code` alongside the human-readable `decision_reason`. Codes distinguish policy-off scans, missing resident indexes, unsupported query shapes, missing graph bindings, cost-gate declines, selected indexes, engine mismatches, and unavailable index paths. `index_policy=force` bypasses only the cost gate for an index-coverable shape. Unsupported or unavailable shapes still scan and return the same result. Tests cover no resident index, missing graph columns, automatic cost decline, forced selection, policy-off diagnostics, and scan/index parity.
 - **Static enforcement for the type-hygiene defect classes that keep failing code review**: five recurring review rejections -- unannotated function contracts, dynamic attribute access, `Any` + call-site `cast()`, unparameterized `list`/`dict`, and a bare `str` where a `Literal` belongs -- are now checked by tooling instead of by a human. `bin/ci_type_hygiene_guard.py` is a stdlib-only AST pass over `graphistry/` (tests excluded, matching `mypy.ini`) wired into `bin/lint.sh`, so it runs in the existing `python-lint-types` matrix on py3.8-3.14 with no new workflow and no new job; it takes ~1.4s and returns byte-identical counts on all seven interpreters. Enforcement is a **per-file count ratchet** against `bin/ci_type_hygiene_baseline.json` -- a file may not gain findings and a file absent from the baseline must have zero -- because the honest measurement is that the codebase carries 1704 missing annotations, 1605 `Any`-in-annotation sites and 959 `cast()` calls, and a rule that flags those as errors today is a rule nobody can turn on. Two checks are effectively hard rules rather than ratchets because their real counts are tiny: `plottable-setattr` has **3** sites repo-wide (one of them is the open bug #1825, where caching by `setattr` onto the caller's `Plottable` keyed on `id()` returned a stale answer after an in-place frame mutation), and `plottable-attr-write` -- the `param.attr = ...` form the same hazard takes when nobody writes `setattr` -- has 39, all in five files. A blanket `getattr`/`setattr` ban was measured and rejected: 300 non-test `getattr` sites are overwhelmingly legitimate optional-attribute reads, and the leak requires a *write*, so only writes onto a parameter annotated as a `Plottable` are flagged. Class 5 gets the narrowest defensible rule rather than a plausible one: `vocab-str-param` knows exactly six parameter names (`table`, `kind`, `direction`, `how`, `mode`, `engine`) whose vocabularies this repo has already committed to as `Literal` aliases, and covers 42 sites; the general "this `str` should be a `Literal`" question was prototyped as a call-site/comparison heuristic, measured at roughly 80% precision over 44 hits, and deliberately **not** shipped -- a column name is legitimately `str`, and a rule that needs manual triage every PR is worse than no rule. Ruff gains `B009`/`B010` (constant-name `getattr`/`setattr`) as genuine errors, with today's 31 offenders across 14 files seeded into `per-file-ignores` and retired one file at a time by `ruff check --fix`. Findings that are genuinely correct are annotated in place with `# hygiene-ok: <check> -- <reason>` rather than by raising a baseline cap. Conventions and escape hatches documented in DEVELOP.md "Type Hygiene Guard". No production code changed.
 - **Native polars `rows(binding_ops=...)` for UNBOUNDED directed variable-length patterns (`-[*]->` / `-[*0..]->`) (#1709)**: the Cypher multi-alias bindings table already lowered natively for fixed-length and *bounded* variable-length segments, but an unbounded fixed-point segment declined with `NotImplementedError: polars engine does not yet natively support cypher row op 'rows'`. This was the last shape blocking `engine='polars'` on LDBC SNB interactive-short-6 (`MATCH (m:Message)-[:REPLY_OF*0..]->(p:Post)<-[:CONTAINER_OF]-(f:Forum)-[:HAS_MODERATOR]->(mod)`), the one interactive-short query polars could not answer. It now runs natively: a dedup-by-node frontier walk finds the exhaustion depth, then the SAME lazy bounded pair-join loop the `-[*1..k]->` arm uses materializes one row per distinct edge SEQUENCE (Cypher path multiplicity, parallel edges included) — no pandas bridge, no `to_pandas()` round trip. A cycle reachable from the seed means infinitely many paths; that raises the same E108 "require terminating variable-length segments" error pandas raises — same exception class and same `.code` on both engines — and is detected before the path expansion blows up rather than after, bounded by the REACHABLE node count so an unreachable remainder of the graph costs nothing. Still declining honestly (NIE, never a silent answer): UNDIRECTED unbounded (`-[*]-`, needs the min_hops == 1 multiplicity reconstruction plus backtrack-aware termination — pandas rejects it outright), aliased variable-length relationships (pandas rejects those too), unbounded segments WITHOUT `to_fixed_point` (pandas silently truncates at a bound this lowering cannot reconstruct), unbounded segments with `min_hops >= 2` (`-[*2..]->`: pandas' step pairs are pruned by min_hops against a dedup-by-node eccentricity, which this raw-edge reconstruction cannot reproduce — serving it would return a different count with no error), and `to_fixed_point` combined with an explicit bound (declined on master too; it is not Cypher-reachable, since the parser only sets the flag for `*` / `*k..` where there is no maximum, but through the AST surface it hits the same reconstruction gap for `min_hops >= 3`). Cross-engine parity is the gate: differential fuzz vs the pandas oracle over random DAGs and cyclic graphs with self-loops and parallel edges, plus pinned IS6/zero-hop/multiplicity/cycle tests.
 

graphistry/compute/gfql/index/api.py

@@ -25,7 +25,7 @@
 from .policy import IndexPolicy, validate_index_policy
 from .types import (
     AdjacencyIndexKind, EdgeIndexDirection, HopDirection, IndexKind,
-    IndexTrace, IndexTraceStep,
+    IndexDecisionCode, IndexTrace, IndexTraceStep,
 )
 
 # Private Plottable attachment keys. Keep access behind helpers.
@@ -140,6 +140,7 @@ def _record_indexed_traversal(
         "hop_details": [] if hop_details is None else hop_details,
         "path": path,
         "decision_reason": reason,
+        "decision_code": "index_selected" if served else "index_path_unavailable",
     }))
 
 
@@ -514,6 +515,8 @@ def maybe_index_hop(
     Cost gate: only route to the index when (a) a valid matching index is resident
     (or buildable under auto/force), (b) the query is covered, (c) the frontier is
     not so large that a full scan is cheaper. Correctness is identical either way.
+    "force" bypasses only the cost gate for a covered query. It still falls back to
+    scan when the index cannot serve the query.
     """
     resolved_policy: IndexPolicy = validate_index_policy(policy) or "use"
 
@@ -533,16 +536,18 @@ def maybe_index_hop(
         except (AttributeError, TypeError, ValueError):
             pass
 
-    def _bail(reason: str) -> Optional[Plottable]:
+    def _bail(reason: str, decision_code: IndexDecisionCode) -> Optional[Plottable]:
         if trace:
-            _record(cast(IndexTraceStep, {**diag, "path": "scan", "decision_reason": reason}))
+            _record(cast(IndexTraceStep, {
+                **diag, "path": "scan", "decision_reason": reason, "decision_code": decision_code,
+            }))
         return None
 
     if resolved_policy == "off":
-        return _bail("policy=off")
+        return _bail("policy=off", "policy_off")
     registry = get_registry(g)
     if registry.is_empty() and resolved_policy not in ("auto", "force"):
-        return _bail("no resident index (policy=use)")
+        return _bail("no resident index (policy=use)", "no_resident_index")
 
     min_hops = cast(Optional[int], rest.get("min_hops"))
     max_hops = cast(Optional[int], rest.get("max_hops"))
@@ -575,18 +580,18 @@ def _bail(reason: str) -> Optional[Plottable]:
         target_wave_front=target_wave_front,
         return_as_wave_front=return_as_wave_front,
     ):
-        return _bail("query not index-coverable")
+        return _bail("query not index-coverable", "not_index_coverable")
     assert nodes is not None
 
     node_col = g._node
     src, dst = g._source, g._destination
     if node_col is None or src is None or dst is None or g._edges is None or g._nodes is None:
-        return _bail("graph missing node/edge columns")
+        return _bail("graph missing node/edge columns", "missing_graph_columns")
 
     if resolved_policy in ("auto", "force"):
         registry = _ensure_indexes(g, registry, direction, engine, resolved_policy, nodes, src, dst, node_col)
     if registry.is_empty():
-        return _bail("no index available (build declined)")
+        return _bail("no index available (build declined)", "index_build_declined")
 
     # Cost gate: if the frontier covers a large fraction of distinct sources, the
     # scan path is competitive — fall back (avoids index overhead on bulk-ish hops).
@@ -612,7 +617,7 @@ def _bail(reason: str) -> Optional[Plottable]:
             if idx0.n_keys > 0 and frontier_n >= frac * idx0.n_keys:
                 return _bail(
                     f"frontier {frontier_n} >= {frac}*n_keys "
-                    f"({frac * idx0.n_keys:.0f}) -> scan cheaper"
+                    f"({frac * idx0.n_keys:.0f}) -> scan cheaper", "scan_cost"
                 )
         except (AttributeError, TypeError, ValueError):
             pass
@@ -629,12 +634,20 @@ def _bail(reason: str) -> Optional[Plottable]:
         edge_match=cast(Optional[dict], rest.get("edge_match")),
     )
     if trace:
+        engine_mismatch_reason = (
+            _engine_mismatch_reason(registry, direction, engine) if result is None else None
+        )
         _record(cast(IndexTraceStep, {
             **diag, "hops": eff_hops,
             "path": "index" if result is not None else "scan",
             "decision_reason": (
                 "frontier below cost gate -> index" if result is not None
-                else _engine_mismatch_reason(registry, direction, engine) or "index path not applicable -> scan"
+                else engine_mismatch_reason or "index path not applicable -> scan"
+            ),
+            "decision_code": (
+                "index_selected" if result is not None
+                else "engine_mismatch" if engine_mismatch_reason is not None
+                else "index_path_unavailable"
             ),
         }))
     return result

graphistry/compute/gfql/index/explain.py

@@ -13,7 +13,7 @@
 from graphistry.compute.gfql.query_types import GFQLQuery
 from .api import index_trace, show_indexes
 from .policy import IndexPolicy, validate_index_policy
-from .types import IndexTraceStep
+from .types import IndexDecisionCode, IndexTraceStep
 
 if TYPE_CHECKING:
     from graphistry.compute.ComputeMixin import ComputeMixin
@@ -29,6 +29,7 @@ class GfqlExplainReport(TypedDict):
     est_result_rows: Optional[int]
     chosen_direction: Optional[str]
     decision_reason: Optional[str]
+    decision_code: Optional[IndexDecisionCode]
     error: Optional[str]
 
 
@@ -54,6 +55,8 @@ def gfql_explain(
     # of seeds; `est_result_rows` = estimated fanout (Σ seed degree, free from CSR).
     ref = [s for s in steps if s.get("path") == "index"] or list(steps)
     last = ref[-1] if ref else {}
+    if not last and resolved_policy == "off":
+        last = {"decision_reason": "policy=off", "decision_code": "policy_off"}
     resident_names = cast(List[str], resident["name"].tolist() if len(resident) else [])
     return {
         "engine": eng.value,
@@ -65,5 +68,6 @@ def gfql_explain(
         "est_result_rows": cast(Optional[int], last.get("est_result_rows")),
         "chosen_direction": cast(Optional[str], last.get("direction")),
         "decision_reason": cast(Optional[str], last.get("decision_reason")),
+        "decision_code": last.get("decision_code"),
         "error": error,
     }

graphistry/compute/gfql/index/policy.py

@@ -10,7 +10,10 @@
 def validate_index_policy(policy: Optional[str]) -> Optional[IndexPolicy]:
     """Validate a public ``index_policy`` value.
 
-    ``None`` means the caller did not override the default planner behavior.
+    ``None`` means the caller did not override the default planner behavior. "force"
+    is a performance and diagnostic opt-in: it builds missing indexes and bypasses the
+    cost gate for an index-coverable query. It does not require index coverage and never
+    changes query results. A query that the index cannot serve still uses the scan path.
     """
     if policy is None:
         return None

graphistry/compute/gfql/index/types.py

@@ -21,6 +21,17 @@
 
 
 IndexPath = Literal["scan", "index"]
+IndexDecisionCode = Literal[
+    "policy_off",
+    "no_resident_index",
+    "not_index_coverable",
+    "missing_graph_columns",
+    "index_build_declined",
+    "scan_cost",
+    "index_selected",
+    "engine_mismatch",
+    "index_path_unavailable",
+]
 
 
 # One column's constraint in an ``edge_match``/filter dict — exactly the runtime shapes
@@ -54,6 +65,7 @@ class IndexTraceStep(TypedDict, total=False):
     frontier_n: int
     path: IndexPath
     decision_reason: str
+    decision_code: IndexDecisionCode
     n_keys: int
     seed_deg_sum: Optional[int]
     est_result_rows: Optional[int]

graphistry/tests/compute/gfql/index/test_index.py

@@ -5,14 +5,15 @@
 importorskip so GPU lanes run only where available.
 """
 import importlib
+from copy import copy
 
 import numpy as np
 import pandas as pd
 import pytest
 
 import graphistry
 from graphistry.compute.ast import n, e_forward, e_reverse
-from graphistry.compute.gfql.index.api import _engine_mismatch_reason
+from graphistry.compute.gfql.index.api import _engine_mismatch_reason, maybe_index_hop
 from graphistry.compute.gfql.index import (
     CreateIndex, DropIndex, ShowIndexes, index_op_from_json, parse_index_ddl,
     get_registry,
@@ -230,12 +231,84 @@ def test_index_policy_force_and_explain(graph, engine):
     chain = [n({"id": 0}), e_forward(hops=1)]
     rep_off = graph.gfql_explain(chain, index_policy="off", engine=engine)
     assert rep_off["used_index"] is False
+    assert rep_off["decision_code"] == "policy_off"
+    assert rep_off["decision_reason"] == "policy=off"
     rep_force = graph.gfql_explain(chain, index_policy="force", engine=engine)
     assert rep_force["used_index"] is True
+    assert rep_force["decision_code"] == "index_selected"
     # results identical regardless of policy
     r_scan = graph.gfql(chain, engine=engine)
     r_force = graph.gfql(chain, index_policy="force", engine=engine)
     assert _sig(r_scan) == _sig(r_force)
+    assert rep_force["error"] is None
+
+
+def test_maybe_index_hop_reports_no_resident_index(graph):
+    """A direct planner caller gets a named scan decline when no index is resident."""
+    from graphistry.Engine import Engine
+    from graphistry.compute.gfql.index import index_trace
+
+    with index_trace() as steps:
+        result = maybe_index_hop(
+            graph,
+            Engine.PANDAS,
+            nodes=pd.DataFrame({"id": [0]}),
+            hops=1,
+            direction="forward",
+            return_as_wave_front=False,
+            policy="use",
+        )
+
+    assert result is None
+    assert steps[-1]["decision_code"] == "no_resident_index"
+
+
+def test_maybe_index_hop_reports_missing_graph_columns(graph):
+    """A resident index cannot make an unbound graph use a different query path."""
+    from graphistry.Engine import Engine
+    from graphistry.compute.gfql.index import index_trace
+
+    unbound = copy(graph.gfql_index_all(engine="pandas"))
+    unbound._nodes = None
+    with index_trace() as steps:
+        result = maybe_index_hop(
+            unbound,
+            Engine.PANDAS,
+            nodes=pd.DataFrame({"id": [0]}),
+            hops=1,
+            direction="forward",
+            return_as_wave_front=False,
+            policy="use",
+        )
+
+    assert result is None
+    assert steps[-1]["decision_code"] == "missing_graph_columns"
+
+
+def test_maybe_index_hop_reports_auto_build_decline_with_scan_parity(graph):
+    """An unselective auto request declines the build and preserves scan results."""
+    from graphistry.Engine import Engine
+    from graphistry.compute.gfql.index import index_trace
+
+    seeds = graph._nodes.copy()
+    with index_trace() as steps:
+        result = maybe_index_hop(
+            graph,
+            Engine.PANDAS,
+            nodes=seeds,
+            hops=1,
+            direction="forward",
+            return_as_wave_front=False,
+            policy="auto",
+        )
+
+    auto_graph = graph.bind()
+    auto_graph._gfql_index_policy = "auto"
+    auto_result = auto_graph.hop(nodes=seeds, engine="pandas", hops=1)
+    scan_result = graph.hop(nodes=seeds, engine="pandas", hops=1)
+    assert result is None
+    assert steps[-1]["decision_code"] == "index_build_declined"
+    assert _sig(auto_result) == _sig(scan_result)
 
 
 @pytest.mark.parametrize("engine", ENGINES)
@@ -261,6 +334,7 @@ def test_explain_exposes_planner_diagnostics(graph, engine):
     gi = graph.gfql_index_all(engine=engine)
     rep_off = gi.gfql_explain(chain, index_policy="off", engine=engine)
     assert rep_off["used_index"] is False
+    assert rep_off["decision_code"] == "policy_off"
     assert rep_off["decision_reason"] == "policy=off", rep_off
 
 
@@ -506,7 +580,7 @@ def test_index_min_two_bounded_range_scans_pandas(graph):
         indexed = _force(graph, "pandas").hop(nodes=seeds, engine="pandas", **kwargs)
     assert _sig(base) == _sig(indexed)
     assert not any(step["path"] == "index" for step in steps), steps
-    assert any(step["decision_reason"] == "query not index-coverable" for step in steps), steps
+    assert any(step["decision_code"] == "not_index_coverable" for step in steps), steps
 
 @pytest.mark.parametrize("engine", ENGINES)
 def test_index_duplicate_node_ids(engine):
@@ -791,6 +865,9 @@ def topd(df):
                e_forward({"etype": 2}, hops=1)]
 _UNTYPED_CHAIN = [n({"id": 100}), e_forward(hops=1)]
 _MEMBER_CHAIN = [n({"id": 100}), e_forward({"etype": [0, 1]}, hops=1)]
+_RANGE_CHAIN = [n({"id": 100}), e_forward(
+    min_hops=1, max_hops=2, output_min_hops=2, output_max_hops=2,
+)]
 
 
 @pytest.mark.parametrize("engine", ENGINES)
@@ -832,6 +909,23 @@ def test_chain_membership_edge_match_stays_on_scan(typed_graph, engine):
     assert rep["used_index"] is False, (engine, rep)
 
 
+def test_chain_range_with_auto_labels_stays_on_scan(typed_graph):
+    """A Cypher range needs per-depth records, so it must decline until indexed."""
+    engine = "pandas"
+    from graphistry.compute.gfql.index import index_trace
+
+    gi = typed_graph.gfql_index_all(engine=engine)
+    base = typed_graph.gfql(_RANGE_CHAIN, index_policy="off", engine=engine)
+    with index_trace() as steps:
+        indexed = gi.gfql(_RANGE_CHAIN, index_policy="force", engine=engine)
+    rep = gi.gfql_explain(_RANGE_CHAIN, index_policy="force", engine=engine)
+    assert _sig_typed(base) == _sig_typed(indexed)
+    assert rep["used_index"] is False, (engine, rep)
+    assert not any(step["path"] == "index" for step in steps), steps
+    assert rep["decision_code"] == "not_index_coverable", rep
+    assert any(step["decision_code"] == "not_index_coverable" for step in steps), steps
+
+
 def test_rebind_edges_revalidates_after_shallow_augmentation():
     """rebind_edges re-points the edge adjacency index at a shallow-copied frame that
     merely ADDS a column (the chain's synthetic edge id), so get_valid recognizes it."""