docs(changelog): CSR index — our reproduced numbers + adversarial hardening note

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: lmeyerov

CHANGELOG.md

@@ -9,7 +9,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 <!-- Do Not Erase This Section - Used for tracking unreleased changes -->
 
 ### Added
-- **GFQL physical adjacency indexes — pay-as-you-go seeded-traversal acceleration**: Opt-in CSR adjacency / node-id indexes that turn seeded `hop()`/`gfql()` traversal from an O(E) full edge scan into an O(degree) `searchsorted` gather. Sidecar over edge **row positions** (never reorders `.edges`/`.nodes`), fingerprint-validated so a `.edges()` rebind safely invalidates a stale index (treated as absent — never a wrong answer). Three uniform surfaces driving one registry: **Python** (`g.create_index('edge_out_adj'|'edge_in_adj'|'node_id')`, `drop_index`, `show_indexes`, `gfql_index_edges`/`gfql_index_all`, `gfql_explain`), **Cypher DDL** (`CREATE/DROP GFQL INDEX FOR <kind>`, `SHOW GFQL INDEXES` — the mandatory `GFQL` token disambiguates from standard property `CREATE INDEX`), and the **JSON wire protocol** (`{"type":"CreateIndex",...}` ops + `index_policy` in the request envelope). Optimizer policy `gfql(..., index_policy='use'|'auto'|'force'|'off')` (default `use` = use-resident-never-build). Engine-polymorphic (numpy host arrays for pandas/polars, **cupy on-device for cudf**); the seeded fast path is hooked at every scan site (`compute/hop.py`, the lazy polars hop, and the polars single-hop chain fast path) and falls back to the scan/join path for any uncovered feature (edge/source/dest match, `target_wave_front`, `min_hops>1`, labeling). Differential-parity verified (indexed subgraph == scan subgraph) across pandas/cudf/polars/polars-gpu × forward/reverse/undirected × 1–3 hop × wavefront. **Perf (dgx-spark, deg-8, warm median):** seeded latency is **flat in graph size** — GFQL-pandas SEL1 0.110/0.109/0.113 ms at 0.8M/8M/80M edges (vs an O(E) scan of 17.7/128/1361 ms) — and **beats kuzu (CSR) and neo4j by 12–30×** on CPU (e.g. 10M nodes/80M edges: GFQL-pandas 0.113 ms vs kuzu 2.64 ms). Build cost (one O(E log E) sort) is amortized over queries; `index_policy='auto'` builds only when the planner predicts selectivity. cuDF/polars-GPU are flat but floored by ~2–3 ms GPU kernel-launch overhead (selective traversal is an indexing problem, not a compute one). No change to default (un-indexed) behavior.
+- **GFQL physical adjacency indexes — pay-as-you-go seeded-traversal acceleration**: Opt-in CSR adjacency / node-id indexes that turn seeded `hop()`/`gfql()` traversal from an O(E) full edge scan into an O(degree) `searchsorted` gather. Sidecar over edge **row positions** (never reorders `.edges`/`.nodes`), fingerprint-validated so a `.edges()` rebind safely invalidates a stale index (treated as absent — never a wrong answer). Three uniform surfaces driving one registry: **Python** (`g.create_index('edge_out_adj'|'edge_in_adj'|'node_id')`, `drop_index`, `show_indexes`, `gfql_index_edges`/`gfql_index_all`, `gfql_explain`), **Cypher DDL** (`CREATE/DROP GFQL INDEX FOR <kind>`, `SHOW GFQL INDEXES` — the mandatory `GFQL` token disambiguates from standard property `CREATE INDEX`), and the **JSON wire protocol** (`{"type":"CreateIndex",...}` ops + `index_policy` in the request envelope). Optimizer policy `gfql(..., index_policy='use'|'auto'|'force'|'off')` (default `use` = use-resident-never-build). Engine-polymorphic (numpy host arrays for pandas/polars, **cupy on-device for cudf**); the seeded fast path is hooked at every scan site (`compute/hop.py`, the lazy polars hop, and the polars single-hop chain fast path) and falls back to the scan/join path for any uncovered feature (edge/source/dest match, `target_wave_front`, `min_hops>1`, labeling). Differential-parity verified (indexed subgraph == scan subgraph) across pandas/cudf/polars/polars-gpu × forward/reverse/undirected × 1–3 hop × wavefront, hardened by an adversarial review that found and fixed several index-vs-scan divergences the original scenarios missed (`max_hops` honored; duplicate node ids, edge endpoints absent from the node table, int32/int64 seed-vs-key dtype mismatches, and stale `.edges()` rebinds all now match scan or fall back — never a wrong answer), with object-identity fingerprinting (recycle-proof) and 6 added differential regression tests. **Perf (dgx-spark, deg-8, warm median; every measured cell guarded so the index path was actually taken AND the index result equals the scan result):** seeded latency is **flat in graph size** — GFQL-pandas 1-hop 0.124/0.122 ms at 0.8M/8M nodes (6.4M/64M edges) while the O(E) scan grows 105 → 1045 ms — and **beats kuzu (CSR) and neo4j by 9–28×** on CPU at 0.8M nodes (1-hop: GFQL-pandas 0.123 ms vs kuzu 1.15 ms vs neo4j 1.45 ms; 1–2-hop: 0.150 ms vs 4.25 ms vs 2.54 ms, matched answer counts). Build cost (one O(E log E) sort) is amortized over queries; `index_policy='auto'` builds only when the planner predicts selectivity. cuDF/polars-GPU are flat but floored by ~3 ms (cuDF) GPU kernel-launch overhead — selective traversal is an indexing problem, not a compute one (CPU wins it). No change to default (un-indexed) behavior.
 - **GFQL native Polars engine — traversals (`engine='polars'`)**: Added a native, vectorized Polars execution engine for the core GFQL traversals `hop()` and `chain()`, dispatched at the engine boundary so the production pandas/cuDF paths are untouched. `Engine.POLARS` is opt-in (explicit `engine='polars'`); `engine='auto'` with Polars input still coerces to pandas as before. Covers forward/reverse/undirected single-hop traversal, directed multi-hop chains, node/edge filter dicts and predicates (lowered to Polars expressions), `edge_match`/`source_node_match`/`destination_node_match`, `target_wave_front`, and alias names; the BFS advances via semi/anti joins (no per-row Python work). Validated by differential parity against the pandas engine (hop + chain test suites plus a randomized fuzzer) and benchmarked vs pandas (`benchmarks/gfql/pandas_vs_polars.py`) — Polars wins at scale (up to ~2.5x on multi-edge chains at millions of edges; crossover ~50–100k rows). Variable-length/multi-hop edges, undirected edges in multi-edge chains, hop labels, and node `query=` raise `NotImplementedError` for now (use `engine='pandas'`).
 - **GFQL native Polars engine — cypher row pipeline (`engine='polars'`)**: Extended the Polars engine to the Cypher `MATCH … RETURN` row surface, natively vectorized. **NO CHEATING:** the polars engine never silently falls back to the pandas engine — every query runs natively on polars or raises an honest `NotImplementedError` pointing at `engine='pandas'` (falling back to pandas would misrepresent pandas performance as polars; only a human may consent to a bridge). `chain_polars` splits boundary `call()` ops (mirroring the pandas `_handle_boundary_calls`) and runs each trailing row op per-op native or raises. **Native polars** (no pandas round-trip): frame ops (`rows`/`limit`/`skip`/`distinct`/`drop_cols`), `select`/`with_`/`return_` projection (a conservative cypher-expr-AST → `pl.Expr` lowering covering property access, arithmetic, comparison, boolean, literals), `order_by` (`.sort`), `group_by` (`count`/`sum`/`avg`/`min`/`max`), `unwind` (literal-list cross-join), the result projection for property/expr columns, and entity-text `RETURN n` rendering for int/string/bool nodes (`pl.concat_str`). **Honestly deferred** (raise `NotImplementedError`, no pandas fallback): multi-entity `rows(binding_ops=…)`, cross-entity same-path `WHERE` (`DFSamePathExecutor`), float/temporal/nested entity-text, and exotic expressions (CASE/list/map/temporal, `collect` aggregates) — these are the forward native-engineering targets. Validated by differential parity vs pandas including a TCK-style conformance lane (`test_engine_polars_cypher_conformance.py`: native-only curated corpus + seeded fuzzer + NULL/3-valued-logic graph + entity-text escaping, plus a `DEFERRED` list asserting deferred queries raise rather than silently bridge) and benchmarked (`benchmarks/gfql/cypher_row_pipeline.py`). **Perf (interleaved, 1M nodes, each engine on its native-frame graph, all fully native):** polars wins **5.6–38×** across the surface — `RETURN n` ~38×, `ORDER BY` ~17×, `WHERE`+`ORDER BY`+`LIMIT` ~14×, traversals 6–7.5×, projections/aggregations/`DISTINCT` 5.6–6.9×. cuDF/pandas paths untouched.
 - **GFQL lazy Polars engine + GPU target (`engine='polars-gpu'`, cudf_polars)**: The Polars traversal engine now builds a single deferred `pl.LazyFrame` plan per single-hop and materializes `out_edges`+`out_nodes` in ONE `collect_all` on a chosen **execution target** (CPU or GPU). `engine='polars-gpu'` (`Engine.POLARS_GPU`, explicit opt-in only — AUTO never selects it) runs that same lazy plan on the RAPIDS cudf_polars backend (`pl.GPUEngine(raise_on_fail=True)` — NO-CHEATING: a GPU-incapable plan node **raises** rather than silently running on CPU and being reported as a GPU result; see Fixed). The collect-once design is what makes GPU pay off: a benchmark showed per-op eager GPU collect was a *regression* (repeated H2D), while collect-once is a **2.84× single-hop GPU win @1M** with CPU parity. Frames stay `pl.DataFrame` (handled like `POLARS` everywhere); the target is carried by a context var set at the chain/hop dispatch boundary, so `engine='polars'` (CPU) is byte-for-byte unchanged. Validated by differential parity `engine='polars-gpu' == engine='polars'` across the cypher conformance corpus + traversals (`test_engine_polars_gpu.py`, skips when no cudf_polars/GPU). Multi-hop and the chain forward/backward fusion (where the GPU win currently dilutes) are follow-up optimizations.