From 59c8b0a4117ad6cfe06e3bc22cb2fd0646c54d35 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 28 Jun 2026 21:35:45 -0700 Subject: [PATCH 01/20] docs(gfql): engine-selection guide (pandas/polars/cuDF/polars-gpu) + motivating comparison New persona-tested "Choosing a GFQL Engine" page (gfql/engines.rst): the four interchangeable engines, the one-keyword engine='polars' speedup (11-47x over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (shape x size x hardware -> engine) with crossover/work-bound/memory-pressure/GPU-or-error footnotes, cuDF-vs-polars-gpu disambiguation (eager vs fused-lazy; cuDF not deprecated), an honest "when NOT to use Polars", the differential-parity guarantee, and methodology + reproducer scripts. Also: rewrote the top of gfql/performance.rst to lead with the engine comparison (de-marketed the prose flagged by the skeptic persona), wired the page into the GFQL toctree + recommended paths, and added polars/polars-gpu to the engine examples in quick.rst and about.rst (docs previously mentioned only pandas/cuDF). Driven by 4-persona doc user-testing (pandas data scientist, RAPIDS/cuDF user, performance engineer, skeptical evaluator). Docs-only; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 3 + docs/source/gfql/about.rst | 23 ++- docs/source/gfql/engines.rst | 275 +++++++++++++++++++++++++++++++ docs/source/gfql/index.rst | 4 +- docs/source/gfql/performance.rst | 49 +++++- docs/source/gfql/quick.rst | 19 ++- 6 files changed, 355 insertions(+), 18 deletions(-) create mode 100644 docs/source/gfql/engines.rst diff --git a/CHANGELOG.md b/CHANGELOG.md index dafe1eab68..4bac3c70e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,9 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Development] +### Documentation +- **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (11–47× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the ~1M crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). + ### Added - **GFQL viz-filter-pipeline acceptance suite + regression benchmark (viz-filter L3)**: `test_viz_pipeline_conformance.py` — curated full-panel pipelines (node+edge filters, exclusion-dominates composition, `(pred OR IS NULL)` keep-null leaves, both EXISTS prune-isolated flavors, searchAny composition, deterministic paging), graph-state prune shapes with exact node+edge pins, a 40-seed panel-state fuzzer with an independent plain-pandas second oracle, and a case/regex/unicode trick matrix (ß/İ full-case-mapping pins, metachar literal-vs-regex, null cells, Categorical) — all parity-or-NIE across pandas/cuDF/polars/polars-gpu. `benchmarks/gfql/viz_filter_pipeline.py` — six streamgl-viz panel scenarios (filters, keep-self GRAPH prune, EXISTS prune, node/edge search, combined) at 100K/1M/10M with native-frame-per-engine fairness, an NIE-tolerant matrix, and JSON receipts (first receipt: 100k × 4 engines, everything within the ~350ms interactive reference except pandas combined). Documented findings from the first runs: the same-path WHERE route dedupes parallel edges (diverges from the panel algebra's edge multiplicity — pinned + tracked), and edge-alias searchAny declines on polars (tracked). - **GFQL Cypher `searchAny(entity, term[, opts])` cross-column search predicate + `g.search_nodes()`/`g.search_edges()` (viz-filter L2, native on all four engines)**: True where ANY of the entity's columns matches the term — the streamgl-viz inspector's table-search semantics as a composable WHERE predicate: OR across columns; case-insensitive substring by DEFAULT (case-folded, never regex — the common call avoids every engine regex limit); regex opt-in obeying the same per-engine decline rules as `=~`; dtype gate AS SEMANTICS (string columns always; integer columns iff the term is a numeric literal, per the inspector's `/^[0-9.-]+$/` gate; floats/dates/booleans reachable via the explicit `columns:` list). Options map `{caseSensitive, regex, columns}` is strict-validated (unknown keys error listing the valid ones); unbound aliases and missing explicit columns error clearly; null cells never match. Lowered like the pattern-predicate markers (a `search_any` row op + fresh marker column), so it composes through AND/OR/NOT and different node/edge terms coexist in one pipeline. Per-column matching reuses the parity-hardened `Contains` predicate on pandas/cuDF and a lowercase-fold/any_horizontal lowering on polars; oracle-pinned + 4-engine parity-or-NIE conformance cases. Python twins `g.search_nodes(term, columns=, case_sensitive=, regex=)` / `g.search_edges(...)` filter their own table and return a Plottable (polars-frame twins decline honestly for now — use the cypher op). Honest declines (NIE, use engine='pandas'): edge-alias searchAny on polars, and explicit columns beyond string/int/bool dtypes on polars AND cuDF (incl. floats: repr diverges across engines — dgx-probed). diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index fa57ea92e7..a1bfa0fdcb 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -372,21 +372,30 @@ GFQL is optimized for GPU acceleration using ``cudf`` and ``rapids``. When using - GFQL detects ``cudf`` dataframes and runs the query on the GPU. - Achieves significant performance improvements on large datasets. -7. Forcing GPU Mode -~~~~~~~~~~~~~~~~~~~~ +7. Selecting an Engine (CPU and GPU) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You can explicitly set the engine to ensure GPU execution. +You can explicitly set the execution engine. The same query returns identical +results on every engine — see :doc:`Choosing an Engine `. -**Example: Force GFQL to use GPU engine** +**Example: CPU columnar speedup (no GPU)** :: - g_result = g_gpu.gfql([ ... ], engine='cudf') + g_result = g.gfql([ ... ], engine='polars') # 11-47x over pandas on real graphs + +**Example: Force GFQL to use a GPU engine** + +:: + + g_result = g_gpu.gfql([ ... ], engine='cudf') # NVIDIA GPU, eager + g_result = g_gpu.gfql([ ... ], engine='polars-gpu') # NVIDIA GPU, fused plan **Explanation:** -- ``engine='cudf'`` forces the use of the GPU-accelerated engine. -- Useful when you want to ensure the query runs on the GPU. +- ``engine='polars'`` runs the columnar CPU engine — the biggest win without a GPU. +- ``engine='cudf'`` / ``'polars-gpu'`` force GPU-accelerated execution. +- Useful when you want to ensure the query runs on a specific engine. Integration with PyData Ecosystem --------------------------------- diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst new file mode 100644 index 0000000000..f9173cc547 --- /dev/null +++ b/docs/source/gfql/engines.rst @@ -0,0 +1,275 @@ +.. _gfql-engines: + +Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU +======================================================== + +GFQL runs the **same query** on four interchangeable execution engines. You pick +the engine with one keyword — ``engine=`` on :func:`gfql`/:func:`hop`/:func:`chain` — +and GFQL returns **identical results** on every one (differential parity is a release +gate). Pick the engine that fits your hardware and workload; nothing else changes. + +The one-line speedup +-------------------- + +On real graphs, switching the default ``pandas`` engine to the columnar **Polars** +engine is a one-keyword change — no GPU, same results: + +.. code-block:: python + + g.gfql(query) # engine='pandas' (default) + g.gfql(query, engine='polars') # 11-47x faster on real graphs, no GPU, identical results + +Your existing pandas (or cuDF) graph works as-is: the input frames are accepted and +coerced once; results come back as Polars frames. The only change is the keyword. + +The four engines +---------------- + +.. list-table:: + :header-rows: 1 + :widths: 16 14 18 12 40 + + * - Engine + - Hardware + - Frame type + - Opt-in? + - In one line + * - ``pandas`` + - CPU + - ``pandas`` + - default + - Universal default; best on small/interactive graphs. + * - ``polars`` + - CPU + - ``polars`` + - explicit + - Columnar + fused lazy plan; the CPU speed win, **no GPU needed**. + * - ``cudf`` + - NVIDIA GPU + - ``cudf`` + - explicit + - RAPIDS GPU, eager op-by-op; great for one very large materialization. + * - ``polars-gpu`` + - NVIDIA GPU + - ``polars`` + - explicit + - The Polars fused plan executed on GPU (cudf_polars); fastest on heavy multi-hop. + +``engine='auto'`` resolves to ``cudf`` for cuDF input and ``pandas`` otherwise. **AUTO +never selects Polars or Polars-GPU** — they are explicit opt-in (see *Why opt-in?* below). + +Motivating comparison (real graphs) +----------------------------------- + +Same query, same answers, four engines. Warm-median latency on **Orkut** (3.1M nodes / +**117M edges**, SNAP), measured on a single machine: + +.. list-table:: + :header-rows: 1 + :widths: 34 16 16 16 16 + + * - Workload (Orkut, 117M edges) + - ``pandas`` + - ``polars`` + - ``cudf`` + - ``polars-gpu`` + * - 1-hop from 10K seeds + - 2613 ms + - **68 ms** + - 1005 ms + - 63 ms + * - 2-hop from 10K seeds + - 18161 ms + - 2695 ms + - 2774 ms + - **1518 ms** + * - Full out-degree aggregation + - 799 ms + - 205 ms + - 314 ms + - **167 ms** + +*Warm median, identical result rows across all four engines. Reproducer:* +``benchmarks/gfql/index_bulk_olap_bench.py``. *See Methodology below.* + +Reading the table: + +- **Polars-CPU beats pandas 11-47x** on bulk traversal and ~4x on aggregation — **with no + GPU**. On the 1-hop workload it is ~38x faster than pandas (68 ms vs 2613 ms). +- **Polars-CPU also beats cuDF** on these shapes (68 ms vs 1005 ms on 1-hop). cuDF runs + GFQL *eagerly*, op by op (a kernel launch + a materialized intermediate per hop), while + Polars builds **one fused lazy plan and collects once**. The fused plan wins until the + work is large enough to amortize GPU launch costs. +- **Polars-GPU is fastest on heavy multi-hop** (2-hop: 1518 ms) and on aggregation — the + same fused plan, executed on the GPU. +- On a smaller graph (**LiveJournal**, 35M edges) the pattern holds: 1-hop from 10K seeds is + pandas 1129 ms → polars **37 ms** (~30x). On a real **LDBC SNB sf1** filter/lookup + workload, Polars is **18-106x** over pandas. + +.. note:: + There is **no universal winner** — route by workload shape and size (next section). + Below ~1M edges, ``pandas`` often wins because Polars/GPU pay a fixed startup overhead + that the small amount of work cannot amortize. + +Decision matrix +--------------- + +.. list-table:: + :header-rows: 1 + :widths: 30 16 18 22 14 + + * - Workload shape + - Size (edges) + - Hardware + - Recommended engine + - Notes + * - Filter / ``WHERE`` / aggregation + - > ~1M + - CPU + - ``polars`` + - 18-106x over pandas (LDBC sf1) [F1] + * - Bulk 1-hop frontier expansion + - > ~1M + - CPU + - ``polars`` + - 11-47x pandas, 6-18x cuDF + * - Heavy multi-hop (2-hop+) + - large + - GPU + - ``polars-gpu`` + - fastest until extreme materialization [F3] + * - Full-graph aggregation + - 100M+ + - GPU + - ``polars-gpu`` / ``cudf`` + - GPU work-bound [F2] + * - One very large single materialization + - 80M+ output rows + - GPU + - ``cudf`` + - Polars-GPU can hit memory pressure here [F3] + * - Anything + - < ~1M + - CPU + - ``pandas`` + - fixed Polars/GPU overhead dominates [F1] + * - Selective / seeded traversal + - any + - CPU + - ``pandas``/``polars`` + **CSR index** + - O(degree), not an engine choice [F5] + +**[F1] Crossover ~1M edges.** Below it pandas often wins (Polars/GPU fixed overhead). +Above it Polars/GPU pull away. Some fast-path shapes (filter/lookup) cross over below 100K. + +**[F2] GPU is work-bound, not size-bound.** A GPU wins when there is enough work to amortize +its ~3 ms kernel-launch floor: big frontiers, dense joins, full-graph aggregation. Tiny or +seeded work finishes faster on CPU. + +**[F3] Polars-GPU memory pressure.** On an extreme single materialization (~85M output rows, +2-hop from 100K seeds on Orkut) raw ``cudf`` leads (6.0 s) and ``polars-gpu`` slips (8.6 s) +as its in-memory GPU executor comes under memory pressure. Prefer ``cudf`` for that regime. + +**[F4] Polars-GPU is GPU-or-error.** It never silently falls back to CPU and reports the +result as a GPU run (see *Honesty* below). + +**[F5] Selective traversal is an indexing problem, not an engine choice.** A seeded ``hop`` +from a few nodes is fastest with the opt-in **CSR adjacency index** (``g.create_index(...)``, +``index_policy=``), which turns the O(E) scan into an O(degree) gather — on CPU, independent +of ``engine=``. See the index API in :doc:`/api/gfql/index`. + +cuDF vs Polars-GPU +------------------ + +Both run on an NVIDIA GPU, so which do you use? + +- **cuDF is not deprecated.** It remains a first-class, supported engine and is the right + choice for one very large materialization (footnote F3). +- **They execute differently.** ``cudf`` runs GFQL eagerly — each hop is a separate kernel + launch with a materialized intermediate. ``polars-gpu`` runs the **same fused lazy plan as + the CPU Polars engine**, collected once on the GPU. Fusing the plan is why ``polars-gpu`` + leads on heavy multi-hop and why even **CPU Polars often beats eager cuDF** on bulk work. +- **Frame type.** ``cudf`` operates on ``cudf.DataFrame``; ``polars-gpu`` operates on + ``polars.DataFrame`` (only the lazy ``.collect()`` runs on the GPU). Either way, a graph + built from pandas frames is accepted and coerced for you — only the keyword changes. +- **Install.** ``cudf`` and ``polars-gpu`` both need the RAPIDS GPU stack; ``polars-gpu`` + additionally uses ``cudf_polars``. ``polars`` (CPU) only needs ``pip install polars``. + +When **not** to use Polars +-------------------------- + +Honesty matters more than a bigger number: + +- **Small graphs (< ~1M edges):** ``pandas`` often wins; the Polars/GPU startup overhead is + not worth it (footnote F1). +- **A few exotic Cypher features** are not yet native on Polars (e.g. cross-entity same-path + ``WHERE``, some temporal/entity-text forms). They raise an honest ``NotImplementedError`` + pointing at ``engine='pandas'`` — GFQL **never** silently bridges Polars to pandas, because + that would misreport pandas performance as Polars (see *Honesty*). +- **One extreme materialization (80M+ output rows):** prefer ``cudf`` over ``polars-gpu`` + (footnote F3). +- **vs graph databases:** GFQL-Polars beats embedded kuzu on frontier expansion (up to ~87x + in our runs) and beats Neo4j+GDS end-to-end (:doc:`benchmark_filter_pagerank`). The honest + boundary: kuzu's worst-case-optimal joins target **cyclic / multi-way join** patterns + (triangles, cliques) that we have **not** yet benchmarked, and kuzu may lead there. + +Parity and honesty +------------------ + +- **Identical results across engines.** Differential parity — every engine's output must match + the pandas oracle — is a release gate, exercised across forward/reverse/undirected, 1-3 hop, + filters, and aggregations. +- **No silent fallback (NO-CHEATING).** The Polars engine runs natively or raises + ``NotImplementedError`` — it never quietly converts to pandas. ``polars-gpu`` is + **GPU-or-error**: if any step of the plan cannot run on the GPU it raises (pointing at + ``engine='polars'``) rather than silently running on CPU and labelling it a GPU result. + So any latency you measure is real work on the engine you asked for. + +Methodology +----------- + +- Host: ``dgx-spark`` (GB10 Grace-Blackwell), RAPIDS container + ``graphistry/test-rapids-official:26.02-gfql-polars``. +- Datasets: `SNAP `_ **com-LiveJournal** (35M edges), + **com-Orkut** (117M edges); **LDBC SNB sf1** for the filter/lookup figures. +- Measurement: **warm median** of repeated runs after warmups; every reported cell is + **guarded** — the result rows are verified identical across engines before any timing is kept. +- Reproduce: ``benchmarks/gfql/index_bulk_olap_bench.py`` (engine comparison), + ``benchmarks/gfql/pandas_vs_polars.py``, and ``benchmarks/gfql/index_vs_kuzu_prepared.py`` + (vs kuzu). Numbers on this page are rendered from saved runs; the page does not re-run them. + +Install +------- + +.. code-block:: bash + + pip install graphistry # base; pandas engine works out of the box + pip install graphistry polars # adds the CPU 'polars' engine + # 'cudf' and 'polars-gpu' require the NVIDIA RAPIDS stack (GPU); + # 'polars-gpu' additionally uses cudf_polars. + +Then change one keyword — your existing graph and query are unchanged: + +.. code-block:: python + + import graphistry + g = graphistry.edges(df, 'src', 'dst') # your existing pandas/cuDF graph + g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars') # CPU columnar + g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars-gpu') # same plan on GPU + +Why opt-in? +----------- + +Polars and Polars-GPU are explicit (``engine='polars'`` / ``'polars-gpu'``; ``auto`` never +picks them) for two honest reasons: (1) below the ~1M-edge crossover they can be *slower* than +pandas, so a blanket default would regress small/interactive workloads; and (2) a few exotic +Cypher features still require ``engine='pandas'`` and raise rather than silently bridge. Opting +in keeps the default behavior unchanged and makes the performance trade-off explicit. + +See also +-------- + +- :doc:`performance` — GFQL performance overview +- :doc:`benchmark_filter_pagerank` — end-to-end CPU/GPU vs Neo4j+GDS +- :doc:`/api/gfql/index` — CSR adjacency index for seeded traversal +- :doc:`remote` — run GFQL on a remote GPU diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index 8362feb6f7..1418cf0a20 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -36,7 +36,8 @@ Recommended paths: - New to GFQL: :doc:`overview` -> :doc:`quick` -> :doc:`where` -> :doc:`return` - Running Cypher syntax in GFQL: :doc:`cypher` -> :doc:`quick` -> :doc:`return` -> :doc:`spec/cypher_mapping` -- Performance path (intro -> GPU -> remote GPU): :doc:`about` -> :doc:`performance` -> :doc:`remote` +- Faster on CPU (no GPU): :doc:`engines` -> :doc:`performance` (one keyword, ``engine='polars'``, 11-47x over pandas) +- Performance path (intro -> engine choice -> GPU -> remote GPU): :doc:`about` -> :doc:`engines` -> :doc:`performance` -> :doc:`remote` - Translating existing Cypher to native GFQL: :doc:`spec/cypher_mapping` - Building agents/integrations: :doc:`spec/language` + :doc:`spec/python_embedding` + :doc:`spec/wire_protocol` @@ -50,6 +51,7 @@ See also: about overview remote + Choosing an Engine GFQL CPU & GPU Acceleration End-to-End Benchmark translate diff --git a/docs/source/gfql/performance.rst b/docs/source/gfql/performance.rst index 1c7334840b..2400075053 100644 --- a/docs/source/gfql/performance.rst +++ b/docs/source/gfql/performance.rst @@ -1,9 +1,48 @@ .. _gfql-performance: -GFQL Performance: Unleashing Vectorization and GPU Power for Scalable Graph Analytics -====================================================================================== - -GFQL, developed by Graphistry, rethinks graph analytics by harnessing vectorization and GPU acceleration. As datasets grow from thousands to billions of rows, traditional tools struggle to keep up without significant infrastructure investment. GFQL is rewriting the story. Start small with a quick `pip install graphistry` on your CPU system, and scale more smoothly by leveraging the power of vectorization and GPUs to handle historically tricky datasets. +GFQL Performance: Vectorization and GPU Acceleration +==================================================== + +Engine speedups at a glance +--------------------------- + +GFQL runs the **same query** on four interchangeable engines — ``pandas`` (default), +``polars`` (CPU, columnar), ``cudf`` (NVIDIA GPU), and ``polars-gpu`` (GPU) — and returns +**identical results** on each (differential parity is a release gate). The biggest, easiest +win is one keyword, **no GPU required**: + +.. code-block:: python + + g.gfql(query) # engine='pandas' (default) + g.gfql(query, engine='polars') # 11-47x faster on real graphs, same results + +Warm-median latency, same query, identical result rows (**Orkut**, 117M edges, SNAP): + +.. list-table:: + :header-rows: 1 + :widths: 40 15 15 15 15 + + * - Workload (117M edges) + - ``pandas`` + - ``polars`` + - ``cudf`` + - ``polars-gpu`` + * - 1-hop from 10K seeds + - 2613 ms + - **68 ms** + - 1005 ms + - 63 ms + * - Full out-degree aggregation + - 799 ms + - 205 ms + - 314 ms + - **167 ms** + +There is **no universal winner**: below ~1M edges ``pandas`` often wins, and the right GPU +engine depends on the workload. See :doc:`engines` for the full decision matrix, the honest +"when *not* to use Polars", the cuDF-vs-Polars-GPU comparison, and the methodology + reproducer +scripts behind these numbers. The end-to-end CPU/GPU-vs-Neo4j benchmark is in +:doc:`benchmark_filter_pagerank`. Built from Real-World Necessity ------------------------------- @@ -17,7 +56,7 @@ A New Era of Graph Analytics Graphistry has a history of award-winning open source data visualization and GPU acceleration engines. With GFQL, we bring our lessons learned to graph querying and analysis for real-time insights on datasets both big and small. Unlike traditional graph databases that process one path at a time, GFQL traverses entire collections simultaneously. Similar to best-of-class analytical CPU databases like Clickhouse and Google BigQuery, our vectorized approach maximizes throughput to drastically reduce query time. -When coupled with GPU acceleration, GFQL's performance reaches Graph 500 levels with even the cheapest cloud GPUs. Modern GPUs execute tens of thousands of threads in parallel, and GFQL is designed to fully saturate this capability. Whether you're traversing graphs with billions of edges or running complex algorithms, GFQL transforms previously impractical tasks into manageable ones. +When coupled with GPU acceleration, GFQL handles hundred-million-edge traversals in interactive time on a single GPU (see :doc:`engines` and :doc:`benchmark_filter_pagerank` for measured numbers). Modern GPUs execute tens of thousands of threads in parallel, and GFQL is designed to fully saturate this capability. Whether you're traversing graphs with billions of edges or running complex algorithms, GFQL transforms previously impractical tasks into manageable ones. Three Simple Ideas Behind GFQL's Performance diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 2b2d63fd6e..66cb0c5544 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -27,7 +27,7 @@ Basic Usage :meth:`gfql ` sequences multiple matchers for more complex patterns of paths and subgraphs - **query**: Sequence of graph node/edge matchers and optional row-pipeline call steps (for example, `rows()`, `where_rows()`, `return_()`, `order_by()`, `limit()`), or an equivalent GFQL chain object. -- **engine**: Optional execution engine. Engine is typically not set, defaulting to `'auto'`. Use `'cudf'` for GPU acceleration and `'pandas'` for CPU. +- **engine**: Optional execution engine. Engine is typically not set, defaulting to `'auto'`. Use `'polars'` for a CPU columnar speedup (11-47x over pandas, no GPU), `'cudf'` or `'polars-gpu'` for NVIDIA GPU acceleration, and `'pandas'` for the default CPU path. See :doc:`Choosing an Engine `. Native GFQL chains are typed Python inputs. Pass the list, dict envelope, or ``Chain`` object itself; strings passed to ``g.gfql(...)`` are interpreted as @@ -400,14 +400,23 @@ Combined Examples n(query="status == 'active'") ]) -GPU Acceleration ----------------- +Engine Selection (CPU and GPU) +------------------------------ -- **Enable GPU mode:** +The same query runs on four interchangeable engines with identical results. Pick one +with ``engine=``. See :doc:`Choosing an Engine ` for the full decision matrix. + +- **CPU columnar speedup (no GPU):** ``'polars'`` — 11-47x over pandas on real graphs. .. code-block:: python - g.gfql([...], engine='cudf') + g.gfql([...], engine='polars') # keep your existing pandas frames; just the keyword changes + +- **NVIDIA GPU:** ``'cudf'`` (eager) or ``'polars-gpu'`` (fused plan on GPU). + + .. code-block:: python + + g.gfql([...], engine='polars-gpu') - **Example with cuDF DataFrames:** From 075eb957f269b2c3fcbcfa6406028823d1913431 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 28 Jun 2026 21:43:49 -0700 Subject: [PATCH 02/20] =?UTF-8?q?docs(gfql):=20round-2=20user-testing=20fi?= =?UTF-8?q?xes=20=E2=80=94=20de-market=20perf=20tail,=20honest=20numbers,?= =?UTF-8?q?=20output-frame=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies fixes from a second persona user-testing pass on the rendered docs: - performance.rst: removed the surviving marketing tail (the skeptic persona's #1 residual) — "A New Era", "Empower Your Data Journey", "Join the Community", and the NVIDIA-investment-implies-performance line — replaced with a tight, de-superlatived "How GFQL is fast" (the real mechanisms) + a focused Next Steps. - engines.rst: added the cuDF-WINS row to the comparison table (2-hop/100K seeds, ~85M output rows: cuDF 6.0s) so cuDF winning is visible without reading footnotes (RAPIDS persona); added a prominent note that result frames match the engine (polars-gpu/polars return polars.DataFrame; .to_pandas() to convert) — the pandas+RAPIDS personas' top practical gotcha; fixed the LDBC sf1 figure attribution (it is from a separate benchmark, not the cited Orkut/LiveJournal source-of-truth) to keep every on-page number traceable; added run counts + unified-memory note to Methodology (perf-engineer persona). Docs-only; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 38 +++++++++---- docs/source/gfql/performance.rst | 91 +++++++------------------------- 2 files changed, 47 insertions(+), 82 deletions(-) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index f9173cc547..a4c82452e5 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -20,7 +20,13 @@ engine is a one-keyword change — no GPU, same results: g.gfql(query, engine='polars') # 11-47x faster on real graphs, no GPU, identical results Your existing pandas (or cuDF) graph works as-is: the input frames are accepted and -coerced once; results come back as Polars frames. The only change is the keyword. +coerced once; the only change is the keyword. + +.. note:: + **Result frames match the engine.** With ``engine='polars'`` or ``'polars-gpu'`` the + output is Polars — ``result._nodes`` and ``result._edges`` are ``polars.DataFrame`` (and + ``cudf.DataFrame`` for ``engine='cudf'``). If downstream code is pandas-specific (``.iloc``, + ``.loc``, ``groupby().apply()``), call ``result._nodes.to_pandas()`` to convert back. The four engines ---------------- @@ -88,6 +94,11 @@ Same query, same answers, four engines. Warm-median latency on **Orkut** (3.1M n - 205 ms - 314 ms - **167 ms** + * - 2-hop from 100K seeds (~85M output rows) + - 28822 ms + - 8215 ms + - **6002 ms** + - 8559 ms *Warm median, identical result rows across all four engines. Reproducer:* ``benchmarks/gfql/index_bulk_olap_bench.py``. *See Methodology below.* @@ -100,11 +111,15 @@ Reading the table: GFQL *eagerly*, op by op (a kernel launch + a materialized intermediate per hop), while Polars builds **one fused lazy plan and collects once**. The fused plan wins until the work is large enough to amortize GPU launch costs. -- **Polars-GPU is fastest on heavy multi-hop** (2-hop: 1518 ms) and on aggregation — the - same fused plan, executed on the GPU. +- **Polars-GPU is fastest on heavy multi-hop** (2-hop from 10K seeds: 1518 ms) and on + aggregation — the same fused plan, executed on the GPU. +- **cuDF wins the one extreme case** — a 2-hop from 100K seeds materializing ~85M output rows + (6.0 s) — where raw GPU throughput on a single massive join overtakes everything and + Polars-GPU comes under memory pressure (footnote F3). - On a smaller graph (**LiveJournal**, 35M edges) the pattern holds: 1-hop from 10K seeds is - pandas 1129 ms → polars **37 ms** (~30x). On a real **LDBC SNB sf1** filter/lookup - workload, Polars is **18-106x** over pandas. + pandas 1129 ms → polars **37 ms** (~30x). Filter- and lookup-heavy workloads favor Polars + even more strongly — a separate **LDBC SNB sf1** benchmark shows order-of-magnitude gains + (tens of × over pandas; see ``benchmarks/gfql/`` and the GFQL benchmark notes). .. note:: There is **no universal winner** — route by workload shape and size (next section). @@ -127,7 +142,7 @@ Decision matrix - > ~1M - CPU - ``polars`` - - 18-106x over pandas (LDBC sf1) [F1] + - order-of-magnitude over pandas [F1] * - Bulk 1-hop frontier expansion - > ~1M - CPU @@ -228,12 +243,15 @@ Parity and honesty Methodology ----------- -- Host: ``dgx-spark`` (GB10 Grace-Blackwell), RAPIDS container +- Host: ``dgx-spark`` (GB10 Grace-Blackwell, unified memory — the F3 memory-pressure + boundary is partly a property of this box), RAPIDS container ``graphistry/test-rapids-official:26.02-gfql-polars``. - Datasets: `SNAP `_ **com-LiveJournal** (35M edges), - **com-Orkut** (117M edges); **LDBC SNB sf1** for the filter/lookup figures. -- Measurement: **warm median** of repeated runs after warmups; every reported cell is - **guarded** — the result rows are verified identical across engines before any timing is kept. + **com-Orkut** (117M edges). The order-of-magnitude filter/lookup figure is from a separate + **LDBC SNB sf1** benchmark, not the table above. +- Measurement: **warm median** after 2 warmups (5 timed runs on Orkut, 8 on LiveJournal); + every reported cell is **guarded** — the result rows are verified identical across engines + before any timing is kept. - Reproduce: ``benchmarks/gfql/index_bulk_olap_bench.py`` (engine comparison), ``benchmarks/gfql/pandas_vs_polars.py``, and ``benchmarks/gfql/index_vs_kuzu_prepared.py`` (vs kuzu). Numbers on this page are rendered from saved runs; the page does not re-run them. diff --git a/docs/source/gfql/performance.rst b/docs/source/gfql/performance.rst index 2400075053..fa7d4e3194 100644 --- a/docs/source/gfql/performance.rst +++ b/docs/source/gfql/performance.rst @@ -44,88 +44,35 @@ engine depends on the workload. See :doc:`engines` for the full decision matrix, scripts behind these numbers. The end-to-end CPU/GPU-vs-Neo4j benchmark is in :doc:`benchmark_filter_pagerank`. -Built from Real-World Necessity -------------------------------- +How GFQL is fast +---------------- -GFQL was born out of the challenges our team faced across many graph customer projects over the last 10 years. Projects often start with manageable datasets, and as they scale up, require tools that can grow without imposing prohibitive costs or complexities. Likewise, traditional graph solutions often require adding additional storage tier infrastructure and systems of record that duplicate a team's existing standard databases and warehouses: Too many projects died from premature distractions and complexities here. +Three design choices explain the numbers above: -We have `long recognizing the untapped potential of CPUs and GPUs in the compute tier `_ and the lack of effective libraries to leverage them for graph analytics. GFQL fills this gap. We designed GFQL to integrate seamlessly with the graph and dataframe ecosystem, providing a much easier, unified, and scalable solution while eliminating the need for hazardous storage tier detours. +**Collection-oriented execution.** GFQL evaluates whole collections of nodes and edges at +once (set-at-a-time), rather than walking one path at a time like traditional Cypher/Gremlin +engines. A traversal advances by joining edge tables, so the work vectorizes. -A New Era of Graph Analytics ----------------------------- +**Vectorized columnar processing.** Data is processed in columnar batches on top of +`Apache Arrow `_, which keeps the CPU path fast and makes moving +data between systems cheap. The ``polars`` engine additionally builds **one fused lazy plan +and collects once**, which is why it outruns both pandas and eager cuDF on bulk work. -Graphistry has a history of award-winning open source data visualization and GPU acceleration engines. With GFQL, we bring our lessons learned to graph querying and analysis for real-time insights on datasets both big and small. Unlike traditional graph databases that process one path at a time, GFQL traverses entire collections simultaneously. Similar to best-of-class analytical CPU databases like Clickhouse and Google BigQuery, our vectorized approach maximizes throughput to drastically reduce query time. +**Massive parallelism on GPUs.** On an NVIDIA GPU (``cudf`` / ``polars-gpu``), the same +vectorized work saturates tens of thousands of threads — paying off when there is enough +work to amortize kernel-launch cost (large frontiers, dense joins, full-graph aggregation). -When coupled with GPU acceleration, GFQL handles hundred-million-edge traversals in interactive time on a single GPU (see :doc:`engines` and :doc:`benchmark_filter_pagerank` for measured numbers). Modern GPUs execute tens of thousands of threads in parallel, and GFQL is designed to fully saturate this capability. Whether you're traversing graphs with billions of edges or running complex algorithms, GFQL transforms previously impractical tasks into manageable ones. - - -Three Simple Ideas Behind GFQL's Performance ---------------------------------------------- - -At the core of GFQL's performance are three pioneering techniques: - -**Collection-Oriented Algorithms** - -GFQL operates on entire collections of nodes and edges simultaneously, different from older commercial Cypher and Gremlin graph query engines that process one path at a time. The collection-oriented approach, inspired by our research at UC Berkeley and our experience with GPUs, maximizes data throughput and minimizes computational overhead. Small queries stay interactive, and large-scale graph analytics is now more efficient than ever before. - -**Vectorized Columnar Processing** - -GFQL processes data in large, parallel batches using columnar data structures. This method optimizes memory usage and computational efficiency, significantly speeding up data handling compared to traditional row-based systems. Natively integrating with cutting-edge technologies like `Apache Arrow `_, this approach ensures high performance even on CPUs, and unusually fast speeds for moving large data across systems. - -**Massive Parallelism with GPUs** - -Designed to saturate the tens of thousands of threads in modern GPUs, GFQL enables rapid processing of complex graph queries. This massive parallelism allows GFQL to handle tasks that are impractical on typical CPU systems, such as real-time traversals that touch hundreds of millions of edges and compute on them. - - -Seamless Scalability from CPUs to GPUs --------------------------------------- - -GFQL allows you to start analyzing graphs on standard CPUs without specialized hardware. As your data grows, you can transition to GPU acceleration without changing your code. GFQL intelligently utilizes available hardware to optimize performance, ensuring efficient resource use whether you're on a single machine or across a cluster. - -By eliminating the need for additional infrastructure, GFQL reduces time and expense, allowing you to focus on extracting insights from your data. This seamless scalability ensures that as your projects evolve, GFQL adapts to meet your needs. - -Optimized for Analytical Workloads ----------------------------------- - -GFQL excels in scenarios requiring deep analytical capabilities. It is designed for: - -- **Graph ETL and Analytics**: Efficiently process and transform large volumes of graph data. -- **Machine Learning and AI**: Accelerate graph-based ML and AI tasks, leveraging GPUs for training and inference. -- **Visualization**: Power high-performance graph visualizations, enabling real-time interaction with complex datasets. - -By focusing on these areas, GFQL meets the demands of modern data projects, from initial exploration to advanced analysis, without the overhead typically associated with large-scale analytics. +Start on CPU with no special hardware, and move to a GPU engine by changing one keyword when +your workload grows into GPU territory. See :doc:`engines` for exactly when each engine wins. .. note:: Same-path constraints (``where``) can be more expensive on dense graphs. Prefer selective per-step predicates and see :doc:`/gfql/where` for details. -Built on Graphistry's Expertise -------------------------------- - -Graphistry's reputation for leveraging GPUs and vectorization in data analytics is well-established. GFQL embodies this expertise, filling gaps in the graph and dataframe ecosystem by providing tools that maximize GPU utilization and integrate with open-source technologies like Apache Arrow. Our collaboration with `NVIDIA `_, including their investment into our team, ensures that GFQL benefits from optimized kernel methods for top-tier performance. - -Empower Your Data Journey -------------------------- - -With GFQL, you can start quickly, scale more smoothly, and leverage cutting-edge performance. It empowers you to: - -- Begin analyzing graphs immediately on your existing hardware -- Grow from CPU to GPU processing without code changes -- Handle datasets ranging from thousands to billions of edges efficiently - -Whether you're analyzing social networks, investigating cybersecurity threats, or exploring intricate datasets, GFQL transforms how you work with graph data, making complex analytics accessible and efficient. - -Join the Graphistry Community ------------------------------ - -We invite you to become part of our community dedicated to advancing graph analytics through innovation in vectorization and GPU computing. Let's keep pushing the boundaries of what's possible! - ---- - Next Steps ---------- -- **Explore GFQL**: Dive deeper into GFQL's capabilities in :ref:`10min-gfql`. -- **Get Started with PyGraphistry**: Follow the :ref:`10min-pygraphistry` to setup and experience the performance firsthand. -- **Learn About Vectorization and GPUs**: Understand the partner ecosystem technologies behind GFQL by exploring `Apache Arrow `_ and `NVIDIA RAPIDS `_. -- **Connect with Us**: Join our :ref:`community` to share insights and collaborate with others pushing the boundaries of graph analytics. +- **Choose an engine**: :doc:`engines` — the full decision matrix, methodology, and reproducers. +- **End-to-end benchmark**: :doc:`benchmark_filter_pagerank` — CPU/GPU vs Neo4j+GDS. +- **Explore GFQL**: :ref:`10min-gfql`. **Get started**: :ref:`10min-pygraphistry`. +- **Ecosystem**: `Apache Arrow `_ and `NVIDIA RAPIDS `_. From 29b969b8682a9db25a9717181df26a9d4be0bf10 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 28 Jun 2026 22:07:25 -0700 Subject: [PATCH 03/20] =?UTF-8?q?docs(gfql):=20apply=20test-amplification?= =?UTF-8?q?=20=C2=A70=20protocol=20findings=20(clean-room)=20+=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the repo's documented user-testing protocol (test-amplification SKILL §0 "User-Workflow Exploration") clean-room — two passes (need-finding vs original docs, then QA on the produced docs) — and applied the deltas it surfaced that the earlier ad-hoc persona pass missed: Completeness (Pass A): finished the engine enumerations the ad-hoc pass deferred — overview.rst now names all four engines + auto's resolution rule + the opt-in/no-silent- fallback contract (was "GFQL automatically executes on GPU", which implied silent selection); notebooks/gpu.rst now points GPU readers to the engines page. Accuracy/QA (Pass B): reconciled the recurring "11-47x" headline to what the on-page table supports (-> "up to ~38x", Orkut 1-hop, traceable) across 9 sites; fixed cuDF "6-18x" -> "~15x (Orkut 1-hop)"; corrected a wrong "polars (CPU) is GPU-or-error" claim (only polars-gpu is — CPU polars raises NotImplementedError); dropped the deprecated `chain` from the engines.rst entrypoint line (gfql/hop only); scoped the ~87x kuzu claim to LiveJournal + named its reproducer; stopped the CSR-index footnote from over-promising an API page that doesn't document it yet; cited the orphaned [F4] footnote. Docs-only; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/about.rst | 2 +- docs/source/gfql/engines.rst | 21 +++++++++++---------- docs/source/gfql/index.rst | 2 +- docs/source/gfql/overview.rst | 14 +++++++++----- docs/source/gfql/performance.rst | 2 +- docs/source/gfql/quick.rst | 4 ++-- docs/source/notebooks/gpu.rst | 5 +++++ 7 files changed, 30 insertions(+), 20 deletions(-) diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index a1bfa0fdcb..8d471a103b 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -382,7 +382,7 @@ results on every engine — see :doc:`Choosing an Engine `. :: - g_result = g.gfql([ ... ], engine='polars') # 11-47x over pandas on real graphs + g_result = g.gfql([ ... ], engine='polars') # up to ~38x over pandas on real graphs **Example: Force GFQL to use a GPU engine** diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index a4c82452e5..2ad31a7ff1 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -4,9 +4,9 @@ Choosing a GFQL Engine: pandas, Polars, cuDF, Polars-GPU ======================================================== GFQL runs the **same query** on four interchangeable execution engines. You pick -the engine with one keyword — ``engine=`` on :func:`gfql`/:func:`hop`/:func:`chain` — -and GFQL returns **identical results** on every one (differential parity is a release -gate). Pick the engine that fits your hardware and workload; nothing else changes. +the engine with one keyword — ``engine=``, accepted uniformly by ``g.gfql()`` and +``g.hop()`` — and GFQL returns **identical results** on every one (differential parity +is a release gate). Pick the engine that fits your hardware and workload; nothing else changes. The one-line speedup -------------------- @@ -17,7 +17,7 @@ engine is a one-keyword change — no GPU, same results: .. code-block:: python g.gfql(query) # engine='pandas' (default) - g.gfql(query, engine='polars') # 11-47x faster on real graphs, no GPU, identical results + g.gfql(query, engine='polars') # up to ~38x faster on real graphs, no GPU, identical results Your existing pandas (or cuDF) graph works as-is: the input frames are accepted and coerced once; the only change is the keyword. @@ -105,7 +105,7 @@ Same query, same answers, four engines. Warm-median latency on **Orkut** (3.1M n Reading the table: -- **Polars-CPU beats pandas 11-47x** on bulk traversal and ~4x on aggregation — **with no +- **Polars-CPU beats pandas up to ~38x** on bulk traversal and ~4x on aggregation — **with no GPU**. On the 1-hop workload it is ~38x faster than pandas (68 ms vs 2613 ms). - **Polars-CPU also beats cuDF** on these shapes (68 ms vs 1005 ms on 1-hop). cuDF runs GFQL *eagerly*, op by op (a kernel launch + a materialized intermediate per hop), while @@ -147,12 +147,12 @@ Decision matrix - > ~1M - CPU - ``polars`` - - 11-47x pandas, 6-18x cuDF + - up to ~38x pandas, ~15x cuDF (Orkut 1-hop) * - Heavy multi-hop (2-hop+) - large - GPU - ``polars-gpu`` - - fastest until extreme materialization [F3] + - fastest until extreme materialization [F3]; GPU-or-error [F4] * - Full-graph aggregation - 100M+ - GPU @@ -191,7 +191,7 @@ result as a GPU run (see *Honesty* below). **[F5] Selective traversal is an indexing problem, not an engine choice.** A seeded ``hop`` from a few nodes is fastest with the opt-in **CSR adjacency index** (``g.create_index(...)``, ``index_policy=``), which turns the O(E) scan into an O(degree) gather — on CPU, independent -of ``engine=``. See the index API in :doc:`/api/gfql/index`. +of ``engine=``. (A dedicated index guide is in progress; the methods live under the GFQL API.) cuDF vs Polars-GPU ------------------ @@ -224,7 +224,8 @@ Honesty matters more than a bigger number: - **One extreme materialization (80M+ output rows):** prefer ``cudf`` over ``polars-gpu`` (footnote F3). - **vs graph databases:** GFQL-Polars beats embedded kuzu on frontier expansion (up to ~87x - in our runs) and beats Neo4j+GDS end-to-end (:doc:`benchmark_filter_pagerank`). The honest + on LiveJournal 1-hop in our runs — reproducer ``benchmarks/gfql/index_vs_kuzu_prepared.py``), + and separately beats Neo4j+GDS end-to-end (:doc:`benchmark_filter_pagerank`). The honest boundary: kuzu's worst-case-optimal joins target **cyclic / multi-way join** patterns (triangles, cliques) that we have **not** yet benchmarked, and kuzu may lead there. @@ -289,5 +290,5 @@ See also - :doc:`performance` — GFQL performance overview - :doc:`benchmark_filter_pagerank` — end-to-end CPU/GPU vs Neo4j+GDS -- :doc:`/api/gfql/index` — CSR adjacency index for seeded traversal +- :doc:`/api/gfql/index` — GFQL API reference - :doc:`remote` — run GFQL on a remote GPU diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index 1418cf0a20..43b74cc8bb 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -36,7 +36,7 @@ Recommended paths: - New to GFQL: :doc:`overview` -> :doc:`quick` -> :doc:`where` -> :doc:`return` - Running Cypher syntax in GFQL: :doc:`cypher` -> :doc:`quick` -> :doc:`return` -> :doc:`spec/cypher_mapping` -- Faster on CPU (no GPU): :doc:`engines` -> :doc:`performance` (one keyword, ``engine='polars'``, 11-47x over pandas) +- Faster on CPU (no GPU): :doc:`engines` -> :doc:`performance` (one keyword, ``engine='polars'``, up to ~38x over pandas) - Performance path (intro -> engine choice -> GPU -> remote GPU): :doc:`about` -> :doc:`engines` -> :doc:`performance` -> :doc:`remote` - Translating existing Cypher to native GFQL: :doc:`spec/cypher_mapping` - Building agents/integrations: :doc:`spec/language` + :doc:`spec/python_embedding` + :doc:`spec/wire_protocol` diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index 1a12f475e1..f08d531464 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -316,9 +316,11 @@ Key advantages of GFQL Let: Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -GFQL is optimized to take advantage of GPU acceleration using `cudf` and RAPIDS. When you use GPU dataframes, GFQL automatically executes queries on the GPU for massive speedups. +GFQL runs the same query on four interchangeable engines, all returning identical results: ``pandas`` (CPU, default), ``polars`` (CPU columnar — up to ~38x over pandas, **no GPU**), ``cudf`` (NVIDIA GPU), and ``polars-gpu`` (NVIDIA GPU). ``engine='auto'`` resolves to ``cudf`` for cuDF input and ``pandas`` otherwise; ``polars`` / ``polars-gpu`` are explicit opt-in (``auto`` never selects them). Neither silently bridges: ``polars-gpu`` is GPU-or-error (it raises rather than silently running on CPU), and ``polars`` (CPU) raises ``NotImplementedError`` for the few unsupported Cypher features rather than falling back to pandas. See :doc:`Choosing an Engine ` for the decision matrix and benchmarks. -**Automatic GPU Acceleration** +When you use cuDF (GPU) dataframes with ``engine='auto'``, GFQL executes queries on the GPU for massive speedups. + +**Automatic GPU Acceleration (cuDF)** Example: Run GFQL queries with GPU dataframes. @@ -338,13 +340,15 @@ Example: Run GFQL queries with GPU dataframes. g_result = g_gpu.gfql([ ... ]) # Your GFQL query here print('Number of resulting edges:', len(g_result._edges)) -**Forcing GPU Mode** +**Selecting an Engine Explicitly** -Example: Explicitly set the engine to ensure GPU execution. +Example: set the engine for a CPU columnar speedup or to force a specific GPU engine. .. code-block:: python - g_result = g_gpu.gfql([ ... ], engine='cudf') + g_result = g.gfql([ ... ], engine='polars') # CPU columnar, no GPU + g_result = g_gpu.gfql([ ... ], engine='cudf') # NVIDIA GPU, eager + g_result = g_gpu.gfql([ ... ], engine='polars-gpu') # NVIDIA GPU, fused plan Run Remotely ~~~~~~~~~~~~~ diff --git a/docs/source/gfql/performance.rst b/docs/source/gfql/performance.rst index fa7d4e3194..92d6961f39 100644 --- a/docs/source/gfql/performance.rst +++ b/docs/source/gfql/performance.rst @@ -14,7 +14,7 @@ win is one keyword, **no GPU required**: .. code-block:: python g.gfql(query) # engine='pandas' (default) - g.gfql(query, engine='polars') # 11-47x faster on real graphs, same results + g.gfql(query, engine='polars') # up to ~38x faster on real graphs, same results Warm-median latency, same query, identical result rows (**Orkut**, 117M edges, SNAP): diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 66cb0c5544..ed469f27b3 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -27,7 +27,7 @@ Basic Usage :meth:`gfql ` sequences multiple matchers for more complex patterns of paths and subgraphs - **query**: Sequence of graph node/edge matchers and optional row-pipeline call steps (for example, `rows()`, `where_rows()`, `return_()`, `order_by()`, `limit()`), or an equivalent GFQL chain object. -- **engine**: Optional execution engine. Engine is typically not set, defaulting to `'auto'`. Use `'polars'` for a CPU columnar speedup (11-47x over pandas, no GPU), `'cudf'` or `'polars-gpu'` for NVIDIA GPU acceleration, and `'pandas'` for the default CPU path. See :doc:`Choosing an Engine `. +- **engine**: Optional execution engine. Engine is typically not set, defaulting to `'auto'`. Use `'polars'` for a CPU columnar speedup (up to ~38x over pandas, no GPU), `'cudf'` or `'polars-gpu'` for NVIDIA GPU acceleration, and `'pandas'` for the default CPU path. See :doc:`Choosing an Engine `. Native GFQL chains are typed Python inputs. Pass the list, dict envelope, or ``Chain`` object itself; strings passed to ``g.gfql(...)`` are interpreted as @@ -406,7 +406,7 @@ Engine Selection (CPU and GPU) The same query runs on four interchangeable engines with identical results. Pick one with ``engine=``. See :doc:`Choosing an Engine ` for the full decision matrix. -- **CPU columnar speedup (no GPU):** ``'polars'`` — 11-47x over pandas on real graphs. +- **CPU columnar speedup (no GPU):** ``'polars'`` — up to ~38x over pandas on real graphs. .. code-block:: python diff --git a/docs/source/notebooks/gpu.rst b/docs/source/notebooks/gpu.rst index 52293d5575..eea06fcef4 100644 --- a/docs/source/notebooks/gpu.rst +++ b/docs/source/notebooks/gpu.rst @@ -1,6 +1,11 @@ GPU ========================== +GFQL has two NVIDIA GPU engines: ``engine='cudf'`` (RAPIDS, eager) and +``engine='polars-gpu'`` (the fused lazy Polars plan on GPU). See +:doc:`Choosing a GFQL Engine ` for which to use and how they compare to the +CPU ``pandas`` / ``polars`` engines. + .. toctree:: :maxdepth: 2 :caption: GPU compute with Nvidia RAPIDS From 9d10b87c6f5ec514f88602bb29f62e3425f17afc Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 28 Jun 2026 22:43:39 -0700 Subject: [PATCH 04/20] =?UTF-8?q?docs(gfql):=20address=20the=20existing-Po?= =?UTF-8?q?lars-user=20persona=20(=C2=A70=20round-003)=20=E2=80=94=20silen?= =?UTF-8?q?t-coercion=20warning?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A §0 user-testing pass on polars-centric personas found a real P0 gap: nothing in the docs spoke to a user who is ALREADY on Polars, and the silent default-path downgrade was never warned. A graph built from polars.DataFrame run with the default engine='auto' is coerced to pandas (auto -> cudf for cuDF input, pandas for everything else incl. Polars; it never selects the Polars engine), so result._nodes comes back pandas and downstream pl.* breaks at runtime. Fixes: - engines.rst: a `.. warning::` "Already a Polars user? pass engine='polars' — the default does not" with a pl.DataFrame in -> engine='polars' -> pl.DataFrame out worked example; co-located the "catch" (crossover + NotImplementedError) under the one-liner. - overview.rst: spelled out that auto coerces a Polars-frame graph to pandas unless you pass engine='polars'. - Added Polars to the accepted-input lists in engines.rst / overview.rst / about.rst (was "pandas, cuDF" only). Artifact: plans/gfql-engine-docs/rounds/round-003/user_testing_playbook.md. Docs-only. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/about.rst | 2 +- docs/source/gfql/engines.rst | 22 +++++++++++++++++++--- docs/source/gfql/overview.rst | 4 ++-- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index 8d471a103b..bfb32d6c24 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -27,7 +27,7 @@ GFQL fills a critical gap in the data community by providing an in-process, high **Key Benefits:** -- **Dataframe-Native:** Works directly with Pandas, cuDF, and other dataframe libraries. +- **Dataframe-Native:** Works directly with Pandas, Polars, cuDF, and other dataframe libraries. - **High Performance:** Optimized for both CPU and GPU execution. - **Ease of Use:** No need for external databases or new infrastructure. - **Interoperability:** Integrates with the Python data science ecosystem, including PyGraphistry for visualization. diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 2ad31a7ff1..895f765f14 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -19,8 +19,24 @@ engine is a one-keyword change — no GPU, same results: g.gfql(query) # engine='pandas' (default) g.gfql(query, engine='polars') # up to ~38x faster on real graphs, no GPU, identical results -Your existing pandas (or cuDF) graph works as-is: the input frames are accepted and -coerced once; the only change is the keyword. +Your existing pandas, Polars, or cuDF graph works as-is: the input frames are accepted and +coerced once; the only change is the keyword. The catch: below ~1M edges ``pandas`` can be +faster (Polars has fixed startup overhead), and a few exotic Cypher features still require +``engine='pandas'`` — see *When not to use Polars* below. + +.. warning:: + **Already a Polars user? Pass** ``engine='polars'`` **— the default does not.** With the + default ``engine='auto'``, a graph built from ``polars.DataFrame`` is **silently coerced to + pandas** (``auto`` resolves to ``cudf`` for cuDF input and ``pandas`` for everything else, + *including Polars*; it never selects the Polars engine). To stay native end-to-end, pass + ``engine='polars'`` explicitly: + + .. code-block:: python + + import polars as pl, graphistry + g = graphistry.edges(edges_pl, 'src', 'dst').nodes(nodes_pl, 'id') # polars frames + out = g.gfql(query) # auto -> coerced to PANDAS (out._nodes is pandas!) + out = g.gfql(query, engine='polars') # native Polars in and out (out._nodes is polars) .. note:: **Result frames match the engine.** With ``engine='polars'`` or ``'polars-gpu'`` the @@ -272,7 +288,7 @@ Then change one keyword — your existing graph and query are unchanged: .. code-block:: python import graphistry - g = graphistry.edges(df, 'src', 'dst') # your existing pandas/cuDF graph + g = graphistry.edges(df, 'src', 'dst') # your existing pandas, Polars, or cuDF graph g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars') # CPU columnar g.gfql("MATCH (a)-[e]->(b) RETURN b", engine='polars-gpu') # same plan on GPU diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index f08d531464..c4ce1aaa89 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -24,7 +24,7 @@ GFQL addresses a critical gap in the data community by providing an in-process g Key Features ~~~~~~~~~~~~~ -- **Dataframe-Native Integration**: Works directly with Pandas, cuDF, and Apache Arrow dataframes. +- **Dataframe-Native Integration**: Works directly with Pandas, Polars, cuDF, and Apache Arrow dataframes. - **High Performance**: Optimized for both CPU and GPU execution, capable of processing billions of edges. - **Ease of Use**: Install via `pip` and start querying without the need for external databases. - **Seamless Visualization**: Integrated with PyGraphistry for GPU-accelerated graph visualization. @@ -316,7 +316,7 @@ Key advantages of GFQL Let: Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -GFQL runs the same query on four interchangeable engines, all returning identical results: ``pandas`` (CPU, default), ``polars`` (CPU columnar — up to ~38x over pandas, **no GPU**), ``cudf`` (NVIDIA GPU), and ``polars-gpu`` (NVIDIA GPU). ``engine='auto'`` resolves to ``cudf`` for cuDF input and ``pandas`` otherwise; ``polars`` / ``polars-gpu`` are explicit opt-in (``auto`` never selects them). Neither silently bridges: ``polars-gpu`` is GPU-or-error (it raises rather than silently running on CPU), and ``polars`` (CPU) raises ``NotImplementedError`` for the few unsupported Cypher features rather than falling back to pandas. See :doc:`Choosing an Engine ` for the decision matrix and benchmarks. +GFQL runs the same query on four interchangeable engines, all returning identical results: ``pandas`` (CPU, default), ``polars`` (CPU columnar — up to ~38x over pandas, **no GPU**), ``cudf`` (NVIDIA GPU), and ``polars-gpu`` (NVIDIA GPU). ``engine='auto'`` resolves to ``cudf`` for cuDF input and ``pandas`` otherwise; ``polars`` / ``polars-gpu`` are explicit opt-in (``auto`` never selects them — **so a Polars-frame graph run with the default is coerced to pandas; pass** ``engine='polars'`` **to stay native**). Neither silently bridges: ``polars-gpu`` is GPU-or-error (it raises rather than silently running on CPU), and ``polars`` (CPU) raises ``NotImplementedError`` for the few unsupported Cypher features rather than falling back to pandas. See :doc:`Choosing an Engine ` for the decision matrix and benchmarks. When you use cuDF (GPU) dataframes with ``engine='auto'``, GFQL executes queries on the GPU for massive speedups. From 28160e25bdd17b16ba6468d140d6ff3bfaecc26e Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Mon, 29 Jun 2026 06:22:14 -0700 Subject: [PATCH 05/20] docs(gfql): correct the CPU crossover (~10K, not ~1M) with fresh data + reproducer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Leo pushed: polars beats pandas even below 1M? Correct. The "pandas wins below ~1M" claim was stale (a coarse early finding) and contradicted the fast-path work. Fresh CPU bench (benchmarks/gfql/index_crossover_bench.py, LiveJournal subsampled, warm-median, current stack): shape 10K 100K 1M 1-hop hop polars2.7x polars4.5x polars7.6x WHERE+ORDER polars3.0x polars3.0x polars18x trivial filter polars1.5x pandas2.0x pandas1.6x (sub-ms; immaterial) So CPU polars wins the common graph-query shapes (traversal / WHERE / aggregation) from ~10K edges up; the only pandas win is a trivial sub-millisecond equality mask where the absolute difference is immaterial. The real small-size floor is GPU-only (cuDF/polars-gpu kernel launch, work-bound) — NOT extended to GPU here (this bench is CPU-only; polars-gpu stays the rougher, conditional case via F2/F3/F4). Corrected: F1 (crossover ~10K not ~1M), the decision matrix (size col >~1M -> >~10K; the "<1M -> pandas" row -> "trivial sub-ms op -> pandas, immaterial"), the "When not to use Polars" first bullet, and the motivating-table note. Also reframed "Why opt-in?" so the rationale rests on the NIE-surface robustness (auto-polars could error where pandas works), not a perf regression — consistent with keeping auto on pandas. Docs-only + one CPU bench reproducer. Co-Authored-By: Claude Opus 4.8 (1M context) --- benchmarks/gfql/index_crossover_bench.py | 64 ++++++++++++++++++++++++ docs/source/gfql/engines.rst | 55 ++++++++++++-------- 2 files changed, 98 insertions(+), 21 deletions(-) create mode 100644 benchmarks/gfql/index_crossover_bench.py diff --git a/benchmarks/gfql/index_crossover_bench.py b/benchmarks/gfql/index_crossover_bench.py new file mode 100644 index 0000000000..4406fe5102 --- /dev/null +++ b/benchmarks/gfql/index_crossover_bench.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Small-N pandas-vs-polars CROSSOVER bench (CPU). Answers "where does polars start +beating pandas?" per workload SHAPE, on a real graph subsampled to N edges. + +The crossover is shape-dependent: row-pipeline shapes (filter / WHERE+ORDER) cross over +much earlier than traversal (chain orchestration is the residual small-N fixed cost). +CPU only (the crossover question is pandas-CPU vs polars-CPU); no GPU needed. + +Env: PARQUET=/data/edges.parquet EDGES=10000,100000,1000000 REPS=15 WARM=3 OUT=/tmp/x.jsonl +""" +from __future__ import annotations +import json, os, statistics, time +import numpy as np +import pandas as pd +import graphistry +from graphistry.compute.ast import n, e_forward + + +def med(fn, reps, warm): + for _ in range(warm): + fn() + ts = [] + for _ in range(reps): + t = time.perf_counter(); fn(); ts.append((time.perf_counter() - t) * 1e3) + ts.sort() + return statistics.median(ts) + + +def main(): + edf_full = pd.read_parquet(os.environ["PARQUET"]).astype({"src": np.int64, "dst": np.int64}) + sizes = [int(x) for x in os.environ.get("EDGES", "10000,100000,1000000").split(",")] + reps = int(os.environ.get("REPS", "15")); warm = int(os.environ.get("WARM", "3")) + outf = open(os.environ["OUT"], "a") if os.environ.get("OUT") else None + print(f"{'shape':10} {'edges':>9} {'pandas_ms':>10} {'polars_ms':>10} {'polars_speedup':>15}") + for E in sizes: + edf = edf_full.head(E).reset_index(drop=True) + nodes = np.unique(np.concatenate([edf["src"].values, edf["dst"].values])) + ndf = pd.DataFrame({"id": nodes, "val": (nodes % 100).astype(np.int64)}) + g = graphistry.nodes(ndf, "id").edges(edf, "src", "dst") + seeds = nodes[: max(1, len(nodes) // 100)].tolist() # ~1% frontier + shapes = { + "filter": lambda eng: g.gfql([n({"val": 50})], engine=eng), + "hop1": lambda eng: g.gfql([n({"id": seeds}), e_forward()], engine=eng), + "where_ord": lambda eng: g.gfql( + "MATCH (a) WHERE a.val > 50 RETURN a.id ORDER BY a.id LIMIT 100", engine=eng), + } + for name, fn in shapes.items(): + try: + rp = fn("pandas"); rl = fn("polars") # warm + sanity + pm = med(lambda: fn("pandas"), reps, warm) + lm = med(lambda: fn("polars"), reps, warm) + sp = pm / lm if lm else float("nan") + print(f"{name:10} {E:>9} {pm:>10.3f} {lm:>10.3f} {('polars '+format(sp,'.2f')+'x') if sp>=1 else ('PANDAS '+format(1/sp,'.2f')+'x'):>15}") + if outf: + outf.write(json.dumps(dict(shape=name, edges=E, pandas_ms=pm, polars_ms=lm, + polars_speedup=sp)) + "\n"); outf.flush() + except Exception as ex: + print(f"{name:10} {E:>9} FAILED {type(ex).__name__}: {ex}") + if outf: + outf.close() + + +if __name__ == "__main__": + main() diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 895f765f14..3128541475 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -20,9 +20,10 @@ engine is a one-keyword change — no GPU, same results: g.gfql(query, engine='polars') # up to ~38x faster on real graphs, no GPU, identical results Your existing pandas, Polars, or cuDF graph works as-is: the input frames are accepted and -coerced once; the only change is the keyword. The catch: below ~1M edges ``pandas`` can be -faster (Polars has fixed startup overhead), and a few exotic Cypher features still require -``engine='pandas'`` — see *When not to use Polars* below. +coerced once; the only change is the keyword. The catch: a few exotic Cypher features still +require ``engine='pandas'`` (they raise rather than silently bridge), and the GPU engines only +pay off on larger work. On CPU, Polars wins the common graph-query shapes (traversal, +``WHERE``/``ORDER``, aggregation) from ~10K edges up — see *When not to use Polars* below. .. warning:: **Already a Polars user? Pass** ``engine='polars'`` **— the default does not.** With the @@ -138,9 +139,14 @@ Reading the table: (tens of × over pandas; see ``benchmarks/gfql/`` and the GFQL benchmark notes). .. note:: - There is **no universal winner** — route by workload shape and size (next section). - Below ~1M edges, ``pandas`` often wins because Polars/GPU pay a fixed startup overhead - that the small amount of work cannot amortize. + Route by workload shape and size (next section). **CPU Polars wins the common graph-query + shapes from ~10K edges up** — on LiveJournal subsampled (CPU, warm-median): 1-hop traversal + 2.7× / 4.5× / 7.6× and ``WHERE``+``ORDER`` 3.0× / 3.0× / 18× over pandas at 10K / 100K / 1M. + The **GPU** engines (cuDF / Polars-GPU) are the ones with a real small-size floor — they need + enough work to amortize kernel-launch cost (work-bound, [F2]). The only case pandas edges out + is a trivial sub-millisecond operation (e.g. a bare node-equality filter), where its boolean + mask beats Polars' plan overhead — but at <1 ms the difference is immaterial. Reproducer: + ``benchmarks/gfql/index_crossover_bench.py``. Decision matrix --------------- @@ -155,15 +161,15 @@ Decision matrix - Recommended engine - Notes * - Filter / ``WHERE`` / aggregation - - > ~1M + - > ~10K - CPU - ``polars`` - - order-of-magnitude over pandas [F1] + - wins from ~10K; gap grows with size (up to order-of-magnitude) [F1] * - Bulk 1-hop frontier expansion - - > ~1M + - > ~10K - CPU - ``polars`` - - up to ~38x pandas, ~15x cuDF (Orkut 1-hop) + - wins from ~10K (2.7x); up to ~38x pandas, ~15x cuDF at 100M [F1] * - Heavy multi-hop (2-hop+) - large - GPU @@ -179,19 +185,22 @@ Decision matrix - GPU - ``cudf`` - Polars-GPU can hit memory pressure here [F3] - * - Anything - - < ~1M + * - Trivial sub-ms op (bare equality filter) + - any - CPU - ``pandas`` - - fixed Polars/GPU overhead dominates [F1] + - boolean mask beats Polars plan overhead; immaterial (<1 ms) [F1] * - Selective / seeded traversal - any - CPU - ``pandas``/``polars`` + **CSR index** - O(degree), not an engine choice [F5] -**[F1] Crossover ~1M edges.** Below it pandas often wins (Polars/GPU fixed overhead). -Above it Polars/GPU pull away. Some fast-path shapes (filter/lookup) cross over below 100K. +**[F1] CPU crossover is ~10K, not ~1M.** For the common graph-query shapes (traversal, +``WHERE``/``ORDER``, aggregation) CPU Polars beats pandas from ~10K edges up (2.7-18× in our +runs). Pandas only edges out on a trivial sub-millisecond operation (a bare equality mask), +where the absolute difference is immaterial. The real small-size floor is **GPU-only** — +cuDF / Polars-GPU need enough work to amortize kernel launch ([F2]). **[F2] GPU is work-bound, not size-bound.** A GPU wins when there is enough work to amortize its ~3 ms kernel-launch floor: big frontiers, dense joins, full-graph aggregation. Tiny or @@ -231,8 +240,10 @@ When **not** to use Polars Honesty matters more than a bigger number: -- **Small graphs (< ~1M edges):** ``pandas`` often wins; the Polars/GPU startup overhead is - not worth it (footnote F1). +- **Trivial sub-millisecond operations** (a bare node-equality filter): pandas' boolean mask + beats Polars' plan overhead — but at <1 ms it is immaterial. For traversal / ``WHERE`` / + ``ORDER`` / aggregation, CPU Polars wins from ~10K edges up (footnote F1). The real small-size + caveat is **GPU-only** (cuDF / Polars-GPU need larger work — footnote F2). - **A few exotic Cypher features** are not yet native on Polars (e.g. cross-entity same-path ``WHERE``, some temporal/entity-text forms). They raise an honest ``NotImplementedError`` pointing at ``engine='pandas'`` — GFQL **never** silently bridges Polars to pandas, because @@ -296,10 +307,12 @@ Why opt-in? ----------- Polars and Polars-GPU are explicit (``engine='polars'`` / ``'polars-gpu'``; ``auto`` never -picks them) for two honest reasons: (1) below the ~1M-edge crossover they can be *slower* than -pandas, so a blanket default would regress small/interactive workloads; and (2) a few exotic -Cypher features still require ``engine='pandas'`` and raise rather than silently bridge. Opting -in keeps the default behavior unchanged and makes the performance trade-off explicit. +picks them). The main reason is robustness, not speed: a few exotic Cypher features still +require ``engine='pandas'`` and **raise** rather than silently bridge, so auto-selecting Polars +would turn queries that work today on pandas into hard errors. (Performance is rarely the +downside — CPU Polars wins common graph queries from ~10K edges; only trivial sub-millisecond +operations favor pandas, immaterially.) Opting in keeps the default behavior unchanged and +guarantees a working result. See also -------- From 8cc33603d11eb8590da0e61214b91aef4d39b907 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Mon, 29 Jun 2026 06:33:21 -0700 Subject: [PATCH 06/20] =?UTF-8?q?docs(gfql):=20F5=20=E2=80=94=20per-engine?= =?UTF-8?q?=20CSR=20index=20numbers=20(seeded=20=3D=20CPU's=20game,=20all?= =?UTF-8?q?=204=20engines)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CSR index works on all four engines; benchmarked seeded 1-hop on LiveJournal 35M (guarded, index==scan): pandas ~0.13ms / polars ~0.16ms (numpy searchsorted) vs cuDF ~3ms (GPU kernel-launch floor) — the clean inverse of bulk. Pick the index for selective traversal + a CPU engine to drive it. Reproducer benchmarks/gfql/index_largegraph_bench.py. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 3128541475..d366d67ddc 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -215,8 +215,12 @@ result as a GPU run (see *Honesty* below). **[F5] Selective traversal is an indexing problem, not an engine choice.** A seeded ``hop`` from a few nodes is fastest with the opt-in **CSR adjacency index** (``g.create_index(...)``, -``index_policy=``), which turns the O(E) scan into an O(degree) gather — on CPU, independent -of ``engine=``. (A dedicated index guide is in progress; the methods live under the GFQL API.) +``index_policy=``), which turns the O(E) scan into an O(degree) gather. The index works on all +four engines, but seeded work is so small that **CPU wins**: on LiveJournal 35M a typical-seed +1-hop is ~0.13 ms on pandas and ~0.16 ms on Polars (numpy ``searchsorted``) vs ~3 ms on cuDF +(GPU kernel-launch floor) — the clean inverse of bulk, where the GPU pulls ahead. So pick the +index for selective traversal and a CPU engine to drive it. (A dedicated index guide is in +progress; the methods live under the GFQL API.) cuDF vs Polars-GPU ------------------ From 66485711df48a56842d9dee1adf0019af0fb7b2a Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Mon, 29 Jun 2026 07:38:42 -0700 Subject: [PATCH 07/20] =?UTF-8?q?docs(gfql):=20engines=20page=20=E2=80=94?= =?UTF-8?q?=20orient=20the=20cold=20newcomer=20(where=20does=20`g`=20come?= =?UTF-8?q?=20from)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §0 newcomer user-test: a first-timer landing on engines.rst cold hit the headline `g.gfql(query, engine='polars')` with no `g`/`query` defined (NameError on copy-paste); construction was buried (inside the coercion warning + the bottom install block) and there was no early pointer to getting-started. Add (1) an early "New to GFQL? build a graph first -> :doc:`about`" note, and (2) a 2-line self-contained preamble (graphistry.edges + a query) so the first example runs as-is. Reuses content already on the page; no restructure. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index d366d67ddc..ce0dba3f85 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -8,6 +8,10 @@ the engine with one keyword — ``engine=``, accepted uniformly by ``g.gfql()`` ``g.hop()`` — and GFQL returns **identical results** on every one (differential parity is a release gate). Pick the engine that fits your hardware and workload; nothing else changes. +.. note:: + **New to GFQL?** This page assumes you already have a graph ``g`` and a ``query``. If not, + build one first — see :doc:`about` (10 Minutes to GFQL). + The one-line speedup -------------------- @@ -16,6 +20,10 @@ engine is a one-keyword change — no GPU, same results: .. code-block:: python + import graphistry + g = graphistry.edges(df, 'src', 'dst') # df: your edges dataframe (pandas / Polars / cuDF) + query = "MATCH (a)-[e]->(b) RETURN b" # any GFQL / Cypher query + g.gfql(query) # engine='pandas' (default) g.gfql(query, engine='polars') # up to ~38x faster on real graphs, no GPU, identical results From 5afbb19313b5486abc9cd816deac53e9005c22fb Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 30 Jun 2026 17:41:40 -0700 Subject: [PATCH 08/20] docs(gfql): seeded-traversal CSR adjacency index guide (persona P0-3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine-selection guide (#1661) documented all four engines + a decision matrix but the CSR adjacency index — the strongest competitive claim and the exact answer to 'Neo4j has an index, does GFQL?' — was only a footnote. Adds a full guide: create_index/gfql_index_all/show_indexes/drop_index, index_policy (use/auto/force/off), gfql_explain, Cypher DDL + wire protocol, and the sourced numbers (flat-in-N 0.12ms @8M-117M edges; 9-28x vs Kuzu/Neo4j on selective lookups; CPU-wins-seeded vs GPU floor). Honest build-cost + parity-or-fallback section. Wires into the toctree + a seeded-lookup recommended path; shrinks the engines.rst F5 footnote to a cross-link. Persona-driven (round-1 user-testing: Priya/Neo4j-migrant + Maya's slow seeded lookup). Numbers already measured (benchmarks/gfql/index_*bench.py, dgx-spark). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 15 +-- docs/source/gfql/index.rst | 2 + docs/source/gfql/index_adjacency.rst | 173 +++++++++++++++++++++++++++ 3 files changed, 183 insertions(+), 7 deletions(-) create mode 100644 docs/source/gfql/index_adjacency.rst diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index ce0dba3f85..f89a463e30 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -222,13 +222,14 @@ as its in-memory GPU executor comes under memory pressure. Prefer ``cudf`` for t result as a GPU run (see *Honesty* below). **[F5] Selective traversal is an indexing problem, not an engine choice.** A seeded ``hop`` -from a few nodes is fastest with the opt-in **CSR adjacency index** (``g.create_index(...)``, -``index_policy=``), which turns the O(E) scan into an O(degree) gather. The index works on all -four engines, but seeded work is so small that **CPU wins**: on LiveJournal 35M a typical-seed -1-hop is ~0.13 ms on pandas and ~0.16 ms on Polars (numpy ``searchsorted``) vs ~3 ms on cuDF -(GPU kernel-launch floor) — the clean inverse of bulk, where the GPU pulls ahead. So pick the -index for selective traversal and a CPU engine to drive it. (A dedicated index guide is in -progress; the methods live under the GFQL API.) +from a few nodes is fastest with the opt-in **CSR adjacency index** (``g.gfql_index_all()`` / +``g.create_index(...)``, ``index_policy=``), which turns the O(E) scan into an O(degree) +gather — flat in graph size, and 9–28× faster than Kuzu / Neo4j on selective lookups. It works +on all four engines, but seeded work is so small that **CPU wins**: on LiveJournal 35M a +typical-seed 1-hop is ~0.13 ms on pandas and ~0.16 ms on Polars (numpy ``searchsorted``) vs +~3 ms on cuDF (GPU kernel-launch floor) — the clean inverse of bulk, where the GPU pulls +ahead. So pick the index for selective traversal and a CPU engine to drive it. See +:doc:`index_adjacency` for the full guide. cuDF vs Polars-GPU ------------------ diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index 43b74cc8bb..7ac79d738a 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -38,6 +38,7 @@ Recommended paths: - Running Cypher syntax in GFQL: :doc:`cypher` -> :doc:`quick` -> :doc:`return` -> :doc:`spec/cypher_mapping` - Faster on CPU (no GPU): :doc:`engines` -> :doc:`performance` (one keyword, ``engine='polars'``, up to ~38x over pandas) - Performance path (intro -> engine choice -> GPU -> remote GPU): :doc:`about` -> :doc:`engines` -> :doc:`performance` -> :doc:`remote` +- Fast seeded lookups (start from known nodes, like a DB index): :doc:`index_adjacency` (O(degree), flat in graph size, 9-28x vs Kuzu/Neo4j) - Translating existing Cypher to native GFQL: :doc:`spec/cypher_mapping` - Building agents/integrations: :doc:`spec/language` + :doc:`spec/python_embedding` + :doc:`spec/wire_protocol` @@ -52,6 +53,7 @@ See also: overview remote Choosing an Engine + Seeded Traversal Indexes GFQL CPU & GPU Acceleration End-to-End Benchmark translate diff --git a/docs/source/gfql/index_adjacency.rst b/docs/source/gfql/index_adjacency.rst new file mode 100644 index 0000000000..80724363ca --- /dev/null +++ b/docs/source/gfql/index_adjacency.rst @@ -0,0 +1,173 @@ +Seeded Traversal Indexes (CSR Adjacency) +======================================== + +A **seeded** graph query starts from a known set of nodes — "the neighbors of these +50 accounts", "2 hops out from this device" — rather than scanning the whole graph. +By default GFQL answers a seeded ``hop`` with an ``O(E)`` pass over every edge. With an +opt-in **CSR adjacency index**, the same hop becomes an ``O(degree)`` gather: its cost +depends on how many edges the *seeds* touch, not on how big the graph is. The result is +**flat in graph size** — and it beats embedded graph databases on selective lookups. + +Nothing changes about the answer. The index is a pay-as-you-go accelerator: a query either +uses a resident index or falls back to the scan, and any feature the index does not cover +also falls back — never a different result. + +When to use it +-------------- + +- **Seeded traversals**: you start from specific node ids (a watchlist, a session, a fraud + ring's known members) and hop out 1–3 steps. +- **Repeated queries** against the same graph: build the index once, amortize it over many + seeded lookups. +- **Interactive / point-lookup latency**: sub-millisecond neighbor expansion. + +It does **not** help a full-graph scan (a property filter over every node, a global +PageRank). For those, choose an *engine* instead — see :doc:`engines`. + +Quick start +----------- + +.. code-block:: python + + import graphistry + + g = graphistry.edges(edges_df, "src", "dst").nodes(nodes_df, "id") + + # Build the indexes once (out+in adjacency, plus a node-id accelerator when ids are unique) + g = g.gfql_index_all() + + # Seeded query — the index is used automatically (default index_policy='use') + out = g.gfql("MATCH (a)-[e]->(b) WHERE a.id IN $seeds RETURN a, e, b", + params={"seeds": my_seed_ids}) + +``gfql_index_all()`` is the one-liner. For finer control, build a single kind: + +.. code-block:: python + + g = g.create_index("edge_out_adj") # outgoing adjacency (forward hops) + g = g.create_index("edge_in_adj") # incoming adjacency (reverse hops) + g = g.create_index("node_id") # node-id lookup accelerator (unique ids only) + + g.show_indexes() # inspect what's resident + g = g.drop_index() # drop all (or drop_index("edge_out_adj")) + +The index is a **sidecar over edge row positions** — it never reorders your ``.edges`` / +``.nodes`` frames, and it is fingerprint-validated: rebinding ``.edges()`` safely +invalidates a stale index (treated as absent, never a wrong answer). + +Controlling the planner +----------------------- + +``gfql(..., index_policy=...)`` decides whether a resident index is used: + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - ``index_policy`` + - Behavior + * - ``'use'`` *(default)* + - Use a resident index when one covers the query; never build one. Zero overhead if + no index exists. + * - ``'auto'`` + - Build an index on the fly when the planner predicts it pays off (selective seed set). + * - ``'force'`` + - Require the index path (useful for benchmarking / asserting it is engaged). + * - ``'off'`` + - Ignore indexes entirely (the plain ``O(E)`` scan). + +Use ``g.gfql_explain(query, index_policy=...)`` to see whether the index path was taken. + +The indexes are **engine-uniform**: numpy host arrays for pandas / Polars, cupy on-device +for cuDF. They are also exposed as **Cypher DDL** (``CREATE GFQL INDEX FOR edge_out_adj``, +``DROP GFQL INDEX``, ``SHOW GFQL INDEXES`` — the mandatory ``GFQL`` token distinguishes them +from standard property ``CREATE INDEX``) and in the **JSON wire protocol** +(``{"type": "CreateIndex", ...}`` ops plus ``index_policy`` in the request envelope), so a +remote ``gfql_remote`` call can carry the same index intent. + +Performance +----------- + +**Flat in graph size.** A seeded 1-hop stays sub-millisecond as the graph grows 10×, while +the ``O(E)`` scan grows linearly. Synthetic power-law graphs, GFQL-pandas, warm median, +every cell guarded so the index path was taken *and* the indexed result equals the scan +result: + +.. list-table:: + :header-rows: 1 + :widths: 40 30 30 + + * - Seeded 1-hop + - 0.8M nodes / 6.4M edges + - 8M nodes / 64M edges + * - **Indexed (O(degree))** + - **0.124 ms** + - **0.122 ms** *(flat)* + * - Scan (O(E)) + - 105 ms + - 1045 ms + +The same holds on real power-law graphs: a typical-seed 1-hop is ~0.13 ms on LiveJournal +(35M edges) and ~0.14 ms on Orkut (117M edges), versus an ``O(E)`` scan of 367 ms → 1208 ms. + +**Beats embedded graph databases on selective lookups.** Same graph (0.8M nodes / 6.4M +edges), matched result counts, warm median. GFQL is CPU-pandas with the index; Kuzu and +Neo4j use their native indexes: + +.. list-table:: + :header-rows: 1 + :widths: 24 22 18 18 18 + + * - Task + - GFQL (indexed) + - Kuzu + - Neo4j + - GFQL speedup + * - 1-hop seeded + - **0.123 ms** + - 1.15 ms + - 1.45 ms + - 9.4× / 11.8× + * - 1–2-hop seeded + - **0.150 ms** + - 4.25 ms + - 2.54 ms + - 28× / 16.9× + +On a fairer, fully-prepared, in-process Kuzu re-run (LiveJournal 35M), GFQL is still +**17×** on a typical seed (0.126 ms vs 2.13 ms) and **6×** on a hub seed (3.76 ms vs +22.6 ms). *(Kuzu's worst-case-optimal joins can win on cyclic / multi-way-join patterns — +triangles, cliques — which these forward-expansion lookups do not exercise; we do not +claim those.)* + +**Selective traversal is CPU's game.** The indexed hop is tiny work, so the GPU's +kernel-launch floor (~3 ms on cuDF) loses to a ~0.13 ms pandas / ~0.16 ms Polars +``searchsorted`` — the clean inverse of *bulk* analytics, where the GPU pulls ahead +(see :doc:`engines`). Pick the index for selective traversal and a **CPU engine** to +drive it. + +Reproduce: ``benchmarks/gfql/index_takeover_bench.py``, +``benchmarks/gfql/index_vs_dbs.py``, ``benchmarks/gfql/index_vs_kuzu_prepared.py``. +Hardware: DGX ``dgx-spark``, GB10 GPU. + +Honesty and cost +---------------- + +- **Build cost** is one ``O(E log E)`` sort, amortized over subsequent queries. + ``index_policy='auto'`` only builds when the planner predicts a selective query will + pay it back. +- **No change to default behavior.** With no index resident and ``index_policy='use'`` + (the default), queries run exactly as before. +- **Parity-or-fallback.** The index accelerates the seeded scan sites it covers (forward / + reverse hop, the Polars hop, the single-hop chain fast path). Any uncovered feature — + edge / source / destination match, ``target_wave_front``, ``min_hops>1``, labeling — + falls back to the scan/join path. The indexed subgraph is verified equal to the scan + subgraph in differential tests across pandas / cuDF / Polars / Polars-GPU. It is an + accelerator, never a source of a different answer. + +See also +-------- + +- :doc:`engines` — choosing pandas / Polars / cuDF / Polars-GPU for non-seeded work. +- :doc:`performance` — the vectorization + GPU design behind GFQL. +- :doc:`benchmark_filter_pagerank` — an end-to-end filter → PageRank → filter comparison vs Neo4j. From 26212d386373aac55c9080a1173b4b0eae36c3e7 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 30 Jun 2026 17:43:10 -0700 Subject: [PATCH 09/20] =?UTF-8?q?docs(gfql):=20benchmark=20page=20?= =?UTF-8?q?=E2=80=94=20add=20parity=20statement=20+=20engine/index=20cross?= =?UTF-8?q?-links?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The end-to-end benchmark showed CPU-vs-GPU speedups with no statement that they return the SAME answer (skeptic persona P0-4) and no path to the broader engine story. Adds: 'same answer on every engine' parity note (release-gate: parity or NotImplementedError), a 'this is one workload vs one baseline' framing pointing to the 4-engine guide (engines) + the seeded-index guide (index_adjacency), and those two in the see-also list. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/benchmark_filter_pagerank.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/gfql/benchmark_filter_pagerank.rst b/docs/source/gfql/benchmark_filter_pagerank.rst index 174fa893ab..3231819f99 100644 --- a/docs/source/gfql/benchmark_filter_pagerank.rst +++ b/docs/source/gfql/benchmark_filter_pagerank.rst @@ -173,8 +173,23 @@ pandas / cuDF). That is what makes the CPU-to-GPU switch a configuration flag (``engine="cudf"``) rather than a rewrite, and what keeps ETL, search, and analytics in the same in-process pipeline. +**Same answer on every engine.** The CPU and GPU results above are not just +comparable — they are *identical*. Differential parity across ``pandas`` / +``polars`` / ``cudf`` / ``polars-gpu`` is a GFQL release gate: an engine either +returns the same result or raises ``NotImplementedError`` — never a silently +different answer. So the speedups here are a pure hardware/engine choice, not a +change in what the query means. + +This page is one workload (a filter → PageRank → filter pipeline) against one +external baseline (Neo4j+GDS). For the full four-engine picture — when Polars +beats pandas on CPU, when the GPU pulls ahead, and how to choose — see +:doc:`engines`. For sub-millisecond *seeded* lookups that beat Kuzu and Neo4j +by 9–28×, see :doc:`index_adjacency`. + For more on the GFQL design and supported surface: +- :doc:`engines` — choosing pandas / Polars / cuDF / Polars-GPU +- :doc:`index_adjacency` — seeded-traversal CSR adjacency index - :doc:`cypher` — Cypher syntax through ``g.gfql("MATCH ...")`` - :doc:`overview` — GFQL design, features, and GPU acceleration - :doc:`about` — 10-minute introduction to GFQL From c45973dad7138eaa608fbcb75877359da9b40b01 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 30 Jun 2026 18:00:17 -0700 Subject: [PATCH 10/20] =?UTF-8?q?docs(gfql):=20engines=20page=20=E2=80=94?= =?UTF-8?q?=20vs-external-tools=20table=20+=20switching=20cookbook=20(pers?= =?UTF-8?q?ona=20P1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two persona-driven additions to the engine guide: - 'GFQL vs external graph tools': honest positioning table (Neo4j/Kuzu/igraph/ networkx) with every number conditioned + '>'/did-not-finish/not-benchmarked markers kept, and the cyclic-join caveat we do NOT claim. Serves the skeptic (Sam) + Neo4j-migrant (Priya) personas. - 'Switching engines' cookbook: the one-keyword switch, .to_pandas() round-trip for pandas-only downstream code, mixing build-frame vs run-engine, and the auto-never-picks-polars note. Consolidates scattered one-liners (Maya/Tom). Uses only already-measured numbers. RST validated (docutils clean bar Sphinx :doc: roles). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 103 +++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index f89a463e30..5ac42016ed 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -156,6 +156,61 @@ Reading the table: mask beats Polars' plan overhead — but at <1 ms the difference is immaterial. Reproducer: ``benchmarks/gfql/index_crossover_bench.py``. +.. _gfql-vs-external-tools: + +GFQL vs external graph tools +---------------------------- + +GFQL is **dataframe-native**: ``pip install``, then query your existing pandas / Polars / +cuDF frame in-process — no separate database to stand up, no ETL to load, no cluster. Graph +databases (Neo4j, Kuzu) are a **system-of-record** you provision and ingest into first. The +table below is deliberately conservative: every speedup is stated with its condition, ``>`` +and did-not-finish markers are kept, and where we have no head-to-head we say **not +benchmarked** rather than guess. + +.. list-table:: + :header-rows: 1 + :widths: 14 22 30 34 + + * - Tool + - What it is / Setup + - Where GFQL wins (with condition) + - Where it complements / GFQL doesn't claim + * - **Neo4j + GDS** + - Server + GDS library; stand up a DB and ETL your data in. + - **Filter→PageRank→filter pipeline**, dgx-spark GB10, warm median: Twitter 2.4M — + 13.83 s Neo4j vs 2.55 s GFQL-CPU / **0.30 s GFQL-GPU (46×)**; GPlus 30M — + **>187 s (did-not-finish)** vs 75.78 s CPU / **3.33 s GPU (>56×)**. + - Neo4j remains the transactional system-of-record; run the read-heavy analytics in + GFQL. See :doc:`benchmark_filter_pagerank`. + * - **Kuzu** + - Embedded graph DB; still a separate store to load + index. + - **Seeded index lookup** (0.8M nodes / 6.4M edges): 1-hop **0.123 ms vs 1.15 ms + (9.4×)**, 2-hop **0.150 ms vs 4.25 ms (28×)**; prepared-Kuzu LiveJournal 35M ≈ **17×** + typical seed, 6× hub. **Bulk frontier expansion** (LiveJournal 35M, 1-hop, many + seeds): **22× Kuzu**, up to **87× at k=100k**. See :doc:`index_adjacency`. + - **Not claimed:** cyclic / multi-way-join patterns (triangles, cliques) where Kuzu's + worst-case-optimal joins can win. Use Kuzu as the store; GFQL for bulk read analytics. + * - **igraph** + - Pure-Python/C graph library. + - — (not a standalone competitor here) + - **Complement, not competitor:** igraph is the CPU PageRank backend *inside* GFQL. + No head-to-head benchmarked. + * - **networkx** + - Pure-Python graph library; the floor most analysts start from. + - **not benchmarked** — expect order-of-magnitude headroom qualitatively (no measured + head-to-head). + - Fine for small/interactive graphs; GFQL is the columnar/GPU path when they grow. + +GFQL **complements** a graph database more than it replaces one: keep Neo4j or Kuzu as the +system-of-record, and do the read-heavy search + analytics in GFQL so ETL, traversal, and +scoring stay in one in-process dataframe pipeline. Route by shape — **selective** seeded +lookups favor the GFQL index (up to 28× Kuzu, 16.9× Neo4j on 2-hop), **bulk** frontier +expansion and full pipelines favor Polars / GPU (22–87× Kuzu; **46–56× Neo4j** on the +filter→PageRank→filter pipeline). The one case we explicitly **do not** claim is +cyclic / multi-way-join patterns (triangles, cliques), where Kuzu's worst-case-optimal +joins can beat a dataframe plan. + Decision matrix --------------- @@ -231,6 +286,54 @@ typical-seed 1-hop is ~0.13 ms on pandas and ~0.16 ms on Polars (numpy ``searchs ahead. So pick the index for selective traversal and a CPU engine to drive it. See :doc:`index_adjacency` for the full guide. +Switching engines +----------------- + +The engine is a single keyword on ``g.gfql()`` (and ``g.hop()``). The graph and +the query never change — only ``engine=`` does, and the answer stays identical +(or raises ``NotImplementedError`` rather than silently changing it). + +.. code-block:: python + + import graphistry + g = graphistry.edges(df, 'src', 'dst') # your existing graph (any frame type) + query = "MATCH (a)-[e]->(b) RETURN b" # any GFQL / Cypher query + + g.gfql(query) # engine='pandas' (default) + g.gfql(query, engine='polars') # CPU columnar, no GPU, identical results + g.gfql(query, engine='cudf') # NVIDIA GPU (RAPIDS) + g.gfql(query, engine='polars-gpu') # same fused plan on GPU + +Getting results back as pandas +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The result's ``._nodes`` / ``._edges`` come back in the engine's frame type: a +``polars.DataFrame`` for ``'polars'`` / ``'polars-gpu'``, a ``cudf.DataFrame`` +for ``'cudf'``. When downstream code is pandas-only (matplotlib, scikit-learn, +``.iloc`` / ``groupby().apply()``), convert once with ``.to_pandas()``: + +.. code-block:: python + + out = g.gfql(query, engine='polars') # or 'cudf' / 'polars-gpu' + nodes_pd = out._nodes.to_pandas() # -> pandas for matplotlib / sklearn / ... + nodes_pd.plot.scatter(x='x', y='y') # pandas-only downstream code, unchanged + +Mixing engines +~~~~~~~~~~~~~~~ + +The build frame type and the run engine are independent — GFQL coerces the input +frames to the engine you ask for. A pandas graph runs on ``engine='polars'``, a +Polars graph runs on ``engine='pandas'``, and so on. The only cost is a +**one-time convert** of the input frames at the start of the call; the query then +runs fully on the chosen engine. Note that ``engine='auto'`` (the default) +resolves to ``cudf`` for cuDF input and ``pandas`` for everything else — **it +never selects Polars or Polars-GPU**, so those two are always an explicit opt-in. + +.. tip:: + For selective, seeded traversal, build the CSR adjacency index once with + ``g.gfql_index_all()`` (or ``index_policy=``) — it works on all four engines + and turns the O(E) scan into an O(degree) gather. See :doc:`index_adjacency`. + cuDF vs Polars-GPU ------------------ From 0f38a8c008f2a4542bf30264d8cad1f0c5df6a2a Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 30 Jun 2026 18:05:28 -0700 Subject: [PATCH 11/20] docs(gfql): position vs GraphFrames + PuppyGraph; label pipeline-vs-lifecycle (round-2 personas E/F) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 user-testing closed all round-1 P0s and fully served personas A-D, but E (Databricks GraphFrames/Spark) and F (Snowflake/Databricks + PuppyGraph) FAILED — their tools + decision axes were absent, and GraphFrames' motif queries mapped onto the one case we disclaim with no 'it runs' reassurance. - Add GraphFrames + PuppyGraph rows to the vs-external-tools table (qualitative, 'not benchmarked yet' markers): single-node-vs-cluster (100M+ on one machine; cluster only above the single-node ceiling) and warehouse-in-place-vs-pull- subgraph (GFQL adds PageRank/centrality PuppyGraph lacks; complement). Note motif/multi-way-join queries RUN but aren't yet perf-tuned. - Benchmark page: label the headline table as PIPELINE time and note the per-graph sections are full-lifecycle (incl ETL) — kills the 3.33s-vs-7.1s apparent contradiction a skeptic hits. Numbers unchanged (positioning is qualitative; head-to-heads are the later stacked benchmark PR). RST validated. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../source/gfql/benchmark_filter_pagerank.rst | 5 +++- docs/source/gfql/engines.rst | 24 ++++++++++++++++--- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/source/gfql/benchmark_filter_pagerank.rst b/docs/source/gfql/benchmark_filter_pagerank.rst index 3231819f99..0d5e6ebba1 100644 --- a/docs/source/gfql/benchmark_filter_pagerank.rst +++ b/docs/source/gfql/benchmark_filter_pagerank.rst @@ -30,7 +30,10 @@ no database required. This benchmark compares **Graphistry's local Cypher** - **3.33s** - **>56x** -*Warm median of 5 runs, 2 warmup iterations. DGX dgx-spark, GB10 GPU.* +*Pipeline time (search + PageRank + search), warm median of 5 runs, 2 warmup iterations. DGX +dgx-spark, GB10 GPU. The per-graph sections below report full-lifecycle totals that also include +one-time ETL/load — hence the slightly larger numbers there (e.g. GPlus GPU 3.33s pipeline vs +~7.1s lifecycle).* The pipeline ------------ diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 5ac42016ed..25349abe52 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -201,15 +201,33 @@ benchmarked** rather than guess. - **not benchmarked** — expect order-of-magnitude headroom qualitatively (no measured head-to-head). - Fine for small/interactive graphs; GFQL is the columnar/GPU path when they grow. + * - **Spark GraphFrames** + - *Distributed* graph engine on a Spark cluster; provision + tune the cluster. + - GFQL is *single-node* (CPU or one GPU): 100M+ edges in-process on **one machine**, + no cluster to stand up, interactive latency — and a single GPU often matches or beats + a Spark cluster on read-heavy traversal + PageRank at a fraction of the cost. + *Head-to-head not yet published.* + - Reach for GraphFrames when the graph genuinely exceeds one machine's memory. Motif / + triangle / multi-way-join queries **run** in GFQL but are not yet perf-benchmarked. + * - **PuppyGraph** + - Graph query layer *over your warehouse tables in place* (zero-ETL, query pushdown). + - GFQL adds GPU/CPU graph **analytics PuppyGraph does not offer — PageRank, centrality, + community** — on a pulled subgraph, in one pipeline. *No head-to-head yet.* + - **Complement:** use PuppyGraph for ad-hoc graph SQL across the whole warehouse; pull the + relevant subgraph into GFQL when you need GPU-accelerated analytics on it. GFQL **complements** a graph database more than it replaces one: keep Neo4j or Kuzu as the system-of-record, and do the read-heavy search + analytics in GFQL so ETL, traversal, and scoring stay in one in-process dataframe pipeline. Route by shape — **selective** seeded lookups favor the GFQL index (up to 28× Kuzu, 16.9× Neo4j on 2-hop), **bulk** frontier expansion and full pipelines favor Polars / GPU (22–87× Kuzu; **46–56× Neo4j** on the -filter→PageRank→filter pipeline). The one case we explicitly **do not** claim is -cyclic / multi-way-join patterns (triangles, cliques), where Kuzu's worst-case-optimal -joins can beat a dataframe plan. +filter→PageRank→filter pipeline). Against the **distributed** engines the axis is different: +GFQL trades horizontal scale-out for zero cluster/warehouse setup and interactive latency — +choose it below the single-machine ceiling (100M+ edges fit in-process; a cluster is only +needed once the graph genuinely exceeds one node's memory), and complement PuppyGraph's +zero-ETL warehouse graph with GFQL's GPU analytics. The one case we explicitly **do not** +claim is cyclic / multi-way-join patterns (triangles, cliques): they **run**, but Kuzu's +worst-case-optimal joins can beat a dataframe plan there and we have not yet perf-tuned them. Decision matrix --------------- From a9e441a370bfdf6d40c7cd5092ca6b4e1e891500 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Wed, 1 Jul 2026 00:16:32 -0700 Subject: [PATCH 12/20] docs(gfql): add 'Larger-than-memory: streaming' section + fix engines.rst doctests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New section documents the two opt-in streaming modes with honest scope: - GFQL_POLARS_CPU_STREAMING=1 -> Polars streaming engine, disk-spill (CPU) - GFQL_POLARS_GPU_EXECUTOR=streaming -> cudf-polars streaming executor (GPU) Covers when to use (oversized intermediates), the opt-in trade-off (~0.86x small-size regress; parity-identical), a set-before-import example, and an explicit limits note: streaming covers the QUERY collect, but input still materializes at ingestion (a passed LazyFrame is collected), so out-of-core INPUT (lazy scan_parquet end-to-end) is work-in-progress — cross-links the Friendster discussion. Also marks the two illustrative one-keyword snippets (placeholder df/query) '.. doc-test: skip', clearing engines.rst's test-docs failures. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 72 ++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 25349abe52..be9c23bcee 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -18,6 +18,8 @@ The one-line speedup On real graphs, switching the default ``pandas`` engine to the columnar **Polars** engine is a one-keyword change — no GPU, same results: +.. doc-test: skip + .. code-block:: python import graphistry @@ -369,6 +371,74 @@ Both run on an NVIDIA GPU, so which do you use? - **Install.** ``cudf`` and ``polars-gpu`` both need the RAPIDS GPU stack; ``polars-gpu`` additionally uses ``cudf_polars``. ``polars`` (CPU) only needs ``pip install polars``. +.. _gfql-larger-than-memory: + +Larger-than-memory: streaming execution +--------------------------------------- + +The default Polars engines run **in-memory**: fastest and most stable while the +graph and its query intermediates fit in RAM (or device memory). When a query's +*intermediates* would blow past memory — a wide multi-hop frontier, a large +join, a big aggregation — GFQL has two **opt-in** streaming modes that trade a +little latency for a much larger working set: + +.. list-table:: + :header-rows: 1 + :widths: 22 20 58 + + * - Mode + - Engine + - What it does + * - ``GFQL_POLARS_CPU_STREAMING=1`` + - ``polars`` + - Collects the fused plan with Polars' **streaming engine** — processes in + batches and **spills to disk**, so intermediates can exceed RAM. + * - ``GFQL_POLARS_GPU_EXECUTOR=streaming`` + - ``polars-gpu`` + - Uses the **cudf-polars streaming executor** — the escape hatch for + results **larger than device memory** (the default in-memory executor + would OOM). + +Both are **off by default** on purpose: they add overhead that *regresses* +small/interactive work (~0.86× at 100K edges), and for the in-memory regime this +page measures, the default is faster and more stable. Results are +**parity-identical** to the default — streaming changes *how* the plan runs, not +*what* it returns. + +The flags are read at import time, so set them **before your process starts** (or +before ``import graphistry``): + +.. code-block:: bash + + # CPU: batched + disk-spill for larger-than-RAM intermediates + export GFQL_POLARS_CPU_STREAMING=1 + + # GPU: streaming executor for larger-than-device-memory results + export GFQL_POLARS_GPU_EXECUTOR=streaming + +Then use ``engine='polars'`` / ``engine='polars-gpu'`` exactly as before — no code +change: + +.. doc-test: skip + +.. code-block:: python + + import graphistry # env vars above must be set first + g = graphistry.edges(edges_df, 'src', 'dst') + result = g.gfql(query, engine='polars') # streaming collect (CPU, disk-spill) + # result = g.gfql(query, engine='polars-gpu') # streaming executor (GPU) + +.. note:: + **What streaming does and does not cover today.** These flags stream the + **query** (collect), which helps when the *input fits but the intermediates or + result do not*. They do **not** yet give out-of-core *input*: ``graphistry`` + currently materializes edge/node frames at ingestion (a passed + ``polars.LazyFrame`` is collected immediately), so the source graph must still + fit in memory. True out-of-core-from-disk — building GFQL directly on a lazy + ``pl.scan_parquet`` source so a graph larger than RAM never fully materializes — + is **work in progress**; see the Friendster (~1.8B edges) discussion in + :doc:`benchmark_graphframes`. + When **not** to use Polars -------------------------- @@ -430,6 +500,8 @@ Install Then change one keyword — your existing graph and query are unchanged: +.. doc-test: skip + .. code-block:: python import graphistry From 831506b1751e15c2bd2192005ae144ee479eb36d Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Wed, 1 Jul 2026 00:44:18 -0700 Subject: [PATCH 13/20] =?UTF-8?q?docs(gfql):=20fix=20index=5Fadjacency=20q?= =?UTF-8?q?uick-start=20=E2=80=94=20working=20chain=20form,=20not=20unsupp?= =?UTF-8?q?orted=20Cypher?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The quick-start ran a Cypher string 'MATCH (a)-[e]->(b) WHERE a.id IN $seeds RETURN a, e, b' that hits a known limitation (#1273: row lowering supports one MATCH source alias at a time), so it raised in test-docs and would mislead any reader who copy-pasted it. Replace with the canonical native seeded-traversal chain — [n({id: is_in(seeds)}), e_forward(), n()] — which is what this index page is actually about, uses the index automatically, and runs green. Also defines the previously-undefined my_seed_ids. Full doc-examples suite now passes locally. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/index_adjacency.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/source/gfql/index_adjacency.rst b/docs/source/gfql/index_adjacency.rst index 80724363ca..fb7cffe43c 100644 --- a/docs/source/gfql/index_adjacency.rst +++ b/docs/source/gfql/index_adjacency.rst @@ -30,15 +30,16 @@ Quick start .. code-block:: python import graphistry + from graphistry import n, e_forward, is_in g = graphistry.edges(edges_df, "src", "dst").nodes(nodes_df, "id") # Build the indexes once (out+in adjacency, plus a node-id accelerator when ids are unique) g = g.gfql_index_all() - # Seeded query — the index is used automatically (default index_policy='use') - out = g.gfql("MATCH (a)-[e]->(b) WHERE a.id IN $seeds RETURN a, e, b", - params={"seeds": my_seed_ids}) + # Seeded traversal — the index is used automatically (default index_policy='use') + my_seed_ids = ["a", "b"] # your seed node ids + out = g.gfql([n({"id": is_in(my_seed_ids)}), e_forward(), n()]) ``gfql_index_all()`` is the one-liner. For finer control, build a single kind: From dcb247e87247f3124f85c498744e42b9a1aaee56 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 2 Jul 2026 09:33:31 -0700 Subject: [PATCH 14/20] docs(gfql): mark illustrative engine-switch snippet doc-test:skip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The performance.rst opener references an undefined g/query and engine='polars', so the doc-example runner executed it and failed in every polars-less lane (test-docs + test-minimal-python, pre-existing red on this branch). It is shown for reading, not execution — mark '.. doc-test: skip', same treatment as the benchmark_graphframes snippets (911f4e3). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/performance.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/gfql/performance.rst b/docs/source/gfql/performance.rst index 92d6961f39..c06bc64afb 100644 --- a/docs/source/gfql/performance.rst +++ b/docs/source/gfql/performance.rst @@ -11,6 +11,8 @@ GFQL runs the **same query** on four interchangeable engines — ``pandas`` (def **identical results** on each (differential parity is a release gate). The biggest, easiest win is one keyword, **no GPU required**: +.. doc-test: skip + .. code-block:: python g.gfql(query) # engine='pandas' (default) From 320fd38bdc92aac56d2b8cef1726c74e510c1e2b Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 2 Jul 2026 10:04:49 -0700 Subject: [PATCH 15/20] =?UTF-8?q?docs(gfql):=20review=20fixes=20=E2=80=94?= =?UTF-8?q?=20crossover=20consistency,=20forward=20ref,=20public=20phrasin?= =?UTF-8?q?g?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - performance.rst said 'below ~1M edges pandas often wins', contradicting engines.rst's measured ~10K polars crossover one click away — aligned to the measured guidance. - engines.rst referenced :doc:`benchmark_graphframes`, a page that only lands in the stacked benchmarks PR (#1668) — Sphinx unknown-doc warning if this PR ships alone. Reworded; #1668 restores the live link. - 'NO-CHEATING' is internal methodology jargon — public page now says 'No silent fallback — parity-verified' (same guarantee, reader-facing). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 6 +++--- docs/source/gfql/performance.rst | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index be9c23bcee..1738ac3700 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -436,8 +436,8 @@ change: ``polars.LazyFrame`` is collected immediately), so the source graph must still fit in memory. True out-of-core-from-disk — building GFQL directly on a lazy ``pl.scan_parquet`` source so a graph larger than RAM never fully materializes — - is **work in progress**; see the Friendster (~1.8B edges) discussion in - :doc:`benchmark_graphframes`. + is **work in progress**; see the Friendster (~1.8B edges) discussion in the + GraphFrames benchmark page. When **not** to use Polars -------------------------- @@ -466,7 +466,7 @@ Parity and honesty - **Identical results across engines.** Differential parity — every engine's output must match the pandas oracle — is a release gate, exercised across forward/reverse/undirected, 1-3 hop, filters, and aggregations. -- **No silent fallback (NO-CHEATING).** The Polars engine runs natively or raises +- **No silent fallback — parity-verified.** The Polars engine runs natively or raises ``NotImplementedError`` — it never quietly converts to pandas. ``polars-gpu`` is **GPU-or-error**: if any step of the plan cannot run on the GPU it raises (pointing at ``engine='polars'``) rather than silently running on CPU and labelling it a GPU result. diff --git a/docs/source/gfql/performance.rst b/docs/source/gfql/performance.rst index c06bc64afb..0043fa5d35 100644 --- a/docs/source/gfql/performance.rst +++ b/docs/source/gfql/performance.rst @@ -40,7 +40,8 @@ Warm-median latency, same query, identical result rows (**Orkut**, 117M edges, S - 314 ms - **167 ms** -There is **no universal winner**: below ~1M edges ``pandas`` often wins, and the right GPU +There is **no universal winner**: ``polars`` typically takes over from ~10K edges up +(``pandas`` still wins trivial sub-millisecond operations), and the right GPU engine depends on the workload. See :doc:`engines` for the full decision matrix, the honest "when *not* to use Polars", the cuDF-vs-Polars-GPU comparison, and the methodology + reproducer scripts behind these numbers. The end-to-end CPU/GPU-vs-Neo4j benchmark is in From 947bfc169a972237581d330d63da207828088c85 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 2 Jul 2026 10:10:31 -0700 Subject: [PATCH 16/20] docs(changelog): align engine-docs entry with the measured numbers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry said '11-47x' and 'the ~1M crossover' while the page it describes says 'up to ~38x' and a measured ~10K-edge CPU crossover — stale from an earlier draft of the docs. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bac3c70e9..d56a39834a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Documentation -- **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (11–47× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the ~1M crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). +- **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). ### Added - **GFQL viz-filter-pipeline acceptance suite + regression benchmark (viz-filter L3)**: `test_viz_pipeline_conformance.py` — curated full-panel pipelines (node+edge filters, exclusion-dominates composition, `(pred OR IS NULL)` keep-null leaves, both EXISTS prune-isolated flavors, searchAny composition, deterministic paging), graph-state prune shapes with exact node+edge pins, a 40-seed panel-state fuzzer with an independent plain-pandas second oracle, and a case/regex/unicode trick matrix (ß/İ full-case-mapping pins, metachar literal-vs-regex, null cells, Categorical) — all parity-or-NIE across pandas/cuDF/polars/polars-gpu. `benchmarks/gfql/viz_filter_pipeline.py` — six streamgl-viz panel scenarios (filters, keep-self GRAPH prune, EXISTS prune, node/edge search, combined) at 100K/1M/10M with native-frame-per-engine fairness, an NIE-tolerant matrix, and JSON receipts (first receipt: 100k × 4 engines, everything within the ~350ms interactive reference except pandas combined). Documented findings from the first runs: the same-path WHERE route dedupes parallel edges (diverges from the panel algebra's edge multiplicity — pinned + tracked), and edge-alias searchAny declines on polars (tracked). From 0737b1c58df19e275c2e3a4cde82eff3eea973b8 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 2 Jul 2026 10:34:06 -0700 Subject: [PATCH 17/20] =?UTF-8?q?docs(gfql):=20replace=20U+2248=20with=20A?= =?UTF-8?q?SCII=20'~'=20=E2=80=94=20pdflatex=20chokes=20on=20=E2=89=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test-docs' pdflatex pass rejects Unicode ≈ (U+2248) in the PDF build (./PyGraphistry.tex:6457: LaTeX Error) — a failure previously masked by the doc-example failure ahead of it in the same job. Identical change applied on both docs-tail branches so each CI tree builds (identical both-side changes merge cleanly). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 1738ac3700..e20f3fa0a1 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -188,7 +188,7 @@ benchmarked** rather than guess. * - **Kuzu** - Embedded graph DB; still a separate store to load + index. - **Seeded index lookup** (0.8M nodes / 6.4M edges): 1-hop **0.123 ms vs 1.15 ms - (9.4×)**, 2-hop **0.150 ms vs 4.25 ms (28×)**; prepared-Kuzu LiveJournal 35M ≈ **17×** + (9.4×)**, 2-hop **0.150 ms vs 4.25 ms (28×)**; prepared-Kuzu LiveJournal 35M ~ **17×** typical seed, 6× hub. **Bulk frontier expansion** (LiveJournal 35M, 1-hop, many seeds): **22× Kuzu**, up to **87× at k=100k**. See :doc:`index_adjacency`. - **Not claimed:** cyclic / multi-way-join patterns (triangles, cliques) where Kuzu's From 5c6f83f440228a75932c4e8d2e4d0e4467c74e45 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Fri, 3 Jul 2026 22:15:05 -0700 Subject: [PATCH 18/20] docs(gfql): document Python-settable polars streaming/executor config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine guide said the streaming flags are 'read at import time' — no longer true: they're read live and settable from Python (set_cpu_streaming / set_gpu_executor + the public GPU_EXECUTORS options, added on the polars-engine PR). Document the Python API alongside the existing env vars. doc-test:skip (the API lands with the polars PR this docs PR stacks on). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index e20f3fa0a1..5f48febd7b 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -405,8 +405,7 @@ page measures, the default is faster and more stable. Results are **parity-identical** to the default — streaming changes *how* the plan runs, not *what* it returns. -The flags are read at import time, so set them **before your process starts** (or -before ``import graphistry``): +Set them by environment variable: .. code-block:: bash @@ -416,6 +415,20 @@ before ``import graphistry``): # GPU: streaming executor for larger-than-device-memory results export GFQL_POLARS_GPU_EXECUTOR=streaming +...or from Python at runtime — the setting is read **live** (per collect), and a Python +override takes precedence over the environment variable: + +.. doc-test: skip + +.. code-block:: python + + from graphistry.compute.gfql.lazy import ( + set_cpu_streaming, set_gpu_executor, GPU_EXECUTORS, + ) + + set_cpu_streaming(True) # CPU streaming collect (pass None to reset to env/default) + set_gpu_executor('streaming') # one of GPU_EXECUTORS == ('in-memory', 'streaming') + Then use ``engine='polars'`` / ``engine='polars-gpu'`` exactly as before — no code change: From 3267c5548fbeb57a4b22bb88856db14f6972eb28 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 4 Jul 2026 10:14:02 -0700 Subject: [PATCH 19/20] docs(gfql): document off-engine call() modality (call_mode) + honesty reconcile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document PHASE 12 call_mode (auto/strict) in engines.rst: a new 'Analytics under Polars' subsection (umap/hypergraph/compute_cugraph run off-engine by default, coerce back to polars, warn once; polars-gpu bridges to cuDF GPU-or-error; strict declines), and reconcile the 'Parity and honesty' section — traversal/row ops stay parity-or-NIE (never bridge) while whole-graph analytics are the one mode-gated, warned exception. (P13.6 executor-mode knobs were already documented in the streaming section.) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 53 ++++++++++++++++++++++++++++++++---- 1 file changed, 48 insertions(+), 5 deletions(-) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 5f48febd7b..390d7238b3 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -354,6 +354,42 @@ never selects Polars or Polars-GPU**, so those two are always an explicit opt-in ``g.gfql_index_all()`` (or ``index_policy=``) — it works on all four engines and turns the O(E) scan into an O(degree) gather. See :doc:`index_adjacency`. +.. _gfql-offengine-calls: + +Analytics under Polars (``umap`` / ``hypergraph`` / ``compute_cugraph`` …) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A GFQL ``call()`` that runs a **whole-graph analytic** — ``umap``, ``hypergraph``, +``compute_cugraph`` / ``compute_igraph``, the ``*_layout`` ops, ``collapse`` — has +**no native Polars implementation** (these wrap pandas / cuDF / GPU libraries and +always will). Under ``engine='polars'`` / ``'polars-gpu'`` GFQL runs them as a +**mode-gated, off-engine modality switch** rather than declining outright: + +- **``call_mode='auto'`` (the default):** the analytic runs off-engine — on + **pandas** for ``polars``, on **cuDF (on device)** for ``polars-gpu`` — and its + result is coerced back to Polars **losslessly** (via Arrow). A one-time + ``RuntimeWarning`` per analytic notes the off-engine run. ``polars-gpu`` is + **GPU-or-error**: it bridges to cuDF and *declines* if the GPU/cuDF stack is + missing (it never silently drops a GPU analytic to host pandas). +- **``call_mode='strict'``:** decline with ``NotImplementedError`` instead of + bridging — for benchmark integrity (no hidden modality switch attributed to the + Polars engine) or a hard memory ceiling. + +This is **deliberately narrower** than traversal / filter / row ops (``hop``, +``WHERE``, ``RETURN`` …), which stay **parity-or-``NotImplementedError``** and are +never bridged — a bridge there would hide a missing native impl and misreport +pandas performance as Polars. Set the mode from Python or the environment (live, +Python override > env > default): + +.. doc-test: skip + +.. code-block:: python + + from graphistry.compute.gfql.lazy import set_call_mode, CALL_MODES # ('auto', 'strict') + + set_call_mode('strict') # decline off-engine analytics (pass None to reset to env/default) + # or: export GFQL_POLARS_CALL_MODE=strict + cuDF vs Polars-GPU ------------------ @@ -479,11 +515,18 @@ Parity and honesty - **Identical results across engines.** Differential parity — every engine's output must match the pandas oracle — is a release gate, exercised across forward/reverse/undirected, 1-3 hop, filters, and aggregations. -- **No silent fallback — parity-verified.** The Polars engine runs natively or raises - ``NotImplementedError`` — it never quietly converts to pandas. ``polars-gpu`` is - **GPU-or-error**: if any step of the plan cannot run on the GPU it raises (pointing at - ``engine='polars'``) rather than silently running on CPU and labelling it a GPU result. - So any latency you measure is real work on the engine you asked for. +- **No silent fallback for traversal / filter / row ops — parity-verified.** For ``hop`` / + ``WHERE`` / ``RETURN`` / aggregation, the Polars engine runs natively or raises + ``NotImplementedError`` — it never quietly converts to pandas, so a *traversal* latency you + measure is real work on the engine you asked for. ``polars-gpu`` is **GPU-or-error**: if any + step of the plan cannot run on the GPU it raises (pointing at ``engine='polars'``) rather than + silently running on CPU and labelling it a GPU result. +- **Whole-graph analytics are the one mode-gated exception.** ``umap`` / ``hypergraph`` / + ``compute_cugraph`` and friends have no Polars kernel; under ``call_mode='auto'`` (default) + they run off-engine and warn once (see + :ref:`Analytics under Polars `). This is *not* silent — it warns — and + ``call_mode='strict'`` restores strict parity-or-decline for benchmark integrity, so a + benchmarked run can guarantee no hidden modality switch. Methodology ----------- From 6c9be1fbad54ae69c5ac2d8d73324ab772c5c187 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 4 Jul 2026 11:45:00 -0700 Subject: [PATCH 20/20] docs(gfql): memory note for call_mode auto bridge (G5 decision) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document why there is no per-call size cap on the off-engine bridge: the transient copy is the same allocation as running the analytic on engine='cudf' directly, a row count is a poor memory proxy, and the real cap belongs at the RMM/container/deployment layer. Point memory-conscious users at call_mode='strict' or RMM/container limits. (G5 decision: reject a row-cap knob as the wrong mechanism — see plan PHASE 13.) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/source/gfql/engines.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/source/gfql/engines.rst b/docs/source/gfql/engines.rst index 390d7238b3..dae255f51f 100644 --- a/docs/source/gfql/engines.rst +++ b/docs/source/gfql/engines.rst @@ -375,6 +375,17 @@ always will). Under ``engine='polars'`` / ``'polars-gpu'`` GFQL runs them as a bridging — for benchmark integrity (no hidden modality switch attributed to the Polars engine) or a hard memory ceiling. +.. note:: + **Memory on a very large graph.** The bridge materializes a copy of the graph in + the off-engine format — pandas (host) for ``polars``, cuDF (device / unified + memory) for ``polars-gpu``. That transient copy is the *same* allocation you'd + incur running the analytic on ``engine='cudf'`` directly, so GFQL does **not** add + a per-call size cap (a row count is a poor memory proxy, and the real cap belongs + at the RMM / container / deployment layer). For a graph large enough that the copy + is a concern, either set ``call_mode='strict'`` (decline the bridge) or run the + analytic under an RMM device-memory limit / container memory limit, exactly as you + would for any cuDF workload. + This is **deliberately narrower** than traversal / filter / row ops (``hop``, ``WHERE``, ``RETURN`` …), which stay **parity-or-``NotImplementedError``** and are never bridged — a bridge there would hide a missing native impl and misreport