Commit 96e1d5f
docs(gfql): benchmark GFQL vs Spark GraphFrames (L4, single-node, honest)
New page docs/source/gfql/benchmark_graphframes.rst comparing GFQL polars
(CPU) and polars-gpu (GPU) against Spark GraphFrames (local[*], single node)
on filter / 1-2 hop / PageRank over SNAP LiveJournal (35M) and Orkut (117M),
with a committed reproducible harness (benchmarks/gfql/bench_graphframes.py).
Findings, stated honestly (numbers = median of 5 after 2 warmups; result-size
parity enforced per task):
- filter/traversal: GFQL wins 2-43x even on CPU (no JVM/scheduler/shuffle
overhead; single-node columnar is the right tool for sub-second graph queries).
- PageRank: mixed and disclosed — GFQL's CPU/igraph path is SLOWER than
GraphFrames (0.23-0.33x); only the GPU/cugraph path wins (~10-15x). Guidance:
reach for the GPU engine for whole-graph analytics.
- PageRank cross-engine parity verified: Spearman rho = 1.00, top-100 overlap
100/100 across igraph/cugraph/GraphFrames (saved artifact).
- Friendster (1.8B edges): documented single-node memory ceiling — every engine
(GFQL pandas-load OOM, GFQL cudf-lean swap, GraphFrames thrash) exceeds one
119GB node; reported as a wall, not dropped.
Harness: shared --filter-threshold for bit-identical filter parity; node-count
parity for hops; guarded per-cell (OOM/skip continues); warm-median + cold-load;
GraphFrames 0.8.4-spark3.5-s_2.12 / PySpark 3.5.1. Results rendered from saved
JSON (_static/graphframes/). engines.rst head-to-head row updated to link this
page; toctree entry added. Persona-tested (Raj/Sam/Lena): maxIter/tolerance
disclosed, single-node ceiling stated, blocked-not-interleaved noted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent c7f3af4 commit 96e1d5f
7 files changed
Lines changed: 1334 additions & 3 deletions
File tree
- benchmarks/gfql
- docs/source/gfql
- _static/graphframes
0 commit comments