Skip to content

Commit 96e1d5f

Browse files
lmeyerovclaude
andcommitted
docs(gfql): benchmark GFQL vs Spark GraphFrames (L4, single-node, honest)
New page docs/source/gfql/benchmark_graphframes.rst comparing GFQL polars (CPU) and polars-gpu (GPU) against Spark GraphFrames (local[*], single node) on filter / 1-2 hop / PageRank over SNAP LiveJournal (35M) and Orkut (117M), with a committed reproducible harness (benchmarks/gfql/bench_graphframes.py). Findings, stated honestly (numbers = median of 5 after 2 warmups; result-size parity enforced per task): - filter/traversal: GFQL wins 2-43x even on CPU (no JVM/scheduler/shuffle overhead; single-node columnar is the right tool for sub-second graph queries). - PageRank: mixed and disclosed — GFQL's CPU/igraph path is SLOWER than GraphFrames (0.23-0.33x); only the GPU/cugraph path wins (~10-15x). Guidance: reach for the GPU engine for whole-graph analytics. - PageRank cross-engine parity verified: Spearman rho = 1.00, top-100 overlap 100/100 across igraph/cugraph/GraphFrames (saved artifact). - Friendster (1.8B edges): documented single-node memory ceiling — every engine (GFQL pandas-load OOM, GFQL cudf-lean swap, GraphFrames thrash) exceeds one 119GB node; reported as a wall, not dropped. Harness: shared --filter-threshold for bit-identical filter parity; node-count parity for hops; guarded per-cell (OOM/skip continues); warm-median + cold-load; GraphFrames 0.8.4-spark3.5-s_2.12 / PySpark 3.5.1. Results rendered from saved JSON (_static/graphframes/). engines.rst head-to-head row updated to link this page; toctree entry added. Persona-tested (Raj/Sam/Lena): maxIter/tolerance disclosed, single-node ceiling stated, blocked-not-interleaved noted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent c7f3af4 commit 96e1d5f

7 files changed

Lines changed: 1334 additions & 3 deletions

File tree

0 commit comments

Comments
 (0)