Skip to content

Commit ceb9a44

Browse files
Lingkai981mlk12333Copilot
authored
feat(analytical): Graph-Analytics-Benchmarks (#4625)
<!-- Thanks for your contribution! please review https://github.com/alibaba/GraphScope/blob/main/CONTRIBUTING.md before opening an issue. --> ## What do these changes do? This PR contributes the open-sourced artifacts of our SIGMOD 2025 paper [*Revisiting Graph Analytics Benchmark*](https://doi.org/10.1145/3725345). Specifically, it adds: - **Failure-Free Trial Data Generator (FFT-DG)** for generating benchmark datasets with controlled scale, density, and diameter. - **LLM-based API usability evaluation framework**, packaged with Docker, enabling reproducible usability studies without costly human evaluation. - **Performance evaluation scripts and assets** for running benchmarks on multiple graph platforms (GraphX, PowerGraph, Flash, Grape, Pregel+, Ligra, G-thinker) in Kubernetes + Docker environments. These artifacts ensure reproducibility of the paper’s results and provide the community with a standardized, extensible benchmark suite for cross-platform graph analytics. ## Related issue number N/A Fixes --------- Co-authored-by: LingkaiMeng <2722694948@qq.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 431f4b2 commit ceb9a44

9 files changed

Lines changed: 2224 additions & 0 deletions

‎analytical_engine/benchmarks/framework/.env.example‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
OPENAI_API_KEY="your_openai_api_key_here"

‎analytical_engine/benchmarks/framework/README.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Graph-Analytics-Benchmarks
2+
3+
**Graph-Analytics-Benchmarks** accompanies the paper [*Revisiting Graph Analytics Benchmark*](https://doi.org/10.1145/3725345), published in *Proceedings of the ACM on Management of Data (SIGMOD 2025)*.
4+
This project introduces a new benchmark that overcomes limitations of existing suites and enables apples-to-apples comparisons across graph platforms.
5+
6+
---
7+
8+
9+
10+
## Table of Contents
11+
12+
1. [Overview](#overview)
13+
2. [Quick Start](#quick-start)
14+
- [Prerequisites](#prerequisites)
15+
- [Installation & Setup](#installation--setup)
16+
- [Commands](#commands)
17+
- [Global help](#global-help)
18+
- [`datagen` Data Generation](#datagen-data-generation)
19+
- [Requirements](#requirements)
20+
- [Usage](#usage)
21+
- [Example](#example)
22+
- [`llm-eval` LLM Usability Evaluation](#llm-eval-llm-usability-evaluation)
23+
- [Requirements](#requirements-1)
24+
- [Usage](#usage-1)
25+
- [Example](#example-1)
26+
- [`perf-eval` Performance Evaluation](#perf-eval-performance-evaluation)
27+
- [Requirements](#requirements-2)
28+
- [Usage](#usage-2)
29+
- [Examples](#examples)
30+
4. [Cite This Work](#cite-this-work)
31+
32+
33+
34+
35+
36+
## Overview
37+
38+
**Graph-Analytics-Benchmarks** accompanies the paper [“Revisiting Graph Analytics Benchmark”](https://doi.org/10.1145/3725345), which introduces a new benchmark suite for cross-platform graph analytics.
39+
The paper argues that existing suites (e.g., LDBC Graphalytics) fall short in fully capturing differences across platforms, and proposes a benchmark enabling fair, scalable, and reproducible comparisons.
40+
41+
**This repository provides three main components:**
42+
- **Failure-Free Trial Data Generator (FFT-DG)**:
43+
A lightweight, failure-immune data generator with independent control over **scale**, **density**, and **diameter**. Supports multiple output formats (including weighted/unweighted edge lists).
44+
- **LLM-based API Usability Evaluation**:
45+
A multi-level LLM framework for automatically generating and evaluating algorithm implementations across platforms, producing multi-dimensional quality scores and replacing costly human studies (packaged in Docker for one-command execution).
46+
- **Performance Evaluation Scripts**:
47+
Reproducible experiment setup in **Kubernetes + Docker**, with distributed jobs scheduled by the **Kubeflow MPI Operator (MPIJob)**. Provides unified reporting on **timing, throughput (edges/s), scalability, and robustness**.
48+
49+
**Algorithm Coverage (8 representative algorithms):**
50+
PageRank (PR), Single-Source Shortest Path (SSSP), Triangle Counting (TC), Betweenness Centrality (BC), K-Core (KC), Community Detection (CD), Label Propagation (LPA), Weakly Connected Components (WCC).
51+
52+
**Supported Platforms and Execution Modes:**
53+
- **Kubernetes + MPI**: Flash, Ligra, Grape
54+
- **Kubernetes + MPI + Hadoop**: Pregel+, Gthinker, PowerGraph
55+
- **Spark-based**: GraphX (requires Spark 2.4.x / Scala 2.11 / Hadoop 2.7 / Java 8)
56+
57+
**Intended Audience:**
58+
Researchers, practitioners, and educators in graph systems and distributed computing who require reproducible, apples-to-apples comparisons and system tuning under consistent conditions.
59+
60+
> For citation details, see [Cite This Work](#cite-this-work).
61+
62+
63+
## Quick Start
64+
65+
66+
### Prerequisites
67+
68+
- **Python 3.x+** (recommended: 3.10/3.11)
69+
- **Docker** (required for `llm-eval` and platform images)
70+
- **Kubernetes + MPI Operator + Hadoop** (for distributed experiments)
71+
- (GraphX only) **Java 8 / Scala 2.11 / Spark 2.4.x (Hadoop 2.7)**
72+
73+
### Global help
74+
```bash
75+
python3 gab_cli.py --help
76+
```
77+
78+
### `datagen` Data Generation
79+
80+
#### Requirements
81+
We provide a lightweight C++ program (download from [Data_Generator.zip](https://graphscope.oss-cn-beijing.aliyuncs.com/benchmark_datasets/Data_Generator.zip)) to generate data. Download and unzip to the `Data_Generator/` folder.
82+
83+
#### Usage
84+
```bash
85+
python3 gab_cli.py datagen --platform <platform> --scale <scale> --feature <feature>
86+
```
87+
- **--platform**: Target graph system (e.g., `flash`, `ligra`, `grape`, `gthinker`, `pregel+`, `powergraph`, `graphx`)
88+
- **--scale**: Graph scale (e.g., `8`, `9`, `10`, or custom value)
89+
- **--feature**: Data generation feature (`Standard`, `Density`, or `Diameter`)
90+
91+
#### Example
92+
```bash
93+
python3 gab_cli.py datagen --scale 9 --platform flash --feature Standard
94+
```
95+
---
96+
97+
### `llm-eval` LLM Usability Evaluation
98+
99+
#### Requirements
100+
- `OPENAI_API_KEY` set in `.env.example` or system environment.
101+
102+
#### Usage
103+
```bash
104+
python3 gab_cli.py llm-eval --platform <platform> --algorithm <algorithm>
105+
```
106+
- **--platform**: Target graph system (e.g., `flash`, `ligra`, `grape`, `gthinker`, `pregel+`, `powergraph`, `graphx`)
107+
- **--algorithm**: Algorithm to evaluate usability (e.g., `pagerank`, `sssp`, `triangle`, `bc`, `cd`, `lpa`, `kclique`, `cc`)
108+
109+
#### Example
110+
```bash
111+
python3 gab_cli.py llm-eval --platform flash --algorithm pagerank
112+
```
113+
114+
---
115+
116+
### `perf-eval` Performance Evaluation
117+
118+
#### Requirements
119+
- All performance experiments are conducted in a properly configured **Kubernetes + MPI Operator (MPIJob) + Hadoop** environment to ensure reproducibility and consistency.
120+
- **GraphX** experiments additionally require a properly configured **Spark environment** (Spark 2.4.x, Scala 2.11, Hadoop 2.7, Java 8).
121+
122+
#### Usage
123+
```bash
124+
python3 gab_cli.py perf-eval --platform <platform> --algorithm <algorithm> [--path <dataset_file>] [--spark-master <spark-master>]
125+
```
126+
- **--platform**: Target graph system (e.g., `flash`, `ligra`, `grape`, `gthinker`, `pregel+`, `powergraph`, `graphx`)
127+
- **--algorithm**: Algorithm to run (e.g., `pagerank`, `sssp`, `triangle`, `bc`, `cd`, `lpa`, `kclique`, `cc`)
128+
- **--path**: Path to input dataset file or directory.
129+
> **All machines in the cluster must have the dataset available at the same path.**
130+
> If not specified, a default test dataset will be used.
131+
132+
> **Note:**
133+
> - For **flash**, specify a **directory** containing the dataset files.
134+
> - For **grape**, provide the **prefix** for `.e` and `.v` files.
135+
> - For other platforms, specify the **complete dataset file**.
136+
> See the [sample datasets]() for details.
137+
- **--spark-master**: Spark master URL (only needed for **GraphX** experiments)
138+
139+
#### Examples
140+
```bash
141+
# Run PageRank on Flash
142+
python3 gab_cli.py perf-eval --platform flash --algorithm pagerank --path sample_data/flash_sample_graph/
143+
144+
# Run PageRank on Grape
145+
python3 gab_cli.py perf-eval --platform grape --algorithm pagerank --path sample_data/grape_sample_graph
146+
147+
# Run PageRank on Ligra
148+
python3 gab_cli.py perf-eval --platform ligra --algorithm pagerank --path sample_data/ligra_sample_graph.txt
149+
150+
# Run Triangle Counting on GraphX
151+
python3 gab_cli.py perf-eval --platform graphx --algorithm triangle --path sample_data/graphx_sample_graph.txt --spark-master spark://spark-master:7077
152+
```
153+
---
154+
155+
156+
## Cite This Work
157+
158+
If you use this artifact, please cite the paper:
159+
160+
```bibtex
161+
@article{meng2025revisiting,
162+
title={Revisiting Graph Analytics Benchmark},
163+
author={Meng, Lingkai and Shao, Yu and Yuan, Long and Lai, Longbin and Cheng, Peng and Li, Xue and Yu, Wenyuan and Zhang, Wenjie and Lin, Xuemin and Zhou, Jingren},
164+
journal={Proceedings of the ACM on Management of Data},
165+
volume={3},
166+
number={3},
167+
pages={1--28},
168+
year={2025},
169+
publisher={ACM New York, NY, USA}
170+
}
171+
```
172+
173+
---

0 commit comments

Comments
 (0)