Skip to content

Commit 9c031e2

Browse files
joseph-isaacsclaude
joseph-isaacs
authored and
claude
committed
Link per-suite explainer docs from CI benchmark PR comments
Each "Benchmarks: <suite>" PR comment now carries a book icon link in its title pointing at a markdown explainer that lives next to the suite definition (vortex-bench/sql/* for SQL suites, benchmarks/* for the random-access and compression binaries). The doc path is a new "doc" field on each benchmark matrix entry, rendered into the comment title by the Compare results steps. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
1 parent c981534 commit 9c031e2

15 files changed

Lines changed: 262 additions & 2 deletions

File tree

‎.github/workflows/bench-pr.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,11 @@ jobs:
3030
benchmark:
3131
- id: random-access-bench
3232
name: Random Access
33+
doc: benchmarks/random-access-bench/README.md
3334
build_args: "--features lance"
3435
- id: compress-bench
3536
name: Compression
37+
doc: benchmarks/compress-bench/README.md
3638
steps:
3739
- uses: runs-on/action@v2
3840
if: github.event.pull_request.head.repo.fork == false
@@ -113,7 +115,7 @@ jobs:
113115
python3 scripts/s3-download.py s3://vortex-ci-benchmark-results/data.json.gz data.json.gz --no-sign-request
114116
gzip -d -c data.json.gz > base.json
115117
116-
echo '# Benchmarks: ${{ matrix.benchmark.name }}' > comment.md
118+
echo '# Benchmarks: ${{ matrix.benchmark.name }}${{ matrix.benchmark.doc && format(' [📖]({0}/{1}/blob/develop/{2})', github.server_url, github.repository, matrix.benchmark.doc) || '' }}' > comment.md
117119
echo '' >> comment.md
118120
uv run --no-project scripts/compare-benchmark-jsons.py base.json results.json "${{ matrix.benchmark.name }}" \
119121
>> comment.md

‎.github/workflows/bench.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,7 @@ jobs:
183183
"id": "vortex-queries",
184184
"subcommand": "vortex",
185185
"name": "Vortex queries",
186+
"doc": "vortex-bench/sql/vortex/README.md",
186187
"data_formats": ["parquet", "vortex"],
187188
"pr_targets": [
188189
{"engine": "datafusion", "format": "parquet"},

‎.github/workflows/sql-benchmarks.yml‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ on:
2020
"id": "clickbench-nvme",
2121
"subcommand": "clickbench",
2222
"name": "Clickbench on NVME",
23+
"doc": "vortex-bench/sql/clickbench.md",
2324
"data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"],
2425
"pr_targets": [
2526
{"engine": "datafusion", "format": "parquet"},
@@ -43,6 +44,7 @@ on:
4344
"id": "clickbench-sorted-nvme",
4445
"subcommand": "clickbench-sorted",
4546
"name": "Clickbench Sorted on NVME",
47+
"doc": "vortex-bench/sql/clickbench.md#sorted-variant",
4648
"data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"],
4749
"pr_targets": [
4850
{"engine": "datafusion", "format": "parquet"},
@@ -66,6 +68,7 @@ on:
6668
"id": "tpch-nvme",
6769
"subcommand": "tpch",
6870
"name": "TPC-H SF=1 on NVME",
71+
"doc": "vortex-bench/sql/tpch/README.md",
6972
"data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"],
7073
"pr_targets": [
7174
{"engine": "datafusion", "format": "arrow"},
@@ -95,6 +98,7 @@ on:
9598
"id": "tpch-s3",
9699
"subcommand": "tpch",
97100
"name": "TPC-H SF=1 on S3",
101+
"doc": "vortex-bench/sql/tpch/README.md",
98102
"local_dir": "vortex-bench/data/tpch/1.0",
99103
"remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/1.0/",
100104
"data_formats": ["parquet", "vortex", "vortex-compact"],
@@ -121,6 +125,7 @@ on:
121125
"id": "tpch-nvme-10",
122126
"subcommand": "tpch",
123127
"name": "TPC-H SF=10 on NVME",
128+
"doc": "vortex-bench/sql/tpch/README.md",
124129
"data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"],
125130
"pr_targets": [
126131
{"engine": "datafusion", "format": "arrow"},
@@ -150,6 +155,7 @@ on:
150155
"id": "tpch-s3-10",
151156
"subcommand": "tpch",
152157
"name": "TPC-H SF=10 on S3",
158+
"doc": "vortex-bench/sql/tpch/README.md",
153159
"local_dir": "vortex-bench/data/tpch/10.0",
154160
"remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/10.0/",
155161
"data_formats": ["parquet", "vortex", "vortex-compact"],
@@ -176,6 +182,7 @@ on:
176182
"id": "tpcds-nvme",
177183
"subcommand": "tpcds",
178184
"name": "TPC-DS SF=1 on NVME",
185+
"doc": "vortex-bench/sql/tpcds/README.md",
179186
"data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"],
180187
"pr_targets": [
181188
{"engine": "datafusion", "format": "parquet"},
@@ -201,6 +208,7 @@ on:
201208
"id": "statpopgen",
202209
"subcommand": "statpopgen",
203210
"name": "Statistical and Population Genetics",
211+
"doc": "vortex-bench/sql/statpopgen.md",
204212
"local_dir": "vortex-bench/data/statpopgen",
205213
"data_formats": ["parquet", "vortex", "vortex-compact"],
206214
"pr_targets": [
@@ -219,6 +227,7 @@ on:
219227
"id": "fineweb",
220228
"subcommand": "fineweb",
221229
"name": "FineWeb NVMe",
230+
"doc": "vortex-bench/sql/fineweb.md",
222231
"data_formats": ["parquet", "vortex", "vortex-compact"],
223232
"pr_targets": [
224233
{"engine": "datafusion", "format": "parquet"},
@@ -242,6 +251,7 @@ on:
242251
"id": "fineweb-s3",
243252
"subcommand": "fineweb",
244253
"name": "FineWeb S3",
254+
"doc": "vortex-bench/sql/fineweb.md",
245255
"local_dir": "vortex-bench/data/fineweb",
246256
"remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/fineweb/",
247257
"data_formats": ["parquet", "vortex", "vortex-compact"],
@@ -267,6 +277,7 @@ on:
267277
"id": "polarsignals",
268278
"subcommand": "polarsignals",
269279
"name": "PolarSignals Profiling",
280+
"doc": "vortex-bench/sql/polarsignals.md",
270281
"data_formats": ["vortex"],
271282
"pr_targets": [
272283
{"engine": "datafusion", "format": "vortex"}
@@ -280,6 +291,7 @@ on:
280291
"id": "appian-nvme",
281292
"subcommand": "appian",
282293
"name": "Appian on NVME",
294+
"doc": "vortex-bench/sql/appian/README.md",
283295
"data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"],
284296
"pr_targets": [
285297
{"engine": "datafusion", "format": "parquet"},
@@ -310,6 +322,7 @@ on:
310322
"id": "clickbench-nvme",
311323
"subcommand": "clickbench",
312324
"name": "Clickbench on NVME",
325+
"doc": "vortex-bench/sql/clickbench.md",
313326
"data_formats": ["parquet", "vortex"],
314327
"pr_targets": [
315328
{"engine": "datafusion", "format": "parquet"},
@@ -329,6 +342,7 @@ on:
329342
"id": "clickbench-sorted-nvme",
330343
"subcommand": "clickbench-sorted",
331344
"name": "Clickbench Sorted on NVME",
345+
"doc": "vortex-bench/sql/clickbench.md#sorted-variant",
332346
"data_formats": ["parquet", "vortex"],
333347
"pr_targets": [
334348
{"engine": "datafusion", "format": "parquet"},
@@ -348,6 +362,7 @@ on:
348362
"id": "tpch-nvme",
349363
"subcommand": "tpch",
350364
"name": "TPC-H SF=1 on NVME",
365+
"doc": "vortex-bench/sql/tpch/README.md",
351366
"data_formats": ["parquet", "vortex"],
352367
"pr_targets": [
353368
{"engine": "datafusion", "format": "arrow"},
@@ -371,6 +386,7 @@ on:
371386
"id": "tpch-s3",
372387
"subcommand": "tpch",
373388
"name": "TPC-H SF=1 on S3",
389+
"doc": "vortex-bench/sql/tpch/README.md",
374390
"local_dir": "vortex-bench/data/tpch/1.0",
375391
"remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/1.0/",
376392
"data_formats": ["parquet", "vortex"],
@@ -393,6 +409,7 @@ on:
393409
"id": "tpch-nvme-10",
394410
"subcommand": "tpch",
395411
"name": "TPC-H SF=10 on NVME",
412+
"doc": "vortex-bench/sql/tpch/README.md",
396413
"data_formats": ["parquet", "vortex"],
397414
"pr_targets": [
398415
{"engine": "datafusion", "format": "arrow"},
@@ -416,6 +433,7 @@ on:
416433
"id": "tpcds-nvme",
417434
"subcommand": "tpcds",
418435
"name": "TPC-DS SF=1 on NVME",
436+
"doc": "vortex-bench/sql/tpcds/README.md",
419437
"data_formats": ["parquet", "vortex"],
420438
"pr_targets": [
421439
{"engine": "datafusion", "format": "parquet"},
@@ -435,6 +453,7 @@ on:
435453
"id": "statpopgen",
436454
"subcommand": "statpopgen",
437455
"name": "Statistical and Population Genetics",
456+
"doc": "vortex-bench/sql/statpopgen.md",
438457
"local_dir": "vortex-bench/data/statpopgen",
439458
"data_formats": ["parquet", "vortex"],
440459
"pr_targets": [
@@ -451,6 +470,7 @@ on:
451470
"id": "fineweb",
452471
"subcommand": "fineweb",
453472
"name": "FineWeb NVMe",
473+
"doc": "vortex-bench/sql/fineweb.md",
454474
"data_formats": ["parquet", "vortex"],
455475
"pr_targets": [
456476
{"engine": "datafusion", "format": "parquet"},
@@ -470,6 +490,7 @@ on:
470490
"id": "fineweb-s3",
471491
"subcommand": "fineweb",
472492
"name": "FineWeb S3",
493+
"doc": "vortex-bench/sql/fineweb.md",
473494
"local_dir": "vortex-bench/data/fineweb",
474495
"remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/fineweb/",
475496
"data_formats": ["parquet", "vortex"],
@@ -491,6 +512,7 @@ on:
491512
"id": "polarsignals",
492513
"subcommand": "polarsignals",
493514
"name": "PolarSignals Profiling",
515+
"doc": "vortex-bench/sql/polarsignals.md",
494516
"data_formats": ["vortex"],
495517
"pr_targets": [
496518
{"engine": "datafusion", "format": "vortex"}
@@ -678,7 +700,7 @@ jobs:
678700
python3 scripts/s3-download.py s3://vortex-ci-benchmark-results/data.json.gz data.json.gz --no-sign-request
679701
gzip -d -c data.json.gz > base.json
680702
681-
echo "# Benchmarks: ${{ matrix.name }}" > comment.md
703+
echo "# Benchmarks: ${{ matrix.name }}${{ matrix.doc && format(' [📖]({0}/{1}/blob/develop/{2})', github.server_url, github.repository, matrix.doc) || '' }}" > comment.md
682704
echo '' >> comment.md
683705
uv run --no-project scripts/compare-benchmark-jsons.py base.json results.json "${{ matrix.name }}" \
684706
>> comment.md

‎.github/workflows/sql-vortex-pr.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ jobs:
2929
"id": "vortex-queries",
3030
"subcommand": "vortex",
3131
"name": "Vortex queries",
32+
"doc": "vortex-bench/sql/vortex/README.md",
3233
"data_formats": ["parquet", "vortex"],
3334
"pr_targets": [
3435
{"engine": "datafusion", "format": "parquet"},

‎benchmarks/compress-bench/README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Compression benchmark
2+
3+
Measures compression and decompression throughput, plus resulting file sizes, for Vortex
4+
versus Parquet (and optionally Lance) across a range of datasets: NYC taxi data, several
5+
[Public BI](https://github.com/cwida/public_bi_benchmark) tables (Arade, Bimbo,
6+
CMSprovider, Euro2016, Food, HashTags), TPC-H `l_comment` variants, and synthetic nested
7+
data. This is the workload behind the `Compression` PR comment.
8+
9+
See [`src/main.rs`](./src/main.rs) for the dataset list and CLI flags (`--formats`,
10+
`--datasets`, `--ops compress,decompress`).
11+
12+
## Running locally
13+
14+
```bash
15+
cargo run -p compress-bench --profile release_debug
16+
```

‎benchmarks/random-access-bench/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Random Access benchmark
2+
3+
Measures point-lookup latency: fetching individual rows by index from a file, rather than
4+
scanning it. This is the workload behind the `Random Access` PR comment.
5+
6+
Two access patterns are generated with a fixed seed (see [`src/main.rs`](./src/main.rs)):
7+
8+
- **correlated**: several clusters of consecutive indices scattered across the dataset,
9+
simulating lookups with spatial locality;
10+
- **uniform**: indices drawn from a Poisson process spread uniformly across the dataset,
11+
simulating lookups with no locality.
12+
13+
Each pattern runs over four datasets (`taxi`, `feature-vectors`, `nested-lists`,
14+
`nested-structs`) in Parquet, Lance, and Vortex, both with a cached open file handle and
15+
reopening the file per lookup. CI drives the full matrix via
16+
[`scripts/random-access-split.py`](../../scripts/random-access-split.py).
17+
18+
## Running locally
19+
20+
```bash
21+
cargo run -p random-access-bench --profile release_debug --features lance
22+
```

‎vortex-bench/REUSE.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@ path = "**/*.csv"
1010
SPDX-FileCopyrightText = "Copyright the Vortex contributors"
1111
SPDX-License-Identifier = "Apache-2.0"
1212

13+
# Benchmark suite explainer docs linked from CI PR comments.
14+
[[annotations]]
15+
path = "sql/**/*.md"
16+
SPDX-FileCopyrightText = "Copyright the Vortex contributors"
17+
SPDX-License-Identifier = "CC-BY-4.0"
18+
1319
# `insta` snapshot files do not allow leading comment lines.
1420
[[annotations]]
1521
path = "src/snapshots/**.snap"

‎vortex-bench/sql/appian/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Appian benchmark
2+
3+
Mirrors the queries from DuckDB's in-tree
4+
[`appian_benchmarks`](https://github.com/duckdb/duckdb/tree/main/benchmark/appian_benchmarks)
5+
suite — a real-world business-application workload (customers, orders, addresses) with
6+
selective filters and joins over camelCase-named columns.
7+
8+
The eight queries live in this directory (`q1.sql` ... `q8.sql`). Upstream ships the data
9+
as a single `.duckdb` blob (~593 MB); the harness downloads it once and projects each
10+
table into Parquet with lowercased column names so that DataFusion and DuckDB resolve the
11+
verbatim queries identically — see [`src/appian`](../../src/appian) for the details.
12+
13+
## CI variant
14+
15+
CI runs this suite from local NVMe as the `Appian on NVME` PR comment, comparing
16+
DataFusion and DuckDB over Parquet and Vortex files (plus a native DuckDB baseline).
17+
18+
## Running locally
19+
20+
```bash
21+
vx-bench run appian --engine datafusion,duckdb --format parquet,vortex
22+
```

‎vortex-bench/sql/clickbench.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# ClickBench benchmark
2+
3+
[ClickBench](https://github.com/ClickHouse/ClickBench) is ClickHouse's web-analytics
4+
benchmark: 43 queries over a single wide `hits` table (~100M rows of real-ish traffic
5+
data). It is heavy on aggregations, `GROUP BY`s over high-cardinality columns, and
6+
selective string filters, and is the main "wide table scan" workload in CI.
7+
8+
The queries live in [`clickbench_queries.sql`](./clickbench_queries.sql) (one query per
9+
line, numbered from Q0 in file order). The harness lives in
10+
[`src/clickbench`](../src/clickbench).
11+
12+
## CI variant
13+
14+
CI runs this suite from local NVMe as the `Clickbench on NVME` PR comment, comparing
15+
DataFusion and DuckDB over Parquet, Vortex, and vortex-compact files (plus a native
16+
DuckDB baseline).
17+
18+
## Sorted variant
19+
20+
`Clickbench Sorted on NVME` runs the same table sorted by event time, split into 100
21+
shards whose filenames are shuffled so engines cannot rely on file order, exercising sort
22+
pushdown and zone-map-style pruning on a subset of the queries. See
23+
[`ClickBenchSortedBenchmark`](../src/clickbench/benchmark.rs).
24+
25+
## Running locally
26+
27+
```bash
28+
vx-bench run clickbench --engine datafusion,duckdb --format parquet,vortex
29+
vx-bench run clickbench-sorted --engine datafusion,duckdb --format parquet,vortex
30+
```

‎vortex-bench/sql/fineweb.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# FineWeb benchmark
2+
3+
A string-heavy benchmark over a sample shard of the HuggingFace
4+
[FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) web-crawl dataset
5+
(URLs, dates, and full page text). The dataset exercises dictionary and FSST string
6+
compression heavily, and the hand-crafted queries in [`fineweb.sql`](./fineweb.sql)
7+
(numbered from Q0 in file order) focus on string predicates: equality filters, `LIKE`
8+
prefix and containment patterns, and aggregations over string columns.
9+
10+
The harness lives in [`src/fineweb`](../src/fineweb).
11+
12+
## CI variants
13+
14+
CI runs this suite from local NVMe (`FineWeb NVMe`) and from S3 (`FineWeb S3`), comparing
15+
DataFusion and DuckDB over Parquet, Vortex, and vortex-compact files.
16+
17+
## Running locally
18+
19+
```bash
20+
vx-bench run fineweb --engine datafusion,duckdb --format parquet,vortex
21+
```

0 commit comments

Comments
 (0)