Skip to content

Commit 5e9547d

Browse files
JingsongLiJingsongLi
committed
docs: add TPC-DS SF100 performance report
1 parent 6824972 commit 5e9547d

4 files changed

Lines changed: 328 additions & 0 deletions

File tree

‎docs/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,16 @@ mkdocs build
5353
```
5454

5555
The output will be in the `docs/site/` directory.
56+
57+
## Performance Report
58+
59+
The TPC-DS SF100 performance report is maintained as
60+
`docs/src/benchmark.md` and is included in the MkDocs navigation. Preview it
61+
as part of the documentation site:
62+
63+
```bash
64+
cd docs
65+
mkdocs serve
66+
```
67+
68+
Then open <http://127.0.0.1:8000/benchmark/>.

‎docs/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ nav:
4949
- Home: index.md
5050
- Getting Started: getting-started.md
5151
- SQL Integration: sql.md
52+
- Performance: benchmark.md
5253
- Go Integration: go-binding.md
5354
- Python Integration: python-binding.md
5455
- Architecture: architecture.md

‎docs/src/benchmark.md‎

Lines changed: 305 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,305 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Paimon Rust at Native Speed
21+
22+
## TPC-DS SF100 read performance with DataFusion
23+
24+
!!! warning "Benchmark disclosure"
25+
26+
This is a TPC-DS-derived non-TPC benchmark. It is not an official TPC
27+
result and must not be compared with official TPC results.
28+
29+
This experiment compares Paimon Rust with DataFusion's native Parquet reader
30+
using the same DataFusion 54 SQL, planner, execution operators, query files,
31+
runtime limits, and local SSD. It measures 24 unpartitioned, append-only
32+
Paimon tables imported from the SF100 Parquet source.
33+
34+
## Result
35+
36+
Each query received one warmup followed by three measured iterations. The
37+
table reports the median of three workload totals, where each workload total
38+
is the sum of `execution_ms` for the same measured-iteration index across the
39+
96 queries that completed in every measured iteration for both sources.
40+
41+
| Source | Median workload execution time | Three-run range |
42+
| --- | ---: | ---: |
43+
| Paimon Rust | 317.61 s | 314.64–317.76 s |
44+
| Native Parquet | 290.53 s | 289.69–290.75 s |
45+
46+
Paimon Rust completes the workload within 9.3% of DataFusion's native Parquet
47+
reader. For a table-format reader that also interprets Paimon metadata and
48+
preserves Paimon's correctness boundaries, this is near-native performance.
49+
50+
The two layouts do not use the same compression codec. The source Parquet
51+
files use SNAPPY, while the Paimon rewrite uses the higher-compression ZSTD
52+
codec. Paimon's data files occupy 27.98 GB instead of 36.45 GB, a 23.25%
53+
reduction in physical storage. ZSTD saves storage and read I/O at the cost of
54+
more CPU-intensive decompression than SNAPPY, so the runtime comparison
55+
includes that tradeoff; it does not isolate Paimon metadata overhead from
56+
codec cost.
57+
58+
At the query level, Paimon has the lower median execution time on
59+
23 of the 96 comparable queries. For the 14 queries whose native-Parquet
60+
median is at least five seconds, the sum of per-query medians is 184.82 s for
61+
Paimon and 179.29 s for Parquet, a gap of 3.1%. The smaller gap on the long
62+
queries shows closer alignment as query runtime grows.
63+
64+
The detailed per-iteration JSON reports are not committed with this article
65+
to keep the documentation lightweight. During validation, all 288 measured
66+
output-row counts matched between the two sources, and the 96 normal queries
67+
completed without warmup failures, execution errors, or spill events on
68+
either source. The table above summarizes the validated reports using the
69+
aggregation method described in this section.
70+
71+
### Resource-limit queries
72+
73+
Q67, Q78, and Q97 are excluded from the workload aggregate because they hit
74+
the same 32 GiB memory-pool limit on both sources. Repeating an out-of-memory
75+
query in the same DataFusion session can leave the benchmark runtime unable
76+
to make progress, so these three queries were run once in separate focused
77+
processes; the other 96 queries use the full 1+3 protocol.
78+
79+
## Scope
80+
81+
The result applies to the append-only SF100 read path tested here. The loader
82+
creates unpartitioned append-only Paimon tables, so this experiment does
83+
**not** measure the runtime cost of primary-key merging, schema evolution,
84+
deletion vectors, row lineage, or non-Parquet file formats.
85+
86+
Paimon Rust supports or is developing those broader table-format capabilities,
87+
and its read path includes correctness safeguards for them. They should be
88+
evaluated with dedicated workloads before making performance claims about
89+
their overhead.
90+
91+
This is also an end-to-end source comparison rather than a catalog-overhead
92+
microbenchmark. Importing the source into Paimon rewrites the physical file
93+
layout: the original dataset contains 24 Parquet files and the Paimon copy
94+
contains 125 Parquet data files. It also changes compression from SNAPPY to
95+
ZSTD, so this is intentionally not a codec-matched microbenchmark.
96+
97+
## Engineering behind the read path
98+
99+
The SF100 result follows a set of changes that align Paimon scans with
100+
DataFusion's execution model:
101+
102+
- Paimon string and binary columns map to Arrow `Utf8View` and `BinaryView` at
103+
the DataFusion boundary.
104+
- `read.batch-size` reaches raw readers, primary-key readers, and
105+
data-evolution readers.
106+
- row counts, null counts, column bounds, and compressed sizes are exposed
107+
with explicit precision rather than being overstated as exact.
108+
- Paimon predicates provide conservative partition, file, and row-group
109+
pruning where they can represent the expression.
110+
- a format-neutral Arrow row-filter interface carries supported DataFusion
111+
expressions into the Parquet decoder for late materialization; the parent
112+
DataFusion filter is retained as the exact correctness filter.
113+
- primary-key, data-evolution, and `_ROW_ID` paths use conservative boundaries
114+
where pushdown could change merge results or physical row positions.
115+
116+
Only the append-only Parquet-backed read path is benchmarked by this report;
117+
the last point describes a correctness capability, not a measured result.
118+
119+
## Multi-engine direction
120+
121+
### DataFusion
122+
123+
DataFusion is the first complete SQL integration, from `SQLContext` and
124+
`TableProvider` through statistics and physical runtime-filter pruning.
125+
126+
### StarRocks
127+
128+
Work continues on the Paimon connector, row lineage, and vector-search path.
129+
Paimon Rust's native reader and FFI provide a foundation for deeper
130+
integration.
131+
132+
### Apache Doris
133+
134+
The community Paimon write architecture reserves a Rust FFI backend, providing
135+
a possible path toward removing the JVM bridge and reducing data-exchange
136+
costs.
137+
138+
### Milvus
139+
140+
Integration work is also underway for Milvus. Paimon Rust already exposes a
141+
materialized vector-search API through its C FFI: callers can select a vector
142+
column, provide a query vector, filter and limit the search, pass index
143+
options, and consume the result as streaming Arrow record batches. This gives
144+
Milvus a native path to Paimon-managed vector data without requiring a JVM
145+
bridge. The integration is an ecosystem direction and is not measured by this
146+
TPC-DS report.
147+
148+
## Reproduction
149+
150+
### Revisions and software
151+
152+
| Component | Version or revision |
153+
| --- | --- |
154+
| paimon-rust benchmark binary | `7afff74a20c7dee2fe7dc1f862ce7b8b74bf6cd2` |
155+
| Release binary SHA-256 | `1ebda8878fd2b7f4880a3fa2afaa14520e4cbc40a16352315b6a63b8aee643e7` |
156+
| DataFusion | `54.0.0` |
157+
| paimon-rust crate | `0.3.0` development build |
158+
| Query files | `delta-io/delta-rs` at `0f68868d1dbbe77fa4e99c96df49ae121f8974e4` |
159+
| Data generator | DuckDB `v1.5.0` (`3a3967aa81`) TPC-DS extension |
160+
161+
The benchmark binary was rebuilt from a clean Cargo target directory:
162+
163+
```bash
164+
cargo clean
165+
cargo build --release -p paimon-tpcds-bench
166+
```
167+
168+
### Dataset and physical layout
169+
170+
| Item | Value |
171+
| --- | ---: |
172+
| Scale factor | 100 |
173+
| Source Parquet files | 24 |
174+
| Source Parquet compression | SNAPPY |
175+
| Source Parquet physical bytes | 36,449,466,908 |
176+
| Paimon data files | 125 |
177+
| Paimon data-file compression | ZSTD |
178+
| Paimon data-file physical bytes | 27,975,615,232 |
179+
| Paimon physical data-file reduction | 8,473,851,676 bytes / 23.25% |
180+
| All files in Paimon warehouse | 269 files / 27,975,785,742 bytes |
181+
| Paimon table mode | unpartitioned, append-only |
182+
183+
The source data was generated and exported with this command:
184+
185+
```bash
186+
TPCDS_ROOT=/path/to/tpcds-benchmark/sf100
187+
DUCKDB_SPILL_ROOT=/path/to/tpcds-benchmark/spill/duckdb-sf100
188+
189+
./duckdb "${TPCDS_ROOT}/tpcds.duckdb" -c \
190+
"SET threads=12; \
191+
SET memory_limit='32GB'; \
192+
SET temp_directory='${DUCKDB_SPILL_ROOT}'; \
193+
CALL dsdgen(sf=100); \
194+
EXPORT DATABASE '${TPCDS_ROOT}/parquet' \
195+
(FORMAT PARQUET);"
196+
```
197+
198+
The Paimon copy was loaded with one writer partition. Query execution still
199+
used 12 target partitions.
200+
201+
```bash
202+
target/release/paimon-tpcds-bench load \
203+
--data "${TPCDS_ROOT}/parquet" \
204+
--warehouse "${TPCDS_ROOT}/paimon-layout-target1-full" \
205+
--database tpcds \
206+
--if-exists error \
207+
--target-partitions 1 \
208+
--memory-limit-gib 32
209+
```
210+
211+
### Machine and runtime
212+
213+
| Item | Value |
214+
| --- | --- |
215+
| Operating system | macOS 26.3.1, build 25D2128 |
216+
| CPU | Apple M4 Pro, 12 physical cores |
217+
| Memory | 48 GiB physical / 32 GiB DataFusion memory pool |
218+
| Storage | Apple SSD AP1024Z, local APFS/NVMe |
219+
| DataFusion target partitions | 12 |
220+
| Spill directory | none for query runs |
221+
| Cache protocol | one warmup per query, then three measured iterations |
222+
| Source order | Alternated by query batch; see below |
223+
224+
The OS page cache was not evicted, so these are warm-cache measurements. To
225+
reduce systematic source-order bias, the first source alternates by query
226+
batch:
227+
228+
| Batch | Queries | Execution order |
229+
| --- | --- | --- |
230+
| 01 | Q1–Q10 | Paimon, Parquet |
231+
| 02 | Q11–Q20 | Parquet, Paimon |
232+
| 03 | Q21–Q30 | Paimon, Parquet |
233+
| 04 | Q31–Q40 | Parquet, Paimon |
234+
| 05 | Q41–Q50 | Paimon, Parquet |
235+
| 06 | Q51–Q60 | Parquet, Paimon |
236+
| 07 | Q61–Q66 | Paimon, Parquet |
237+
| 08 | Q68–Q77 | Parquet, Paimon |
238+
| 09 | Q79–Q88 | Paimon, Parquet |
239+
| 10 | Q89–Q96 | Parquet, Paimon |
240+
| 11 | Q98–Q99 | Paimon, Parquet |
241+
242+
### Measured commands
243+
244+
The following functions reproduce the arguments used for every normal batch;
245+
set the two root paths for the local environment:
246+
247+
```bash
248+
TPCDS_ROOT=/path/to/tpcds-benchmark/sf100
249+
QUERY_ROOT=/path/to/delta-rs/crates/benchmarks/queries/tpcds
250+
251+
run_paimon() {
252+
target/release/paimon-tpcds-bench run \
253+
--source paimon \
254+
--warehouse "${TPCDS_ROOT}/paimon-layout-target1-full" \
255+
--database tpcds \
256+
--queries "${QUERY_ROOT}" \
257+
--output "paimon-publication-batch-$1.json" \
258+
--query "$2" \
259+
--warmup 1 \
260+
--iterations 3 \
261+
--target-partitions 12 \
262+
--memory-limit-gib 32
263+
}
264+
265+
run_parquet() {
266+
target/release/paimon-tpcds-bench run \
267+
--source parquet \
268+
--data "${TPCDS_ROOT}/parquet" \
269+
--warehouse "${TPCDS_ROOT}/parquet-catalog-review" \
270+
--database tpcds \
271+
--queries "${QUERY_ROOT}" \
272+
--output "parquet-publication-batch-$1.json" \
273+
--query "$2" \
274+
--warmup 1 \
275+
--iterations 3 \
276+
--target-partitions 12 \
277+
--parquet-pushdown-filters \
278+
--memory-limit-gib 32
279+
}
280+
281+
run_paimon 01 1-10; run_parquet 01 1-10
282+
run_parquet 02 11-20; run_paimon 02 11-20
283+
run_paimon 03 21-30; run_parquet 03 21-30
284+
run_parquet 04 31-40; run_paimon 04 31-40
285+
run_paimon 05 41-50; run_parquet 05 41-50
286+
run_parquet 06 51-60; run_paimon 06 51-60
287+
run_paimon 07 61-66; run_parquet 07 61-66
288+
run_parquet 08 68-77; run_paimon 08 68-77
289+
run_paimon 09 79-88; run_parquet 09 79-88
290+
run_parquet 10 89-96; run_paimon 10 89-96
291+
run_paimon 11 98-99; run_parquet 11 98-99
292+
```
293+
294+
Q67, Q78, and Q97 were each run in a fresh process with the same source
295+
arguments and runtime settings, replacing the batch arguments with
296+
`--query <query> --warmup 0 --iterations 1`. Both sources hit the 32 GiB
297+
memory-pool limit for all three queries.
298+
299+
## Interpretation limits
300+
301+
This result is one machine, one data layout, and a warm-cache protocol. It is
302+
useful as a reproducible engineering checkpoint, not a universal performance
303+
ranking. A broader study should repeat complete benchmark processes, separate
304+
cold and warm cache states, and include object storage and multi-node
305+
environments.

‎docs/src/index.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,15 @@ Key features:
3232
- REST Catalog integration
3333
- Apache DataFusion integration for SQL queries
3434

35+
## Performance
36+
37+
In the TPC-DS SF100 benchmark, Paimon Rust completes the 96-query workload
38+
within 9.3% of DataFusion's native Parquet reader. The gap narrows to 3.1% for
39+
queries whose native-Parquet median is at least five seconds, while Paimon's
40+
ZSTD data files use 23.25% less physical storage than the SNAPPY source files.
41+
42+
[Read the full TPC-DS SF100 performance report](benchmark.md)
43+
3544
## Status
3645

3746
The project is under active development (0.3.0 in development).

0 commit comments

Comments
 (0)