docs: add TPC-DS SF100 performance report

[JingsongLi]

Summary

Publish an auditable TPC-DS SF100 read-performance report comparing Paimon Rust with DataFusion's native Parquet reader. Across the 96 queries that complete under the shared 32 GiB runtime limit, Paimon finishes the workload within 9.3% of native Parquet; the gap is 3.1% for queries whose native-Parquet median is at least five seconds.

The source Parquet files use SNAPPY, while the Paimon rewrite uses ZSTD and occupies 23.25% less physical data-file storage. The timing comparison therefore includes the higher-compression codec's additional decompression CPU cost as well as its storage and I/O savings.

Changes

• replace the standalone HTML article with an English Markdown page integrated into the MkDocs theme, navigation, and documentation homepage
• summarize the validated measurements and protocol without committing the large per-iteration JSON reports
• document the exact benchmark revision, binary checksum, query revision, dataset layout, machine, cache policy, alternating source order, commands, and aggregation method
• document the SNAPPY-to-ZSTD compression change: Paimon data files use 27,975,615,232 bytes versus 36,449,466,908 bytes for the source, a 23.25% reduction, with more CPU-intensive decompression
• describe the StarRocks, Apache Doris, and Milvus integration directions, including the materialized vector-search C FFI path for Milvus
• scope the conclusion to the tested unpartitioned append-only Parquet-backed read path and describe other Paimon capabilities without presenting them as measured

Testing

• cargo clean
• cargo build --release -p paimon-tpcds-bench
• run all 96 comparable queries for both sources with one warmup and three measured iterations
• run Q67, Q78, and Q97 separately for both sources and retain their resource-limit errors
• verify query IDs, iteration counts, zero normal-query errors, zero warmup failures, zero normal-query spills, and all 288 output-row-count comparisons
• verify all 24 source files use SNAPPY and all 125 Paimon data files use ZSTD from their Parquet metadata
• recalculate the published workload medians from the local benchmark reports
• cargo fmt --all -- --check
• python3 scripts/validate_asf_yaml.py
• cd docs && mkdocs build --strict

Notes

This is a TPC-DS-derived non-TPC benchmark, not an official TPC result. Q67, Q78, and Q97 are excluded from the aggregate because both sources exhaust the same 32 GiB DataFusion memory pool.

[leaves12138]

The standalone page builds correctly, but the public performance conclusion needs a stronger evidence chain before publication. I found three blocking issues around measurement stability, reproducibility, and benchmark scope.

[leaves12138]

Blocking: this result uses 0 warmups and a single measured iteration, while benchmarks/tpcds/README.md's published procedure uses --warmup 1 --iterations 3. The headline gap is only 1.39 seconds over roughly 515 seconds (0.27%), which can easily be smaller than run-to-run or cache noise; one observation cannot establish parity. Please rerun both sources for multiple measured iterations under the same cache policy (ideally alternating source order), publish an aggregate such as the median plus dispersion, and derive the headline numbers from that dataset.

[leaves12138]

Blocking: the page labels this a locally reproducible experiment, but the PR contains neither raw benchmark JSON nor enough metadata to reproduce or verify the numbers. The benchmark README explicitly requires the exact paimon-rust commit, generated-data scale and generator version, file counts and physical bytes, OS/CPU/memory/storage model, and exact commands for every published report. Please add those details and link or commit the two raw result files so reviewers can verify the totals, failed queries, and the 30/96 count.

[leaves12138]

Blocking accuracy issue: this benchmark harness imports the data into unpartitioned append-only Paimon tables (benchmarks/tpcds/README.md), so the run does not exercise primary-key merging, schema evolution, deletion vectors, row lineage, or multiple file formats. This sentence currently presents the result as evidence that those semantics carry no material performance penalty. Please scope the conclusion to the append-only SF100 read path tested here, and describe the broader features as capabilities rather than benchmarked outcomes.

[leaves12138]

The previous blocking concerns have been addressed. The benchmark protocol, reproducibility details, and scope are now sufficiently documented. The uncommitted raw JSON is acceptable given its size; a compact summary artifact would be a non-blocking future improvement.