Skip to content

Commit ac4353b

Browse files
Ignacio Van Droogenbroeckclaude
andcommitted
Update benchmark docs with corrected ClickBench results
- Update cold run: 36.43s → 120.25s (with proper cache flushing) - Add warm run results: 35.70s - Update comparison table: Focus on QuestDB and TimescaleDB - Remove VictoriaLogs (not time-series DB) - Update performance ratios: 1.80x faster than QuestDB, 9.39x faster than TimescaleDB - Add storage efficiency comparisons - Replace detailed query table with summary statistics - Update GitHub link to corrected results file - Emphasize ClickBench compliance (cache flushing, query caching disabled) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 9e400f8 commit ac4353b

1 file changed

Lines changed: 32 additions & 76 deletions

File tree

docs/performance/benchmarks.md

Lines changed: 32 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,21 @@ Arc has been benchmarked using [ClickBench](https://github.com/ClickHouse/ClickB
88

99
## ClickBench Results
1010

11-
**Arc is the fastest time-series database**, completing all 43 queries on ClickBench in **107.66 seconds total** (36.43s cold run).
11+
**Arc is the fastest time-series database**, completing all 43 queries on ClickBench with **120.25s cold run** and **35.70s warm run**.
1212

1313
### Official Rankings (Cold Run)
1414

1515
| Database | Cold Run | vs Arc | Architecture |
1616
|----------|----------|--------|--------------|
17-
| **Arc** | **36.43s** | **1.0x** | DuckDB + Parquet + HTTP API |
18-
| VictoriaLogs | 113.8s | 3.1x slower | LogsQL engine |
19-
| QuestDB | 223.2s | 6.1x slower | Columnar time-series |
20-
| Timescale Cloud | 626.6s | 17.2x slower | PostgreSQL extension |
21-
| TimescaleDB | 1022.5s | 28.1x slower | PostgreSQL extension |
17+
| **Arc** | **120.25s** | **1.0x** | DuckDB + Parquet + HTTP API |
18+
| QuestDB | 216.30s | 1.80x slower | Columnar time-series |
19+
| TimescaleDB | 1,128.87s | 9.39x slower | PostgreSQL extension |
2220

2321
:::tip Key Achievements
24-
- **Cold Run**: 36.43s (fastest cold start among time-series databases)
25-
- **Complete Benchmark**: 107.66s total for all 43 queries (3 runs each)
26-
- **Arc is the only time-series database to complete ClickBench in under 2 minutes via HTTP API**
22+
- **Cold Run**: 120.25s (fastest cold start among time-series databases)
23+
- **Warm Run**: 35.70s (with proper filesystem cache)
24+
- **1.80x faster than QuestDB**, 9.39x faster than TimescaleDB
25+
- **5.29x less storage** than QuestDB (13.76 GB vs 72.84 GB)
2726
:::
2827

2928
## Test Environment
@@ -53,60 +52,19 @@ Arc has been benchmarked using [ClickBench](https://github.com/ClickHouse/ClickB
5352

5453
## ClickBench Query Results
5554

56-
All 43 analytical queries completed successfully. Results show **3 runs** per query:
57-
58-
### Query Performance (seconds)
59-
60-
| Query | Run 1 | Run 2 | Run 3 | Best | Notes |
61-
|-------|-------|-------|-------|------|-------|
62-
| Q0 | 0.3385 | 0.2606 | 0.3211 | 0.2606 | Simple aggregation |
63-
| Q1 | 0.5951 | 0.5608 | 0.5928 | 0.5608 | COUNT with GROUP BY |
64-
| Q2 | 0.5631 | 0.3030 | 0.1436 | 0.1436 | Aggregation with filter |
65-
| Q3 | 0.1057 | 0.0905 | 0.0764 | 0.0764 | Fast filter scan |
66-
| Q4 | 0.3500 | 0.3335 | 0.3234 | 0.3234 | Multiple GROUP BY |
67-
| Q5 | 0.5696 | 0.5386 | 0.5515 | 0.5386 | Complex aggregation |
68-
| Q6 | 0.0598 | 0.0590 | 0.0593 | 0.0590 | Selective filter |
69-
| Q7 | 0.0564 | 0.0548 | 0.0550 | 0.0548 | Simple scan |
70-
| Q8 | 0.4429 | 0.4716 | 0.4487 | 0.4429 | JOIN operation |
71-
| Q9 | 0.5682 | 0.5676 | 0.5602 | 0.5602 | Heavy aggregation |
72-
| Q10 | 0.1413 | 0.1428 | 0.1408 | 0.1408 | String operations |
73-
| Q11 | 0.1875 | 0.2139 | 0.1815 | 0.1815 | Complex filter |
74-
| Q12 | 0.5742 | 0.5466 | 0.5648 | 0.5466 | Window functions |
75-
| Q13 | 0.9176 | 0.8787 | 0.8699 | 0.8699 | Multiple JOINs |
76-
| Q14 | 0.5764 | 0.5977 | 0.6207 | 0.5764 | Subqueries |
77-
| Q15 | 0.3892 | 0.4011 | 0.4074 | 0.3892 | DISTINCT operations |
78-
| Q16 | 1.0798 | 1.0383 | 1.0153 | 1.0153 | Heavy computation |
79-
| Q17 | 0.7985 | 0.7727 | 0.7853 | 0.7727 | String matching |
80-
| Q18 | 3.3340 | 3.3020 | 3.3478 | 3.3020 | Complex analytics |
81-
| Q19 | 0.0757 | 0.0683 | 0.0570 | 0.0570 | Simple filter |
82-
| Q20 | 1.0360 | 0.9106 | 0.9079 | 0.9079 | Aggregation pipeline |
83-
| Q21 | 0.8482 | 0.8400 | 0.8520 | 0.8400 | GROUP BY with filter |
84-
| Q22 | 1.7228 | 1.6782 | 1.7208 | 1.6782 | Multiple aggregations |
85-
| Q23 | 0.5097 | 0.5317 | 0.5237 | 0.5097 | Complex WHERE |
86-
| Q24 | 0.1973 | 0.2058 | 0.2073 | 0.1973 | Simple aggregation |
87-
| Q25 | 0.3004 | 0.2941 | 0.2923 | 0.2923 | String operations |
88-
| Q26 | 0.1375 | 0.1461 | 0.1384 | 0.1375 | Fast lookup |
89-
| Q27 | 0.9975 | 0.9866 | 0.9847 | 0.9847 | Complex JOIN |
90-
| Q28 | 9.1263 | 9.1334 | 9.1713 | 9.1263 | Heavy analytics |
91-
| Q29 | 0.0787 | 0.0802 | 0.0787 | 0.0787 | Simple filter |
92-
| Q30 | 0.7854 | 0.6878 | 0.5742 | 0.5742 | Scan with filter |
93-
| Q31 | 0.6781 | 0.6799 | 0.6920 | 0.6781 | Aggregation |
94-
| Q32 | 1.9562 | 1.9239 | 1.9322 | 1.9239 | Window functions |
95-
| Q33 | 2.3368 | 2.2877 | 2.3325 | 2.2877 | Complex analytics |
96-
| Q34 | 2.3724 | 2.3640 | 2.3611 | 2.3611 | Multiple GROUP BY |
97-
| Q35 | 0.5792 | 0.7450 | 0.5765 | 0.5765 | Aggregation pipeline |
98-
| Q36 | 0.1609 | 0.1560 | 0.1666 | 0.1560 | Simple scan |
99-
| Q37 | 0.1366 | 0.1455 | 0.1282 | 0.1282 | Fast filter |
100-
| Q38 | 0.1007 | 0.1072 | 0.0992 | 0.0992 | Selective scan |
101-
| Q39 | 0.2687 | 0.2780 | 0.2750 | 0.2687 | String operations |
102-
| Q40 | 0.0651 | 0.0633 | 0.0686 | 0.0633 | Simple lookup |
103-
| Q41 | 0.0757 | 0.0642 | 0.0626 | 0.0626 | Fast aggregation |
104-
| Q42 | 0.2365 | 0.2269 | 0.2251 | 0.2251 | Final query |
105-
106-
**Total Cold Run (First Run)**: 36.43s
107-
**Average Query Time**: 0.847s
108-
**Fastest Query**: 0.0548s (Q7)
109-
**Slowest Query**: 9.1263s (Q28)
55+
All 43 analytical queries completed successfully with proper cache flushing compliance:
56+
57+
**Performance Summary:**
58+
- **Total Cold Run**: 120.25s
59+
- **Total Warm Run**: 35.70s
60+
- **Cold/Warm Ratio**: 3.37x (proper cache flushing verification)
61+
- **Storage Size**: 13.76 GB (Parquet with Snappy compression)
62+
63+
**Query Execution:**
64+
- All 43 queries passed successfully
65+
- Tested via HTTP REST API with JSON output format
66+
- Filesystem cache properly flushed before cold runs
67+
- Query result caches disabled per ClickBench compliance
11068

11169
## Why Arc is Fast
11270

@@ -167,22 +125,20 @@ Arc excels at:
167125

168126
## Comparison with Other Databases
169127

170-
### vs VictoriaLogs (3.1x faster)
171-
172-
- **Architecture**: LogsQL vs SQL
173-
- **Storage**: Custom format vs Parquet
174-
- **Query interface**: Custom API vs HTTP REST + SQL
175-
176-
### vs QuestDB (6.1x faster)
128+
### vs QuestDB (1.80x faster cold, 1.20x faster warm)
177129

178130
- **Architecture**: Custom columnar engine vs DuckDB
179-
- **Storage**: Proprietary vs Parquet
180-
- **Query complexity**: Limited SQL vs Full SQL
131+
- **Storage**: Proprietary (72.84 GB) vs Parquet (13.76 GB)
132+
- **Cold Run**: Arc 120.25s vs QuestDB 216.30s
133+
- **Warm Run**: Arc 35.70s vs QuestDB 42.70s
134+
- **Storage Efficiency**: Arc uses 5.29x less storage
181135

182-
### vs TimescaleDB (28.1x faster)
136+
### vs TimescaleDB (9.39x faster cold, 12.39x faster warm)
183137

184138
- **Architecture**: PostgreSQL extension vs Purpose-built
185-
- **Storage**: Row-based vs Columnar
139+
- **Storage**: Row-based (19.30 GB) vs Columnar (13.76 GB)
140+
- **Cold Run**: Arc 120.25s vs TimescaleDB 1,128.87s
141+
- **Warm Run**: Arc 35.70s vs TimescaleDB 442.34s
186142
- **Query engine**: General-purpose vs Analytics-optimized
187143

188144
## Write Performance
@@ -248,8 +204,8 @@ All benchmarks are reproducible. See [Running Benchmarks](/arc/performance/runni
248204

249205
### Download Results
250206

251-
- [c6a.4xlarge_cache_disabled.json](https://github.com/Basekick-Labs/ClickBench/blob/main/arc/results/c6a.4xlarge_cache_disabled.json)
252-
- [Full ClickBench Results](https://clickhouse.com/benchmark)
207+
- [c6a.4xlarge.json](https://github.com/Basekick-Labs/ClickBench/blob/main/arc/results/c6a.4xlarge.json) - Corrected results with proper cache flushing
208+
- [Full ClickBench Results](https://benchmark.clickhouse.com)
253209

254210
## What This Means
255211

0 commit comments

Comments
 (0)