Skip to content

Commit 656f54d

Browse files
tae898tae898
committed
docs: update heap size recommendations in benchmark scripts for clarity
1 parent 32968be commit 656f54d

2 files changed

Lines changed: 148 additions & 2 deletions

File tree

‎bindings/python/examples/benchmark-vector/README.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,154 @@
1111
- Take the duration with a grain of salt, since there are other processes running on the machine. RSS and DB size are more stable. 4 threads were allocated per task, but there aren't always the same number of tasks runing in parallel, so effective CPU usage may vary.
1212
- If not mentioned, `MAX_CONNECTIONS` is fixed as 12, `BEAM_WIDTHS` as 64, and `OVERQUERY_FACTORS` as 1
1313

14+
### Commit/Date: main @ a75817367 (2026-01-28 10:11:01 +0100)
15+
16+
We ran the same 1024-dimensional MSMARCO vectors with `INT8` quantization.
17+
Config (all runs): `store_vectors_in_graph=False`, `add_hierarchy=True`,
18+
`max_connections=16`, `beam_width=100`, `overquery_factor=1`, `batch_size=10000`.
19+
These runs include both index build and search timings.
20+
21+
**Index build (INT8, 1000 queries, Recall@50):**
22+
23+
#### MSMARCO-1M
24+
25+
| heap | ingest_s | peak_rss_mb | db_size_mb | total_duration |
26+
| :--- | -------: | ----------: | ---------: | :------------- |
27+
| 4g | 66.278 | 4460.43 | 6753.94 | 31m 49.438s |
28+
29+
#### MSMARCO-2M
30+
31+
| heap | ingest_s | peak_rss_mb | db_size_mb | total_duration |
32+
| :--- | -------: | ----------: | ---------: | :------------- |
33+
| 8g | 130.745 | 8624.22 | 13507.5 | 1h 0m 51.475s |
34+
35+
#### MSMARCO-4M
36+
37+
| heap | ingest_s | peak_rss_mb | db_size_mb | total_duration |
38+
| :--- | -------: | ----------: | ---------: | :------------- |
39+
| 16g | 298.401 | 16956.9 | 27014.9 | 2h 23m 31.456s |
40+
41+
#### MSMARCO-8M
42+
43+
| heap | ingest_s | peak_rss_mb | db_size_mb | total_duration |
44+
| :--- | -------: | ----------: | ---------: | :------------- |
45+
| 32g | 620.635 | 33592.9 | 54024.1 | 5h 42m 31.988s |
46+
47+
**Lessons learned:**
48+
49+
- 1M: at least 4G
50+
- 2M: at least 8G
51+
- 4M: at least 16G
52+
- 8M: at least 32G
53+
54+
These are near bare minimums; lower heaps may work if vector dimensions are reduced.
55+
56+
**Search study (INT8, 1000 queries, Recall@50):**
57+
58+
#### MSMARCO-1M
59+
60+
| heap | overquery_factor | recall@50 | latency_ms_mean | latency_ms_p95 | search_s | peak_rss_mb | db_size_mb | total_duration (s) |
61+
| :--- | ---------------: | --------: | --------------: | -------------: | -------: | ----------: | ---------: | -----------------: |
62+
| 1g | 16 | 0.993 | 387 | 713 | 387 | 1300 | 6753 | 393 |
63+
| 1g | 8 | 0.990 | 216 | 391 | 217 | 1274 | 6753 | 223 |
64+
| 1g | 4 | 0.985 | 131 | 239 | 132 | 1259 | 6753 | 137 |
65+
| 1g | 2 | 0.973 | 86 | 172 | 86 | 1249 | 6753 | 92 |
66+
| 1g | 1 | 0.942 | 60 | 141 | 61 | 1236 | 6753 | 68 |
67+
| 2g | 16 | 0.993 | 216 | 384 | 216 | 2337 | 6753 | 221 |
68+
| 2g | 8 | 0.990 | 133 | 252 | 133 | 2303 | 6753 | 139 |
69+
| 2g | 4 | 0.985 | 82 | 173 | 83 | 2284 | 6753 | 88 |
70+
| 2g | 2 | 0.973 | 53 | 119 | 53 | 2269 | 6753 | 58 |
71+
| 2g | 1 | 0.942 | 34 | 80 | 35 | 2252 | 6753 | 43 |
72+
| 4g | 16 | 0.993 | 90 | 169 | 90 | 4400 | 6753 | 95 |
73+
| 4g | 8 | 0.990 | 53 | 117 | 54 | 4369 | 6753 | 59 |
74+
| 4g | 4 | 0.985 | 31 | 80 | 31 | 4348 | 6753 | 36 |
75+
| 4g | 2 | 0.973 | 19 | 61 | 2 | 4330 | 6753 | 25 |
76+
| 4g | 1 | 0.942 | 12 | 42 | 13 | 4312 | 6753 | 19 |
77+
| 8g | 16 | 0.993 | 32 | 48 | 32 | 8570 | 6753 | 37 |
78+
| 8g | 8 | 0.990 | 17 | 25 | 17 | 8540 | 6753 | 21 |
79+
| 8g | 4 | 0.985 | 9 | 13 | 9 | 8522 | 6753 | 14 |
80+
| 8g | 2 | 0.973 | 5 | 6 | 5 | 8510 | 6753 | 9 |
81+
| 8g | 1 | 0.942 | 3 | 6 | 3 | 8456 | 6753 | 10 |
82+
83+
#### MSMARCO-2M
84+
85+
| heap | overquery_factor | recall@50 | latency_ms_mean | latency_ms_p95 | search_s | peak_rss_mb | db_size_mb | total_duration (s) |
86+
| :--- | ---------------: | --------: | --------------: | -------------: | -------: | ----------: | ---------: | -----------------: |
87+
| 1g | 16 | 0.991 | 897 | 1577 | 898 | 1215 | 13507 | 916 |
88+
| 1g | 8 | 0.987 | 464 | 827 | 465 | 1278 | 13507 | 483 |
89+
| 1g | 4 | 0.982 | 277 | 490 | 278 | 1277 | 13507 | 296 |
90+
| 1g | 2 | 0.971 | 167 | 342 | 167 | 1265 | 13507 | 180 |
91+
| 1g | 1 | 0.940 | 9 | 229 | 96 | 1255 | 13507 | 109 |
92+
| 2g | 16 | 0.991 | 511 | 849 | 511 | 2355 | 13507 | 528 |
93+
| 2g | 8 | 0.987 | 280 | 492 | 280 | 2325 | 13507 | 298 |
94+
| 2g | 4 | 0.982 | 167 | 336 | 167 | 2305 | 13507 | 184 |
95+
| 2g | 2 | 0.971 | 101 | 234 | 101 | 2286 | 13507 | 119 |
96+
| 2g | 1 | 0.940 | 69 | 204 | 69 | 2267 | 13507 | 94 |
97+
| 4g | 16 | 0.991 | 319 | 551 | 319 | 4407 | 13507 | 339 |
98+
| 4g | 8 | 0.987 | 181 | 321 | 181 | 4377 | 13507 | 200 |
99+
| 4g | 4 | 0.982 | 118 | 228 | 119 | 4357 | 13507 | 138 |
100+
| 4g | 2 | 0.971 | 71 | 148 | 72 | 4338 | 13507 | 90 |
101+
| 4g | 1 | 0.940 | 43 | 114 | 43 | 4323 | 13507 | 65 |
102+
| 8g | 16 | 0.991 | 135 | 263 | 135 | 8561 | 13507 | 151 |
103+
| 8g | 8 | 0.987 | 80 | 184 | 81 | 8530 | 13507 | 98 |
104+
| 8g | 4 | 0.982 | 46 | 126 | 47 | 8511 | 13507 | 64 |
105+
| 8g | 2 | 0.971 | 27 | 56 | 28 | 8498 | 13507 | 45 |
106+
| 8g | 1 | 0.940 | 15 | 30 | 16 | 8488 | 13507 | 38 |
107+
108+
#### MSMARCO-4M
109+
110+
| heap | overquery_factor | recall@50 | latency_ms_mean | latency_ms_p95 | search_s | peak_rss_mb | db_size_mb | total_duration (s) |
111+
| :--- | ---------------: | --------: | --------------: | -------------: | -------: | ----------: | ---------: | -----------------: |
112+
| 1g | 16 | 0.99 | 2084 | 3747 | 2084 | 1214 | 27014 | 2135 |
113+
| 1g | 8 | 0.985 | 1276 | 2195 | 1277 | 1193 | 27014 | 1322 |
114+
| 1g | 4 | 0.976 | 686 | 1206 | 686 | 1192 | 27014 | 734 |
115+
| 1g | 2 | 0.963 | 443 | 824 | 443 | 1237 | 27014 | 488 |
116+
| 1g | 1 | 0.938 | 269 | 495 | 270 | 1233 | 27014 | 312 |
117+
| 2g | 16 | 0.99 | 534 | 910 | 535 | 2373 | 27014 | 566 |
118+
| 2g | 8 | 0.985 | 297 | 505 | 298 | 2346 | 27014 | 330 |
119+
| 2g | 4 | 0.976 | 179 | 358 | 179 | 2331 | 27014 | 207 |
120+
| 2g | 2 | 0.963 | 106 | 275 | 107 | 2320 | 27014 | 131 |
121+
| 2g | 1 | 0.938 | 66 | 222 | 67 | 2308 | 27014 | 103 |
122+
| 4g | 16 | 0.99 | 385 | 681 | 385 | 4453 | 27014 | 411 |
123+
| 4g | 8 | 0.985 | 208 | 422 | 209 | 4423 | 27014 | 235 |
124+
| 4g | 4 | 0.976 | 129 | 320 | 129 | 4403 | 27014 | 156 |
125+
| 4g | 2 | 0.963 | 78 | 160 | 78 | 4385 | 27014 | 106 |
126+
| 4g | 1 | 0.938 | 49 | 109 | 49 | 4363 | 27014 | 85 |
127+
| 8g | 16 | 0.99 | 279 | 446 | 280 | 8603 | 27014 | 307 |
128+
| 8g | 8 | 0.985 | 166 | 285 | 166 | 8572 | 27014 | 193 |
129+
| 8g | 4 | 0.976 | 101 | 204 | 102 | 8549 | 27014 | 127 |
130+
| 8g | 2 | 0.963 | 62 | 148 | 63 | 8530 | 27014 | 89 |
131+
| 8g | 1 | 0.938 | 37 | 103 | 37 | 8514 | 27014 | 71 |
132+
133+
#### MSMARCO-8M
134+
135+
| heap | overquery_factor | recall@50 | latency_ms_mean | latency_ms_p95 | search_s | peak_rss_mb | db_size_mb | total_duration (s) |
136+
| :--- | ---------------: | --------: | --------------: | -------------: | -------: | ----------: | ---------: | -----------------: |
137+
| 2g | 16 | 0.980 | 1210 | 1975 | 1211 | 2385 | 54024 | 1270 |
138+
| 2g | 8 | 0.974 | 716 | 1172 | 716 | 2343 | 54024 | 834 |
139+
| 2g | 4 | 0.966 | 435 | 744 | 436 | 2317 | 54024 | 491 |
140+
| 2g | 2 | 0.947 | 272 | 428 | 272 | 2305 | 54024 | 327 |
141+
| 2g | 1 | 0.911 | 173 | 372 | 173 | 2296 | 54024 | 242 |
142+
| 4g | 16 | 0.980 | 359 | 625 | 359 | 4491 | 54024 | 406 |
143+
| 4g | 8 | 0.974 | 201 | 435 | 202 | 4465 | 54024 | 246 |
144+
| 4g | 4 | 0.966 | 123 | 352 | 123 | 4442 | 54024 | 169 |
145+
| 4g | 2 | 0.947 | 74 | 294 | 75 | 4422 | 54024 | 133 |
146+
| 4g | 1 | 0.911 | 47 | 247 | 47 | 4411 | 54024 | 112 |
147+
| 8g | 16 | 0.980 | 267 | 558 | 267 | 8650 | 54024 | 309 |
148+
| 8g | 8 | 0.974 | 140 | 311 | 140 | 8621 | 54024 | 180 |
149+
| 8g | 4 | 0.966 | 85 | 183 | 85 | 8600 | 54024 | 127 |
150+
| 8g | 2 | 0.947 | 54 | 137 | 54 | 8582 | 54024 | 96 |
151+
| 8g | 1 | 0.911 | 34 | 100 | 35 | 8548 | 54024 | 79 |
152+
153+
**Lesson learned:**
154+
155+
- Vector search needs less heap than index build.
156+
- 1M: 1G works; at least 1G recommended.
157+
- 2M: 1G works; at least 2G recommended.
158+
- 4M: 1G works; at least 2G recommended.
159+
- 8M: 1G OOM; 2G works; at least 4G recommended.
160+
- Build on a more powerful machine; search deployment can run on less powerful hardware.
161+
14162
### Commit/Date: main @ 6ef8858 (Thu Jan 15 16:40:51 2026 -0500)
15163

16164
- This commit adds Product Quantization (PQ) support to JVector index.

‎bindings/python/examples/benchmark-vector/run_arcadedb_sweep.sh‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,14 +33,12 @@ HEAP_SIZES=()
3333
# 2M: at least 8G
3434
# 4M: at least 16G
3535
# 8M: at least 32G
36-
# 16M: 32G tesing ... I wish I can do 64G but not available right now.
3736

3837
# ofc vector search doesn't need as much heap as building the vector index.
3938
# 1M: 1g works, at least 1g recommended
4039
# 2M: 1g works, at least 2g recommended
4140
# 4M: 1g works, at least 2g recommended
4241
# 8M: 1g doesn't work (OoM), 2g works, at least 4g recommended
43-
# 16M: testing
4442

4543
if [[ -z "$PRESET_JVM_ARGS" ]]; then
4644
# Default heap sweep per dataset (override by setting HEAP_SIZES="8g,12g,16g")

0 commit comments

Comments
 (0)