Skip to content

Commit 2488fac

Browse files
tae898tae898
committed
feat: Add detailed benchmarking results for MSMARCO-1M queries
1 parent cf9aab4 commit 2488fac

2 files changed

Lines changed: 41 additions & 0 deletions

File tree

‎bindings/python/examples/benchmark-vector/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Benchmarking Arcadedb's Vector Index Build and Search (LSM + JVector) Performance
2+
3+
## MSMARCO Dataset
4+
5+
- Data prepared with
6+
[convert-msmacro-parquet-to-shards.py](./convert-msmacro-parquet-to-shards.py).
7+
[Download Cohere MSMARCO
8+
v2.1](https://huggingface.co/datasets/Cohere/msmarco-v2.1-embed-english-v3) parquet
9+
shards, normalize to float32, write flat f32 shards, and build exact GT for 1K queries
10+
(top-50).
11+
- Benchmarks here use the 1M subset. For production/RAG we should target 10M+ vectors;
12+
GT and shards are already computed—ask if you want the bundle to rerun.
13+
14+
### Commit/Date: main @ d8098d7 (Wed Jan 14 15:20:25 2026 -0500)
15+
16+
#### MSMARCO-1M (1000 queries, Recall@50)
17+
18+
| quantization | store_vectors_in_graph | add_hierarchy | ingest_s | ingest_rss_mb | warmup_s | warmup_rss_mb | search_s | search_rss_mb | recall@50_before_close | open_db_s | warmup_after_reopen_s | search_after_reopen_s | recall@50_after_reopen | peak_rss_mb | db_size_mb | total_duration |
19+
| :----------- | :--------------------- | :------------ | -------: | ------------: | -------: | ------------: | -------: | ------------: | ---------------------: | --------: | --------------------: | --------------------: | ---------------------: | ----------: | ---------: | :------------- |
20+
| NONE | False | True | 70 | 8708 | 7139.74 | 152 | 6 | 17 | 0.9101 | 1 | 7 | 9 | 0.9101 | 9354 | 9650 | 2h 0m |
21+
| INT8 | False | False | 71 | 8825 | 3865.7 | 140 | 27 | 10 | 0.9072 | 15 | 5 | 65 | 0.9072 | 9458 | 10633 | 1h 7m |
22+
| NONE | True | False | 67 | 8699 | 6561.28 | 147 | 16 | 10 | 0.9085 | 4 | 13 | 58 | 0.9049 | 9352 | 9645 | 1h 52m |
23+
| NONE | False | False | 66 | 8707 | 6590.55 | 171 | 13 | 16 | 0.8994 | 3 | 13 | 23 | 0.8994 | 9380 | 9645 | 1h 51m |
24+
25+
##### Findings
26+
27+
- Memory: JVM heap capped at 8GB, yet RSS (Resident Set Size) peaks 9.3–9.5GB in all runs; forcing 4GB causes OOM. Even a 1M dataset pushes outside heap, suggesting off-heap/native graph build and mmap traffic dominate.
28+
- Storage: Each run writes ~1.0GB `*.lsmvecidx` + ~5.6GB bucket + ~3.9GB `*.vecgraph`; vectors are effectively stored twice (bucket + graph) because `store_vectors_in_graph=False` is ignored—LSMVectorIndexGraphFile still serializes inline vectors. This doubles disk and keeps RSS high when mapping the graph file.
29+
- Lazy build + rebuild: Graph is built only after the first search, so the first query does all construction (long warmup). Post-ingest mutations set `graphState=MUTABLE`, and the search path currently rebuilds on the very next query since it only checks `mutationsSinceSerialize>0`; the configured threshold (GlobalConfiguration default 100) is bypassed. Pure queries do not increment the counter, so 1,000 searches alone never trigger rebuilds.
30+
- Persistence: Close/reopen shows no rebuild because the Jan 14, 2026 engine fix now persists and reloads the graph successfully. The reopen warmup is mostly graph load, not rebuild.
31+
- Hierarchy: `add_hierarchy` raises build time modestly (~+9m: 2h00 vs. 1h51) but improves recall (0.9101 vs. 0.8994) and cuts search time materially (6s vs. 13–16s across 1K queries); likely fewer hops during graph search.
32+
- Quantization (int8): Ingest time drops sharply (1h07 vs. ~1h51–2h00) with comparable recall (0.9072 vs. 0.8994 baseline). However RSS does not improve and db size increases (10.6GB vs. 9.6GB), likely because vectors are duplicated in the graph and/or stored as float alongside the int8 quantized form.
33+
- JVector knobs: `MAX_CONNECTIONS=12` and `BEAM_WIDTH=64` held constant; higher will improve recall at higher build cost. JVector lacks `efSearch`, so overquery (>k then rerank) is the lever; overquery factor was 1 here to simplify results.

‎bindings/python/examples/benchmark-vector/summaries/arcadedb_msmarco_MSMARCO-1M.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# ArcadeDB MSMARCO MSMARCO-1M (1000 queries, Recall@50)
2+
3+
| quantization | store_vectors_in_graph | add_hierarchy | max_connections | beam_width | overquery_factor | load_corpus_s | load_corpus_rss_mb | ingest_s | ingest_rss_mb | build_s | build_rss_mb | warmup_s | warmup_rss_mb | search_s | search_rss_mb | recall@50_before_close | close_db_s | close_db_rss_mb | open_db_s | open_db_rss_mb | warmup_after_reopen_s | warmup_after_reopen_rss_mb | search_after_reopen_s | search_after_reopen_rss_mb | recall@50_after_reopen | peak_rss_mb | db_size_mb | total_duration |
4+
|:---------------|:-------------------------|:----------------|------------------:|-------------:|-------------------:|----------------:|---------------------:|-----------:|----------------:|----------:|---------------:|-----------:|----------------:|-----------:|----------------:|-------------------------:|-------------:|------------------:|------------:|-----------------:|------------------------:|-----------------------------:|------------------------:|-----------------------------:|-------------------------:|--------------:|-------------:|:-----------------|
5+
| NONE | False | True | 12 | 64 | 1 | 0 | 0 | 70.115 | 8708.92 | 1.003 | 46.355 | 7139.74 | 152.625 | 6.557 | 17.27 | 0.9101 | 0.015 | -0.113 | 1.937 | 0.254 | 7.22 | 1.102 | 9.971 | 16.68 | 0.9101 | 9354.03 | 9650.44 | 2h 0m 36.559s |
6+
| INT8 | False | False | 12 | 64 | 1 | 0 | 0 | 71.864 | 8825.92 | 1.387 | 54.582 | 3865.7 | 140.305 | 27.124 | 10.75 | 0.9072 | 0.104 | 0.176 | 15.561 | 4.301 | 5.088 | -0.152 | 65.328 | 11.852 | 0.9072 | 9458.73 | 10633.9 | 1h 7m 32.156s |
7+
| NONE | True | False | 12 | 64 | 1 | 0 | 0 | 67.068 | 8699.28 | 0.999 | 63.719 | 6561.28 | 147.508 | 16.731 | 10.727 | 0.9085 | 0.033 | 0.121 | 4.237 | 3.109 | 13.047 | -0.176 | 58.171 | 4.031 | 0.9049 | 9352.84 | 9645.44 | 1h 52m 1.563s |
8+
| NONE | False | False | 12 | 64 | 1 | 0 | 0 | 66.561 | 8707.15 | 0.863 | 40.184 | 6590.55 | 171.473 | 13.247 | 16.387 | 0.8994 | 0.029 | 0.062 | 3.569 | 1.121 | 13.58 | 0.938 | 23.175 | 31.762 | 0.8994 | 9380.1 | 9645.44 | 1h 51m 51.577s |

0 commit comments

Comments
 (0)