Skip to content

Commit d502cd2

Browse files
committed
docs: Update benchmarking findings in README with detailed analysis and next steps
1 parent 2488fac commit d502cd2

1 file changed

Lines changed: 29 additions & 7 deletions

File tree

  • bindings/python/examples/benchmark-vector

bindings/python/examples/benchmark-vector/README.md

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,32 @@
2424

2525
##### Findings
2626

27-
- Memory: JVM heap capped at 8GB, yet RSS (Resident Set Size) peaks 9.3–9.5GB in all runs; forcing 4GB causes OOM. Even a 1M dataset pushes outside heap, suggesting off-heap/native graph build and mmap traffic dominate.
28-
- Storage: Each run writes ~1.0GB `*.lsmvecidx` + ~5.6GB bucket + ~3.9GB `*.vecgraph`; vectors are effectively stored twice (bucket + graph) because `store_vectors_in_graph=False` is ignored—LSMVectorIndexGraphFile still serializes inline vectors. This doubles disk and keeps RSS high when mapping the graph file.
29-
- Lazy build + rebuild: Graph is built only after the first search, so the first query does all construction (long warmup). Post-ingest mutations set `graphState=MUTABLE`, and the search path currently rebuilds on the very next query since it only checks `mutationsSinceSerialize>0`; the configured threshold (GlobalConfiguration default 100) is bypassed. Pure queries do not increment the counter, so 1,000 searches alone never trigger rebuilds.
30-
- Persistence: Close/reopen shows no rebuild because the Jan 14, 2026 engine fix now persists and reloads the graph successfully. The reopen warmup is mostly graph load, not rebuild.
31-
- Hierarchy: `add_hierarchy` raises build time modestly (~+9m: 2h00 vs. 1h51) but improves recall (0.9101 vs. 0.8994) and cuts search time materially (6s vs. 13–16s across 1K queries); likely fewer hops during graph search.
32-
- Quantization (int8): Ingest time drops sharply (1h07 vs. ~1h51–2h00) with comparable recall (0.9072 vs. 0.8994 baseline). However RSS does not improve and db size increases (10.6GB vs. 9.6GB), likely because vectors are duplicated in the graph and/or stored as float alongside the int8 quantized form.
33-
- JVector knobs: `MAX_CONNECTIONS=12` and `BEAM_WIDTH=64` held constant; higher will improve recall at higher build cost. JVector lacks `efSearch`, so overquery (>k then rerank) is the lever; overquery factor was 1 here to simplify results.
27+
- Memory: JVM heap capped at 8GB, yet RSS (Resident Set Size) peaks 9.3–9.5GB in all
28+
runs; forcing 4GB causes OOM. Even a 1M dataset pushes outside heap, suggesting
29+
off-heap/native graph build and mmap traffic dominate.
30+
- Storage: Each run writes ~1.0GB `*.lsmvecidx` + ~5.6GB bucket + ~3.9GB `*.vecgraph`;
31+
vectors are effectively stored twice (bucket + graph) because
32+
`store_vectors_in_graph=False` is ignored—LSMVectorIndexGraphFile still serializes
33+
inline vectors. This doubles disk and keeps RSS high when mapping the graph file.
34+
- Lazy build + rebuild: Graph is built only after the first search, so the first query
35+
does all construction (long warmup). Post-ingest mutations set `graphState=MUTABLE`,
36+
and the search path currently rebuilds on the very next query since it only checks
37+
`mutationsSinceSerialize>0`; the configured threshold (GlobalConfiguration default
38+
100) is bypassed. Pure queries do not increment the counter, so 1,000 searches alone
39+
never trigger rebuilds.
40+
- Persistence: Close/reopen shows no rebuild because the Jan 14, 2026 engine fix now
41+
persists and reloads the graph successfully. The reopen warmup is mostly graph load,
42+
not rebuild.
43+
- Hierarchy: `add_hierarchy` raises build time modestly (~+9m: 2h00 vs. 1h51) but
44+
improves recall (0.9101 vs. 0.8994) and cuts search time materially (6s vs. 13–16s
45+
across 1K queries); likely fewer hops during graph search.
46+
- Quantization (int8): Ingest time drops sharply (1h07 vs. ~1h51–2h00) with comparable
47+
recall (0.9072 vs. 0.8994 baseline). However RSS does not improve and db size
48+
increases (10.6GB vs. 9.6GB), likely because vectors are duplicated in the graph
49+
and/or stored as float alongside the int8 quantized form.
50+
- JVector knobs: `MAX_CONNECTIONS=12` and `BEAM_WIDTH=64` held constant; higher will
51+
improve recall at higher build cost. JVector lacks `efSearch`, so overquery (>k then
52+
rerank) is the lever; overquery factor was 1 here to simplify results.
53+
- Next steps: The ArcadeDB team is looking into the vector duplication and
54+
rebuild-threshold issues. Once fixes land, we will rerun on the 1M set for
55+
verification and then move to the 10M benchmark.

0 commit comments

Comments
 (0)