Skip to content

Commit 4bbaf95

Browse files
varshith-Gitvarshith-Git
committed
feat: IVF auto-scaling, snapshot persistence, CI fixes, dynamic PyPI badge
- IVF centroid auto-scale: n_list=sqrt(N), n_probe=sqrt(n_list); VALORI_IVF_N_LIST/N_PROBE overrides - Fix encrypted/shred events bypassing the BLAKE3 audit committer - Fix created_at not removed on record delete - Cap batch_seen at 65536; remove dead wal_accumulator field - Snapshot now persists created_at (CRTS) and BM25 corpus (BCRP) sections - Dockerfile: add valori-mcp stub so docker build no longer fails (health check fix) - release.yml: fix flatten-dist same-file error with inode-based cp guard - deny.toml: ignore RUSTSEC-2026-0190 (anyhow unsound, no patch yet) - README: replace ASCII art with Mermaid architecture diagram - valoricore_readme.md: switch version badge to dynamic PyPI badge
1 parent 42c7684 commit 4bbaf95

35 files changed

Lines changed: 2000 additions & 399 deletions

‎.github/workflows/release.yml‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -145,16 +145,18 @@ jobs:
145145

146146
- name: Flatten dist directory
147147
run: |
148+
# download-artifact@v4 nests each artifact as dist/<artifact-name>/<file>.
149+
# When the artifact name matches the file name the src and dst are the
150+
# same inode — skip the copy in that case to avoid the "same file" error.
148151
find dist/ -mindepth 2 -type f | while read f; do
149152
dest="dist/$(basename "$f")"
150-
# realpath comparison avoids the "same file" error when the artifact
151-
# subdirectory name matches the filename (download-artifact@v4 nests
152-
# artifacts as dist/<name>/<name> when they share the same name).
153-
if [ "$(realpath "$f")" != "$(realpath "$dest" 2>/dev/null || echo "")" ]; then
154-
mv "$f" "$dest"
153+
src_inode=$(stat -c '%i' "$f" 2>/dev/null || stat -f '%i' "$f")
154+
dst_inode=$(stat -c '%i' "$dest" 2>/dev/null || stat -f '%i' "$dest" 2>/dev/null || echo "none")
155+
if [ "$src_inode" != "$dst_inode" ]; then
156+
cp "$f" "$dest"
155157
fi
156158
done
157-
find dist/ -mindepth 1 -maxdepth 1 -type d -empty -delete
159+
find dist/ -mindepth 1 -maxdepth 1 -type d -delete
158160
159161
- name: Install cosign
160162
uses: sigstore/cosign-installer@v3

‎CHANGELOG.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,29 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
66

77
## [Unreleased]
88

9+
### Fixed
10+
- **Snapshot `CapacityExceeded` at scale** — `encode_state` rewritten from a
11+
fixed `&mut [u8]` buffer to a growable `&mut Vec<u8>`. Snapshots above ~250K
12+
records (any dimension) previously failed with `Kernel(CapacityExceeded)`
13+
because the V6 schema added 10 bytes/record that the buffer-size formula did
14+
not account for. Verified end-to-end at 1M records (515 MB snapshot in 1.2 s).
15+
The encoder is now structurally incapable of this error. Stays `no_std`.
16+
- **WAL loss on clean teardown** — added `impl Drop for Engine` and
17+
`impl Drop for EventCommitter` to flush the batched write buffer on scope
18+
exit. A clean shutdown could previously lose up to `flush_every` buffered
19+
events; recovery tests found 0 events after a simulated crash.
20+
21+
### Added
22+
- **IVF centroid auto-scaling** (`n_list = max(16, sqrt(N))`, `n_probe = max(1, sqrt(n_list))`) — fixes a 153× QPS regression from 10K to 1M records. Centroids now scale with dataset size so average bucket size stays O(sqrt(N)) and scan cost is O(sqrt(N)) not O(N). Manual override via `VALORI_IVF_N_LIST` / `VALORI_IVF_N_PROBE` disables auto-scaling. Added `IvfIndex::needs_rebuild(count)` hook (returns true when online inserts exceed 2× the build size).
23+
- **`encode_capacity_hint(state)`** — V6-correct pre-allocation estimate so the
24+
snapshot `Vec` avoids repeated reallocation on the hot path.
25+
- **SIMD L2 distance** (`l2_sq_i32`) — NEON (aarch64) + AVX2 (x86_64) paths with
26+
scalar fallback; identical integer result on every path (determinism
27+
preserved), purely a speedup.
28+
- **Benchmark suite** — `benchmarks/local_perf.py` (B1–B7) + `RESULTS_1M.md`,
29+
with a full performance section and HNSW-above-50K / small-batch warnings in
30+
the root `README.md`.
31+
932
## [0.2.3] — 2026-06-29
1033

1134
### Security

‎CLAUDE.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,9 @@ Backward-compat: V5 snapshots restore into an empty namespace registry (all reco
228228
| `VALORI_EVENT_LOG_PATH` | — | Audit log path (omit = in-memory only) |
229229
| `VALORI_SNAPSHOT_PATH` | — | Snapshot file path |
230230
| `VALORI_AUTH_TOKEN` | — | Bearer token (omit = no auth) |
231-
| `VALORI_INDEX` | brute | `brute` or `hnsw` |
231+
| `VALORI_INDEX` | brute | `brute`, `hnsw`, or `ivf` |
232+
| `VALORI_IVF_N_LIST` | auto | IVF centroid count. Absent = auto-scale: `max(16, sqrt(N))` computed at each `build()`. Setting this disables auto-scale. |
233+
| `VALORI_IVF_N_PROBE` | auto | IVF probe count. Absent = auto-scale: `max(1, sqrt(n_list))`. Setting this disables auto-scale. |
232234
| `VALORI_DECAY_HALF_LIFE_SECS` | — | Phase C4.1 default decay half-life for search ranking; per-request `decay_half_life_secs` overrides. Omit/0 = no decay |
233235
| `VALORI_EMBED_PROVIDER` | — | Phase I2: `ollama` / `openai` / `custom`; absent = embedding disabled; enables `POST /v1/ingest` |
234236
| `VALORI_EMBED_MODEL` | provider default | Embed model name (e.g. `nomic-embed-text`, `text-embedding-3-small`) |

‎Dockerfile‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,12 @@ COPY crates/valori-consensus/Cargo.toml crates/valori-consensus/
3232
# valori-ffi and embedded are non-default workspace members but must be present
3333
# for workspace resolution even though valori-node never depends on them.
3434
COPY crates/valori-ffi/Cargo.toml crates/valori-ffi/
35+
COPY crates/valori-mcp/Cargo.toml crates/valori-mcp/
3536
COPY embedded/Cargo.toml embedded/
3637

3738
# Stub src files to allow `cargo build` to populate the dep cache.
38-
# valori-ffi / embedded stubs are inert (not in valori-node's dep graph).
39-
RUN for crate in valori-kernel valori-node valori-wire valori-cli valori-verify valori-consensus valori-ffi; do \
39+
# valori-ffi / valori-mcp / embedded stubs are inert (not in valori-node's dep graph).
40+
RUN for crate in valori-kernel valori-node valori-wire valori-cli valori-verify valori-consensus valori-ffi valori-mcp; do \
4041
mkdir -p crates/$crate/src && \
4142
printf 'pub fn stub() {}\n' > crates/$crate/src/lib.rs && \
4243
printf 'fn main() {}\n' > crates/$crate/src/main.rs; \

‎README.md‎

Lines changed: 133 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -50,27 +50,62 @@ Every byte of state is recovered from the append-only, BLAKE3-chained event log
5050

5151
## Where Valori Sits in Your Stack
5252

53-
```
54-
┌─────────────────────────────────────────────────────────────────────┐
55-
│ Your AI Application │
56-
│ LangChain · LlamaIndex · OpenAI Agents · Custom Orchestrators │
57-
└────────────────────────┬────────────────────────────────────────────┘
58-
│ Python SDK / HTTP / PyO3 FFI
59-
┌────────────────────────▼────────────────────────────────────────────┐
60-
│ VALORI │
61-
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
62-
│ │ Vector │ │ Knowledge │ │ Cryptographic │ │
63-
│ │ Memory │ │ Graph │ │ Audit Trail │ │
64-
│ │ (HNSW/Brute)│ │ (same store)│ │ (BLAKE3 + replay) │ │
65-
│ └──────────────┘ └──────────────┘ └──────────────────────┘ │
66-
│ ┌──────────────────────────────────────────────────────────────┐ │
67-
│ │ Q16.16 Fixed-Point Kernel (no_std / no_alloc) │ │
68-
│ │ bit-identical results on x86 · ARM · RISC-V · Cortex-M4 │ │
69-
│ └──────────────────────────────────────────────────────────────┘ │
70-
│ ┌───────────────────────┐ ┌──────────────────────────────────┐ │
71-
│ │ Standalone Node │ │ 3- or 5-Node Raft Cluster │ │
72-
│ └───────────────────────┘ └──────────────────────────────────┘ │
73-
└─────────────────────────────────────────────────────────────────────┘
53+
```mermaid
54+
flowchart TB
55+
subgraph APP["Your AI Application"]
56+
direction LR
57+
A1["LangChain · LlamaIndex"]
58+
A2["OpenAI Agents · Orchestrators"]
59+
A3["MCP Clients · Claude · Cursor"]
60+
end
61+
62+
subgraph ACCESS["Access Layer"]
63+
direction LR
64+
S1["Python SDK"]
65+
S2["HTTP REST"]
66+
S3["PyO3 FFI\nin-process"]
67+
S4["MCP stdio\nvalori-mcp"]
68+
end
69+
70+
subgraph VALORI[" VALORI "]
71+
direction TB
72+
73+
subgraph CAPS["Capabilities"]
74+
direction LR
75+
C1["Vector Memory\nHNSW · IVF · Brute-force"]
76+
C2["Knowledge Graph\nGraphRAG · Tree-RAG · Community"]
77+
C3["Cryptographic Audit\nBLAKE3 chain · receipts"]
78+
C4["Self-Maintaining Memory\ndecay · consolidate · contradict"]
79+
end
80+
81+
subgraph KERN["Q16.16 Fixed-Point Kernel · no_std · WASM-safe"]
82+
direction LR
83+
K1["x86"] ~~~ K2["ARM"] ~~~ K3["RISC-V"] ~~~ K4["Cortex-M4"]
84+
end
85+
86+
subgraph DEPLOY["Deployment"]
87+
direction LR
88+
D1["Standalone Node"]
89+
D2["3 / 5-Node Raft Cluster"]
90+
end
91+
end
92+
93+
subgraph STORAGE["Durable Storage"]
94+
direction LR
95+
ST1["events.log\nBLAKE3 WAL"]
96+
ST2["Snapshot\nVAL1 V6"]
97+
ST3["S3 · MinIO · R2"]
98+
end
99+
100+
APP --> ACCESS --> VALORI --> STORAGE
101+
102+
style APP fill:#0f172a,color:#e2e8f0,stroke:#475569
103+
style ACCESS fill:#0f172a,color:#e2e8f0,stroke:#475569
104+
style VALORI fill:#1e1b4b,color:#e2e8f0,stroke:#6366f1,stroke-width:2px
105+
style CAPS fill:#1e1b4b,color:#e2e8f0,stroke:#4338ca
106+
style KERN fill:#312e81,color:#c7d2fe,stroke:#818cf8
107+
style DEPLOY fill:#1e1b4b,color:#e2e8f0,stroke:#4338ca
108+
style STORAGE fill:#0f172a,color:#e2e8f0,stroke:#475569
74109
```
75110

76111
---
@@ -100,6 +135,83 @@ Every byte of state is recovered from the append-only, BLAKE3-chained event log
100135

101136
---
102137

138+
## Performance
139+
140+
Measured on Apple Silicon M-series · release build · k=10.
141+
Reproduce: `python3 benchmarks/local_perf.py --million`
142+
143+
### Batch insert throughput by embedding model
144+
145+
| Model | Dim | Batch 100 | Batch 1,000 | Batch 10,000 |
146+
|---|---|---|---|---|
147+
| baseline / custom | 128 | 20,800 rec/s | 98,150 rec/s | **177,705 rec/s** |
148+
| nomic-embed-text · all-MiniLM-L6-v2 | 384 | 18,431 rec/s | 62,719 rec/s | **81,971 rec/s** |
149+
| BGE-base · E5-base · bert-base | 768 | 14,284 rec/s | 36,815 rec/s | **47,143 rec/s** |
150+
| OpenAI ada-002 · text-embedding-3-small | 1,536 | 9,734 rec/s | 19,929 rec/s | **25,196 rec/s** |
151+
152+
> **Batch size warning:** `insert_batch` with fewer than 100 records is **slower than a plain `insert` loop** — per-call overhead dominates at small sizes. Always use batches of ≥ 100; the sweet spot is 1,000–10,000.
153+
154+
### HNSW search latency by embedding model (10K records, k=10)
155+
156+
| Model | Dim | HNSW p50 | HNSW QPS | Brute p50 | Brute QPS |
157+
|---|---|---|---|---|---|
158+
| baseline / custom | 128 | 0.050 ms | 19,759 q/s | 1.224 ms | 810 q/s |
159+
| nomic-embed-text · all-MiniLM-L6-v2 | 384 | 0.146 ms | 4,486 q/s | 3.329 ms | 273 q/s |
160+
| BGE-base · E5-base · bert-base | 768 | 0.269 ms | 3,674 q/s | 7.338 ms | 135 q/s |
161+
| OpenAI ada-002 · text-embedding-3-small | 1,536 | 0.523 ms | 1,897 q/s | 14.923 ms | 66 q/s |
162+
163+
> **Index selection warning:** Brute force is O(N) — latency grows linearly with dataset size. It becomes unviable above ~50K records at any dimension. **HNSW is mandatory for production read-heavy workloads above 50K records.** Build cost is paid once and survives snapshot/restore; search stays sub-millisecond regardless of dataset size.
164+
165+
### Search latency vs dataset size (HNSW, dim=128, k=10)
166+
167+
| Records | p50 | p99 | QPS |
168+
|---|---|---|---|
169+
| 1,000 | 0.05 ms | — | ~20,000 q/s |
170+
| 10,000 | 0.05 ms | 0.069 ms | 19,759 q/s |
171+
| 1,000,000 | **0.107 ms** | 0.138 ms | **9,199 q/s** |
172+
173+
→ **Sub-millisecond search at 1 million records.**
174+
175+
### Search latency vs dataset size (bruteforce, dim=128, k=10)
176+
177+
| Records | p50 | p95 | p99 | QPS |
178+
|---|---|---|---|---|
179+
| 1,000 | 0.129 ms | 0.131 ms | 0.135 ms | 7,820 q/s |
180+
| 10,000 | 1.224 ms | 1.285 ms | 1.354 ms | 810 q/s |
181+
| 50,000 | 10.129 ms | 10.735 ms | 11.336 ms | 98 q/s |
182+
| 1,000,000 | 247.815 ms | 288.795 ms | 308.291 ms | 3 q/s |
183+
184+
### Index comparison @ 1 million records (dim=128, k=10)
185+
186+
| Index | Build time | p50 | p99 | QPS |
187+
|---|---|---|---|---|
188+
| **HNSW** | 4.4 min (one-time) | **0.107 ms** | **0.138 ms** | **9,199 q/s** |
189+
| IVF | 28 s | 58.35 ms | 66.05 ms | 16 q/s |
190+
| Brute force | 27 s | 247.41 ms | 297.01 ms | 4 q/s |
191+
192+
### Snapshot timing
193+
194+
| Records | Dim | Size | `snapshot()` | `restore()` | `save_snapshot()` |
195+
|---|---|---|---|---|---|
196+
| 10,000 | 128 | 5.2 MB | 2.2 ms | 4.3 ms | 4.7 ms |
197+
| 10,000 | 384 | 14.9 MB | 5.9 ms | 6.0 ms | 12.4 ms |
198+
| 10,000 | 768 | 29.6 MB | 10.0 ms | 16.3 ms | 20.6 ms |
199+
| 10,000 | 1,536 | 58.9 MB | 18.8 ms | 29.5 ms | 44.7 ms |
200+
| 50,000 | 128 | 25.8 MB | 10.1 ms | 21.6 ms | 26.7 ms |
201+
202+
### Batch size sweet spot (dim=128, bruteforce, 10K total records)
203+
204+
| Batch size | Throughput |
205+
|---|---|
206+
| 1 (single inserts) | 2,512 rec/s |
207+
| 10 | 1,936 rec/s ⚠️ slower than single |
208+
| 100 | 14,561 rec/s |
209+
| 500 | 60,805 rec/s |
210+
| 1,000 | 95,147 rec/s |
211+
| **10,000** | **174,963 rec/s** |
212+
213+
---
214+
103215
## Get Started
104216

105217
> **New contributor?** `bash dev-setup.sh` — one script installs Rust, the wasm32 target, Python SDK, and UI deps with OS detection and version gates. See [Build from Source](#build-from-source) and [CONTRIBUTING.md](CONTRIBUTING.md).

‎benchmarks/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,36 @@ Built in direct response to the 2026-06-12 review (Mayur, Rahul). Three asks, th
1212

1313
| Script | What it measures | Run it |
1414
|---|---|---|
15+
| [`local_perf.py`](local_perf.py) | Insert throughput, search latency, index comparison, snapshot — up to 1M records, no server required | `python3 benchmarks/local_perf.py --million` |
1516
| [`run_benchmark.py`](run_benchmark.py) | Three-arm RAG quality (float32 vs Q16.16 vs Q16.16+graph) | `python3 benchmarks/run_benchmark.py` |
1617
| [`multi_arch_hash.py`](multi_arch_hash.py) | Identical BLAKE3 state hash across CPU architectures | `python3 benchmarks/multi_arch_hash.py --url http://localhost:3000` |
1718
| [`q16_precision.py`](q16_precision.py) | Recall@10 of Q16.16 vs float32 ground truth at 384/768/1536/3072 dims | `python3 benchmarks/q16_precision.py --dim 384 [--st] [--openai]` |
1819

20+
### local_perf.py — usage
21+
22+
```bash
23+
# Standard run (up to 50K records, ~2 min):
24+
python3 benchmarks/local_perf.py
25+
26+
# Quick run (10K max, ~30 sec):
27+
python3 benchmarks/local_perf.py --quick
28+
29+
# Full 1 million record run (~10–15 min):
30+
python3 benchmarks/local_perf.py --million
31+
32+
# Save results as markdown:
33+
python3 benchmarks/local_perf.py --million --out benchmarks/RESULTS_1M.md
34+
```
35+
36+
**Prerequisites** — install the release wheel first (unoptimized dev build gives 10–30× slower numbers):
37+
38+
```bash
39+
cd crates/valori-ffi
40+
maturin build --release
41+
pip install ../../target/wheels/valoricore_ffi-*.whl --force-reinstall
42+
pip install -e python/ # SDK wrapper
43+
```
44+
1945
```bash
2046
# Quick start — run all three against a local node (dim=384)
2147
docker run --rm -d -p 3000:3000 -e VALORI_DIM=384 -e VALORI_MAX_RECORDS=2000 valori-node:latest

‎benchmarks/RESULTS_1M.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Valori Performance Benchmark
2+
3+
> dim=128 · release build · Apple Silicon M-series
4+
> Generated: 2026-06-30
5+
6+
## B1 — Insert throughput (single `insert`, dim={DIM})
7+
8+
| Records | Total time | Throughput |
9+
| ------- | ------------ | ------------ |
10+
| 100 | 7.09 ms | 14,103 rec/s |
11+
| 1,000 | 96.898 ms | 10,320 rec/s |
12+
| 5,000 | 1,147.579 ms | 4,356 rec/s |
13+
14+
## B2 — Batch insert throughput (`insert_batch`, dim=128)
15+
16+
| Batch size | Total time | Throughput |
17+
| ---------- | ---------- | ------------- |
18+
| 10 | 4.655 ms | 2,148 rec/s |
19+
| 100 | 4.808 ms | 20,800 rec/s |
20+
| 1,000 | 10.188 ms | 98,150 rec/s |
21+
| 5,000 | 29.864 ms | 167,427 rec/s |
22+
| 10,000 | 56.273 ms | 177,705 rec/s |
23+
24+
## B3 — Search latency vs dataset size (bruteforce, dim=128, k=10)
25+
26+
| Records | p50 | p95 | p99 | QPS |
27+
| --------- | ---------- | ---------- | ---------- | --------- |
28+
| 1,000 | 0.129 ms | 0.131 ms | 0.135 ms | 7,820 q/s |
29+
| 10,000 | 1.224 ms | 1.285 ms | 1.354 ms | 810 q/s |
30+
| 50,000 | 10.129 ms | 10.735 ms | 11.336 ms | 98 q/s |
31+
| 1,000,000 | 247.815 ms | 288.795 ms | 308.291 ms | 3 q/s |
32+
33+
## B4 — Index type comparison (dim=128, k=10)
34+
35+
| Index | Records | Build time | p50 | p99 | QPS |
36+
| ---------- | --------- | -------------- | ---------- | ---------- | --------- |
37+
| bruteforce | 1,000,000 | 26,636.37 ms | 247.407 ms | 297.008 ms | 4 q/s |
38+
| hnsw | 1,000,000 | 263,779.926 ms | 0.107 ms | 0.138 ms | 9,199 q/s |
39+
| ivf | 1,000,000 | 27,913.735 ms | 58.35 ms | 66.048 ms | 16 q/s |
40+
41+
## B5 — Dimension impact (10K records, bruteforce, k=10)
42+
43+
| Dim | Bytes/record | p50 | p99 | QPS |
44+
| --- | ------------ | -------- | -------- | --------- |
45+
| 32 | 128 B/rec | 0.258 ms | 0.315 ms | 3,802 q/s |
46+
| 128 | 512 B/rec | 0.804 ms | 0.867 ms | 1,233 q/s |
47+
| 384 | 1536 B/rec | 2.834 ms | 3.451 ms | 345 q/s |
48+
49+
## B6 — Snapshot timing
50+
51+
| Records | Size | snapshot() | restore() | save_snapshot() |
52+
| ------- | ---------- | ---------- | --------- | --------------- |
53+
| 10,000 | 5281.5 KB | 2.247 ms | 4.272 ms | 4.704 ms |
54+
| 50,000 | 26375.3 KB | 10.08 ms | 21.569 ms | 26.705 ms |
55+
56+
## B7 — Batch size impact on throughput (dim=128, bruteforce)
57+
58+
| Batch size | # calls | Total time | Throughput |
59+
| ---------- | ------- | ------------ | ------------- |
60+
| 1 | 10000 | 3,979.774 ms | 2,512 rec/s |
61+
| 10 | 1000 | 5,162.906 ms | 1,936 rec/s |
62+
| 100 | 100 | 686.764 ms | 14,561 rec/s |
63+
| 500 | 20 | 164.458 ms | 60,805 rec/s |
64+
| 1,000 | 10 | 105.1 ms | 95,147 rec/s |
65+
| 10,000 | 1 | 57.155 ms | 174,963 rec/s |
66+
67+
## Summary
68+
69+
| Metric | Value |
70+
|---|---|
71+
| Best insert throughput | `insert_batch(10000)` → **177,705 rec/s** |
72+
| Brute search @10K | **1.224 ms** p50 · 810 q/s |
73+
| Brute search @50K | **10.129 ms** p50 · 98 q/s |
74+
| Brute search @1M | **247.815 ms** p50 · 3 q/s |
75+
| HNSW search @1,000,000 | **0.107 ms** p50 · 9,199 q/s |
76+
| IVF search @1,000,000 | **58.35 ms** p50 · 16 q/s |
77+
78+
> Benchmark environment: Apple Silicon M-series · macOS · release build
79+
> Run: `python3 benchmarks/local_perf.py --million`

0 commit comments

Comments
 (0)