You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
QIHSE (Quantum Inspired Hilbert Space Expansion Search) is built for teams that want ANN performance without surrendering
7
-
correctness guarantees. The project is intentionally conservative in its default
8
-
behavior and explicit about when it uses aggressive acceleration. In practice, this
9
-
means you get a small number of clear knobs instead of implicit magic behavior.
9
+
Most vector databases make a quiet deal with you: they will be fast, and you will stop asking whether the results are correct. QIHSE does not make that deal.
10
+
11
+
QIHSE was built for teams that have been burned by silent approximation drift — where a deployment that passed yesterday's tests starts returning subtly wrong top-k because someone tuned a parameter three layers down that you never knew existed. It treats correctness as the default, not an opt-in, and treats speed as something you earn by understanding your data shape, not something you buy with hidden trade-offs.
12
+
13
+
The core promise is simple: **exact float32 search is the only path that does not ask your permission.** Every acceleration layer — graph indexing, scalar quantization, binary compression, sparse inverted indices — is an explicit contract. You decide whether to engage it, the system validates that it is safe for your query shape, and the final ranking is still produced by the same authoritative distance computation that would have run if you had never turned the accelerator on at all. You get to have the conversation about speed *after* you have established that correctness is not on the table.
10
14
11
15
## What makes QIHSE different
12
16
@@ -18,8 +22,15 @@ file-backed lifecycle controls and a query path model designed around two rules:
18
22
-**Only use approximations when they are explicitly requested and validated.**
19
23
Trinary-based acceleration is opt-in and enforced by explicit sidecar contracts.
20
24
21
-
This gives you practical speedups in the common sparse/high-selectivity cases while
22
-
preserving confidence that correctness has not been silently traded away.
25
+
Most vector search libraries are designed around a single happy path: build an approximate index, query it, hope the recall is good enough. QIHSE is designed around a different assumption: you will eventually need to know *exactly* what the right answer is, and when that moment comes, the system should not have painted you into a corner.
26
+
27
+
**Exactness is the default, not a debug mode.** Every query runs through float32 distance unless you explicitly ask for something else. The accelerators — graph indices, quantized sidecars, sparse inverted lists — are candidate *selectors*, not result *producers*. They narrow the field; the exact metric picks the winners. This means you can turn an accelerator on for speed, then turn it off for validation, and expect the same results.
28
+
29
+
**Sidecars are first-class, not afterthoughts.** When you build a graph index or an INT8 quantization table, QIHSE tracks whether that artifact is valid, stale, or corrupt. It does not silently fall back to brute force because a sidecar disappeared. It tells you the sidecar is missing and lets you decide what to do.
30
+
31
+
**The query planner knows when to say no.** The graph index is fast, but it is not always the right tool. QIHSE gates accelerator selection based on query dimensionality, top-k pressure, and dataset scale. Dense queries against small collections do not get pushed through a graph just because one exists. The system falls back to exact search by design when the overhead would not pay off.
32
+
33
+
**Recovery is deterministic, not magical.** There is no background thread you are expected to trust. Snapshot, WAL, replay, checkpoint, compact — these are explicit operations you call when you are ready. If the process crashes, you know exactly what state you will find on restart because you decided when the last snapshot happened.
23
34
24
35
## Unique technical characteristics
25
36
@@ -77,6 +88,18 @@ Maintenance and scheduling calls are available and explicit. There is no require
77
88
that hidden background threads be assumed for basic correctness; your host controls
78
89
the maintenance cadence.
79
90
91
+
### 7) Hierarchical memory storage with automatic hot/cold tiering
92
+
QIHSE tracks per-vector access frequency and temperature, then promotes frequently-accessed
93
+
vectors to faster memory tiers (HBM, NPU cache) and demotes cold vectors to DRAM.
94
+
Access tracking is automatic across all query paths (exact, graph, INT8, sparse).
95
+
Tier assignments are persisted in a `vectors.qtier` sidecar and recovered on restart.
96
+
Configuration via `.qihse.conf` or environment variables:
Run `qihse_vector_db_run_memory_maintenance(db)` explicitly, or let batch search auto-trigger it.
102
+
80
103
## Build and run
81
104
82
105
```bash
@@ -91,14 +114,28 @@ make all
91
114
flowchart TB
92
115
A[Client Process] --> B[Query Ingestion]
93
116
A --> C[Vector Mutations]
117
+
A --> P[Python / CLI Bindings]
94
118
B --> D{qihse_vector_db_search}
95
119
D --> E["Exact float32 rerank path<br/>(default)"]
96
120
D --> F{Query mode}
97
121
F -->|TRINARY_SCALAR| G["qtri sidecar shortlist"]
98
122
F -->|TRINARY_MAGNITUDE| H["qmag sidecar shortlist"]
123
+
F -->|GRAPH| X["Graph index (HNSW)"]
124
+
F -->|INT8| Y["INT8 quantization"]
125
+
F -->|BINARY| Z["Binary quantization"]
126
+
F -->|SPARSE| W["Inverted index (BM25)"]
99
127
G --> E
100
128
H --> E
129
+
X --> E
130
+
Y --> E
131
+
Z --> E
132
+
W --> E
101
133
E --> I["Returned ranked results"]
134
+
I --> Q[Query Result Cache]
135
+
Q --> I
136
+
I --> T[Hierarchical Storage<br/>Hot/cold tiering]
137
+
T --> S[Tier sidecar .qtier]
138
+
S --> T
102
139
C --> J["WAL + snapshot metadata"]
103
140
J --> K["checkpoint/compact"]
104
141
K --> L["Restart-safe snapshot"]
@@ -154,21 +191,28 @@ For rawest speed (at the cost of recall guarantees), use:
154
191
155
192
`QIHSE_VDB_QUERY_TRINARY_MAGNITUDE_BYPASS`.
156
193
157
-
## Core vector DB API surface
194
+
## What is actually in the box
195
+
196
+
**Distance computation that uses your silicon.** On modern x86 CPUs with AVX2, QIHSE automatically selects vectorized implementations of cosine similarity, dot product, and Euclidean distance. On older hardware, it falls back to scalar loops without any code changes or recompilation. You do not configure this. It is simply a property of the hardware you are running on.
197
+
198
+
**A graph index that knows when it is not needed.** The HNSW-style graph index is built and persisted automatically, but it is not used for every query. The system evaluates query dimensionality, top-k pressure, and dataset size before deciding whether the graph will actually be faster than a brute-force scan. For small collections or dense high-dimensional queries, it falls back to exact search — not because the graph is broken, but because the math says brute force is cheaper. The graph state is persisted to `index.qgraph` and loaded on restart, but it is an accelerator, not a crutch.
199
+
200
+
**Quantization that does not quantize your results.** INT8 scalar quantization stores per-dimension min/max scaling factors and compresses vectors to one byte per dimension. Binary quantization goes further, packing each dimension to a single bit. Both are used exclusively as candidate selectors: they produce a shortlist of promising rows, and then the exact float32 metric runs against that shortlist to produce the final ranking. The quantized artifacts are persisted sidecars (`vectors.qint8`, `vectors.qbinary`) and validated on load. If they are stale or corrupt, the system tells you, not your users.
**Sparse vectors handled natively.** If your vectors are mostly zeros — think TF-IDF, think one-hot embeddings, think any high-dimensional space where most dimensions are inactive — QIHSE builds an inverted index with BM25 scoring. The sparse path is not an afterthought or a plugin. It is a first-class query mode, and sparse vectors coexist in the same database as dense ones.
203
+
204
+
**A query cache with teeth.** Repeated identical queries are cached with FNV-1a hashing keyed on vector contents, top-k, and metric choice. The cache is invalidated automatically on any database mutation. There is no stale-cache bug where you delete a vector and still get it in results because the cache did not notice.
205
+
206
+
**Configuration that respects your environment.** Drop a `.qihse.conf` in your working directory or home directory. Set `graph.M`, `cache.max_entries`, `search.default_k` — the usual suspects. Environment variables override file values for containerized deployments. No XML, no YAML, no ceremony.
207
+
208
+
**Python and CLI interfaces.** The core is C, but you do not need to write C to use it. The Python bindings cover the full API, and the CLI tool handles database creation, bulk insertion, index building, and search from the shell.
**AGPL-3.0-or-later. This is strong copyleft. See [LICENSE](LICENSE) before any commercial use.**
282
+
283
+
This project is published as a technical showcase and for home deployment if you so wish, bear me in mind if you want a world class database driving your fancy new framework. Failure to comply will be treated as copyright infringement and pursued to the full extent of the law.
0 commit comments