Skip to content

Commit 42132a0

Browse files
committed
Cap default cluster probe at scale
The default `probe` is a fraction (0.25) of clusters. Clusters grow as ~sqrt(N)/2, so the fraction's probed-list count keeps rising with N while recall@10 saturates well before it. Cap the default at 96 lists (`defaultClusterProbeCap`); the cap binds only above ~400k vectors and an explicit `probe` (fraction or count) bypasses it. Validated on the 3.2M-vector WildChat index (894 clusters): recall@10 is flat at 92.0% from 48 to 320 probed lists, so the cap costs 0pp of top-10 recall vs the uncapped fraction (224 lists) while cutting ~30% of fetches, ~40% of bytes, and ~54% of latency. The sqrt(N)/2 cluster heuristic also holds at this scale.
1 parent 3d516c0 commit 42132a0

6 files changed

Lines changed: 112 additions & 11 deletions

File tree

OPTIMIZE.md

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -140,11 +140,31 @@ N=20k–156k) and tpuf (1024-dim, N=250k/1M), clusters at the shipped √N/2:
140140
Recall knees at ~80 lists (92%). The fraction's last 1pp (92→93%) costs +47
141141
fetches and +11 MB — ~30% more roundtrips and bytes for marginal recall.
142142

143-
**Recommended change:** `probe = min(ceil(fraction × nlist), cap)` with
144-
`cap ≈ 80–96`. The cap only binds above ~400k vectors (where 0.25·√N/2 > 80),
145-
so all current small/medium-N behavior is unchanged; at 1M it trims ~25% of
146-
roundtrips and ~30% of bytes for ~1pp recall. Backward-compatible, low risk.
147-
Open question: exact cap value (80 vs 96) and whether it's user-overridable.
143+
**Implemented:** `probe = min(ceil(fraction × nlist), 96)` (`defaultClusterProbeCap`
144+
in `constants.js`, applied in `ranges.js`). The cap binds only above ~400k
145+
vectors (where 0.25·√N/2 > 96); smaller indexes are unchanged. An explicit
146+
`probe` (fraction or count) bypasses the cap and is honored literally.
147+
148+
**Validated at 3.2M (2026-06-21).** WildChat-4.8M index: 3,199,860 × 1024-dim,
149+
894 clusters (√N/2), cluster sizes 380–26,122 (mean 3,579). 10 queries, exact
150+
brute-force reference. recall@10 is **flat at 92.0% from 48 to 320 probed
151+
lists** — it saturates by ~48 lists; probing more does nothing for top-10:
152+
153+
| lists | fetches | MB | r@10 | r@100 |
154+
|------:|--------:|---:|-----:|------:|
155+
| 48 | 274 | 68 | 92.0% | 86.4% |
156+
| **96 (cap, default)** | **347** | **89** | **92.0%** | **88.0%** |
157+
| 224 (=0.25, uncapped) | 500 | 146 | 92.0% | 89.7% |
158+
| 320 | 599 | 190 | 92.0% | 89.8% |
159+
160+
So at 3.2M the cap costs **0 pp of recall@10** vs the uncapped 0.25 fraction
161+
while cutting ~30% of fetches, ~40% of bytes, and ~54% of latency (513 vs
162+
1038 ms). recall@100 rises slightly with more probes (a top-100 workload
163+
could raise probe explicitly), but for the default top-10 the cap is correct
164+
and even conservative — 48 lists would already serve top-10. Keeping 96
165+
because at 1M the recall@10 knee is higher (~80–96 lists); 96 serves both
166+
scales. **Conclusion: cap appropriate; √N/2 cluster heuristic also confirmed
167+
at 3.2M.** (Caveat: 10 queries → coarse r@10 granularity; trend is clear.)
148168

149169
---
150170

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Vector search over 3,199,860 OpenAI embeddings (1024-dim) of real LLM conversati
3131
|---|---:|---:|---:|---:|---|
3232
| **hypvector** | 13.7 GB | 0.925 | 147 ms | **~$0.32** | none |
3333
| Pinecone | 13.1 GB | 0.920 | 85 ms | $50 min | managed |
34-
| turbopuffer | 13.1 GB | 0.915 | 198 ms | $64 min | managed |
34+
| turbopuffer | 13.1 GB | 0.915 | 198 ms | $16 min | managed |
3535
| S3 Vectors | 13.1 GB | 0.905 | 133 ms | ~$0.79 | serverless |
3636
| pgvector | 41.9 GB | 0.870 | 80 ms | $372 | r5.2xlarge 24/7 |
3737
| Qdrant | 13.1 GB | 0.865 | 70 ms | $186 | r5.xlarge 24/7 |
@@ -252,7 +252,7 @@ Vector search over 837,989 real LLM conversations ([WildChat-1M](https://hugging
252252
| **hypvector** | 3.58 GB | 0.975 | 46 ms <sup>†</sup> | **~$0.08** | none |
253253
| pgvector | 11.5 GB | 0.965 | ~1 ms <sup>†</sup> | $94 | r5.large 24/7 |
254254
| Qdrant | 3.6 GB | 0.965 | 2 ms <sup>†</sup> | $62 | t3.large 24/7 |
255-
| turbopuffer | 3.43 GB | 0.93 | 60 ms <sup>‡</sup> | $64 min | managed |
255+
| turbopuffer | 3.43 GB | 0.93 | 60 ms <sup>‡</sup> | $16 min | managed |
256256
| Pinecone | 3.43 GB | 0.97 | 125 ms <sup>‡</sup> | $50 min | managed |
257257

258258
<sup>†</sup> local compute, no network. <sup>‡</sup> live cloud, includes real internet round-trip.
@@ -306,7 +306,7 @@ hypvector isn't a hosted service. The closest peers are:
306306
|---|---|---:|---:|---:|
307307
| **hypvector** | none, file on S3 | ~500 ms (CloudFront, home WAN) | same, no cache | $0 |
308308
| **LanceDB** (S3 mode) | none, embedded | bandwidth-bound | sub-50 ms (local) | $0 |
309-
| **turbopuffer** | hosted | ~440 ms p90 | ~8 ms | $64 min |
309+
| **turbopuffer** | hosted | ~440 ms p90 | ~8 ms | $16 min |
310310
| **Pinecone Serverless** | hosted | 200 ms – 2 s | 50–100 ms | $0 + per-RU |
311311
| **Cloudflare Vectorize** | hosted (edge) | needs pre-warm | edge-fast | $0 + per-op |
312312

src/constants.js

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@ export const defaultClusterIterations = 6
3030
// file has cluster metadata. Lower = faster but lower recall.
3131
export const defaultClusterProbeFraction = 0.25
3232

33+
// Upper bound on clusters probed under the *default* fraction. Clusters grow
34+
// as ~sqrt(N)/2, so 0.25 x nlist keeps rising with N; measured recall knees
35+
// well before that at scale (~92% at 80-96 lists on 1M x 1024, vs 93% at the
36+
// uncapped 125). Capping the default trims ~25% of roundtrips and ~30% of
37+
// bytes above ~400k vectors for ~1pp recall. Only applies when `probe` is
38+
// left default; an explicit `probe` is honored literally.
39+
export const defaultClusterProbeCap = 96
40+
3341
// When `binary` is not specified at write time, the column is added once
3442
// the corpus is at least this large. Below the threshold, exact full scan
3543
// is fast enough that the rerank path's overhead isn't worth the column.

src/search/ranges.js

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { hammingDistanceBytes } from '../cluster.js'
2-
import { defaultClusterProbeFraction } from '../constants.js'
2+
import { defaultClusterProbeCap, defaultClusterProbeFraction } from '../constants.js'
33

44
/**
55
* @import { HypVectorMetadata } from '../types.js'
@@ -36,9 +36,13 @@ export function selectClusterRowRanges(meta, queryBin, probe) {
3636
const probeFraction = probe === undefined ? defaultClusterProbeFraction : probe
3737
// probe in (0, 1] is a fraction of clusters (1.0 = all clusters);
3838
// probe > 1 is an absolute count.
39-
const targetClusters = probeFraction > 1
39+
let targetClusters = probeFraction > 1
4040
? Math.min(Math.ceil(probeFraction), centroids.length)
4141
: Math.max(1, Math.ceil(centroids.length * probeFraction))
42+
// The default fraction over-probes at large nlist (recall knees well before
43+
// 0.25 x nlist), so cap the *default* to bound roundtrips/bytes at scale.
44+
// An explicit `probe` — fraction or count — is taken literally.
45+
if (probe === undefined) targetClusters = Math.min(targetClusters, defaultClusterProbeCap)
4246

4347
const wanted = clusterDist.slice(0, targetClusters).map(c => c.cluster).sort((a, b) => a - b)
4448
/** @type {{ rowStart: number, rowEnd: number }[]} */

src/types.d.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,9 @@ export interface SearchVectorsOptions {
8888
* actually scans. Can be expressed as:
8989
* - an integer >= 1 (number of clusters)
9090
* - a float in (0, 1] (fraction of total clusters)
91-
* Lower values are faster but reduce recall. Default: 0.25 (scan 25% of clusters).
91+
* Lower values are faster but reduce recall. Default: 0.25 (scan 25% of
92+
* clusters), capped at an absolute ceiling so large indexes don't over-probe;
93+
* the cap applies only to the default — an explicit value is honored as-is.
9294
* Ignored when the file has no centroids.
9395
*/
9496
probe?: number

test/ranges.test.js

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
import { describe, expect, it } from 'vitest'
2+
import { defaultClusterProbeCap } from '../src/constants.js'
3+
import { selectClusterRowRanges } from '../src/search/ranges.js'
4+
5+
/**
6+
* Build synthetic cluster metadata with `nClusters` clusters of one row each,
7+
* so the total rows covered by the returned ranges equals the number of
8+
* clusters probed (independent of how ranges merge).
9+
* @param {number} nClusters
10+
* @returns {import('../src/types.js').HypVectorMetadata}
11+
*/
12+
function makeMeta(nClusters) {
13+
/** @type {Uint8Array[]} */
14+
const centroids = []
15+
for (let c = 0; c < nClusters; c += 1) {
16+
// Distinct 16-bit patterns so cluster ranking by Hamming is well-defined.
17+
centroids.push(Uint8Array.from([c & 0xff, c >> 8 & 0xff]))
18+
}
19+
const clusterCounts = new Uint32Array(nClusters).fill(1)
20+
return {
21+
version: 0, dimension: 16, metric: 'cosine', normalized: true,
22+
hasBinary: true, count: nClusters, clusters: nClusters, centroids, clusterCounts,
23+
}
24+
}
25+
26+
/**
27+
* Sum of rows covered by a list of row ranges.
28+
* @param {{ rowStart: number, rowEnd: number }[]} ranges
29+
* @returns {number}
30+
*/
31+
function rowsCovered(ranges) {
32+
let sum = 0
33+
for (const r of ranges) sum += r.rowEnd - r.rowStart
34+
return sum
35+
}
36+
37+
describe('selectClusterRowRanges probe cap', () => {
38+
const query = Uint8Array.from([0, 0])
39+
40+
it('caps the default fraction at the absolute ceiling for large nlist', () => {
41+
// 0.25 * 500 = 125 clusters, but the default cap (96) should bind.
42+
const ranges = selectClusterRowRanges(makeMeta(500), query, undefined)
43+
expect(rowsCovered(ranges)).toBe(defaultClusterProbeCap)
44+
})
45+
46+
it('does not cap when the default fraction is below the ceiling', () => {
47+
// 0.25 * 100 = 25 clusters, well under the cap — unchanged behavior.
48+
const ranges = selectClusterRowRanges(makeMeta(100), query, undefined)
49+
expect(rowsCovered(ranges)).toBe(25)
50+
})
51+
52+
it('honors an explicit fraction literally (no cap)', () => {
53+
// Explicit 0.25 on 500 clusters → 125, NOT capped.
54+
const ranges = selectClusterRowRanges(makeMeta(500), query, 0.25)
55+
expect(rowsCovered(ranges)).toBe(125)
56+
})
57+
58+
it('honors an explicit absolute count above the cap', () => {
59+
const ranges = selectClusterRowRanges(makeMeta(500), query, 200)
60+
expect(rowsCovered(ranges)).toBe(200)
61+
})
62+
63+
it('never probes more clusters than exist', () => {
64+
const ranges = selectClusterRowRanges(makeMeta(40), query, undefined)
65+
expect(rowsCovered(ranges)).toBe(10) // ceil(0.25 * 40)
66+
})
67+
})

0 commit comments

Comments
 (0)