Skip to content

Commit 0700fe8

Browse files
docs: update embedding benchmarks (3.9.6) (#1034)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: carlos-alm <127798846+carlos-alm@users.noreply.github.com>
1 parent c5fad6d commit 0700fe8

1 file changed

Lines changed: 79 additions & 1 deletion

File tree

generated/benchmarks/EMBEDDING-BENCHMARKS.md

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,94 @@ Hit@N = expected symbol found in top N results.
66

77
| Version | Model | Symbols | Hit@1 | Hit@3 | Hit@5 | Misses | Embed Time |
88
|---------|-------|--------:|------:|------:|------:|-------:|-----------:|
9+
| 3.9.6 | minilm | 1500 | 67.4% | 87.2% | 92.4% | 49 | 125.9s |
10+
| 3.9.6 | jina-small | 1500 | 75.8% | 93.6% | 96.5% | 23 | 253.8s |
11+
| 3.9.6 | jina-base | 1500 | 72.1% | 90.3% | 94.9% | 30 | 1327.4s |
12+
| 3.9.6 | nomic | 1500 | 80.1% | 95.5% | 97.9% | 15 | 1331.2s |
13+
| 3.9.6 | nomic-v1.5 | 1500 | 78.4% | 94.9% | 97.1% | 18 | 1325.3s |
914

1015
### Latest results
1116

12-
**Version:** 3.1.4 | **Strategy:** structured | **Symbols:** 1095 | **Date:** 2026-03-16
17+
**Version:** 3.9.6 | **Strategy:** structured | **Symbols:** 1500 | **Date:** 2026-04-30
1318

1419
| Model | Dim | Context | Hit@1 | Hit@3 | Hit@5 | Hit@10 | Misses | Embed | Search |
1520
|-------|----:|--------:|------:|------:|------:|-------:|-------:|------:|-------:|
21+
| minilm | 384 | 256 | 67.4% | 87.2% | 92.4% | 96.7% | 49 | 125.9s | 130.5s |
22+
| jina-small | 512 | 8192 | 75.8% | 93.6% | 96.5% | 98.5% | 23 | 253.8s | 159.2s |
23+
| jina-base | 768 | 8192 | 72.1% | 90.3% | 94.9% | 98.0% | 30 | 1327.4s | 206.2s |
24+
| nomic | 768 | 8192 | 80.1% | 95.5% | 97.9% | 99.0% | 15 | 1331.2s | 206.0s |
25+
| nomic-v1.5 | 768 | 8192 | 78.4% | 94.9% | 97.1% | 98.8% | 18 | 1325.3s | 207.0s |
1626

1727
<!-- EMBEDDING_BENCHMARK_DATA
1828
[
29+
{
30+
"version": "3.9.6",
31+
"date": "2026-04-30",
32+
"strategy": "structured",
33+
"symbols": 1500,
34+
"models": {
35+
"minilm": {
36+
"dim": 384,
37+
"contextWindow": 256,
38+
"hits1": 1011,
39+
"hits3": 1308,
40+
"hits5": 1386,
41+
"hits10": 1451,
42+
"misses": 49,
43+
"total": 1500,
44+
"embedTimeMs": 125867,
45+
"searchTimeMs": 130509
46+
},
47+
"jina-small": {
48+
"dim": 512,
49+
"contextWindow": 8192,
50+
"hits1": 1137,
51+
"hits3": 1404,
52+
"hits5": 1447,
53+
"hits10": 1477,
54+
"misses": 23,
55+
"total": 1500,
56+
"embedTimeMs": 253823,
57+
"searchTimeMs": 159156
58+
},
59+
"jina-base": {
60+
"dim": 768,
61+
"contextWindow": 8192,
62+
"hits1": 1082,
63+
"hits3": 1355,
64+
"hits5": 1424,
65+
"hits10": 1470,
66+
"misses": 30,
67+
"total": 1500,
68+
"embedTimeMs": 1327410,
69+
"searchTimeMs": 206246
70+
},
71+
"nomic": {
72+
"dim": 768,
73+
"contextWindow": 8192,
74+
"hits1": 1202,
75+
"hits3": 1433,
76+
"hits5": 1469,
77+
"hits10": 1485,
78+
"misses": 15,
79+
"total": 1500,
80+
"embedTimeMs": 1331223,
81+
"searchTimeMs": 206002
82+
},
83+
"nomic-v1.5": {
84+
"dim": 768,
85+
"contextWindow": 8192,
86+
"hits1": 1176,
87+
"hits3": 1423,
88+
"hits5": 1457,
89+
"hits10": 1482,
90+
"misses": 18,
91+
"total": 1500,
92+
"embedTimeMs": 1325336,
93+
"searchTimeMs": 206972
94+
}
95+
}
96+
},
1997
{
2098
"version": "3.1.4",
2199
"date": "2026-03-16",

0 commit comments

Comments
 (0)