Skip to content

Commit 8a17314

Browse files
committed
benchmark: use matched_chunk for content hit scoring
Content hit rate jumps from 14.3% to 85.7% now that we read the matched_chunk field from BM search results (PR #601). Results: R@5=81%, R@10=88%, MRR=0.795, Content Hit=85.7%, P95=868ms
1 parent 1d9e7ee commit 8a17314

File tree

2 files changed

+8
-2
lines changed

2 files changed

+8
-2
lines changed

benchmark/corpus/MEMORY.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
---
2+
title: MEMORY
3+
type: note
4+
permalink: benchmark/memory
5+
---
6+
17
# MEMORY.md - Long-Term Memory
28

39
## About Me
@@ -51,4 +57,4 @@
5157
- Rust compile times are brutal but the binary size payoff is worth it
5258
- SQLite WAL mode is mandatory for concurrent reads during metric aggregation
5359
- Never trust webhook delivery — always implement idempotent handlers
54-
- The onboarding flow is our weakest point right now (users drop off at OAuth)
60+
- The onboarding flow is our weakest point right now (users drop off at OAuth)

benchmark/run.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ async function mcpSearch(query: string, limit = 10): Promise<SearchResult[]> {
131131
file_path: r.file_path || r.permalink || "",
132132
title: r.title || "",
133133
score: r.score || r.similarity || 0,
134-
content: r.content || r.snippet || "",
134+
content: r.matched_chunk || r.content || r.snippet || "",
135135
}))
136136
} catch (err) {
137137
console.error(` MCP search failed for: "${query}"`, err)

0 commit comments

Comments
 (0)