Return matched chunk text in search results instead of truncated note header

[bm-clawd]

Problem

Search results truncate content_snippet to 250 chars via SearchIndexRow.content property (CONTENT_DISPLAY_LIMIT = 250 in src/basic_memory/repository/search_index_row.py:45). This means search always returns the beginning of the note, not the relevant section that matched the query.

Evidence (Benchmark)

Ran 38-query retrieval benchmark against plugin MCP server:

• Recall@5: 94.6% — we find the right notes
• Content Hit Rate: 21.4% — but the returned snippet rarely contains the specific fact

Example: query "What is the team plan pricing?" finds the correct note (2026-02-12.md) but returns:

# 2026-02-12 — Wednesday\n\n## Observations\n- [date] 2026-02-12\n- [type] daily-note\n\n## Standup\n- Maya integrating Linear cycle time API...

The pricing info ($9/mo per seat) is deeper in the note, past the 250-char cutoff.

Root Cause Chain

1. SearchIndexRow.content_snippet stores full content (good)
2. SearchIndexRow.content property truncates to first 250 chars (bad for relevance)
3. SearchResult.content gets the truncated version (src/basic_memory/api/v2/utils.py:183: content=r.content)
4. MCP search tool returns truncated content to agents
5. Agents/benchmarks only see note header, not the relevant section

Proposed Fix

Add matched_chunk field to SearchResult

The vector search pipeline already matches against specific chunks (chunk_key = "{type}:{id}:{chunk_index}"). The matching chunk text contains the relevant section but is discarded before results are returned.

Changes needed:

1. src/basic_memory/schemas/search.py — Add field to SearchResult

class SearchResult(BaseModel):
    # ... existing fields ...
    content: Optional[str] = None
    matched_chunk: Optional[str] = None  # NEW: the chunk text that scored highest in vector search

2. src/basic_memory/repository/search_index_row.py — Add chunk text field

@dataclass
class SearchIndexRow:
    # ... existing fields ...
    matched_chunk_text: Optional[str] = None  # Populated by vector search

3. src/basic_memory/repository/sqlite_search_repository.py — Thread chunk text through vector results

In _vector_search(), the query already JOINs search_vector_chunks. The chunk_text column is available but not selected. Add it to the result:

-- Current (approx line 807):
SELECT c.entity_id, c.chunk_key, vector_matches.distance AS best_distance
FROM ... JOIN search_vector_chunks c ON c.id = vector_matches.rowid

-- Proposed:
SELECT c.entity_id, c.chunk_key, c.chunk_text, vector_matches.distance AS best_distance
FROM ... JOIN search_vector_chunks c ON c.id = vector_matches.rowid

Then populate matched_chunk_text on the SearchIndexRow when building results.

4. src/basic_memory/repository/postgres_search_repository.py — Same change for Postgres

5. src/basic_memory/api/v2/utils.py — Pass through to SearchResult

SearchResult(
    # ... existing fields ...
    content=r.content,
    matched_chunk=r.matched_chunk_text,  # NEW
)

6. Fallback for FTS-only search

When vector search is not used (text-only mode), matched_chunk will be None. Consider a lightweight fallback: find the paragraph in content_snippet that has the best keyword overlap with the query and return that as the snippet.

Impact

• Benchmark content hit rate should jump from 21.4% to 70%+ with chunk-level snippets
• Agent quality improves — agents get the relevant section, not the note header
• No breaking changes — matched_chunk is a new optional field, content behavior unchanged
• Token efficiency — agents get a focused snippet instead of needing to read_note for every search result

Files to Modify

• src/basic_memory/schemas/search.py (SearchResult model)
• src/basic_memory/repository/search_index_row.py (SearchIndexRow dataclass)
• src/basic_memory/repository/sqlite_search_repository.py (vector search query + result mapping)
• src/basic_memory/repository/postgres_search_repository.py (same for postgres)
• src/basic_memory/repository/search_repository_base.py (hybrid merge logic)
• src/basic_memory/api/v2/utils.py (SearchResult construction)
• Tests in tests/repository/ and tests/api/

Benchmark Reference

• Plugin repo: benchmark/run.ts and benchmark/results/bm-mcp-small-results.json
• 38 queries, 9 categories, 11-file corpus
• Full results at: https://github.com/basicmachines-co/openclaw-basic-memory/tree/main/benchmark

Related

• Closes the content quality gap identified in retrieval benchmarks
• Supports SPEC-30 (Graph DB Pro) by establishing baseline metrics
• BUG: Hybrid RRF fusion dilutes vector scores with weak FTS scores #577 (hybrid RRF fusion, now closed) fixed scoring; this fixes snippets

[claude]

Claude finished @bm-clawd's task in 1m 45s —— View job

---

Issue Triage

• Analyze issue
• Read relevant source files
• Check for duplicate issues
• Apply labels
• Provide triage summary

---

Classification

Attribute	Assessment
Type	Enhancement — improving search result content quality
Priority	High — benchmark shows only 21.4% content hit rate; agents are systematically misled by irrelevant note headers
Component	Database (search repositories), MCP search tool
Complexity	Complex — coordinated changes across 6+ files, both SQLite and Postgres backends

No duplicate issues found.

---

Technical Verification

I read all the relevant source files and confirmed the issue analysis is accurate:

Root cause confirmed:

1. _run_vector_query in sqlite_search_repository.py:456 and postgres_search_repository.py:427 already JOIN search_vector_chunks c but only SELECT c.entity_id, c.chunk_key, vector_matches.distance — the chunk_text column is present in the table schema but not included in results.

2. SearchIndexRow.content at search_index_row.py:44-48 truncates to 250 chars from the beginning of the full content_snippet, not from the matched position.

3. api/v2/utils.py:179 maps content=r.content which uses the truncated property.

The fix is straightforward in isolation — adding c.chunk_text to the SELECT is a one-line SQL change in each backend — but threading it through the pipeline correctly requires care:

Key consideration for _search_hybrid: The RRF merge at search_repository_base.py:1121-1127 uses rows_by_id[row.id] = row which overwrites with the last-seen result. When a document appears in both FTS and vector results, the matched_chunk_text from the vector result could be lost if the FTS row wins the overwrite. The merge logic should preserve matched_chunk_text from vector results when building fused rows.

FTS-only fallback: The issue mentions finding the best-matching paragraph for FTS queries (where no vector chunk is selected). This is a worthwhile improvement but can be done as a follow-up — the core fix (vector chunk passthrough) already covers hybrid and vector-only modes.

---

Proposed Implementation Scope

The issue's proposed fix is sound. Files to change:

1. src/basic_memory/repository/sqlite_search_repository.py — Add c.chunk_text to _run_vector_query SELECT (line ~456)
2. src/basic_memory/repository/postgres_search_repository.py — Same for Postgres (line ~427)
3. src/basic_memory/repository/search_index_row.py — Add matched_chunk_text: Optional[str] = None field
4. src/basic_memory/repository/search_repository_base.py — Populate matched_chunk_text from vector results; handle matched_chunk_text preservation in hybrid merge
5. src/basic_memory/schemas/search.py — Add matched_chunk: Optional[str] = None to SearchResult
6. src/basic_memory/api/v2/utils.py — Pass matched_chunk=r.matched_chunk_text in SearchResult construction
7. Tests — Update existing search tests + add coverage for chunk passthrough in both backends

Labels applied: enhancement