Skip to content

Commit e1b5862

Browse files
GoodbyePlanetclaude
GoodbyePlanet
and
claude
committed
docs: document RAG ingestion, retrieval, and configuration
Add docs/ with five topic files covering the ingestion pipeline, dense and sparse embedding paths, hybrid RRF retrieval, and all configuration knobs. Each topic doc includes an Observations section highlighting non-obvious behavior. Link the new docs from the root README. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 582d735 commit e1b5862

8 files changed

Lines changed: 1046 additions & 0 deletions

File tree

‎.ai/plans/2026-06-02-rag-system-documentation.md‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
**Repository:** root
2+
**Status:** Completed (2026-06-02)
3+
**Created:** 2026-06-02
4+
**Author:** lead
5+
6+
# RAG System Documentation
7+
8+
## Goal
9+
10+
Produce comprehensive documentation of the semcode RAG system covering: dense
11+
vector ingestion, sparse vector (BM25) ingestion, hybrid retrieval with Reciprocal
12+
Rank Fusion (RRF), and system configuration. The user is a participant throughout
13+
the investigation — code analysis findings are shared with the user for input and
14+
correction before each draft is written, and each draft is reviewed and approved by
15+
the user before the file is saved to `docs/`. The documentation targets a mixed
16+
audience (newcomers and experienced contributors) and includes observations on
17+
design gaps and improvement opportunities per document.
18+
19+
## Context
20+
21+
- **User preferences:** one file per concern; mixed audience (concept intros for
22+
newcomers, design rationale for experienced contributors); include findings/observations
23+
per doc; human reviews each draft before it is saved; `docs/configuration.md` is a
24+
first-class deliverable, not just inline context.
25+
- **No existing `docs/` folder** — it will be created as part of this plan.
26+
- **Core files:**
27+
- `server/indexer/pipeline.py` — `IndexPipeline`, embedding text builders,
28+
ingestion orchestration, incremental indexing via blob SHA change detection
29+
- `server/store/qdrant.py` — `QdrantStore`, collection creation (named vectors
30+
`text-dense` + `text-sparse`), `upsert_chunks`, `search` with `FusionQuery(RRF)`
31+
- `server/tools/search.py` — MCP tools: `search_code`, `find_symbol`,
32+
`find_usages`, `get_code_context`
33+
- `server/embeddings/base.py` — `EmbeddingProvider` Protocol
34+
- `server/embeddings/factory.py` — registry pattern, singleton provider
35+
- `server/embeddings/bm25.py` — `BM25SparseProvider` using `fastembed` +
36+
`Qdrant/bm25`; uses `split_code_identifiers` pre-processing
37+
- `server/embeddings/code_tokenizer.py` — code identifier splitter for BM25
38+
- `server/embeddings/{jina,jina_api,voyage,openai,ollama}.py` — dense providers
39+
- `server/config.py` — `Settings` (pydantic-settings), `ServiceConfig`,
40+
`EmbeddingsProviderName`
41+
- **Key design decisions visible in the code:**
42+
- Dual-text strategy: dense embeddings use a rich preamble with metadata;
43+
sparse (BM25) uses signature + docstring + source only
44+
- Incremental indexing: blob SHA comparison skips unchanged files
45+
- Deterministic point IDs: `uuid5` from `service:file_path:symbol_name:start_line`
46+
- RRF via Qdrant's native `FusionQuery(Fusion.RRF)` with 2× prefetch limit
47+
- Sparse vector stored in-memory (`on_disk=False`)
48+
- HNSW config: `m=16`, `ef_construct=128`; indexing threshold: 500
49+
- **This is a documentation-only task** — no production code changes.
50+
51+
## Steps
52+
53+
- [x] Clarify requirements with user
54+
- [x] Read codebase (pipeline, store, embeddings, config, tools)
55+
- [x] Write plan
56+
- [ ] Plan review cycle
57+
- [ ] User plan approval
58+
- [ ] Workflow selection
59+
- [x] Task 1: Draft and review ingestion doc
60+
- [x] Task 2: Draft and review dense vectors doc
61+
- [x] Task 3: Draft and review sparse vectors doc
62+
- [x] Task 4: Draft and review retrieval (RRF) doc
63+
- [x] Task 5: Draft and review configuration doc
64+
- [x] Task 6: Draft and review docs/README.md index + root README.md link
65+
- [x] Commit all docs — bf0adea
66+
67+
## Tasks
68+
69+
### Task 1: docs/ingestion.md
70+
71+
Investigate the `IndexPipeline` end-to-end ingestion flow — share key findings
72+
with the user before drafting, then write the document.
73+
74+
**Investigation scope:** how a service is discovered, how files are fetched and
75+
filtered, how symbols are parsed, how both embedding types are generated, and how
76+
results are upserted into Qdrant; incremental indexing (blob SHA change detection);
77+
stale-entry cleanup.
78+
79+
**Process:**
80+
1. Present code analysis findings and observations to the user via AskUserQuestion;
81+
proceed to drafting only after user confirms the findings are accurate.
82+
2. Write the draft; present it to the user for review via AskUserQuestion.
83+
3. Save `docs/ingestion.md` only after the user approves the draft.
84+
85+
Acceptance criteria:
86+
- File `docs/ingestion.md` exists with sections: Overview, Pipeline Stages
87+
(Discovery → Parsing → Embedding → Upsert → Cleanup), Incremental Indexing,
88+
Data Model (what a `CodeSymbol` and its payload look like), Observations
89+
- File written to disk only after user's explicit approval of the draft
90+
- No production code is modified
91+
92+
### Task 2: docs/dense-vectors.md
93+
94+
Investigate how dense embeddings are produced — share key findings with the user
95+
before drafting, then write the document.
96+
97+
**Investigation scope:** `EmbeddingProvider` Protocol, the factory/registry
98+
pattern, five concrete providers (jina local, jina API, voyage, openai, ollama),
99+
the rich preamble text strategy in `_build_embedding_text`, the 6000-char
100+
truncation limit, provider selection at startup.
101+
102+
**Process:**
103+
1. Present code analysis findings and observations to the user via AskUserQuestion;
104+
proceed to drafting only after user confirms the findings are accurate.
105+
2. Write the draft; present it to the user for review via AskUserQuestion.
106+
3. Save `docs/dense-vectors.md` only after the user approves the draft.
107+
108+
Acceptance criteria:
109+
- File `docs/dense-vectors.md` exists with sections: Overview, Provider Protocol,
110+
Embedding Text Strategy (preamble construction), Supported Providers (comparison
111+
table), Provider Selection, Observations
112+
- File written to disk only after user's explicit approval of the draft
113+
- No production code is modified
114+
115+
### Task 3: docs/sparse-vectors.md
116+
117+
Investigate how sparse BM25 embeddings are produced — share key findings with the
118+
user before drafting, then write the document.
119+
120+
**Investigation scope:** `BM25SparseProvider`, `Qdrant/bm25` fastembed model,
121+
`split_code_identifiers` pre-processing (and why it matters for code), distinction
122+
between `passage_embed` and `query_embed`, sparse vector structure
123+
(`indices` + `values`).
124+
125+
**Process:**
126+
1. Present code analysis findings and observations to the user via AskUserQuestion;
127+
proceed to drafting only after user confirms the findings are accurate.
128+
2. Write the draft; present it to the user for review via AskUserQuestion.
129+
3. Save `docs/sparse-vectors.md` only after the user approves the draft.
130+
131+
Acceptance criteria:
132+
- File `docs/sparse-vectors.md` exists with sections: Overview, BM25 for Code
133+
(why BM25 complements dense), Code Tokenizer Pre-processing, Passage vs Query
134+
Embedding, Sparse Vector Structure, Observations
135+
- File written to disk only after user's explicit approval of the draft
136+
- No production code is modified
137+
138+
### Task 4: docs/retrieval-rrf.md
139+
140+
Investigate the hybrid retrieval path — share key findings with the user before
141+
drafting, then write the document.
142+
143+
**Investigation scope:** `QdrantStore.search` dual-prefetch (dense + sparse, each
144+
at 2× limit), Qdrant's native RRF fusion, `find_by_name` fallback (exact scroll vs.
145+
substring scan), four MCP tool entry points (`search_code`, `find_symbol`,
146+
`find_usages`, `get_code_context`), result formatting for MCP clients.
147+
148+
**Process:**
149+
1. Present code analysis findings and observations to the user via AskUserQuestion;
150+
proceed to drafting only after user confirms the findings are accurate.
151+
2. Write the draft; present it to the user for review via AskUserQuestion.
152+
3. Save `docs/retrieval-rrf.md` only after the user approves the draft.
153+
154+
Acceptance criteria:
155+
- File `docs/retrieval-rrf.md` exists with sections: Overview, Hybrid Search
156+
Architecture, RRF Fusion, MCP Tool Interface, Result Formatting, Observations
157+
- File written to disk only after user's explicit approval of the draft
158+
- No production code is modified
159+
160+
### Task 5: docs/configuration.md
161+
162+
Investigate all configuration knobs — share key findings with the user before
163+
drafting, then write the document.
164+
165+
**Investigation scope:** every env var in `Settings` (name, default, which
166+
provider it applies to), `config.yaml` service fields (`name`, `github_repo`,
167+
`github_ref`, `root`, `exclude`), collection settings, transport settings, startup
168+
dimension-mismatch validation.
169+
170+
**Process:**
171+
1. Present code analysis findings and observations to the user via AskUserQuestion;
172+
proceed to drafting only after user confirms the findings are accurate.
173+
2. Write the draft; present it to the user for review via AskUserQuestion.
174+
3. Save `docs/configuration.md` only after the user approves the draft.
175+
176+
Acceptance criteria:
177+
- File `docs/configuration.md` exists with sections: Overview, Environment
178+
Variables (table: var, default, description), config.yaml Structure, Startup
179+
Validation, Observations
180+
- File written to disk only after user's explicit approval of the draft
181+
- No production code is modified
182+
183+
### Task 6: docs/README.md + root README.md link
184+
185+
Write a one-page index that describes the system at a glance (what semcode RAG
186+
is, what it indexes, how it serves queries), lists the five topic docs with
187+
one-line descriptions (ingestion.md, dense-vectors.md, sparse-vectors.md,
188+
retrieval-rrf.md, configuration.md), and provides a quick-start pointer to
189+
`docs/configuration.md`. Then add a link from the root `README.md` into the
190+
new `docs/` tree so readers arriving at the project's entry point can discover
191+
the detailed documentation.
192+
193+
**Process:**
194+
1. Present draft of `docs/README.md` to the user for review via AskUserQuestion.
195+
2. Save `docs/README.md` only after the user approves the draft.
196+
3. Add a "Documentation" section to the root `README.md` linking to
197+
`docs/README.md` with a one-line description; present the addition to the user
198+
before saving.
199+
200+
Acceptance criteria:
201+
- File `docs/README.md` exists with sections: What is semcode RAG, Documentation
202+
Index (links to the five topic docs: ingestion, dense-vectors, sparse-vectors,
203+
retrieval-rrf, configuration), Quick Start
204+
- Root `README.md` contains a link to `docs/README.md`
205+
- Both files written to disk only after user's explicit approval
206+
- No production code is modified
207+
208+
## Decisions
209+
210+
- **One file per concern:** user preference — separate files for ingestion, dense
211+
vectors, sparse vectors, retrieval, and configuration; plus README index.
212+
- **Two-checkpoint workflow per document:** (1) findings are shared with the user
213+
via AskUserQuestion before drafting begins — the user can correct or redirect
214+
the analysis; (2) the draft is presented to the user before the file is written
215+
to disk. This fulfills the "human in every part of the investigation" requirement
216+
at both the analysis and the writing stage.
217+
- **Observations included:** each doc has a findings/observations section surfacing
218+
design gaps and improvement ideas — user preference.
219+
- **Mixed audience:** docs include brief concept intros (what RAG is, what BM25 is)
220+
alongside design-specific detail.
221+
- **docs/ coexists with root README.md:** the existing root `README.md` already
222+
covers the RAG system at a high level (indexing, embedding providers, env vars).
223+
The new `docs/` files provide deeper, structured documentation of each concern.
224+
Rather than rewrite or condense the root README, this plan adds only a
225+
"Documentation" link section pointing to `docs/README.md`. The two sources are
226+
complementary: the root README serves as a project overview and quick-start;
227+
`docs/` serves as the authoritative deep-dive reference.
228+
- **No production code changes:** this plan produces only markdown files in `docs/`
229+
and a minor link addition to the root `README.md`.
230+
231+
## Non-Goals
232+
233+
- Documenting the parser subsystem (`server/parser/`) — out of scope; not part of
234+
the RAG ingestion or retrieval path.
235+
- Documenting git history indexing (`server/indexer/git_history.py`,
236+
`server/store/commit_store.py`) — separate concern from the RAG system.
237+
- Making code changes to fix observed gaps — observations are documented only;
238+
improvements are a separate plan.

‎README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,16 @@ to re-embed everything.
3333

3434
Want to get into more details? [Check out the blog!](blog.md)
3535

36+
## Documentation
37+
38+
In-depth documentation of the RAG system internals lives in [`docs/`](docs/README.md):
39+
40+
- [Ingestion pipeline](docs/ingestion.md) — how code is discovered, parsed, embedded, and stored
41+
- [Dense vectors](docs/dense-vectors.md) — embedding providers and text strategy
42+
- [Sparse vectors](docs/sparse-vectors.md) — BM25 and the code tokenizer
43+
- [Retrieval with RRF](docs/retrieval-rrf.md) — hybrid search and MCP tools
44+
- [Configuration](docs/configuration.md) — all environment variables and config.yaml
45+
3646
## Supported languages
3747

3848
Language is detected automatically from file extension or filename — no configuration needed.

‎docs/README.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# semcode RAG System — Documentation
2+
3+
semcode is an MCP server that provides **hybrid semantic search over code** from GitHub repositories. It parses source files into code symbols (classes, methods, functions) using Tree-sitter, indexes them with both dense and sparse embedding vectors, and retrieves them using Reciprocal Rank Fusion (RRF).
4+
5+
**What it indexes:** code symbols from any configured GitHub repository. Each symbol (class, method, function, interface) is stored as a Qdrant point with its source text, metadata, and two embedding vectors.
6+
7+
**How it serves queries:** AI clients connect via MCP and call tools like `search_code`, `find_symbol`, and `find_usages`. Each search runs a hybrid query — dense semantic vectors find conceptually similar code; BM25 sparse vectors find exact and partial identifier matches. RRF merges both rankings into a single ordered result list.
8+
9+
---
10+
11+
## Documentation Index
12+
13+
| Document | What it covers |
14+
|----------|---------------|
15+
| [ingestion.md](ingestion.md) | End-to-end indexing pipeline: GitHub file discovery, incremental change detection, parsing, embedding, upsert, and stale cleanup |
16+
| [dense-vectors.md](dense-vectors.md) | Dense embedding providers (Jina, Voyage, OpenAI, Ollama), the embedding text strategy, and provider selection |
17+
| [sparse-vectors.md](sparse-vectors.md) | BM25 sparse embeddings, the code identifier tokenizer, and the sparse vector format |
18+
| [retrieval-rrf.md](retrieval-rrf.md) | Hybrid search architecture, RRF fusion, name lookup, and the four MCP tool entry points |
19+
| [configuration.md](configuration.md) | All environment variables, `config.yaml` structure, and startup validation |
20+
21+
---
22+
23+
## Quick Start
24+
25+
1. **Configure services** — copy `config.example.yaml` to `config.yaml` and add your GitHub repositories. See [configuration.md](configuration.md) for all fields.
26+
2. **Set environment variables** — copy `.env.example` to `.env` and set at minimum `GITHUB_TOKEN`. The default embedding provider (`jina`) requires a locally running TEI container; for a hosted alternative, set `EMBEDDINGS_PROVIDER=voyage` and `VOYAGE_API_KEY=...`.
27+
3. **Start Qdrant and the server** — `make docker-up-jina` (local Jina) or `make docker-up-voyage` (Voyage API), then connect your MCP client to `http://localhost:8090`.
28+
29+
For full setup instructions, see the root [README](../README.md).

0 commit comments

Comments
 (0)