|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## What This Project Is |
| 6 | + |
| 7 | +**semcode** is an MCP (Model Context Protocol) server that provides hybrid semantic code search over GitHub repositories. It lets AI clients search codebases using natural language or symbol names by combining dense vector embeddings with BM25 sparse search and Tree-sitter-based code parsing. |
| 8 | + |
| 9 | +## Commands |
| 10 | + |
| 11 | +```bash |
| 12 | +# Install dependencies |
| 13 | +uv sync # production deps |
| 14 | +uv sync --group dev # include test/dev deps |
| 15 | + |
| 16 | +# Run tests |
| 17 | +uv run pytest # all tests |
| 18 | +uv run pytest tests/parser/ # parser tests only |
| 19 | +uv run pytest tests/test_pipeline.py # single test file |
| 20 | + |
| 21 | +# Docker (Qdrant + optional local Jina TEI) |
| 22 | +make docker-up # hosted embeddings provider |
| 23 | +make docker-up-jina # local Jina TEI embeddings |
| 24 | +make docker-logs # all container logs |
| 25 | +make docker-logs-semcode # semcode container only |
| 26 | + |
| 27 | +# Trigger indexing |
| 28 | +make index-code # POST /reindex |
| 29 | +make index-history # POST /reindex-history |
| 30 | + |
| 31 | +# Qdrant utilities |
| 32 | +make qdrant-clean # delete all collections |
| 33 | +make qdrant-dashboard # open Qdrant UI |
| 34 | +``` |
| 35 | + |
| 36 | +## Architecture |
| 37 | + |
| 38 | +All server code lives under `server/`. The system has two main phases: **ingestion** and **retrieval**. |
| 39 | + |
| 40 | +### Ingestion pipeline (`server/indexer/`) |
| 41 | + |
| 42 | +1. **Discovery** — `github_source.py` lists files in a GitHub repo and applies filters from config |
| 43 | +2. **Change detection** — compares Git blob SHAs to skip unchanged files |
| 44 | +3. **Parsing** — `parser/registry.py` routes each file to the correct language parser; each parser uses Tree-sitter to extract `CodeSymbol` objects (functions, classes, methods, etc.) with framework metadata |
| 45 | +4. **Embedding** — `embeddings/factory.py` creates two vectors per symbol: dense (Jina/Voyage/OpenAI/Ollama) + BM25 sparse (`embeddings/bm25.py` with camelCase/snake_case tokenization) |
| 46 | +5. **Upsert** — both vectors stored in Qdrant collection `code_symbols` with rich payload (language, type, file, line, signature, docstring, annotations, parent class, framework metadata) |
| 47 | +6. **Cleanup** — removes stale entries for deleted files/symbols |
| 48 | +7. **Git history** (optional) — `git_history.py` indexes commits into a separate `git_commits` collection (dense-only) |
| 49 | + |
| 50 | +### Retrieval (`server/store/`, `server/tools/`) |
| 51 | + |
| 52 | +- `store/qdrant.py` runs hybrid search via Qdrant `FusionQuery(fusion=RRF)` — combines dense and BM25 sparse results using Reciprocal Rank Fusion |
| 53 | +- `store/commit_store.py` handles commit search (dense-only) |
| 54 | +- `tools/` contains the MCP tool implementations: `search.py`, `index.py`, `history.py`, `admin.py` |
| 55 | + |
| 56 | +### MCP interface (`server/main.py`) |
| 57 | + |
| 58 | +FastMCP serves tools and prompts over stdio, SSE, or HTTP. HTTP indexing endpoints (`POST /reindex`, `POST /reindex-history`) return streaming NDJSON for CI/CD consumption. |
| 59 | + |
| 60 | +### Configuration |
| 61 | + |
| 62 | +`server/config.py` loads `config.yaml` via Pydantic. Services (repos to index), embedding provider, and Qdrant connection are all defined there. |
| 63 | + |
| 64 | +### Parser structure (`server/parser/`) |
| 65 | + |
| 66 | +One file per language (24 languages). Each implements the abstract `CodeParser` from `base.py` and returns a list of `CodeSymbol` dataclasses. When adding a new language: implement the parser, register it in `registry.py`, add fixture files under `tests/fixtures/<language>/`, and add snapshot tests under `tests/parser/`. |
| 67 | + |
| 68 | +### Embedding providers (`server/embeddings/`) |
| 69 | + |
| 70 | +Pluggable via `factory.py` registry. Swap providers in config without reindexing. BM25 sparse vectors are always generated alongside dense vectors. |
| 71 | + |
| 72 | +### Test structure |
| 73 | + |
| 74 | +``` |
| 75 | +tests/ |
| 76 | +├── parser/ # snapshot tests per language (24 files) |
| 77 | +├── fixtures/ # canonical code samples for parser tests |
| 78 | +├── embeddings/ # embedding provider tests |
| 79 | +├── test_pipeline.py # indexing pipeline integration |
| 80 | +├── test_store.py # Qdrant store operations |
| 81 | +├── test_reindex_route.py # HTTP endpoint tests |
| 82 | +└── conftest.py # shared fixtures |
| 83 | +``` |
| 84 | + |
| 85 | +Async tests use `asyncio_mode = "auto"` (pytest-asyncio). HTTP calls are mocked with `respx`. |
0 commit comments