Skip to content

Commit 9f491fe

Browse files
committed
update readme
1 parent 31cea4c commit 9f491fe

1 file changed

Lines changed: 74 additions & 0 deletions

File tree

README.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,80 @@ code-rag-cli
4040
# To use as MCP server see below section
4141
```
4242

43+
## Claude's Opinion About Code-RAG (take it with a huge grain of salt!)
44+
45+
<details>
46+
<summary>I have taken a lot of salt, my mouth is now saltier than the ocean, show me!</summary>
47+
48+
I ran 9 diverse semantic queries to test the tool across different aspects of the codebase. Here's my assessment:
49+
50+
### Strengths
51+
52+
1. Excellent Semantic Understanding
53+
- Query "shared embedding server lifecycle management" correctly identified the core design documentation and implementation (0.54 relevance)
54+
- "syntax-aware chunking for different languages" found both the implementation guide and actual code
55+
- Searches understood concepts, not just keywords - found relevant code even when exact terms didn't match
56+
57+
2. Good Relevance Scoring
58+
- Scores generally correlate well with actual relevance (0.4-0.6+ range for good matches)
59+
- Top results consistently more relevant than lower-ranked ones
60+
- The tool correctly prioritizes implementation code over test fixtures
61+
62+
3. Metadata Quality
63+
- Results include file paths, line numbers, and function/class context
64+
- Example: src/code_rag/api.py:443-462 incremental_reindex() | CodeRAGAPI (0.61)
65+
- This makes it easy to navigate to the exact location
66+
67+
4. Handles Complex Queries Well
68+
- Multi-concept queries like "incremental reindexing detect modified files" found the right code
69+
- Domain-specific terminology works well ("embedding model configuration", "MCP server tools")
70+
71+
### Weaknesses
72+
73+
1. Generic Queries Return Scattered Results
74+
- "error handling and retry logic" (query 5) returned less focused results
75+
- Found timeout handlers, error printing, but no cohesive retry logic
76+
- This is expected for overly broad queries, but worth noting
77+
78+
2. Some Test Code Noise
79+
- "database initialization and connection" returned test fixtures as top results instead of actual implementation
80+
- The mock database in test_mcp_server.py ranked higher (0.45) than real database code (0.35)
81+
- Could be improved with result filtering or boosting production code
82+
83+
3. Limited Context Window
84+
- Results show ~20-30 lines per chunk, which is reasonable but sometimes cuts off mid-implementation
85+
- The expand_context parameter helps but isn't always used
86+
87+
### Specific Observations
88+
89+
#### What Works Great:
90+
- Architecture/design queries: "shared embedding server lifecycle management" → perfect results
91+
- Implementation queries: "syntax-aware chunking" → found both docs and code
92+
- Specific subsystem queries: "incremental reindexing" → found the exact functions
93+
94+
#### What Could Be Better:
95+
- Very broad queries ("error handling") need more filtering
96+
- Test vs. production code disambiguation
97+
- Maybe add file type boosting (prefer src/ over tests/)
98+
99+
### Recommendations
100+
101+
1. Add query guidance in the tool description - Users should know that specific queries work better than generic ones (you already have this in the MCP tool description, which is good)
102+
2. Consider result filtering options - Allow excluding test files or prioritizing certain directories
103+
3. Expand context by default - The expand_context parameter could be enabled by default for better chunk visibility
104+
4. Add result grouping - Group results by file to show "This file has 3 relevant chunks"
105+
106+
### Overall Assessment
107+
108+
**Rating: 8.5/10**
109+
110+
The semantic search works remarkably well for its intended purpose. It successfully finds relevant code based on conceptual queries, not just keyword matching. The relevance scoring is solid, and the metadata makes results actionable.
111+
112+
The main improvement areas are around filtering test code and handling overly broad queries. For a developer using this tool, the key insight is: be specific in your queries. "authentication token refresh logic" will work better than just "authentication."
113+
114+
This is a genuinely useful tool that would save significant time when exploring unfamiliar codebases.
115+
</details>
116+
43117
## Why Use Code-RAG?
44118

45119
- **Understand unfamiliar codebases** - Ask questions instead of reading everything

0 commit comments

Comments
 (0)