Skip to content

Commit 0cae8f7

Browse files
AlexMikhalevAlexMikhalev
authored
feat(mcp): FFF Epic #222 closure - multi_grep, frecency, cursor pagination (#820)
* feat(mcp): add terraphim_multi_grep tool, frecency persistence, cursor pagination - Add terraphim_multi_grep MCP tool: OR-pattern search using Aho-Corasick with KG path-score ordering and constraint filtering - Wire SharedFrecency via FFF_FRECENCY_PATH env var for LMDB-backed persistent access-frequency scoring - Add cursor-based pagination to terraphim_grep and terraphim_multi_grep using base64-encoded file_offset cursors - Update test_find_files.rs for new grep_files signature (cursor param) - Clean compile, clippy-clean, all existing tests pass Refs #222 * fix(clippy): replace sort_by with sort_by_key in markdown-parser Fixes Rust 1.95 clippy unnecessary_sort_by lint on CI. * fix(clippy): replace sort_by with sort_by_key in terraphim_router Fixes Rust 1.95 clippy unnecessary_sort_by lint on CI. * fix(clippy): replace sort_by with sort_by_key in session-analyzer Fixes Rust 1.95 clippy unnecessary_sort_by lint on CI. * fix(clippy): replace sort_by with sort_by_key in terraphim_update Fixes Rust 1.95 clippy unnecessary_sort_by lint on CI. * fix(clippy): fix Rust 1.95 lints in rolegraph examples and task_decomposition - Replace explicit counter loops with enumerate() in examples - Remove useless into_iter() in extend() call - Allow collapsible_match lint in task validation * fix(clippy): fix Rust 1.95 lints in rolegraph examples - Replace explicit counter loops with enumerate() - Allow useless_vec for vec! in for-loop bindings - Remove unused next_id variable * fix(clippy): fix remaining Rust 1.95 lints in session-analyzer and rolegraph examples - sort_by -> sort_by_key in session-analyzer main.rs - Allow useless_vec in knowledge_graph_role_demo example * fix(clippy): replace sort_by with sort_by_key in kg_normalization example * fix(clippy): replace sort_by with sort_by_key in terraphim_persistence * fix(clippy): fix Rust 1.95 lints in agent_evolution and middleware - sort_by -> sort_by_key for Copy-type comparisons - Allow collapsible_match and unnecessary_sort_by for non-Copy types * fix(clippy): fix Rust 1.95 lints in goal_alignment and service - sort_by -> sort_by_key for Copy-type comparisons - Allow collapsible_match and unnecessary_sort_by for complex closures * fix(clippy): replace sort_by with sort_by_key in multi_agent context * fix(clippy): fix Rust 1.95 lints in remaining files - registry.rs: replace sort_by with sort_by_key - validation.rs: collapse if into match guards - handler.rs: allow unnecessary_sort_by for non-Copy rank - service.rs: allow unnecessary_sort_by for tuple field sort - routing.rs: collapse if into match guards - markdown_parser.rs: replace explicit counter with enumerate * fix(clippy): fix remaining sort_by lints in learnings and main - capture.rs: allow unnecessary_sort_by for captured_at and score - main.rs: allow unnecessary_sort_by for pos and rank sorting * fix(clippy): allow unnecessary_sort_by in terraphim_cli service * fix(clippy): allow unnecessary_sort_by in mcp_server scoring * fix(clippy): add unnecessary_sort_by allow for all remaining sort_by calls Batch fix for Rust 1.95 clippy lint across 11 files: - terraphim-session-analyzer, terraphim_agent, terraphim_agent_evolution - terraphim_automata, terraphim_sessions, terraphim_tinyclaw - terraphim_usage
1 parent 8e454cb commit 0cae8f7

45 files changed

Lines changed: 736 additions & 115 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.docs/design-fff-epic-222-closure.md‎

Lines changed: 244 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
# Implementation Plan: Close FFF Epic #222
2+
3+
**Status**: Approved
4+
**Research Doc**: `.docs/research-fff-epic-222-closure.md`
5+
**Author**: opencode (disciplined-design)
6+
**Date**: 2026-04-17
7+
**Estimated Effort**: 4-6 hours
8+
9+
## Overview
10+
11+
### Summary
12+
Complete the 4 remaining items to close FFF Epic #222: add `terraphim_multi_grep` MCP tool, wire SharedFrecency persistence, add cursor pagination, and remove fff-mcp sidecar.
13+
14+
### Approach
15+
Follow the exact pattern established by `terraphim_find_files` and `terraphim_grep`. Three workstreams can execute in parallel since they touch different parts of the MCP server.
16+
17+
### Scope
18+
**In Scope:**
19+
1. Add `terraphim_multi_grep` MCP tool (OR-pattern grep)
20+
2. Wire `SharedFrecency` with configurable LMDB path
21+
3. Add cursor-based pagination to all three tools
22+
4. Remove fff-mcp sidecar from bigbox
23+
5. Close stale sub-issues #225, #226
24+
25+
**Out of Scope:**
26+
- Upstream ExternalScorer trait PR to fff.nvim
27+
- KG content scoring (path-only scoring stays)
28+
- Query completion tracking
29+
30+
**Avoid At All Cost:**
31+
- Refactoring existing find_files/grep implementations (they work)
32+
- Adding configuration file format changes (use env vars/CLI args)
33+
- Abstracting MCP tool registration (YAGNI -- 3 tools is fine)
34+
35+
## Architecture
36+
37+
### Data Flow
38+
```
39+
MCP Client
40+
-> terraphim_find_files (fuzzy match + KG boost)
41+
-> terraphim_grep (content search + KG file ordering + cursor)
42+
-> terraphim_multi_grep (multi-pattern OR search + KG file ordering + cursor)
43+
-> SharedFrecency (LMDB) (persistent access frequency across sessions)
44+
```
45+
46+
### Key Design Decisions
47+
48+
| Decision | Rationale | Alternatives Rejected |
49+
|----------|-----------|----------------------|
50+
| Use `multi_grep_search` directly from fff-core | Already public, no fork needed | Wrapping in our own trait |
51+
| CursorStore as HashMap<String, usize> | Simple, matches fff-mcp pattern | Redis/external store |
52+
| SharedFrecency via configurable env var | `FFF_FRECENCY_PATH` -- zero config change | New config section in TOML |
53+
| Pagination as offset-based | Matches fff-core's `file_offset` in GrepSearchOptions | Keyset pagination |
54+
55+
## File Changes
56+
57+
### Modified Files
58+
| File | Changes |
59+
|------|---------|
60+
| `crates/terraphim_mcp_server/src/lib.rs` | Add multi_grep tool, frecency wiring, cursor store, pagination |
61+
| `crates/terraphim_mcp_server/Cargo.toml` | No changes needed (fff-search already a dep) |
62+
63+
### No new files. No deleted files (sidecar removal is ops, not code).
64+
65+
## Implementation Steps
66+
67+
### Workstream A: terraphim_multi_grep (Parallel -- 1-2h)
68+
69+
**Step A1: Add multi_grep method on McpService**
70+
71+
File: `crates/terraphim_mcp_server/src/lib.rs`
72+
Location: After `grep_files` method (~line 1336)
73+
74+
```rust
75+
pub async fn multi_grep_files(
76+
&self,
77+
patterns: Vec<String>,
78+
path: Option<String>,
79+
constraints: Option<String>,
80+
limit: Option<usize>,
81+
cursor: Option<String>,
82+
output_mode: Option<String>,
83+
) -> Result<CallToolResult, ErrorData> {
84+
let base_path = path.unwrap_or_else(|| ".".to_string());
85+
let max_results = limit.unwrap_or(50);
86+
let files_only = output_mode.as_deref() == Some("files");
87+
88+
// Same FilePicker init as grep_files
89+
let mut picker = FilePicker::new(FilePickerOptions { ... })?;
90+
picker.collect_files()?;
91+
let mut files = picker.get_files().to_vec();
92+
93+
// KG sort (same as grep_files)
94+
if let Some(scorer) = &self.kg_scorer {
95+
files.sort_by(|a, b| scorer.score(b).cmp(&scorer.score(a)));
96+
}
97+
98+
// Parse constraints
99+
let patterns_refs: Vec<&str> = patterns.iter().map(|s| s.as_str()).collect();
100+
let options = GrepSearchOptions { file_offset: cursor_offset, page_limit: max_results, ... };
101+
102+
let result = multi_grep_search(&files, &patterns_refs, &constraints_parsed, &options, &budget, None);
103+
104+
// Format output (same pattern as grep_files)
105+
// Return with next_cursor
106+
}
107+
```
108+
109+
**Step A2: Register tool in get_info()**
110+
111+
Location: After `terraphim_grep` tool entry (~line 1749)
112+
113+
```rust
114+
Tool {
115+
name: "terraphim_multi_grep".into(),
116+
description: "Search file contents for lines matching ANY of multiple patterns (OR logic). ...".into(),
117+
input_schema: serde_json::json!({
118+
"type": "object",
119+
"properties": {
120+
"patterns": { "type": "array", "items": { "type": "string" }, "description": "Patterns to match (OR logic)" },
121+
"path": { "type": "string", "description": "Base directory" },
122+
"constraints": { "type": "string", "description": "File constraints (e.g. '*.rs !test/')" },
123+
"limit": { "type": "integer", "description": "Max results (default 50)" },
124+
"cursor": { "type": "string", "description": "Pagination cursor from previous result" },
125+
"output_mode": { "type": "string", "enum": ["content", "files"], "description": "Output format" }
126+
},
127+
"required": ["patterns"]
128+
}),
129+
}
130+
```
131+
132+
**Step A3: Add match arm in call_tool()**
133+
134+
Location: After `"terraphim_grep"` arm (~line 2159)
135+
136+
```rust
137+
"terraphim_multi_grep" => { ... }
138+
```
139+
140+
### Workstream B: SharedFrecency Wiring (Parallel -- 2-3h)
141+
142+
**Step B1: Add frecency field and initialization**
143+
144+
File: `crates/terraphim_mcp_server/src/lib.rs`
145+
146+
McpService struct already has `frecency` (check). If not, add:
147+
```rust
148+
pub struct McpService {
149+
// ... existing fields ...
150+
frecency: Option<SharedFrecency>,
151+
}
152+
```
153+
154+
Constructor: initialise from env var `FFF_FRECENCY_PATH`:
155+
```rust
156+
let frecency = std::env::var("FFF_FRECENCY_PATH")
157+
.ok()
158+
.map(|path| {
159+
// Init LMDB-backed frecency at path
160+
SharedFrecency::new(&path) // or equivalent from fff-search API
161+
})
162+
.transpose()?;
163+
```
164+
165+
**Step B2: Pass frecency to FilePicker**
166+
167+
In `find_files` and `grep_files` and `multi_grep_files`:
168+
```rust
169+
if let Some(frecency) = &self.frecency {
170+
picker.update_frecency_scores(frecency);
171+
}
172+
```
173+
174+
### Workstream C: Cursor Pagination (Parallel -- 1-2h)
175+
176+
**Step C1: Add CursorStore to McpService**
177+
178+
```rust
179+
pub struct McpService {
180+
// ... existing ...
181+
cursor_store: Arc<Mutex<HashMap<String, usize>>>,
182+
}
183+
```
184+
185+
**Step C2: Add cursor handling to grep_files and multi_grep_files**
186+
187+
Parse `cursor` param -> lookup offset from store.
188+
After results, if more available, generate new cursor token:
189+
```rust
190+
let next_cursor = if result.matches.len() > max_results {
191+
let token = format!("cur_{}", uuid::Uuid::new_v4());
192+
self.cursor_store.lock().unwrap().insert(token.clone(), offset + max_results);
193+
Some(token)
194+
} else {
195+
None
196+
};
197+
```
198+
199+
Include `next_cursor` in the response Content.
200+
201+
### Workstream D: Cleanup (Sequential -- after A,B,C -- 30min)
202+
203+
**Step D1: Close stale Gitea issues**
204+
- Close #225 (research) with comment "Work completed out of order during Phase 3 implementation"
205+
- Close #226 (design) with same comment
206+
207+
**Step D2: Update #224**
208+
- Comment with completion status
209+
- Check off remaining items
210+
211+
**Step D3: Remove fff-mcp sidecar from bigbox**
212+
- `ssh bigbox` -- check if fff-mcp is still running
213+
- Check if any other tool references it
214+
- Stop service, remove from MCP configs
215+
216+
**Step D4: Close epic #222**
217+
218+
## Test Strategy
219+
220+
### Unit Tests (in terraphim_mcp_server)
221+
| Test | Purpose |
222+
|------|---------|
223+
| `test_multi_grep_multiple_patterns` | Verify OR logic returns files matching any pattern |
224+
| `test_multi_grep_no_matches` | Empty result for non-existent patterns |
225+
| `test_cursor_store_round_trip` | Store and retrieve offset |
226+
| `test_cursor_pagination_limit` | Verify next_cursor only when more results exist |
227+
228+
### Integration Tests
229+
- `cargo test -p terraphim_mcp_server` -- existing tests must still pass
230+
- Manual: invoke `terraphim_multi_grep` with patterns `["sort_by", "sort_by_key"]` and verify results
231+
232+
## Execution Order (Max Parallelism)
233+
234+
```
235+
Time Workstream A Workstream B Workstream C Workstream D
236+
---- ----------- ----------- ----------- -----------
237+
T+0 A1: multi_grep B1: frecency init C1: CursorStore
238+
T+1 A2: register tool B2: wire to picker C2: pagination
239+
T+2 A3: call_tool arm (verify)
240+
T+3 (verify + test) (verify + test) (verify + test)
241+
T+4 D1-D4: cleanup
242+
```
243+
244+
All three workstreams are independent -- they can be three separate commits on one branch, or three parallel branches merged sequentially.

‎.docs/research-fff-epic-222-closure.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Research Document: Close FFF Epic #222 -- Remaining Work
2+
3+
**Status**: Approved
4+
**Author**: opencode (disciplined-research)
5+
**Date**: 2026-04-17
6+
**Related**: Gitea #222 (epic), #223 (closed), #224 (Phase 2 remaining), #225/#226 (stale), #227 (Phase 3 done)
7+
8+
## Executive Summary
9+
10+
FFF integration is ~80% complete. Phase 1 (sidecar) and Phase 3 (KG-boosted scoring crate) are done. Four remaining items block epic closure: `terraphim_multi_grep` MCP tool, SharedFrecency persistence, cursor pagination, and sidecar removal. All are well-understood with clear reference implementations in fff-mcp.
11+
12+
## Essential Questions Check
13+
14+
| Question | Answer | Evidence |
15+
|----------|--------|----------|
16+
| Energizing? | Yes | Closes a major epic, cleans up technical debt |
17+
| Leverages strengths? | Yes | Pattern already established in terraphim_grep/find_files |
18+
| Meets real need? | Yes | Agents need multi-pattern search and persistent frecency |
19+
20+
**Proceed**: Yes (3/3)
21+
22+
## Current State Analysis
23+
24+
### Code Locations
25+
| Component | Location | Status |
26+
|-----------|----------|--------|
27+
| terraphim_file_search crate | `crates/terraphim_file_search/` | Done: lib.rs, kg_scorer.rs, config.rs, watcher.rs |
28+
| MCP terraphim_find_files | `crates/terraphim_mcp_server/src/lib.rs:1170-1244` | Done |
29+
| MCP terraphim_grep | `crates/terraphim_mcp_server/src/lib.rs:1250-1336` | Done |
30+
| MCP terraphim_multi_grep | -- | **Missing** |
31+
| SharedFrecency wiring | `crates/terraphim_mcp_server/src/lib.rs:59` | **Not wired** |
32+
| Cursor pagination | `crates/terraphim_mcp_server/src/lib.rs` (next_cursor: None) | **Not implemented** |
33+
| fff-mcp sidecar | bigbox: PID running | **Not removed** |
34+
35+
### Reference: fff-mcp multi_grep Implementation
36+
Location: `~/.cargo/git/checkouts/fff.nvim-14ad43e6a8691b70/efd1552/crates/fff-mcp/src/server.rs:545-594`
37+
38+
Key API: `grep::multi_grep_search(files, &patterns_refs, constraints, &options, budget, None)`
39+
- Takes `Vec<&str>` patterns (OR logic)
40+
- Uses same `GrepSearchOptions` as single grep
41+
- Returns same `GrepResult` type
42+
- Has `CursorStore` for pagination
43+
44+
### Reference: fff-core SharedFrecency
45+
Location: `~/.cargo/git/checkouts/fff.nvim-14ad43e6a8691b70/efd1552/crates/fff-core/src/shared.rs:120`
46+
47+
```rust
48+
pub struct SharedFrecency(pub(crate) Arc<RwLock<Option<FrecencyTracker>>>);
49+
```
50+
- Already exported from `fff-search` crate
51+
- Current terraphim MCP: `SharedFrecency` is imported but field `frecency` not used (0 references to it)
52+
53+
### Existing MCP Tool Pattern (for adding new tools)
54+
1. Add async method on `McpService` (e.g., `find_files`, `grep_files`)
55+
2. Add `Tool` entry in `ServerHandler::get_info()` (~line 1722, 1749)
56+
3. Add match arm in `ServerHandler::call_tool()` (~line 2139, 2159)
57+
58+
## Remaining Work Items (from #224)
59+
60+
| Item | Effort | Dependencies | Parallelizable? |
61+
|------|--------|-------------|-----------------|
62+
| 1. Add terraphim_multi_grep MCP tool | 1-2h | None | Yes |
63+
| 2. Wire SharedFrecency with LMDB persistence | 2-3h | fff-search exposes FrecencyTracker | Yes |
64+
| 3. Add cursor-based pagination | 2-3h | None | Yes |
65+
| 4. Remove standalone fff-mcp sidecar | 30min | Items 1-3 validated | No (last) |
66+
| 5. Close stale sub-issues #225, #226 | 5min | None | Yes |
67+
| 6. Update epic #222 and close #224 | 5min | All above | No (last) |
68+
69+
## Constraints
70+
71+
### Technical
72+
- `fff-search` is a git dependency (branch `feat/external-scorer`) -- cannot modify its API
73+
- `multi_grep_search` is already public in `fff-search::grep` -- no fork needed
74+
- `SharedFrecency` requires an LMDB path -- must be configurable
75+
- MCP server uses `rmcp` 0.9 for protocol -- tool registration pattern is fixed
76+
- `CursorStore` in fff-mcp uses opaque string IDs -- we can replicate or simplify
77+
78+
### Vital Few
79+
80+
| Constraint | Why It's Vital | Evidence |
81+
|------------|----------------|----------|
82+
| multi_grep_search API is public | Enables multi-pattern tool without forking | fff-core/src/grep.rs:870 |
83+
| Existing tools are the pattern | New tools follow find_files/grep pattern | lib.rs:1170-1336 |
84+
| SharedFrecency already imported | Just needs wiring, not new code | lib.rs:7 imports it |
85+
86+
### Eliminated from Scope
87+
88+
| Eliminated Item | Why Eliminated |
89+
|-----------------|----------------|
90+
| Upstream ExternalScorer trait contribution | Not blocking, separate effort |
91+
| KG content scoring (score file contents, not just paths) | Future enhancement, not in epic scope |
92+
| Query completion tracking | Nice-to-have, frecency is sufficient |
93+
94+
## Risks
95+
96+
| Risk | Likelihood | Impact | Mitigation |
97+
|------|------------|--------|------------|
98+
| LMDB path not writable on bigbox | Low | Medium | Configurable path, fallback to temp |
99+
| multi_grep_search API differs from grep_search | Low | Low | Read fff-mcp server.rs reference |
100+
| Cursor pagination state lost on MCP restart | Medium | Low | Expected; cursors are ephemeral |
101+
| fff-mcp sidecar still used by other tools | Medium | Medium | Check before removing |
102+
103+
## Assumptions
104+
105+
| Assumption | Basis | Risk if Wrong |
106+
|------------|-------|---------------|
107+
| `multi_grep_search` has same signature style as `grep_search` | Both in fff-core/src/grep.rs, same author | Low -- signature differs only in `Vec<&str>` vs single pattern |
108+
| SharedFrecency can be initialised with a path | fff-core has `init_db(path)` function | Medium -- may need to check if LMDB is available on target |
109+
| fff-mcp sidecar is not used by other projects | Only terraphim-ai configured it | Medium -- verify before removing |

‎crates/terraphim-markdown-parser/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@ pub fn ensure_terraphim_block_ids(markdown: &str) -> Result<String, MarkdownPars
141141
}
142142

143143
// Apply edits from the end of the buffer to the beginning so byte offsets stay valid.
144-
edits.sort_by(|a, b| b.range.start.cmp(&a.range.start));
144+
edits.sort_by_key(|e| std::cmp::Reverse(e.range.start));
145145
let mut out = markdown.to_string();
146146
for edit in edits {
147147
out.replace_range(edit.range, &edit.replacement);

0 commit comments

Comments
 (0)