Skip to content

Commit 1aee417

Browse files
author
Test User
committed
docs(grep): add terraphim-grep offline setup + ast-grep KG enrichment guide
Lessons-learned how-to for enabling terraphim-grep offline (no server) with a project-specific knowledge graph. Covers: - the stock-install gotcha: code-search (fff-search) is opt-in, so a default build returns 0 chunks for every query (fix: --features code-search) - architecture: chunks come from fff-search; the KG/thesaurus only boosts - .terraphim/ project layout (kg markdown, thesaurus.json format, config) - enriching the thesaurus with ast-grep-mined code anchors (struct/trait/impl identifiers) for precise token matching, with name-selective bucketing - diagnosis table + gotchas checklist Related issue: #3025 (ship code-search by default).
1 parent bb8062a commit 1aee417

1 file changed

Lines changed: 271 additions & 0 deletions

File tree

‎docs/terraphim-grep-offline-setup.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# Enabling terraphim-grep Offline (lessons learned)
2+
3+
**Date:** 2026-06-29
4+
**Scope:** Getting `terraphim-grep` to return real results offline, with a project-specific knowledge graph, no server required.
5+
6+
This is an operational lessons-learned note, not a design doc. It captures the
7+
exact failure mode that makes a freshly-installed `terraphim-grep` return zero
8+
results for every query, and the minimal setup to get KG-driven search working
9+
against any project.
10+
11+
---
12+
13+
## TL;DR — the one thing that bites everyone
14+
15+
A stock `cargo install terraphim_grep` binary **does not search code**.
16+
17+
`terraphim_grep`'s `Cargo.toml` gates the actual code scanner behind an opt-in
18+
feature:
19+
20+
```toml
21+
[features]
22+
code-search = ["dep:fff-search"] # fff-search = the code scanner
23+
default = ["llm"] # <-- code-search is NOT default
24+
```
25+
26+
The default install compiles `search_code()` down to a stub that returns
27+
`Ok(vec![])`:
28+
29+
```rust
30+
// terraphim_grep/src/hybrid_searcher.rs
31+
async fn search_code(..) {
32+
#[cfg(feature = "code-search")] { /* real fff-search */ }
33+
#[cfg(not(feature = "code-search"))] { let _ = (..); Ok(vec![]) } // <- you are here
34+
}
35+
```
36+
37+
So **every query returns `chunks_returned: 0` in `search_latency_ms: 0`**, with
38+
no error. Rebuild with the feature enabled:
39+
40+
```bash
41+
cargo install terraphim_grep --version 1.20.5 --features code-search --force
42+
```
43+
44+
That is the fix. Everything below is diagnosis and project setup.
45+
46+
---
47+
48+
## Symptom → diagnosis → fix
49+
50+
| Symptom | Meaning | Fix |
51+
|--------|---------|-----|
52+
| `chunks_returned: 0`, `search_latency_ms: 0`, no error | fff-search is compiled out (no `code-search` feature) | rebuild with `--features code-search` |
53+
| `chunks_returned > 0` but `kg_hits: 0` | fff-search works; thesaurus not loaded / not matching | pass `--thesaurus`; check synonym terms |
54+
| `Graph has no nodes yet - no documents have been indexed` | rolegraph has no document nodes | **ignore for grep** — the thesaurus-only fallback still boosts; only matters for `terraphim-agent` |
55+
| `No thesaurus specified and could not find default` | `--thesaurus` missing and no auto-discovered file | pass `--thesaurus`, or place `.terraphim/thesaurus-<role>.json` |
56+
| `Failed to compute source hash for ...kg` | broken symlinks in the global KG dir poison the build | `find ~/.config/terraphim/kg -xtype l -delete` |
57+
58+
**The single reliable signal** that fff-search never ran is
59+
`search_latency_ms: 0` combined with `chunks_returned: 0` for a query that
60+
*must* exist in the codebase. Real scans take single-digit-to-hundreds of ms.
61+
62+
---
63+
64+
## Architecture: why it works offline (and what fooled me)
65+
66+
`terraphim-grep` is **offline-first**. The `terraphim-agent` help calls itself
67+
"server-backed", which mislead me into thinking grep also needed the server. It
68+
does not.
69+
70+
```
71+
terraphim-grep "query" --paths . --thesaurus t.json
72+
│
73+
HybridSearcher::search() (parallel tokio tasks)
74+
├── search_code() → fff-search over --paths → code_results (THE chunks)
75+
└── search_kg() → rolegraph query
76+
└─ if graph empty → thesaurus-only fallback
77+
│
78+
boost_chunks_with_kg(code_results, kg_concepts) ← KG only RE-RANKS
79+
│
80+
results
81+
```
82+
83+
Two consequences worth internalising:
84+
85+
1. **Chunks come from fff-search, not from the KG.** The thesaurus/rolegraph
86+
only *boosts* (re-ranks) chunks. So an empty rolegraph is **not** fatal —
87+
there is an explicit thesaurus-only fallback in `search_kg` so boosting still
88+
fires when no documents are indexed.
89+
90+
2. **The rolegraph rabbit hole is a trap.** I burned real time believing the
91+
`Graph has no nodes` message (and the missing `terraphim_server`) was the
92+
blocker. It was a red herring. The real blocker was the feature flag. When
93+
grep returns zero, look at the *code-search feature* first, the graph last.
94+
95+
Source of truth (in the published crate, ~`~/.cargo/registry/src/.../terraphim_grep-<ver>/`):
96+
`src/hybrid_searcher.rs` (`search`, `search_code`, `search_kg`) and `Cargo.toml`.
97+
98+
---
99+
100+
## Project setup: a self-contained `.terraphim/`
101+
102+
Drop this into any project root. No global config edits required.
103+
104+
```
105+
.terraphim/
106+
├── kg/ # KG concepts as markdown
107+
│ ├── provider.md
108+
│ ├── session.md
109+
│ └── ...
110+
├── thesaurus.json # compiled synonym -> {id, nterm} map
111+
└── config.json # (optional) Role for --role-config
112+
```
113+
114+
### KG markdown format
115+
116+
One concept per file. The `synonyms::` line is what the thesaurus is built from:
117+
118+
```markdown
119+
# Provider
120+
121+
Abstract LLM backend implementing the `Provider` trait (`src/provider.rs`).
122+
Description, key files, related concepts.
123+
124+
synonyms:: provider, llm backend, anthropic, openai, gemini, cohere, azure, bedrock, vertex, copilot, kimi
125+
```
126+
127+
### Thesaurus format (must match exactly)
128+
129+
```json
130+
{
131+
"name": "My Project Engineer",
132+
"data": {
133+
"provider": { "id": 100, "nterm": "provider" },
134+
"llm backend": { "id": 100, "nterm": "provider" },
135+
"anthropic": { "id": 100, "nterm": "provider" }
136+
}
137+
}
138+
```
139+
140+
Each synonym maps to `{id, nterm}` where `nterm` is the normalised concept.
141+
All synonyms of one concept share its `id`/`nterm`. Generate it from the
142+
markdown by parsing the `# Title` and the `synonyms::` line — a ~15-line
143+
Python script is enough (lowercase keys, dedupe, sequential ids).
144+
145+
### Generate from KG markdown
146+
147+
```bash
148+
python3 - <<'PY'
149+
import os, json, glob
150+
data, cid = {}, 100
151+
for f in sorted(glob.glob("kg/*.md")):
152+
nterm = os.path.splitext(os.path.basename(f))[0]
153+
txt = open(f, encoding="utf-8").read()
154+
syn = next((l.split("::",1)[1] for l in txt.splitlines()
155+
if l.strip().lower().startswith("synonyms::")), "")
156+
for t in dict.fromkeys([nterm] + [s.strip().lower() for s in syn.split(",") if s.strip()]):
157+
data.setdefault(t.lower(), {"id": cid, "nterm": nterm})
158+
cid += 1
159+
json.dump({"name": "My Project Engineer", "data": data}, open("thesaurus.json","w"), indent=2)
160+
PY
161+
```
162+
163+
---
164+
165+
## Run it
166+
167+
```bash
168+
cd /path/to/project
169+
terraphim-grep "session persistence" \
170+
--paths . --thesaurus .terraphim/thesaurus.json -n 8 -C 2
171+
172+
# structured:
173+
terraphim-grep "provider" --paths . --thesaurus .terraphim/thesaurus.json --json
174+
```
175+
176+
A healthy result:
177+
178+
```json
179+
{ "stats": { "search_latency_ms": 306, "chunks_returned": 5, "kg_hits": 3 },
180+
"concepts": [ {"name":"provider"}, {"name":"sse"} ], "sufficiency": "SearchOnly" }
181+
```
182+
183+
KG-boosted chunks score above the fff baseline (1.0); a chunk whose path/content
184+
matches a thesaurus concept rises to ~3.0, so your project vocabulary directly
185+
shapes ranking.
186+
187+
---
188+
189+
## Gotchas checklist
190+
191+
- [ ] Rebuilt with `--features code-search`? (Default install is useless for search.)
192+
- [ ] `--thesaurus` passed and parsed? (`Loaded thesaurus with N entries` in debug log.)
193+
- [ ] `search_latency_ms > 0`? (Zero = fff-search never ran.)
194+
- [ ] Global KG broken symlinks cleaned? (`find ~/.config/terraphim/kg -xtype l -delete`.)
195+
- [ ] Query terms lowercased in the thesaurus? (Aho-Corasick matching is case-sensitive on the keys.)
196+
197+
---
198+
199+
## Enriching the KG with ast-grep (code anchors)
200+
201+
Prose synonyms only get you so far — a query for `ReadTool` or `AnthropicProvider`
202+
won't boost the right chunks unless those exact code identifiers are in the
203+
thesaurus. Use **ast-grep** to mine real identifiers and add them as anchors.
204+
205+
### Why it helps
206+
207+
KG-boosting (`boost_chunks_with_kg`) raises a chunk's score when its path or
208+
content contains a thesaurus key. Adding real code tokens (`ReadTool`,
209+
`ModelRegistry`, `AuthStorage`, `SseParser`, `AnthropicProvider`) means a query
210+
for that token matches the concept **and** ranks the actual definition file
211+
first. Observed: `ReadTool` → concept `tool`, `tools.rs` boosted to 3.0 vs 1.0
212+
baseline; `AnthropicProvider` → concept `provider`, `providers/anthropic.rs` to
213+
4.0 (three concept hits).
214+
215+
### Extract with ast-grep
216+
217+
```bash
218+
ast-grep run -l Rust -p 'pub struct $NAME { $$$BODY }' src --json=compact > structs.json
219+
ast-grep run -l Rust -p 'pub enum $NAME { $$$BODY }' src --json=compact > enums.json
220+
ast-grep run -l Rust -p 'pub trait $NAME { $$$BODY }' src --json=compact > traits.json
221+
ast-grep run -l Rust -p 'impl $T for $NAME { $$$BODY }' src --json=compact > impls.json
222+
```
223+
224+
Captured names live at `match["metaVariables"]["single"]["NAME"]["text"]` (the
225+
top-level JSON is a **list** of matches; `metaVariables.single`, not a bare
226+
`NAME`).
227+
228+
### Bucket by concept — selectively
229+
230+
Map each identifier to a concept with **name-based** rules. Do **not** use
231+
file-wide catch-alls like `f.startswith("src/providers/")` — they flood the
232+
thesaurus with peripheral request/response types and metrics noise.
233+
234+
```python
235+
rules = {
236+
"provider": lambda n,f: n.endswith("Provider"),
237+
"session": lambda n,f: "Session" in n,
238+
"extension": lambda n,f: "Hostcall" in n or n.startswith("Extension") or "Capability" in n,
239+
"model-registry": lambda n,f: "Model" in n,
240+
"sse": lambda n,f: "Sse" in n or n == "StreamEvent",
241+
"acp": lambda n,f: "Acp" in n,
242+
"auth": lambda n,f: "Auth" in n or "OAuth" in n,
243+
"interactive-tui": lambda n,f: ("Picker" in n or "Selector" in n or n == "PiApp") and "src/interactive" in f,
244+
"tool": lambda n,f: n.endswith("Tool") or n == "ToolRegistry",
245+
"hashline-edit": lambda n,f: n == "HashlineEditTool",
246+
}
247+
```
248+
249+
Append each bucket to the matching `kg/<concept>.md` `synonyms::` line
250+
(idempotently — only add lowercased terms not already present), then regenerate
251+
`thesaurus.json`. A working, idempotent version of this whole loop lives at
252+
`pi_agent_rust`'s `.terraphim/scripts/refresh-anchors.sh` — run it whenever the
253+
code changes to keep anchors aligned.
254+
255+
### Verify
256+
257+
```bash
258+
terraphim-grep 'ReadTool' --paths src --thesaurus .terraphim/thesaurus.json -n 3 --json
259+
# expect: concepts include 'tool', top chunk = tools.rs at score > 1.0
260+
```
261+
262+
---
263+
264+
## Lessons, bluntly
265+
266+
1. **A feature-gated no-op stub is indistinguishable from "working but empty"** unless you read the source. `Ok(vec![])` behind `cfg(not(feature))` returns success with zero items — no error, no warning. Always confirm a search tool actually *scans* (non-zero latency) before debugging results.
267+
2. **"Server-backed" in a sibling tool's tagline does not mean every CLI in the family needs the server.** terraphim-grep is offline-capable; the rolegraph-empty messages are advisory, not blocking.
268+
3. **Match the diagnostic to the layer.** Zero chunks → code scanner. Zero boost → thesaurus. Both are independent; fix them independently.
269+
4. **Keep the project KG in the repo** (`.terraphim/`), version-controlled. It is documentation that doubles as a search index.
270+
5. **The best thesaurus terms are the code's own identifiers.** Prose synonyms match intent; ast-grep-mined struct/trait/impl names match the actual text in chunks. Enrich the KG structurally and keep it fresh with a repeatable script — a hand-maintained thesaurus rots.
271+
6. **Selectivity beats coverage.** File-wide bucketing rules (`f.startswith("src/providers/")`) drown the thesaurus in noise; name-based rules (`n.endswith("Provider")`) keep it signal-rich.

0 commit comments

Comments
 (0)