You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|**Backend**| Optimize | 60 | Final tree optimization |
38
44
39
-
Each stage is independently configurable. The pipeline supports incremental re-indexing via content fingerprinting.
45
+
Each pass is independently configurable. The pipeline supports incremental recompilation via content fingerprinting and checkpoint/resume for fault tolerance.
46
+
47
+
### Agent Acceleration Data
48
+
49
+
The backend passes produce pre-computed acceleration data used by Workers during retrieval:
50
+
51
+
| Data | Pass | Purpose |
52
+
|------|------|---------|
53
+
|**QueryRoutingTable**| Route | Maps intents and concepts to scored target nodes |
|**ContentOverlapMap**| Overlap | Flags duplicate/subset/summary overlap between nodes |
56
+
|**EvidenceScoreMap**| Score | Ranks nodes by information density and data richness |
57
+
58
+
This data is injected as Phase 1.5 hints into the Worker's navigation plan, allowing the LLM to make informed routing decisions without additional navigation steps.
40
59
41
60
## Tree Structure
42
61
@@ -104,7 +123,7 @@ When the user specifies document IDs directly, the Orchestrator skips the analys
104
123
Each Worker navigates a single document's tree to collect evidence through a command-based loop:
105
124
106
125
1.**Bird's-eye** — `ls` the root for an overview
107
-
2.**Plan** — LLM generates a navigation plan based on keyword index hits
126
+
2.**Plan** — LLM generates a navigation plan based on keyword index hits + acceleration data
1. When keyword index hits are available, Workers use `find` with the exact keyword to jump directly to relevant sections
133
152
2. Workers use `ls` when no keyword hints exist or when discovering unknown structure
134
153
3. Workers use `findtree` when the section title pattern is known but not the exact name
154
+
4. Pre-computed acceleration data (routes, scores, chains) is injected as Phase 1.5 hints to guide the Worker toward high-value nodes
135
155
136
156
#### Dynamic Re-planning
137
157
@@ -149,11 +169,11 @@ The system returns raw evidence text — no LLM synthesis or paraphrasing. This
149
169
150
170
## DocCard Catalog
151
171
152
-
When multiple documents are indexed, Vectorless maintains a lightweight `catalog.bin` containing DocCard metadata for each document. This allows the Orchestrator to analyze and select relevant documents without loading the full document trees — a significant optimization for workspaces with many documents.
172
+
When multiple documents are compiled, Vectorless maintains a lightweight `catalog.bin` containing DocCard metadata for each document. This allows the Orchestrator to analyze and select relevant documents without loading the full document trees — a significant optimization for workspaces with many documents.
153
173
154
174
## Cross-Document Graph
155
175
156
-
When multiple documents are indexed, Vectorless automatically builds a relationship graph based on shared keywords and Jaccard similarity. The graph is constructed as a background task after each indexing operation.
176
+
When multiple documents are compiled, Vectorless automatically builds a relationship graph based on shared keywords and Jaccard similarity. The graph is constructed as a background task after each compilation operation.
Checkpointing allows the pipeline to resume from where it left off after an interruption (crash, timeout, process kill). This is critical for large documents where LLM-enhanced compilation can take minutes.
8
+
9
+
## How It Works
10
+
11
+
When `PipelineOptions::checkpoint_dir` is set, the orchestrator saves state to disk after each execution group completes:
12
+
13
+
```text
14
+
Group 0: [ParsePass] → save checkpoint
15
+
Group 1: [BuildPass] → save checkpoint
16
+
Group 2: [ValidatePass, SplitPass] → save checkpoint
17
+
Group 3: [EnhancePass] → save checkpoint ← expensive LLM calls
18
+
...
19
+
```
20
+
21
+
On restart, the orchestrator loads the checkpoint and skips already-completed passes.
22
+
23
+
## What's Stored
24
+
25
+
Each checkpoint contains:
26
+
27
+
```rust
28
+
pubstructPipelineCheckpoint {
29
+
pubdoc_id:String,
30
+
pubsource_hash:String, // SHA-256 of source content
31
+
pubprocessing_version:u32, // Algorithm version
32
+
pubconfig_fingerprint:String, // Hash of PipelineOptions
33
+
pubcompleted_stages:Vec<String>, // Names of completed passes
34
+
pubcontext_data:CheckpointContextData,
35
+
pubtimestamp:DateTime<Utc>,
36
+
}
37
+
38
+
pubstructCheckpointContextData {
39
+
pubraw_nodes:Vec<RawNode>, // From ParsePass
40
+
pubtree:Option<DocumentTree>, // From BuildPass
41
+
pubmetrics:IndexMetrics, // Cumulative metrics
42
+
pubpage_count:Option<usize>,
43
+
publine_count:Option<usize>,
44
+
pubdescription:Option<String>,
45
+
}
46
+
```
47
+
48
+
## Validation
49
+
50
+
Before resuming, the checkpoint is validated against the current input:
Checkpoints are stored as individual JSON files in the checkpoint directory, one per document (keyed by `doc_id`). On successful completion, the checkpoint file is deleted.
0 commit comments