Skip to content

Commit 62f2bea

Browse files
zTgxzTgx
authored
Merge pull request #12 from vectorlessflow/dev
Enhance document reference handling and improve pilot system
2 parents 93e0841 + 6784ad4 commit 62f2bea

48 files changed

Lines changed: 2373 additions & 367 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "vectorless"
3-
version = "0.1.13"
3+
version = "0.1.14"
44
edition = "2024"
55
authors = ["zTgx <beautifularea@gmail.com>"]
66
description = "Hierarchical, reasoning-native document intelligence engine"

‎docs/README.md‎

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,51 @@
11
# Vectorless Documentation
22

3-
## Brand Assets
3+
Welcome to the Vectorless documentation.
44

5-
Logos and icons for use in README, website, and presentations.
5+
## What is Vectorless?
66

7-
- [assets/brand/](assets/brand/) — Logo variants (light, dark, horizontal, icon)
7+
Vectorless is a **reasoning-native document intelligence engine** that uses LLM-powered tree navigation instead of vector embeddings. It preserves document structure and uses intelligent navigation to find relevant content.
88

9-
## Design Documents
9+
## Key Features
1010

11-
System architecture and core mechanism documentation.
11+
- **Dual Pipeline Architecture** - Separate Index and Retrieval pipelines
12+
- **Pilot System** - LLM-guided navigation with layered fallback
13+
- **Multi-Strategy Retrieval** - Keyword, LLM, and Structure-aware strategies
14+
- **Zero Infrastructure** - No vector database, no embeddings
15+
- **Multi-Format Support** - Markdown, PDF, DOCX, HTML
1216

13-
| Document | Description |
14-
|----------|-------------|
15-
| [architecture.svg](design/architecture.svg) | System architecture diagram |
16-
| [recovery.md](design/recovery.md) | Graceful degradation and error recovery strategy |
17+
## Getting Started
1718

18-
## Development Guides
19+
- [Quick Start Guide](guides/quick-start.md) - Get up and running in 5 minutes
1920

20-
Guides for using and contributing to Vectorless.
21+
## Guides
2122

2223
| Guide | Description |
2324
|-------|-------------|
24-
| [deployment.md](guides/deployment.md) | Production deployment checklist |
25+
| [Quick Start](guides/quick-start.md) | Get up and running quickly |
26+
| [Dual Pipeline](guides/dual-pipeline.md) | Understand Index + Retrieval pipelines |
27+
| [Pilot System](guides/pilot-system.md) | LLM-guided navigation |
28+
| [Multi-Strategy Retrieval](guides/multi-strategy.md) | Keyword, LLM, Structure strategies |
29+
30+
## Design Documents
31+
32+
System architecture and core mechanism documentation.
33+
34+
| Document | Description |
35+
|----------|-------------|
36+
| [pilot.md](design/pilot.md) | Pilot system design |
37+
| [content-aggregation.md](design/content-aggregation.md) | Content aggregation design |
38+
| [client-module.md](design/client-module.md) | Client API design |
39+
| [v3.md](design/v3.md) | Version 3 architecture |
2540

2641
## RFCs (Feature Proposals)
2742

2843
Detailed design documents for new features.
2944

3045
| RFC | Title | Status |
3146
|-----|-------|--------|
32-
| [0001](rfcs/0001-docx-parser.md) | DOCX Parser | Proposed |
47+
| [0001](rfcs/0001-docx-parser.md) | DOCX Parser | Implemented |
48+
| [0002](rfcs/0002-html-parser.md) | HTML Parser | Implemented |
3349

3450
### RFC Process
3551

‎docs/guides/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
# Guide
1+
# Vectorless Guides
2+
3+
Practical guides for using Vectorless effectively.

‎docs/guides/dual-pipeline.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Understanding the Dual Pipeline
2+
3+
Vectorless uses a **dual pipeline architecture** that separates document processing from retrieval. This design enables efficient indexing and intelligent retrieval.
4+
5+
## Architecture Overview
6+
7+
```
8+
┌─────────────────────────────────────────────────────────────────────────────┐
9+
│ Vectorless Architecture │
10+
├─────────────────────────────────────────────────────────────────────────────┤
11+
│ │
12+
│ ┌─────────────────────────────┐ ┌─────────────────────────────┐ │
13+
│ │ INDEX PIPELINE │ │ RETRIEVAL PIPELINE │ │
14+
│ │ │ │ │ │
15+
│ │ Parse → Build → Enrich │ │ Analyze → Plan → Search │ │
16+
│ │ ↓ ↓ ↓ │ │ ↓ ↓ ↓ │ │
17+
│ │ Enhance → Optimize → │ │ Evaluate (Sufficiency) │ │
18+
│ │ Persist │ │ ↑_____________│ │ │
19+
│ │ │ │ │ (NeedMoreData)│ │ │
20+
│ └─────────────────────────────┘ └─────────────────────────────┘ │
21+
│ │ ▲ │
22+
│ └──────────── Workspace ─────────────┘ │
23+
│ │
24+
└─────────────────────────────────────────────────────────────────────────────┘
25+
```
26+
27+
## Index Pipeline
28+
29+
The Index Pipeline processes documents and builds a searchable tree structure.
30+
31+
### Stages
32+
33+
| Stage | Purpose |
34+
|-------|---------|
35+
| **Parse** | Extract content from file (MD, PDF, DOCX, HTML) |
36+
| **Build** | Construct hierarchical document tree |
37+
| **Enrich** | Add metadata, TOC, references |
38+
| **Enhance** | Generate summaries (optional) |
39+
| **Optimize** | Prune, compress, optimize tree |
40+
| **Persist** | Save to workspace storage |
41+
42+
### Example
43+
44+
```rust
45+
// Index pipeline is triggered automatically
46+
let doc_id = engine.index(IndexContext::from_path("./manual.md")).await?;
47+
48+
// With summary generation
49+
let doc_id = engine.index(
50+
IndexContext::from_path("./manual.md")
51+
.with_options(IndexOptions::new().with_summaries())
52+
).await?;
53+
```
54+
55+
## Retrieval Pipeline
56+
57+
The Retrieval Pipeline processes queries and retrieves relevant content.
58+
59+
### Stages
60+
61+
| Stage | Purpose |
62+
|-------|---------|
63+
| **Analyze** | Analyze query complexity, extract keywords |
64+
| **Plan** | Select retrieval strategy and algorithm |
65+
| **Search** | Navigate tree to find candidates |
66+
| **Evaluate** | Check sufficiency, aggregate content |
67+
68+
### The Evaluate Stage
69+
70+
The Evaluate stage is crucial - it determines if retrieved content is sufficient:
71+
72+
```text
73+
┌─────────────┐
74+
│ Search │
75+
└──────┬──────┘
76+
│
77+
▼
78+
┌─────────────┐
79+
│ Evaluate │
80+
└──────┬──────┘
81+
│
82+
┌────────────┼────────────┐
83+
│ │ │
84+
▼ ▼ ▼
85+
Sufficient PartialSufficient Insufficient
86+
│ │ │
87+
▼ ▼ ▼
88+
Return More Search Expand Beam
89+
(1 iteration) (2 iterations)
90+
```
91+
92+
### Retrieval Strategies
93+
94+
```rust
95+
// Three built-in strategies:
96+
97+
// 1. Keyword - Fast, exact matching
98+
// 2. LLM - Semantic understanding via Pilot
99+
// 3. Structure - Hierarchy-aware navigation
100+
```
101+
102+
## The Pilot System
103+
104+
Pilot is the "brain" of the Retrieval Pipeline:
105+
106+
- **Query Analysis**: Understands what the user is asking
107+
- **Context Building**: Creates navigation context from TOC
108+
- **Decision Making**: Decides which branches to explore
109+
- **Fallback**: Algorithm takes over when LLM fails
110+
111+
See [The Pilot System](./pilot-system.md) for details.
112+
113+
## Data Flow
114+
115+
```
116+
Document ──► Index Pipeline ──► Workspace
117+
│
118+
Query ──► Retrieval Pipeline ──────────┘
119+
│
120+
▼
121+
RetrievalResult
122+
├── content
123+
├── node_ids
124+
├── confidence
125+
└── trace
126+
```
127+
128+
## Session-Based Operations
129+
130+
For multi-document operations, use sessions:
131+
132+
```rust
133+
// Create a session
134+
let session = engine.session().await;
135+
136+
// Index multiple documents
137+
session.index(IndexContext::from_path("./doc1.md")).await?;
138+
session.index(IndexContext::from_path("./doc2.md")).await?;
139+
140+
// Query across all documents
141+
let results = session.query_all("What is the architecture?").await?;
142+
143+
for result in results {
144+
println!("From {}: {}", result.doc_id, result.content);
145+
}
146+
```
147+
148+
## See Also
149+
150+
- [Multi-Strategy Retrieval](./multi-strategy.md)
151+
- [Content Aggregation](./content-aggregation.md)
152+
- [Sufficiency Checking](./sufficiency.md)

‎docs/guides/quick-start.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Quick Start Guide
2+
3+
Get up and running with Vectorless in 5 minutes.
4+
5+
## Prerequisites
6+
7+
- Rust 1.70+ installed
8+
- An OpenAI API key (or compatible LLM endpoint)
9+
10+
## Installation
11+
12+
Add to your `Cargo.toml`:
13+
14+
```toml
15+
[dependencies]
16+
vectorless = "0.1"
17+
tokio = { version = "1", features = ["full"] }
18+
```
19+
20+
## Basic Usage
21+
22+
```rust
23+
use vectorless::{Engine, IndexContext};
24+
25+
#[tokio::main]
26+
async fn main() -> Result<(), Box<dyn std::error::Error>> {
27+
// 1. Create an engine with OpenAI
28+
let engine = Engine::builder()
29+
.with_workspace("./workspace")
30+
.with_openai(std::env::var("OPENAI_API_KEY")?)
31+
.build()
32+
.await?;
33+
34+
// 2. Index a document
35+
let doc_id = engine.index(IndexContext::from_path("./manual.md")).await?;
36+
println!("Indexed: {}", doc_id);
37+
38+
// 3. Query the document
39+
let result = engine.query(&doc_id, "How do I configure authentication?").await?;
40+
println!("Answer: {}", result.content);
41+
42+
Ok(())
43+
}
44+
```
45+
46+
## Index from Different Sources
47+
48+
```rust
49+
// From file path
50+
let id1 = engine.index(IndexContext::from_path("./doc.pdf")).await?;
51+
52+
// From string content
53+
let html = "<html><body><h1>Title</h1><p>Content</p></body></html>";
54+
let id2 = engine.index(
55+
IndexContext::from_content(html, vectorless::parser::DocumentFormat::Html)
56+
.with_name("webpage")
57+
).await?;
58+
59+
// From bytes (e.g., from HTTP response)
60+
let pdf_bytes = std::fs::read("./document.pdf")?;
61+
let id3 = engine.index(
62+
IndexContext::from_bytes(pdf_bytes, vectorless::parser::DocumentFormat::Pdf)
63+
).await?;
64+
```
65+
66+
## Index Modes
67+
68+
```rust
69+
use vectorless::IndexMode;
70+
71+
// Default: Skip if already indexed
72+
engine.index(IndexContext::from_path("./doc.md")).await?;
73+
74+
// Force: Always re-index
75+
engine.index(
76+
IndexContext::from_path("./doc.md").with_mode(IndexMode::Force)
77+
).await?;
78+
79+
// Incremental: Only re-index if changed
80+
engine.index(
81+
IndexContext::from_path("./doc.md").with_mode(IndexMode::Incremental)
82+
).await?;
83+
```
84+
85+
## Next Steps
86+
87+
- [Understanding the Dual Pipeline](./dual-pipeline.md) - Learn how Vectorless works
88+
- [Indexing Documents](./indexing.md) - Deep dive into document indexing
89+
- [Querying Documents](./querying.md) - Advanced query techniques

0 commit comments

Comments
 (0)