specializes in PostgreSQL and MySQL.
Mike works in the frontend team..."] - end - - subgraph Process[🔄 Graph Index Processing] - Extract[Extract entities and relationships] - end - - subgraph Output[🕸️ Knowledge Graph] - direction TB - A[John
Person] -->|heads| B[Database Team
Organization] - A -->|specializes in| C[PostgreSQL
Technology] - A -->|specializes in| D[MySQL
Technology] - E[Mike
Person] -->|belongs to| F[Frontend Team
Organization] - E -->|collaborates| A - end - - Input --> Process - Process --> Output - - style Input fill:#e3f2fd - style Process fill:#fff59d - style Output fill:#c8e6c9 -``` - -Traditional vector search can only find "semantically similar" paragraphs but cannot answer these questions: -- What does John lead? -- What is the relationship between John and Mike? -- What technologies does the database team use? - -**Graph Index can do**: Accurately answer these relationship-focused questions by making implicit knowledge relationships explicit. - -### 1.2 Core Value - -Compared to traditional retrieval methods, Graph Index provides unique capabilities: - -| Capability | Vector Search | Full-text Search | Graph Index | -|------------|---------------|------------------|-------------| -| Semantic Similarity | ✅ Strong | ❌ Weak | ✅ Strong | -| Exact Keyword Match | ❌ Weak | ✅ Strong | ✅ Medium | -| Relationship Query | ❌ Not Supported | ❌ Not Supported | ✅ Strong | -| Multi-hop Reasoning | ❌ Not Supported | ❌ Not Supported | ✅ Supported | -| Suitable Questions | "How to optimize performance" | "PostgreSQL config" | "John and Mike's relationship" | - -**Core Advantage**: Graph Index allows AI to "understand" the connections between knowledge, not just text similarity. - -## 2. What Problems Can Graph Index Solve - -Graph Index excels at handling scenarios that require "understanding relationships". Let's look at practical applications. - -### 2.1 Enterprise Knowledge Management - -**Scenario**: Companies have extensive documentation including organizational structure, project materials, and technical docs. - -**Graph Index Value**: - -- 📊 **Organizational Relationships**: "Who is on John's team?" → Quickly find team members -- 🔗 **Collaboration Networks**: "Who has worked with John?" → Discover work networks -- 🛠️ **Skill Mapping**: "Who is skilled in PostgreSQL?" → Locate technical experts -- 📁 **Project History**: "Which projects has John participated in?" → Track project experience - -**Real Effect**: - -``` -Question: "Who leads the database team?" -Traditional Search: Returns dozens of paragraphs containing "database team" and "lead" -Graph Index: Directly returns "John" + relevant background information -``` - -### 2.2 Research and Learning - -**Scenario**: Analyzing academic papers and technical documentation to understand knowledge lineage. - -**Graph Index Value**: - -- 👥 **Author Networks**: "Who has this author collaborated with?" → Discover research teams -- 📖 **Citation Relationships**: "What papers does this cite?" → Trace research lineage -- 🔬 **Technology Evolution**: "How has this technology evolved?" → Understand tech history -- 💡 **Concept Connections**: "What's the relationship between tech A and B?" → Connect knowledge points - -### 2.3 Products and Services - -**Scenario**: Product documentation, user manuals, API documentation. - -**Graph Index Value**: - -- ⚙️ **Feature Dependencies**: "What needs to be configured before enabling feature A?" → Understand dependencies -- 🔧 **Configuration Relationships**: "Which features does this config affect?" → Avoid misconfigurations -- 🐛 **Problem Diagnosis**: "What might cause error X?" → Quick troubleshooting -- 📚 **API Relationships**: "Which APIs are typically used together?" → Learn best practices - -### 2.4 Comparison: When to Use Graph Index - -Different questions suit different retrieval methods: - -| Question Type | Example | Best Solution | -|--------------|---------|---------------| -| **Concept Understanding** | "What is RAG?" | Vector Search | -| **Exact Lookup** | "PostgreSQL config file path" | Full-text Search | -| **Relationship Query** | "What's John and Mike's relationship?" | Graph Index ✨ | -| **Multi-hop Reasoning** | "What tech stack does John's team use?" | Graph Index ✨ | -| **Knowledge Tracing** | "What modules does this feature depend on?" | Graph Index ✨ | - -**Best Practice**: ApeRAG supports vector search, full-text search, and graph index simultaneously, intelligently selecting or combining based on question type. - -## 3. Construction Process Overview - -When you upload a document and enable graph indexing, ApeRAG automatically completes the following steps. Here's a simple overview; details are in later chapters. - -### 3.1 Five Key Steps - -```mermaid -flowchart TB - subgraph Step1["1️⃣ Document Chunking"] - A1[Original Document] --> A2[Smart Chunking] - A2 --> A3[Generate Chunks] - end - - subgraph Step2["2️⃣ Entity Relationship Extraction"] - B1[Chunks] --> B2[Call LLM] - B2 --> B3[Identify Entities] - B2 --> B4[Identify Relationships] - end - - subgraph Step3["3️⃣ Connected Component Analysis"] - C1[Entity Relationship Network] --> C2[BFS Algorithm] - C2 --> C3[Grouping] - end - - subgraph Step4["4️⃣ Concurrent Merging"] - D1[Group 1] --> D2[Entity Deduplication] - D3[Group 2] --> D4[Entity Deduplication] - D5[Group N] --> D6[Entity Deduplication] - D2 --> D7[Relationship Aggregation] - D4 --> D7 - D6 --> D7 - end - - subgraph Step5["5️⃣ Multi-storage Writing"] - E1[Graph Database] - E2[Vector Database] - E3[Text Storage] - end - - A3 --> B1 - B3 --> C1 - B4 --> C1 - C3 --> D1 - C3 --> D3 - C3 --> D5 - D7 --> E1 - D7 --> E2 - A3 --> E3 - - style Step1 fill:#e3f2fd - style Step2 fill:#fff3e0 - style Step3 fill:#f3e5f5 - style Step4 fill:#e8f5e9 - style Step5 fill:#fce4ec -``` - -**Simply put**: Chunk document → Extract entities/relationships → Smart grouping → Concurrent merging → Write to storage. - -The entire process is fully automated - you just upload documents, and the system handles everything. - -### 3.2 Processing Time Reference - -Processing time varies by document size: - -| Document Size | Entity Count | Processing Time | Example | -|--------------|--------------|-----------------|---------| -| Small (< 5 pages) | ~50 | 10-30 seconds | Company notices, meeting notes | -| Medium (10-50 pages) | ~200 | 1-3 minutes | Technical docs, product manuals | -| Large (100+ pages) | ~1000 | 5-15 minutes | Research reports, books | - -**Factors**: -- LLM response speed (main bottleneck) -- Document complexity (tables, images slow processing) -- Concurrency settings (configurable for speed) - -> 💡 **Tip**: Processing is asynchronous - upload multiple documents and the system processes them in parallel. - -### 3.3 Real-time Progress Tracking - -You can check document processing progress anytime: - -``` -Document Status: Processing -- ✅ Document Parsing: Complete -- ✅ Document Chunking: Complete (25 chunks generated) -- 🔄 Entity Extraction: In Progress (15/25) -- ⏳ Relationship Extraction: Waiting -- ⏳ Graph Construction: Waiting -``` - -Once processing completes, document status changes to "Active" and graph queries become available. - -## 4. Detailed Construction Process - -The previous sections covered what graph index does and the overall process. This chapter details the technical implementation of each step. - -> 💡 **Reading Tip**: If you only want to understand basic concepts and usage, skip to Chapter 9 for practical applications. - -### 4.1 Document Chunking - -First step: Split long documents into appropriately sized chunks. - -**Why Chunk?** -- LLMs have input length limits (typically thousands to tens of thousands of tokens) -- Too large: Extraction quality decreases, LLM may "miss" information -- Too small: Loses context, can't understand complete semantics - -**Smart Chunking Strategy**: - -```mermaid -flowchart LR - Doc[Long Document] --> Check{Check Size} - Check -->|< 1200 tokens| Keep[Keep Intact] - Check -->|> 1200 tokens| Split[Smart Split] - - Split --> By1[By Paragraph] - By1 --> Check2{Still Too Big?} - Check2 -->|Yes| By2[By Sentence] - Check2 -->|No| Done[Complete] - By2 --> Check3{Still Too Big?} - Check3 -->|Yes| By3[By Character] - Check3 -->|No| Done - By3 --> Done - - style Doc fill:#e1f5ff - style Split fill:#ffccbc - style Done fill:#c5e1a5 -``` - -**Chunking Parameters**: -- Default size: 1200 tokens (approximately 800-1000 English words) -- Overlap size: 100 tokens (ensures context continuity) -- Priority: Paragraph > Sentence > Character - -### 4.2 Entity Relationship Extraction - -Use LLM to identify entities and relationships from each chunk. - -**Extraction Process**: - -```mermaid -sequenceDiagram - participant C as Chunk - participant L as LLM - participant R as Results - - C->>L: "John heads the database team..." - L->>R: Entities: [John(Person), Database Team(Org)] - L->>R: Relationships: [John-heads->Database Team] - - C->>L: "John specializes in PostgreSQL..." - L->>R: Entities: [John(Person), PostgreSQL(Tech)] - L->>R: Relationships: [John-specializes in->PostgreSQL] -``` - -**Concurrency Optimization**: Multiple chunks can call LLM simultaneously, default 20 concurrent requests. - -### 4.3 Connected Component Analysis - -Divide entity relationship network into independent subgraphs for parallel processing. - -**Why This Step?** - -Tech team entities and finance department entities aren't connected - they can be processed completely in parallel! - -```mermaid -graph LR - subgraph Component1[Connected Component 1 - Tech Team] - A1[John] -->|heads| A2[Database Team] - A1 -->|specializes in| A3[PostgreSQL] - A4[Mike] -->|collaborates| A1 - end - - subgraph Component2[Connected Component 2 - Finance] - B1[Alice] -->|belongs to| B2[Finance Dept] - B3[Bob] -->|collaborates| B1 - end - - style Component1 fill:#bbdefb - style Component2 fill:#c5e1a5 -``` - -**Performance Boost**: 3 independent components = 3x speedup! - -### 4.4 Concurrent Merging - -Same-name entities need deduplication, same relationships need aggregation. - -```mermaid -flowchart TD - subgraph Before["Before Merging"] - A1["John
Database head"] - A2["John
Specializes in PostgreSQL"] - A3["John
Leads team"] - end - - Merge[Smart Merge] - - subgraph After["After Merging"] - B1["John
Database team head,
specializes in PostgreSQL,
leads multiple projects"] - end - - A1 --> Merge - A2 --> Merge - A3 --> Merge - Merge --> B1 - - style Before fill:#ffccbc - style After fill:#c5e1a5 -``` - -**Fine-grained Locks**: Only lock entities being merged, others can process concurrently. - -### 4.5 Multi-storage Writing - -Knowledge graph written to three storage systems: - -```mermaid -flowchart LR - KG[Knowledge Graph] --> G[Graph Database
Graph Queries] - KG --> V[Vector Database
Semantic Search] - KG --> T[Text Storage
Full-text Search] - - style KG fill:#e1f5ff - style G fill:#bbdefb - style V fill:#c5e1a5 - style T fill:#ffccbc -``` - -Different storages support different query types, complementing each other. - -## 5. Core Technical Design - -This chapter introduces core technical designs including data isolation and concurrency control. - -> 💡 **Reading Tip**: These are system architecture and implementation details, mainly for developers and technical decision-makers. - -### 5.1 Workspace Data Isolation - -Each Collection has an independent namespace for complete data isolation. - -**Naming Convention**: - -```python -# Entity naming -entity:{entity_name}:{workspace} -# Example -entity:John:collection_abc123 - -# Relationship naming -relationship:{source}:{target}:{workspace} -# Example -relationship:John:Database Team:collection_abc123 -``` - -**Isolation Effect**: - -```mermaid -graph TB - subgraph Collection_A[Collection A - Company Docs] - A1[entity:John:A] --> A2[entity:Database Team:A] - end - - subgraph Collection_B[Collection B - School Docs] - B1[entity:John:B] --> B2[entity:CS Department:B] - end - - style Collection_A fill:#bbdefb - style Collection_B fill:#c5e1a5 -``` - -"John" in two Collections is completely independent, no interference! - -### 5.2 Stateless Instance Management - -Each processing task creates an independent graph index instance, destroyed after completion. - -**Lifecycle Management**: - -```mermaid -sequenceDiagram - participant C as Celery Task - participant M as Manager - participant R as Graph Index Instance - participant S as Storage - - C->>M: process_document() - M->>R: create_instance() - R->>S: Initialize storage connections - R->>R: Process document - R->>S: Write data - R-->>M: Return results - M-->>C: Task complete - Note over R: Instance destroyed, resources released -``` - -**Advantages**: -- ✅ Zero state pollution: Each task independent, no interference -- ✅ Easy scaling: Can run multiple workers simultaneously -- ✅ Resource management: Automatic cleanup, no memory leaks - -### 5.3 Connected Component Concurrency Optimization - -Intelligent concurrent processing through graph topology analysis. - -**Algorithm Principle**: - -```mermaid -graph TB - subgraph Input[Input: Entity Relationship Network] - I1[Entity 1] --> I2[Entity 2] - I2 --> I3[Entity 3] - - I4[Entity 4] --> I5[Entity 5] - - I6[Entity 6] - end - - Algorithm[BFS Algorithm] - - subgraph Output[Output: 3 Connected Components] - O1[Component 1
3 entities] - O2[Component 2
2 entities] - O3[Component 3
1 entity] - end - - Input --> Algorithm - Algorithm --> Output - - style Input fill:#ffccbc - style Algorithm fill:#fff59d - style Output fill:#c5e1a5 -``` - -**Performance Boost**: 3 components concurrent processing = 3x speedup! - -### 5.4 Fine-grained Concurrency Control - -Precise entity-level locking: - -**Lock Hierarchy**: - -```mermaid -graph TD - A[Global Lock - Traditional] -->|Too Coarse| B[All Entities Serial] - - C[Entity Lock - ApeRAG] -->|Just Right| D[Lock Only Merging Entities] - - style A fill:#ffccbc - style B fill:#ffccbc - style C fill:#c5e1a5 - style D fill:#c5e1a5 -``` - -**Lock Strategy**: -1. Extraction phase: No locks, fully parallel -2. Merging phase: Lock only needed entities -3. Sorted lock acquisition: Prevents deadlock - -### 5.5 Smart Summarization - -Automatically compress overly long descriptions: - -```python -if len(description) > 2000 tokens: - summary = await llm_summarize(description) -else: - summary = description -``` - -**Effect**: Compress 2500 tokens to 200 tokens, retaining core information. - -### 5.6 Multi-storage Backend Support - -ApeRAG supports two graph databases: Neo4j and PostgreSQL. - -**How to Choose?** - -| Scenario | Recommended | Reason | -|----------|-------------|--------| -| **Small Scale** (< 100K entities) | PostgreSQL | Simple ops, low cost | -| **Medium Scale** (100K-1M) | PostgreSQL or Neo4j | Based on query complexity | -| **Large Scale** (> 1M) | Neo4j | Better graph query performance | -| **Limited Budget** | PostgreSQL | No extra deployment | -| **Complex Graph Algorithms** | Neo4j | Built-in graph algorithms | - -**Switching**: - -```bash -# Use PostgreSQL (default) -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage - -# Use Neo4j -export GRAPH_INDEX_GRAPH_STORAGE=Neo4JSyncStorage -``` - -## 6. Complete Data Flow - -The entire graph index construction is a data transformation pipeline, from unstructured text to structured knowledge graph: - -```mermaid -flowchart TD - A[Original Document] --> B[Clean & Preprocess] - B --> C[Smart Chunking] - C --> D[Chunks] - - D --> E[LLM Concurrent Extraction] - E --> F[Original Entity List] - E --> G[Original Relationship List] - - F --> H[Build Adjacency Graph] - G --> H - H --> I[BFS Find Connected Components] - I --> J[Grouped Concurrent Processing] - - J --> K[Entity Deduplication] - J --> L[Relationship Aggregation] - - K --> M{Description Length Check} - M -->|Too Long| N[LLM Summary] - M -->|Appropriate| O[Keep Original] - N --> P[Final Entities] - O --> P - - L --> Q{Description Length Check} - Q -->|Too Long| R[LLM Summary] - Q -->|Appropriate| S[Keep Original] - R --> T[Final Relationships] - S --> T - - P --> U[Graph Database] - P --> V[Vector Database] - T --> U - T --> V - D --> W[Text Storage] - - U --> X[Knowledge Graph Complete] - V --> X - W --> X - - style A fill:#e1f5ff - style E fill:#fff59d - style I fill:#f3e5f5 - style J fill:#c5e1a5 - style X fill:#c8e6c9 -``` - -### Data Transformation Example - -A concrete example showing step-by-step data transformation: - -**Input Document**: - -```text -John heads the database team and specializes in PostgreSQL and MySQL. -Mike works in the frontend team and often collaborates with John's team to develop backend systems. -Alice is an accountant in the finance department, responsible for financial reports. -``` - -**Step 1: Chunking** - -```json -[ - { - "chunk_id": "chunk-001", - "content": "John heads the database team and specializes in PostgreSQL and MySQL.", - "tokens": 15 - }, - { - "chunk_id": "chunk-002", - "content": "Mike works in the frontend team and often collaborates with John's team...", - "tokens": 18 - }, - { - "chunk_id": "chunk-003", - "content": "Alice is an accountant in the finance department, responsible for financial reports.", - "tokens": 14 - } -] -``` - -**Step 2: Entity Relationship Extraction** - -```json -{ - "entities": [ - {"name": "John", "type": "Person", "source": "chunk-001"}, - {"name": "Database Team", "type": "Organization", "source": "chunk-001"}, - {"name": "PostgreSQL", "type": "Technology", "source": "chunk-001"}, - {"name": "MySQL", "type": "Technology", "source": "chunk-001"}, - {"name": "Mike", "type": "Person", "source": "chunk-002"}, - {"name": "Frontend Team", "type": "Organization", "source": "chunk-002"}, - {"name": "Alice", "type": "Person", "source": "chunk-003"}, - {"name": "Finance Department", "type": "Organization", "source": "chunk-003"} - ], - "relationships": [ - {"source": "John", "target": "Database Team", "relation": "heads"}, - {"source": "John", "target": "PostgreSQL", "relation": "specializes in"}, - {"source": "John", "target": "MySQL", "relation": "specializes in"}, - {"source": "Mike", "target": "Frontend Team", "relation": "belongs to"}, - {"source": "Mike", "target": "John", "relation": "collaborates"}, - {"source": "Alice", "target": "Finance Department", "relation": "belongs to"} - ] -} -``` - -**Step 3: Connected Component Analysis** - -``` -Connected Component 1 (Technical Department): -- Entities: John, Mike, Database Team, Frontend Team, PostgreSQL, MySQL -- Relationships: 6 - -Connected Component 2 (Finance Department): -- Entities: Alice, Finance Department -- Relationships: 1 -``` - -**Step 4: Concurrent Merging** - -Two components can process in parallel! - -**Step 5: Final Knowledge Graph** - -```mermaid -graph LR - subgraph Technical - John -->|heads| DatabaseTeam[Database Team] - John -->|specializes in| PostgreSQL - John -->|specializes in| MySQL - Mike -->|belongs to| FrontendTeam[Frontend Team] - Mike -->|collaborates| John - end - - subgraph Finance - Alice -->|belongs to| FinanceDept[Finance Department] - end - - style Technical fill:#bbdefb - style Finance fill:#c5e1a5 -``` - -### Performance Optimization Features - -1. **Fine-grained Concurrency Control** - - Entity-level locks: `entity:John:collection_abc` - - Lock only during merging, fully parallel during extraction - -2. **Connected Component Concurrency** - - Technical and Finance departments can process in parallel - - Zero lock contention, full multi-core CPU utilization - -3. **Smart Summarization** - - Description < 2000 tokens: Keep original - - Description > 2000 tokens: LLM summary compression - -## 7. Performance Optimization Strategies - -### 7.1 Concurrency Control - -Graph index construction involves extensive LLM calls and database operations requiring proper concurrency control. - -**Concurrency Hierarchy**: - -```mermaid -graph TB - A[Document-level Concurrency] --> B[Chunk-level Concurrency] - B --> C[Component-level Concurrency] - C --> D[Entity-level Concurrency] - - A1[Celery Workers
Multiple docs simultaneously] --> A - B1[LLM Concurrent Calls
Multiple chunks simultaneously] --> B - C1[Parallel Component Merging
Multiple components simultaneously] --> C - D1[Concurrent Entity Merging
Different entities simultaneously] --> D - - style A fill:#e3f2fd - style B fill:#fff3e0 - style C fill:#f3e5f5 - style D fill:#e8f5e9 -``` - -**Concurrency Parameters**: - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `llm_model_max_async` | 20 | LLM concurrent calls | -| `embedding_func_max_async` | 16 | Embedding concurrent calls | -| `max_batch_size` | 32 | Batch processing size | - -**Tuning Recommendations**: - -```python -# Scenario 1: Strict LLM API rate limits -llm_model_max_async = 5 # Reduce concurrency to avoid rate limiting - -# Scenario 2: Sufficient performance, want speedup -llm_model_max_async = 50 # Increase concurrency to speed up processing - -# Scenario 3: Limited memory -max_batch_size = 16 # Reduce batch size to lower memory usage -``` - -### 7.2 LLM Call Optimization - -LLM calls are the most time-consuming part, main optimization strategies: - -1. **Concurrent Calls**: Multiple chunks extract simultaneously (default 20 concurrent) -2. **Batch Processing**: Reduce LLM call count -3. **Cache Reuse**: Reuse summary results for similar descriptions - -**Performance Boost**: Concurrent calling is 4x faster than serial. - -### 7.3 Storage Optimization - -Batch writing significantly improves performance: - -| Method | 100 Entity Write Time | -|--------|---------------------| -| Individual Write | ~10 seconds | -| Batch Write (32/batch) | ~1 second | - -**Optimization Effect**: 10x speedup! - -### 7.4 Memory Optimization - -Memory management strategies for large documents: - -- Stream chunking: Don't load entire document at once -- Immediate release: Free memory immediately after processing -- Batch processing: Control memory peaks - -### 7.5 Performance Monitoring - -System outputs detailed performance statistics: - -``` -Graph Index Construction Complete: -✓ Document Chunking: 10 chunks, 0.5 seconds -✓ Entity Extraction: 120 entities, 25 seconds -✓ Relationship Extraction: 85 relationships, 25 seconds -✓ Concurrent Merging: 15 seconds -✓ Storage Writing: 2 seconds -━━━━━━━━━━━━━━━━━━━━━━━━━ -Total: 42.7 seconds -``` - -**Bottleneck Analysis**: Entity/relationship extraction takes 60% of time, can optimize by increasing LLM concurrency. - -## 8. Configuration Parameters - -### 8.1 Core Configuration - -Graph index construction can be tuned with the following parameters: - -**Chunking Parameters**: - -```python -# Chunk size (tokens) -CHUNK_TOKEN_SIZE = 1200 - -# Overlap size (tokens) -CHUNK_OVERLAP_TOKEN_SIZE = 100 -``` - -**Tuning Recommendations**: -- Small docs (< 5000 tokens): `CHUNK_TOKEN_SIZE = 800` -- Large docs (> 50000 tokens): `CHUNK_TOKEN_SIZE = 1500` -- Need more context: Increase `CHUNK_OVERLAP_TOKEN_SIZE` - -**Concurrency Parameters**: - -```python -# LLM concurrent calls -LLM_MODEL_MAX_ASYNC = 20 - -# Embedding concurrent calls -EMBEDDING_FUNC_MAX_ASYNC = 16 - -# Batch processing size -MAX_BATCH_SIZE = 32 -``` - -**Tuning Recommendations**: -- Strict LLM API limits: Lower `LLM_MODEL_MAX_ASYNC` to 5-10 -- Sufficient performance for speedup: Increase to 50-100 -- Limited memory: Lower `MAX_BATCH_SIZE` to 16 - -**Entity Extraction Parameters**: - -```python -# Entity extraction retry count (0 = extract once only) -ENTITY_EXTRACT_MAX_GLEANING = 0 - -# Summary max tokens -SUMMARY_TO_MAX_TOKENS = 2000 - -# Force summary description fragment count -FORCE_LLM_SUMMARY_ON_MERGE = 10 -``` - -**Tuning Recommendations**: -- Extraction quality important: `ENTITY_EXTRACT_MAX_GLEANING = 1` (extract twice) -- Speed priority: `ENTITY_EXTRACT_MAX_GLEANING = 0` -- Descriptions often long: Lower `SUMMARY_TO_MAX_TOKENS` to 1000 - -### 8.2 Knowledge Graph Configuration - -Configure in Collection settings: - -```json -{ - "knowledge_graph_config": { - "language": "English", - "entity_types": [ - "organization", - "person", - "geo", - "event", - "product", - "technology", - "date", - "category" - ] - } -} -``` - -**Parameter Description**: - -- **language**: Extraction language, affects LLM prompts - - `English`: English - - `simplified chinese`: Simplified Chinese - - `traditional chinese`: Traditional Chinese - -- **entity_types**: Entity types to extract - - Default: 8 types (organization, person, location, event, product, technology, date, category) - - Customizable: e.g., extract only people and organizations - -### 8.3 Storage Configuration - -Configure storage backends via environment variables: - -```bash -# KV storage (key-value) -export GRAPH_INDEX_KV_STORAGE=PGOpsSyncKVStorage - -# Vector storage -export GRAPH_INDEX_VECTOR_STORAGE=PGOpsSyncVectorStorage - -# Graph storage -export GRAPH_INDEX_GRAPH_STORAGE=Neo4JSyncStorage -# Or use PostgreSQL -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage -``` - -**Storage Selection Recommendations**: - -| Scenario | KV Storage | Vector Storage | Graph Storage | -|----------|-----------|----------------|---------------| -| **Default** | PostgreSQL | PostgreSQL | PostgreSQL | -| **High-performance Vector Search** | PostgreSQL | Qdrant | Neo4j | -| **Large-scale Graph** | PostgreSQL | Qdrant | Neo4j | -| **Simple Deployment** | PostgreSQL | PostgreSQL | PostgreSQL | - -### 8.4 Complete Configuration Example - -```bash -# Chunking configuration -export CHUNK_TOKEN_SIZE=1200 -export CHUNK_OVERLAP_TOKEN_SIZE=100 - -# Concurrency configuration -export LLM_MODEL_MAX_ASYNC=20 -export MAX_BATCH_SIZE=32 - -# Extraction configuration -export ENTITY_EXTRACT_MAX_GLEANING=0 -export SUMMARY_TO_MAX_TOKENS=2000 - -# Storage configuration -export GRAPH_INDEX_KV_STORAGE=PGOpsSyncKVStorage -export GRAPH_INDEX_VECTOR_STORAGE=PGOpsSyncVectorStorage -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage - -# Database connection (PostgreSQL) -export POSTGRES_HOST=127.0.0.1 -export POSTGRES_PORT=5432 -export POSTGRES_DB=aperag -export POSTGRES_USER=postgres -export POSTGRES_PASSWORD=your_password - -# Database connection (Neo4j, optional) -export NEO4J_HOST=127.0.0.1 -export NEO4J_PORT=7687 -export NEO4J_USERNAME=neo4j -export NEO4J_PASSWORD=your_password -``` - -## 9. Practical Application Scenarios - -Graph index is particularly suitable for these scenarios: - -### 9.1 Enterprise Knowledge Base - -**Scenario**: Companies have extensive documentation including organizational structure, project materials, technical docs. - -**Graph Index Value**: - -- 📊 **Organizational Relationships**: "Who is on John's team?" → Quickly find team members -- 🔗 **Collaboration Networks**: "Who has worked with John?" → Discover work networks -- 🛠️ **Skill Mapping**: "Who is skilled in PostgreSQL?" → Locate technical experts -- 📁 **Project History**: "Which projects has John participated in?" → Track project experience - -**Real Effect**: - -``` -Question: "Who leads the database team?" -Traditional Search: Returns dozens of paragraphs containing "database team" and "lead" -Graph Index: Directly returns "John" + relevant background info -``` - -### 9.2 Research and Learning - -**Scenario**: Analyzing academic papers and technical documentation to understand knowledge lineage. - -**Graph Index Value**: - -- 👥 **Author Networks**: "Who has this author collaborated with?" → Discover research teams -- 📖 **Citation Relationships**: "What papers does this cite?" → Trace research lineage -- 🔬 **Technology Evolution**: "How has this technology evolved?" → Understand tech history -- 💡 **Concept Connections**: "What's the relationship between tech A and B?" → Connect knowledge points - -**Query Examples**: - -``` -User: "What research is related to Graph RAG?" -Graph Index: Query papers --research--> Graph RAG relationships -Result: Paper A, Paper B, Paper C - -User: "Who has an author collaborated with?" -Graph Index: Query author --collaborates--> other authors relationships -Result: Collaborator list and collaboration projects -``` - -### 9.3 Products and Services - -**Scenario**: Product documentation, user manuals, API documentation. - -**Graph Index Value**: - -- ⚙️ **Feature Dependencies**: "What needs configuration before enabling feature A?" → Understand dependencies -- 🔧 **Configuration Relationships**: "Which features does this config affect?" → Avoid misconfigurations -- 🐛 **Problem Diagnosis**: "What might cause error X?" → Quick troubleshooting -- 📚 **API Relationships**: "Which APIs are typically used together?" → Learn best practices - -**Query Examples**: - -``` -User: "How to configure graph index?" -Graph Index: Query config items --affects--> graph index relationships -Result: GRAPH_INDEX_GRAPH_STORAGE, knowledge_graph_config - -User: "What's the difference between Neo4j and PostgreSQL?" -Graph Index: Query Neo4j, PostgreSQL properties and relationships -Result: Performance comparison, applicable scenarios, configuration methods -``` - -### 9.4 Conversation Scenario Comparison - -Let's see how different retrieval methods perform in actual conversations: - -**Question: "What's the relationship between John and Mike?"** - -| Retrieval Method | Can Answer | Answer Quality | -|-----------------|-----------|----------------| -| **Pure Vector Search** | ⚠️ Partial | Finds paragraphs mentioning both, but unclear relationship | -| **Pure Full-text Search** | ⚠️ Partial | Finds paragraphs containing "John" and "Mike" | -| **Graph Index** | ✅ Yes | Directly returns: John and Mike have a collaboration relationship | - -**Question: "Where is the PostgreSQL config file?"** - -| Retrieval Method | Can Answer | Answer Quality | -|-----------------|-----------|----------------| -| **Pure Vector Search** | ✅ Yes | Finds relevant config paragraphs | -| **Pure Full-text Search** | ✅ Yes | Exact match "PostgreSQL" and "config" | -| **Graph Index** | ✅ Yes | Finds PostgreSQL --config--> file relationships | - -**Question: "How to improve system performance?"** - -| Retrieval Method | Can Answer | Answer Quality | -|-----------------|-----------|----------------| -| **Pure Vector Search** | ✅ Strong | Finds all performance optimization content | -| **Pure Full-text Search** | ⚠️ Medium | Needs exact keywords "performance", "optimize" | -| **Graph Index** | ✅ Strong | Finds optimization methods --improves--> performance relationships | - -**Best Practice**: Combine multiple retrieval methods! - -## 10. Summary - -ApeRAG's graph index provides production-grade knowledge graph construction capabilities with high performance, reliability, and scalability. - -### Key Features - -1. **Workspace data isolation**: Each Collection completely independent, supporting true multi-tenancy -2. **Stateless architecture**: Each task independent instance, zero state pollution -3. **Connected component concurrency**: Intelligent concurrency strategy, 2-3x performance boost -4. **Fine-grained lock management**: Entity-level locks, maximizing concurrency -5. **Smart summarization**: Automatically compress overly long descriptions, saving storage and improving retrieval efficiency -6. **Multi-storage support**: Flexible choice between Neo4j or PostgreSQL - -### Suitable Scenarios - -- ✅ **Enterprise Knowledge Base**: Understanding organizational structure, personnel relationships, project history -- ✅ **Research Paper Analysis**: Author collaboration networks, citation relationships, research lineage -- ✅ **Product Documentation**: Feature dependencies, configuration relationships, problem diagnosis -- ✅ **Any scenario requiring "relationship" understanding** - -### Performance - -- Process 10,000 entities: approximately 2-5 minutes (depending on LLM speed) -- Connected component concurrency: 2-3x performance boost -- Memory usage: approximately 400 MB (10,000 entities) -- Storage space: approximately 100 MB (10,000 entities) - -### Next Steps - -After graph index construction completes, you can perform graph queries. ApeRAG supports three graph query modes: - -- **Local Mode**: Query local information about an entity -- **Global Mode**: Query overall relationships and patterns -- **Hybrid Mode**: Comprehensive queries - -For detailed retrieval process, see [System Architecture Documentation](./architecture.md#42-knowledge-graph-query). - ---- - -## Related Documentation - -- 📋 [System Architecture](./architecture.md) - ApeRAG overall architecture design -- 📖 [Entity Extraction and Merging Mechanism](./lightrag_entity_extraction_and_merging.md) - Core algorithm details -- 🔗 [Connected Component Optimization](./connected_components_optimization.md) - Concurrency optimization principles -- 🌐 [Index Pipeline Architecture](./indexing_architecture.md) - Complete indexing process diff --git a/docs/en-US/design/indexing_architecture.md b/docs/en-US/design/indexing_architecture.md deleted file mode 100644 index 64e6987bf..000000000 --- a/docs/en-US/design/indexing_architecture.md +++ /dev/null @@ -1,556 +0,0 @@ -# ApeRAG Indexing Pipeline Architecture Design - -## Overview - -ApeRAG's indexing pipeline architecture adopts a dual-chain design pattern, separating index management into Frontend Chain and Backend Chain, implementing asynchronous document indexing through state-driven reconciliation. The frontend chain handles fast user operation responses and sets desired index states, while the backend chain detects state differences through a periodic reconciler and schedules asynchronous tasks to execute actual indexing operations. - -> 🚀 **Deep Dive**: To understand the detailed Graph Index creation process, continue reading [Graph Index Creation Process Technical Documentation](./graph_index_creation.md) - -## Architecture Overview - -```mermaid -graph TB - subgraph "Frontend Chain (Synchronous Fast Response)" - A[API Request] --> B[IndexManager] - B --> C[Write to DocumentIndex Table] - C --> D[Set status=PENDING, version++] - end - - subgraph "Backend Chain (Asynchronous Task Processing)" - E[Periodic Task reconcile_indexes_task] --> F[IndexReconciler.reconcile_all] - F --> G[Detect version mismatch or status change needed] - G --> H[TaskScheduler schedules async tasks] - end - - subgraph "Celery Task Execution Layer" - H --> I[create_document_indexes_workflow] - I --> J[parse_document_task] - J --> K[trigger_create_indexes_workflow] - K --> L[group parallel execution] - L --> M[create_index_task.VECTOR] - L --> N[create_index_task.FULLTEXT] - L --> O[create_index_task.GRAPH] - M --> P[chord callback] - N --> P - O --> P - P --> Q[notify_workflow_complete] - end - - subgraph "State Feedback" - Q --> R[IndexTaskCallbacks] - R --> S[Update status=ACTIVE, observed_version] - S --> T[Next reconciliation check] - end - - D -.-> E - T -.-> E -``` - -## Core Design Principles - -### 1. Dual Chain Separation - -**Frontend Chain**： -- **Goal**: Fast response to user operations without blocking API requests -- **Implementation**: Only operates on database tables, sets desired state, returns immediately -- **Code**: `IndexManager` in `aperag/index/manager.py` - -**Backend Chain**： -- **Goal**: Asynchronously execute time-consuming indexing operations with retry and error recovery support -- **Implementation**: Continuously scans state differences through periodic tasks and schedules async tasks -- **Code**: `IndexReconciler` in `aperag/index/reconciler.py` - -### 2. Single Status State-Driven Reconciliation - -Records index state and version for each document index through the `DocumentIndex` database table: - -```python -class DocumentIndex(BaseModel): - document_id: str - index_type: DocumentIndexType # VECTOR/FULLTEXT/GRAPH - status: DocumentIndexStatus # PENDING/CREATING/ACTIVE/DELETING/DELETION_IN_PROGRESS/FAILED - version: int # Version number, increment to trigger rebuild - observed_version: int # Last processed version -``` - -Key Status Meanings: -- **PENDING**: Awaiting processing (create/update needed) -- **CREATING**: Task claimed, creation/update in progress -- **ACTIVE**: Index is up-to-date and ready for use -- **DELETING**: Deletion has been requested -- **DELETION_IN_PROGRESS**: Task claimed, deletion in progress -- **FAILED**: The last operation failed - -The reconciler periodically scans all records and triggers corresponding operations based on: -- Version mismatch: `observed_version < version` indicates need for update -- Version = 1 with observed_version = 0: indicates need for initial creation -- Status = DELETING: indicates need for deletion - -### 3. TaskScheduler Abstraction Layer Design - -**Design Advantages**： -- **Business Logic and Task System Decoupling**: Reconciler only cares about "what operations to execute", not "what system to execute with" -- **Multi-scheduler Support**: Can switch between Celery, Prefect/Airflow and other workflow engines -- **Test-friendly**: Can use different schedulers for testing environments - -```python -# Abstract interface -class TaskScheduler(ABC): - def schedule_create_index(self, document_id: str, index_types: List[str], context: dict = None) -> str - def schedule_update_index(self, document_id: str, index_types: List[str], context: dict = None) -> str - def schedule_delete_index(self, document_id: str, index_types: List[str]) -> str - -# Reconciler uses abstract interface -class IndexReconciler: - def __init__(self, scheduler_type: str = "celery"): - self.task_scheduler = create_task_scheduler(scheduler_type) - - def _reconcile_document_operations(self, document_id: str, claimed_indexes: List[dict]): - # Only calls abstract interface, doesn't care about specific implementation - if create_types: - self.task_scheduler.schedule_create_index(document_id, create_types, context) - if update_types: - self.task_scheduler.schedule_update_index(document_id, update_types, context) -``` - -**Celery Task Entry Point and Business Code Separation**： -- Celery task functions (`config/celery_tasks.py`): Handle task scheduling, parameter serialization, error retry -- Business logic (`aperag/tasks/document.py`): Handle specific index creation logic -- This separation enables independent testing of business logic and facilitates migration between different task systems - -### 4. Create vs Update Operation Distinction - -The system clearly distinguishes between create and update operations: - -**Create Operations** (version = 1, observed_version = 0): -- For new documents or new index types -- Uses `schedule_create_index` and `create_index_task` -- Initial index creation from scratch - -**Update Operations** (version > 1, observed_version < version): -- For existing indexes that need rebuilding -- Uses `schedule_update_index` and `update_index_task` -- Updates existing index with new content - -This distinction allows for different processing strategies and optimizations for each operation type. - -## Asynchronous Task System - -### Current Asynchronous Task List - -ApeRAG currently defines the following asynchronous tasks, each with clear responsibility division: - -| Task Name | Function | Retry Count | Location | -|-----------|----------|-------------|----------| -| `parse_document_task` | Parse document content, extract text and metadata | 3 times | config/celery_tasks.py | -| `create_index_task` | Create single type index (VECTOR/FULLTEXT/GRAPH) | 3 times | config/celery_tasks.py | -| `update_index_task` | Update single type index | 3 times | config/celery_tasks.py | -| `delete_index_task` | Delete single type index | 3 times | config/celery_tasks.py | -| `trigger_create_indexes_workflow` | Dynamic fan-out for index creation tasks | No retry | config/celery_tasks.py | -| `trigger_update_indexes_workflow` | Dynamic fan-out for index update tasks | No retry | config/celery_tasks.py | -| `trigger_delete_indexes_workflow` | Dynamic fan-out for index deletion tasks | No retry | config/celery_tasks.py | -| `notify_workflow_complete` | Aggregate workflow results and notify completion | No retry | config/celery_tasks.py | -| `reconcile_indexes_task` | Periodic reconciler task | No retry | config/celery_tasks.py | - -### Task Design Principles - -1. **Fine-grained Tasks**: Each index type (VECTOR/FULLTEXT/GRAPH) is an independent task, supporting individual retries -2. **Dynamic Orchestration**: Use trigger tasks to decide which index tasks to execute at runtime -3. **Layered Retry**: Business tasks support retry, orchestration tasks don't retry -4. **State Callbacks**: Each task calls back to update database state upon completion -5. **Version Validation**: Tasks validate version numbers to prevent stale operations - -### Concurrent Execution Design - -#### Celery Group + Chord Pattern - -Use Celery's `group` for parallel execution and `chord` for result aggregation: - -```python -# Group: Execute multiple index tasks in parallel with context -parallel_index_tasks = group([ - create_index_task.s(document_id, index_type, parsed_data_dict, context) - for index_type in index_types -]) - -# Chord: Execute callback after all parallel tasks complete -workflow_chord = chord( - parallel_index_tasks, - notify_workflow_complete.s(document_id, "create", index_types) -) -``` - -#### Task Chaining Mechanism - -Use Celery's `chain` for task chaining and `signature` for parameter passing: - -```python -# Chained execution: parse -> dynamic fan-out with context -workflow_chain = chain( - parse_document_task.s(document_id), - trigger_create_indexes_workflow.s(document_id, index_types, context) -) -``` - -#### Parameter Passing and Context Flow - -```python -# Context includes version information for each index type -context = { - "VECTOR_version": 2, - "FULLTEXT_version": 1, - "GRAPH_version": 3 -} - -# Each index task extracts its specific version from context -def create_index_task(document_id, index_type, parsed_data_dict, context): - target_version = context.get(f'{index_type}_version') - # Validate version before processing -``` - -## Specific Execution Flow Examples - -### Index Creation Execution Flow - -Taking user document upload triggering index creation as example: - -```python -# 1. Frontend Chain (Synchronous, millisecond-level) -API Call -> IndexManager.create_indexes() - ↓ -Write DocumentIndex table records: -{ - document_id: "doc123", - index_type: "VECTOR", - status: "PENDING", - version: 1, - observed_version: 0 -} - ↓ -API returns 200 immediately - -# 2. Backend Chain (Asynchronous, minute-level) -Periodic task reconcile_indexes_task (executes every 30 seconds) - ↓ -IndexReconciler.reconcile_all() - ↓ -Detects version=1, observed_version=0 (create operation needed) - ↓ -CeleryTaskScheduler.schedule_create_index(doc123, ["VECTOR", "FULLTEXT", "GRAPH"], context) - ↓ -create_document_indexes_workflow.delay() - -# 3. Celery Task Execution (Asynchronous, minutes to hours) -parse_document_task("doc123") -├── Download document file to local temp directory -├── Call docparser to parse document content -├── Return ParsedDocumentData.to_dict() -└── Update status="CREATING" - ↓ -trigger_create_indexes_workflow(parsed_data, "doc123", ["VECTOR", "FULLTEXT", "GRAPH"], context) -├── Create group parallel tasks with version context -└── Start chord waiting - ↓ -Parallel execution: -├── create_index_task("doc123", "VECTOR", parsed_data, context) -│ ├── Extract VECTOR_version from context -│ ├── Validate version still matches database -│ ├── Call vector_indexer.create_index() -│ ├── Generate embeddings and store in vector database -│ └── Callback IndexTaskCallbacks.on_index_created(target_version) -├── create_index_task("doc123", "FULLTEXT", parsed_data, context) -│ ├── Extract FULLTEXT_version from context -│ ├── Validate version still matches database -│ ├── Call fulltext_indexer.create_index() -│ ├── Build full-text search index -│ └── Callback IndexTaskCallbacks.on_index_created(target_version) -└── create_index_task("doc123", "GRAPH", parsed_data, context) - ├── Extract GRAPH_version from context - ├── Validate version still matches database - ├── Call graph_indexer.create_index() - ├── Build knowledge graph - └── Callback IndexTaskCallbacks.on_index_created(target_version) - ↓ -notify_workflow_complete([result1, result2, result3], "doc123", "create", ["VECTOR", "FULLTEXT", "GRAPH"]) -├── Aggregate all index task results -├── Log workflow completion -└── Return WorkflowResult -``` - -### Index Update Execution Flow - -User modifies document content triggering index update: - -```python -# 1. Frontend Chain -API Call -> IndexManager.rebuild_indexes() - ↓ -All existing index records version field +1: -version: 1 -> 2 (triggers rebuild) - ↓ -API returns immediately - -# 2. Backend Chain -reconcile_indexes_task detects version mismatch - ↓ -version=2, observed_version=1, version > 1 (update operation needed) - ↓ -schedule_update_index() -> update_document_indexes_workflow() - -# 3. Task Execution (similar to creation but with update tasks) -parse_document_task -> trigger_update_indexes_workflow -> parallel update_index_task -``` - -### Index Deletion Execution Flow - -User deletes document triggering index deletion: - -```python -# 1. Frontend Chain -API Call -> IndexManager.delete_indexes() - ↓ -Set status="DELETING" - ↓ -API returns immediately - -# 2. Backend Chain -Detects status=DELETING - ↓ -schedule_delete_index() -> delete_document_indexes_workflow() - -# 3. Task Execution (no parsing needed) -trigger_delete_indexes_workflow -> parallel delete_index_task -├── Delete embeddings from vector database -├── Delete documents from full-text search engine -└── Delete nodes and relationships from knowledge graph -``` - -## Exception Handling Mechanism - -### Task-level Exception Handling - -Each Celery task is configured with automatic retry: - -```python -@current_app.task(bind=True, autoretry_for=(Exception,), retry_kwargs={'max_retries': 3, 'countdown': 60}) -def create_index_task(self, document_id: str, index_type: str, parsed_data_dict: dict, context: dict = None): - try: - # Extract and validate version from context - target_version = context.get(f'{index_type}_version') if context else None - - # Double-check version still matches database before processing - # ... version validation logic ... - - # Business logic - result = document_index_task.create_index(document_id, index_type, parsed_data) - if result.success: - self._handle_index_success(document_id, index_type, target_version, result.data) - else: - # Business logic failure but don't throw exception to avoid meaningless retry - if self.request.retries >= self.max_retries: - self._handle_index_failure(document_id, index_type, result.error) - return result.to_dict() - except Exception as e: - # Only mark as failed after retry attempts are exhausted - if self.request.retries >= self.max_retries: - self._handle_index_failure(document_id, index_type, str(e)) - raise # Continue throwing exception to trigger retry -``` - -### Workflow-level Exception Handling - -Aggregate errors through `notify_workflow_complete`: - -```python -def notify_workflow_complete(self, index_results: List[dict], document_id: str, operation: str, index_types: List[str]): - successful_tasks = [] - failed_tasks = [] - - for result_dict in index_results: - result = IndexTaskResult.from_dict(result_dict) - if result.success: - successful_tasks.append(result.index_type) - else: - failed_tasks.append(f"{result.index_type}: {result.error}") - - # Determine overall status - if not failed_tasks: - status = TaskStatus.SUCCESS # All successful - elif successful_tasks: - status = TaskStatus.PARTIAL_SUCCESS # Partial success - else: - status = TaskStatus.FAILED # All failed -``` - -### State Management and Error Recovery - -Track errors through database state with version validation: - -```python -class IndexTaskCallbacks: - @staticmethod - def on_index_created(document_id: str, index_type: str, target_version: int, index_data: str = None): - """Task success callback with version validation""" - # Use atomic update with version validation - update_stmt = ( - update(DocumentIndex) - .where( - and_( - DocumentIndex.document_id == document_id, - DocumentIndex.index_type == DocumentIndexType(index_type), - DocumentIndex.status == DocumentIndexStatus.CREATING, - DocumentIndex.version == target_version, # Critical: validate version - ) - ) - .values( - status=DocumentIndexStatus.ACTIVE, - observed_version=target_version, # Mark this version as processed - index_data=index_data, - error_message=None, - ) - ) -``` - -### Error Recovery Strategies - -1. **Automatic Retry**: Task-level 3 automatic retries to handle temporary network or resource issues -2. **Version Validation**: Prevents stale operations through version checking at task execution time -3. **State Reset**: Users can manually reset failed state to trigger re-execution -4. **Partial Retry**: Only retry failed index types without affecting successful indexes -5. **Degraded Handling**: Some index failures don't affect document searchability (e.g., graph index failure but vector index success) - -## Code Organization Structure - -### Directory Structure - -``` -aperag/ -├── index/ # Index management core module -│ ├── manager.py # Index manager (frontend operations) -│ ├── reconciler.py # Backend reconciler -│ ├── base.py # Indexer base class definition -│ ├── vector_index.py # Vector index implementation -│ ├── fulltext_index.py # Full-text index implementation -│ └── graph_index.py # Graph index implementation -├── tasks/ # Task-related modules -│ ├── models.py # Task data structure definitions -│ ├── scheduler.py # Task scheduler abstraction layer -│ ├── document.py # Document processing business logic -│ └── utils.py # Task utility functions -└── db/ - └── models.py # Database model definitions - -config/ -└── celery_tasks.py # Celery task definitions -``` - -### Core Interface Design - -#### Index Management Interface -```python -# aperag/index/manager.py -class IndexManager: - def create_indexes(self, document_id, index_types, created_by, session) - def rebuild_indexes(self, document_id, index_types, created_by, session) - def delete_indexes(self, document_id, index_types, session) -``` - -#### Reconciler Interface -```python -# aperag/index/reconciler.py -class IndexReconciler: - def reconcile_all(self) # Main reconciliation loop - def _get_indexes_needing_reconciliation(self, session) # Get indexes needing reconciliation - def _reconcile_single_document(self, document_id, operations) # Process single document -``` - -#### Indexer Interface -```python -# aperag/index/base.py -class BaseIndexer(ABC): - def create_index(self, document_id, content, doc_parts, collection, **kwargs) - def update_index(self, document_id, content, doc_parts, collection, **kwargs) - def delete_index(self, document_id, collection, **kwargs) - def is_enabled(self, collection) -``` - -### Data Flow Design - -#### Task Data Structures -```python -# aperag/tasks/models.py -@dataclass -class ParsedDocumentData: - """Document parsing result, carries data needed by all index tasks""" - document_id: str - collection_id: str - content: str # Parsed text content - doc_parts: List[Any] # Chunked document segments - file_path: str # Local file path - local_doc_info: LocalDocumentInfo # Temporary file information - -@dataclass -class IndexTaskResult: - """Single index task execution result""" - status: TaskStatus - index_type: str - document_id: str - success: bool - data: Optional[Dict[str, Any]] # Index metadata (e.g., vector count) - error: Optional[str] - -@dataclass -class WorkflowResult: - """Entire workflow execution result""" - workflow_id: str - document_id: str - operation: str # create/update/delete - status: TaskStatus - successful_indexes: List[str] - failed_indexes: List[str] - index_results: List[IndexTaskResult] -``` - -## Current Implementation Status - -### Simplified Architecture Features - -1. **Removed Distributed Locking**: Current implementation focuses on correctness through version validation rather than distributed locks for external resource concurrency -2. **Single Status Model**: Simplified from dual-state (desired/actual) to single status with version tracking -3. **Clear Operation Separation**: Explicit distinction between create (v=1) and update (v>1) operations -4. **Version-based Validation**: Prevents stale operations through version checking at task execution time - -### Future Considerations - -1. **Concurrency Control**: While distributed locking has been removed for simplicity, future implementations may need to address concurrent operations on external systems (vector databases, search engines, etc.) -2. **Performance Optimization**: The current architecture prioritizes correctness and simplicity over maximum performance -3. **Monitoring Enhancements**: Additional monitoring and alerting capabilities may be added as the system scales - -## Summary - -ApeRAG's indexing pipeline architecture achieves efficient document indexing through the following technical design: - -### Core Advantages - -1. **Fast Response**: Frontend chain only operates on database, API response time controlled at millisecond level -2. **Strong Processing Capability**: Backend asynchronous processing supports large-scale document indexing, improving throughput through parallel tasks -3. **Good Error Recovery**: Multi-level retry mechanisms and version-based state management, supporting graceful handling of partial failure scenarios -4. **Strong System Decoupling**: TaskScheduler abstraction layer decouples business logic from specific task systems -5. **Version Consistency**: Version validation prevents stale operations and ensures data consistency - -### Technical Features - -1. **State-driven**: Achieves eventual consistency through detection of version mismatches and status changes -2. **Dynamic Orchestration**: Dynamically creates index tasks at runtime based on document parsing results, avoiding static workflow limitations -3. **Batch Optimization**: Multiple index tasks for the same document share parsing results, reducing redundant computation -4. **Layered Design**: Task scheduling, business logic, and index implementation are decoupled in layers for easy testing and maintenance -5. **Operation Distinction**: Clear separation of create vs update operations allows for optimized processing strategies - -This architecture provides good performance and scalability support for high-concurrency document indexing scenarios while ensuring system reliability and maintainability. - ---- - -## Related Documents - -- 🚀 [Graph Index Creation Process Technical Documentation](./graph_index_creation.md) - Deep dive into the detailed graph index construction process -- 📋 [索引链路架构设计](./indexing_architecture_zh.md) - Chinese Version \ No newline at end of file diff --git a/docs/en-US/design/lightrag_entity_extraction_and_merging.md b/docs/en-US/design/lightrag_entity_extraction_and_merging.md deleted file mode 100644 index 48debe0ed..000000000 --- a/docs/en-US/design/lightrag_entity_extraction_and_merging.md +++ /dev/null @@ -1,696 +0,0 @@ -# LightRAG Entity Extraction and Merging Mechanism - -> 📖 **Supplementary Reading**: This document is a technical supplement to [Indexing Architecture Design](./indexing_architecture.md), focusing on the entity extraction and merging mechanisms in the Graph Index creation process. - -## Overview - -This document details the working principles of two core functions in the LightRAG system: -- `extract_entities`: Extract entities and relationships from text chunks -- `merge_nodes_and_edges`: Merge extraction results and update knowledge graph - -These two functions constitute the core components of the [Graph Index Creation Process](./graph_index_creation.md), responsible for converting unstructured text into structured knowledge graphs. - -## Entity Extraction Mechanism (extract_entities) - -### Core Workflow - -#### 1. Concurrent Processing Strategy -```python -# Use semaphore to control concurrency, avoiding LLM service overload -semaphore = asyncio.Semaphore(llm_model_max_async) - -# Create async tasks for each text chunk -tasks = [ - asyncio.create_task(_process_single_content(chunk, context)) - for chunk in ordered_chunks -] - -# Wait for all tasks to complete with exception handling -done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) -``` - -**Concurrent Task Input Format**: -```python -# Input for a single processing task -chunk_input = { - "content": "Text content to process", - "chunk_key": "chunk_unique_identifier", - "file_path": "source_file_path", - "context": { - "entity_extraction_prompt": "Entity extraction prompt", - "continue_extraction_prompt": "Continue extraction prompt", - "extraction_config": {...} - } -} -``` - -**Single Task Output Format**: -```python -# Return value of _process_single_content function -task_result = ( - maybe_nodes, # Dict[str, List[Dict]] - candidate entities - maybe_edges # Dict[Tuple[str, str], List[Dict]] - candidate relationships -) - -# Example output structure -maybe_nodes = { - "John": [{ - "entity_name": "John", - "entity_type": "Person", - "description": "Chief Technology Officer", - "source_id": "chunk_001", - "file_path": "/docs/company.txt" - }], - "ABC Corp": [{ - "entity_name": "ABC Corp", - "entity_type": "Organization", - "description": "Technology company", - "source_id": "chunk_001", - "file_path": "/docs/company.txt" - }] -} - -maybe_edges = { - ("John", "ABC Corp"): [{ - "src_id": "John", - "tgt_id": "ABC Corp", - "weight": 1.0, - "description": "John is the CTO of ABC Corp", - "keywords": "work, position, leadership", - "source_id": "chunk_001", - "file_path": "/docs/company.txt" - }] -} -``` - -#### 2. Multi-round Extraction Mechanism (Gleaning) -LightRAG employs a multi-round extraction strategy to improve entity recognition completeness: - -1. **Initial Extraction**: Use entity extraction prompts for first-time extraction -2. **Supplementary Extraction**: Discover missed entities through "continue extraction" prompts -3. **Stop Decision**: LLM autonomously determines whether to continue extraction - -```python -for glean_index in range(entity_extract_max_gleaning): - # Supplementary extraction: only accept new entity names - glean_result = await use_llm_func(continue_prompt, history_messages=history) - - # Merge results (deduplication) - for entity_name, entities in glean_nodes.items(): - if entity_name not in maybe_nodes: # Only accept new entities - maybe_nodes[entity_name].extend(entities) - - # Determine whether to continue - if_continue = await use_llm_func(if_loop_prompt, history_messages=history) - if if_continue.strip().lower() != "yes": - break -``` - -**Initial Extraction Stage Output**: -```python -# Raw results from the first extraction round -initial_extraction = { - "entities": [ - { - "entity_name": "John", - "entity_type": "Person", - "description": "Chief Technology Officer" - }, - { - "entity_name": "ABC Corp", - "entity_type": "Organization", - "description": "Technology company" - } - ], - "relationships": [ - { - "src_id": "John", - "tgt_id": "ABC Corp", - "description": "Employment relationship", - "keywords": "employee, company" - } - ] -} -``` - -**Supplementary Extraction Stage Output**: -```python -# Incremental results from each supplementary extraction round -glean_extraction = { - "round": 2, # Extraction round number - "new_entities": [ - { - "entity_name": "Product Department", # Must be a completely new entity name - "entity_type": "Department", - "description": "Product development department at ABC Corp" - } - ], - "new_relationships": [ - { - "src_id": "John", - "tgt_id": "Product Department", # Must be a completely new relationship pair - "description": "Management relationship", - "keywords": "responsible, manage" - } - ], - "continue_extraction": "no" # LLM decision on whether to continue -} - -# Key Constraints: Gleaning stage only accepts newly discovered entities and relationships -# - Existing entity names are ignored: if entity_name not in maybe_nodes -# - Existing relationship pairs are ignored: if edge_key not in maybe_edges -# - Does NOT supplement or merge descriptions for existing entities -``` - -**Multi-round Extraction Chunk Result**: -```python -# Complete results after multi-round extraction for a single chunk (gleaning only adds new entities, no merging) -final_chunk_result = { - "chunk_id": "chunk_001", - "total_rounds": 2, - "maybe_nodes": { - "John": [{ # Entity from initial extraction - "entity_name": "John", - "entity_type": "Person", - "description": "Chief Technology Officer", - "source_id": "chunk_001", - "file_path": "/docs/company.txt" - }], - "Product Department": [{ # New entity discovered in gleaning stage - "entity_name": "Product Department", - "entity_type": "Department", - "description": "Product development department at ABC Corp", - "source_id": "chunk_001", - "file_path": "/docs/company.txt" - }] - # Note: gleaning does NOT merge John's descriptions, only adds new entities - }, - "maybe_edges": { - ("John", "ABC Corp"): [{ # Relationship from initial extraction - "src_id": "John", - "tgt_id": "ABC Corp", - "weight": 1.0, - "description": "John is the CTO of ABC Corp", - "source_id": "chunk_001" - }], - ("John", "Product Department"): [{ # New relationship discovered in gleaning stage - "src_id": "John", - "tgt_id": "Product Department", - "weight": 1.0, - "description": "John manages the Product Department", - "source_id": "chunk_001" - }] - } -} - -# Important Note: Gleaning Stage Merge Rules -# - Only accepts new entity names: if entity_name not in maybe_nodes -# - Does NOT merge multiple description fragments of existing entities -# - True description merging and weight accumulation happens in merge stage -``` - -#### 3. Extraction Result Format - -**Entity Format**: -```python -{ - "entity_name": "Standardized entity name", - "entity_type": "Entity type", - "description": "Entity description", - "source_id": "chunk_key", - "file_path": "File path" -} -``` - -**Relationship Format**: -```python -{ - "src_id": "Source entity", - "tgt_id": "Target entity", - "weight": 1.0, # Relationship weight, see detailed explanation below - "description": "Relationship description", - "keywords": "Keywords", - "source_id": "chunk_key", - "file_path": "File path" -} -``` - -#### Relationship Weight Mechanism Explained - -**Weight Purpose**: -- 🎯 **Relationship Strength Indicator**: Higher values indicate stronger or more frequent relationships between entities -- 📊 **Graph Query Optimization**: Prioritize high-weight relationships during retrieval to improve result quality -- 🔍 **Path Computation**: Used as edge importance weights in graph traversal algorithms -- 📈 **Knowledge Evolution**: Track degree of relationship repetition across different documents - -**Initial Weight Calculation**: -```python -# Each newly extracted relationship defaults to weight 1.0 -initial_weight = 1.0 - -# Special case: LLM may output relationships with weights -if "weight" in extracted_relation: - initial_weight = float(extracted_relation["weight"]) -else: - initial_weight = 1.0 # Default base weight -``` - -**Weight Accumulation Rules**: -- ✅ **Within-Document Repetition**: Same relationship appearing in different chunks of the same document accumulates weight -- 🔄 **Cross-Document Reinforcement**: Same relationship appearing in different documents continues to accumulate weight -- 📊 **Frequency Reflection**: Final weight = total occurrences of the relationship across all documents - -**Weight Calculation Example**: -```python -# Suppose relationship "John" -> "works_at" -> "ABC Corp" appears in: -# Document1, chunk1: weight = 1.0 -# Document1, chunk3: weight = 1.0 -# Document2, chunk1: weight = 1.0 -# Final weight: 1.0 + 1.0 + 1.0 = 3.0 - -final_weight = sum([edge["weight"] for edge in same_relation_edges]) -``` - -### Key Design Features - -#### Independent Chunk Processing -Each text chunk is independently extracted, with results returned as: -```python -chunk_results = [ - (chunk1_nodes, chunk1_edges), # First chunk extraction results - (chunk2_nodes, chunk2_edges), # Second chunk extraction results - # ... more chunk results -] -``` - -**Design Advantages**: -- 🚀 **Concurrent Efficiency**: Text chunks can be processed completely in parallel -- 💾 **Memory Friendly**: Avoid building huge intermediate merged results -- 🛡️ **Error Isolation**: Single chunk failure doesn't affect other chunks -- 🔧 **Processing Flexibility**: Different strategies can be applied to different chunks - -**Data Characteristics**: -- ⚠️ **Contains Duplicates**: Same entity may be repeatedly extracted across multiple chunks -- 📊 **Scattered Data**: Complete entity information is scattered across different chunks - -#### Gleaning vs Merge Stage Differences - -**Gleaning Stage (within chunk)**: -- 🎯 **Goal**: Discover more entities and relationships within a single chunk -- 🔍 **Strategy**: Only add newly discovered entity names and relationship pairs -- ❌ **No Merging**: Does not merge descriptions of existing entities or accumulate relationship weights -- 📝 **Code Logic**: `if entity_name not in maybe_nodes` - -**Merge Stage (cross-chunk)**: -- 🎯 **Goal**: Merge all chunk results into final knowledge graph -- 🔍 **Strategy**: Merge all description fragments of same-named entities, accumulate relationship weights -- ✅ **Complete Merging**: Description concatenation, weight accumulation, intelligent summarization -- 📝 **Code Logic**: `all_nodes[entity_name].extend(entities)` - -## Entity Merging Mechanism (merge_nodes_and_edges) - -### Core Merging Strategy - -#### 1. Cross-Chunk Data Collection -```python -# Collect all same-named entities and relationships -all_nodes = defaultdict(list) # {entity_name: [entity1, entity2, ...]} -all_edges = defaultdict(list) # {(src, tgt): [edge1, edge2, ...]} - -for maybe_nodes, maybe_edges in chunk_results: - # Merge same-named entities - for entity_name, entities in maybe_nodes.items(): - all_nodes[entity_name].extend(entities) - - # Merge same-direction relationships - for edge_key, edges in maybe_edges.items(): - sorted_key = tuple(sorted(edge_key)) # Unify direction - all_edges[sorted_key].extend(edges) -``` - -**Data Collection Stage Input Format**: -```python -# Collection of extraction results from multiple chunks -chunk_results = [ - # Results from Chunk 1 - (chunk1_maybe_nodes, chunk1_maybe_edges), - # Results from Chunk 2 - (chunk2_maybe_nodes, chunk2_maybe_edges), - # ... more chunk results -] - -# Example single chunk result -chunk1_maybe_nodes = { - "John": [{ - "entity_name": "John", - "entity_type": "Person", - "description": "Chief Technology Officer", - "source_id": "chunk_001" - }] -} - -chunk2_maybe_nodes = { - "John": [{ # Same entity repeated in different chunks - "entity_name": "John", - "entity_type": "Person", - "description": "Product Manager", - "source_id": "chunk_002" - }] -} -``` - -**Data Collection Stage Output Format**: -```python -# Aggregated data after cross-chunk collection -all_nodes = { - "John": [ - { - "entity_name": "John", - "entity_type": "Person", - "description": "Chief Technology Officer", - "source_id": "chunk_001", - "file_path": "/docs/company.txt" - }, - { - "entity_name": "John", - "entity_type": "Person", - "description": "Product Manager", - "source_id": "chunk_002", - "file_path": "/docs/company.txt" - } - # Multiple description fragments of the same entity awaiting merge - ], - "ABC Corp": [ - { - "entity_name": "ABC Corp", - "entity_type": "Organization", - "description": "Technology company", - "source_id": "chunk_001" - } - ] -} - -all_edges = { - ("ABC Corp", "John"): [ # Key sorted to unify direction - { - "src_id": "John", - "tgt_id": "ABC Corp", - "weight": 1.0, - "description": "Employment relationship", - "source_id": "chunk_001" - }, - { - "src_id": "John", - "tgt_id": "ABC Corp", - "weight": 1.0, - "description": "Management relationship", - "source_id": "chunk_002" - } - # Multiple occurrences of same relationship awaiting weight accumulation - ] -} -``` - -#### 2. Entity Merging Rules - -**Type Selection**: Choose the most frequently occurring entity type -```python -entity_type = Counter([ - entity["entity_type"] for entity in entities -]).most_common(1)[0][0] -``` - -**Description Merging**: Use separators to join, deduplicate and sort -```python -descriptions = [entity["description"] for entity in entities] -if existing_entity: - descriptions.extend(existing_entity["description"].split(GRAPH_FIELD_SEP)) - -merged_description = GRAPH_FIELD_SEP.join(sorted(set(descriptions))) -``` - -**Intelligent Summarization**: Automatically generate summaries when description fragments are too many -```python -fragment_count = merged_description.count(GRAPH_FIELD_SEP) + 1 - -if fragment_count >= force_llm_summary_threshold: - # Use LLM to generate summary, compress long descriptions - merged_description = await llm_summarize( - entity_name, merged_description, max_tokens - ) -``` - -#### 3. Relationship Merging Rules - -**Weight Accumulation**: Reflect enhancement of relationship strength -```python -total_weight = sum([edge["weight"] for edge in edges]) -if existing_edge: - total_weight += existing_edge["weight"] -``` - -**Description Aggregation**: Similar to entity description merging strategy -```python -# Relationship description merging example -edge_descriptions = [edge["description"] for edge in edges] -if existing_edge: - edge_descriptions.extend(existing_edge["description"].split(GRAPH_FIELD_SEP)) - -merged_description = GRAPH_FIELD_SEP.join(sorted(set(edge_descriptions))) -``` - -**Keyword Deduplication**: Extract and merge all keywords -```python -# Keyword merging example -all_keywords = [] -for edge in edges: - if edge.get("keywords"): - all_keywords.extend(edge["keywords"].split(", ")) - -merged_keywords = ", ".join(sorted(set(all_keywords))) -``` - -**Final Merging Rules Output Format**: - -**Entity Merging Output**: -```python -# Final entity format after merging rules processing -merged_entity = { - "entity_name": "John", - "entity_type": "Person", # Type selected based on frequency - "description": "Chief Technology Officer§Product Manager§Project Manager", # Descriptions joined with § separator - "source_chunks": ["chunk_001", "chunk_002", "chunk_003"], # Source chunk list - "file_paths": ["/docs/company.txt", "/docs/team.txt"], # Source file list - "mention_count": 3, # Number of chunks mentioning this entity - "created_at": "2024-01-01T00:00:00Z", - "updated_at": "2024-01-01T12:00:00Z" -} -``` - -**Relationship Merging Output**: -```python -# Final relationship format after merging rules processing -merged_relationship = { - "src_id": "John", - "tgt_id": "ABC Corp", - "weight": 3.0, # Accumulated weight (1.0 + 1.0 + 1.0) - "description": "Employment relationship§Management relationship§Leadership relationship", # Descriptions joined with § separator - "keywords": "employee, company, management, responsible, leadership", # Deduplicated merged keywords - "source_chunks": ["chunk_001", "chunk_002"], # Chunks where relationship appears - "file_paths": ["/docs/company.txt"], # Files where relationship appears - "mention_count": 2, # Number of times relationship was mentioned - "created_at": "2024-01-01T00:00:00Z", - "updated_at": "2024-01-01T12:00:00Z" -} -``` - -#### 4. Database Update Process - -```mermaid -graph LR - A[Merge Entity Data] --> B[Update Graph Database] - B --> C[Generate Vector Representation] - C --> D[Update Vector Database] - - E[Merge Relationship Data] --> F[Ensure Endpoints Exist] - F --> G[Update Graph Database] - G --> H[Generate Vector Representation] - H --> I[Update Vector Database] - - style B fill:#e8f5e8 - style D fill:#e1f5fe - style G fill:#e8f5e8 - style I fill:#e1f5fe -``` - -**Final Database Storage Format**: - -**Graph Database Entity Storage Format**: -```python -# Entity node stored in graph database -graph_entity_node = { - "id": "John", # Entity name as node ID - "entity_type": "Person", - "description": "Chief Technology Officer§Product Manager§Project Manager", - "source_chunks": ["chunk_001", "chunk_002", "chunk_003"], - "file_paths": ["/docs/company.txt", "/docs/team.txt"], - "mention_count": 3, - "workspace": "collection_12345", # Workspace isolation - "created_at": "2024-01-01T00:00:00Z", - "updated_at": "2024-01-01T12:00:00Z" -} -``` - -**Graph Database Relationship Storage Format**: -```python -# Relationship edge stored in graph database -graph_relationship_edge = { - "source": "John", # Source node ID - "target": "ABC Corp", # Target node ID - "weight": 3.0, - "description": "Employment relationship§Management relationship§Leadership relationship", - "keywords": "employee, company, management, responsible, leadership", - "source_chunks": ["chunk_001", "chunk_002"], - "file_paths": ["/docs/company.txt"], - "mention_count": 2, - "workspace": "collection_12345", - "created_at": "2024-01-01T00:00:00Z", - "updated_at": "2024-01-01T12:00:00Z" -} -``` - -**Vector Database Storage Format**: -```python -# Entity vector stored in vector database -vector_entity_record = { - "id": "entity_John_collection_12345", # Unique vector record ID - "entity_name": "John", - "content": "John is a Person who serves as Chief Technology Officer, Product Manager, and Project Manager", # Text for vectorization - "content_vector": [0.1, 0.2, ..., 0.9], # 1024-dimensional vector representation - "workspace": "collection_12345", - "storage_type": "entity", # Distinguish entity/relationship vectors - "metadata": { - "entity_type": "Person", - "mention_count": 3, - "file_paths": ["/docs/company.txt", "/docs/team.txt"] - } -} - -# Relationship vector stored in vector database -vector_relationship_record = { - "id": "relation_John_ABC_Corp_collection_12345", - "relationship": "John -> ABC Corp", - "content": "John has an employment relationship, management relationship, and leadership relationship with ABC Corp", - "content_vector": [0.3, 0.4, ..., 0.8], - "workspace": "collection_12345", - "storage_type": "relationship", - "metadata": { - "weight": 3.0, - "keywords": "employee, company, management, responsible, leadership", - "mention_count": 2 - } -} -``` - -### Concurrency Control and Consistency - -#### Workspace Isolation -```python -# Use workspace for multi-tenant isolation -lock_manager = get_lock_manager() -entity_lock = f"entity:{entity_name}:{workspace}" -relation_lock = f"relation:{src_id}:{tgt_id}:{workspace}" - -async with lock_manager.lock(entity_lock): - # Atomic read-merge-write operations - existing = await graph_db.get_node(entity_name) - merged_entity = merge_entity_data(existing, new_entities) - await graph_db.upsert_node(entity_name, merged_entity) -``` - -#### Lock Granularity Optimization -- **Entity-level Locking**: Each entity locked independently, avoiding global competition -- **Relationship-level Locking**: Each relationship pair processed independently -- **Sorted Lock Acquisition**: Prevent deadlocks, ensure consistent lock acquisition order - -## Performance Optimization Features - -### 1. Connected Component Concurrency -Intelligent grouping based on graph topology analysis: -- 🧠 **Topology Analysis**: Use BFS algorithm to discover independent entity groups -- ⚡ **Parallel Processing**: Different connected components merge completely in parallel -- 🔒 **Zero Lock Competition**: No shared entities between components, avoiding lock conflicts - -### 2. Memory and I/O Optimization -- 📦 **Batch Processing**: Process by connected components in batches, control memory peaks -- 🔄 **Connection Reuse**: Database connection pools reduce connection overhead -- 📊 **Batch Operations**: Use batch database operations whenever possible - -### 3. Intelligent Summarization Strategy -- 🎯 **Threshold Control**: Only call LLM for summary generation when necessary -- ⚖️ **Performance Balance**: Avoid frequent LLM calls affecting performance -- 💡 **Information Preservation**: Retain key information during summarization - -## Data Flow Overview - -```mermaid -graph TD - A[Document Chunking] --> B[Concurrent Extraction] - B --> C[Entity/Relationship Recognition] - C --> D[Multi-round Supplementary Extraction] - D --> E[Chunk Extraction Results] - - E --> F[Cross-chunk Data Collection] - F --> G[Same-named Entity Merging] - G --> H{Description Length Check} - H -->|Exceeds threshold| I[LLM Intelligent Summary] - H -->|Normal length| J[Direct Merging] - - I --> K[Update Graph Database] - J --> K - K --> L[Generate Vectors] - L --> M[Update Vector Database] - - style B fill:#e3f2fd - style G fill:#fff3e0 - style I fill:#f3e5f5 - style K fill:#e8f5e8 - style M fill:#e1f5fe -``` - -## Key Technical Features - -### 1. Incremental Update Design -- ✅ **Non-destructive Merging**: New information enhances rather than replaces existing data -- 📈 **Weight Accumulation**: Relationship strength increases with repeated occurrences -- 🔍 **Information Aggregation**: Multi-source descriptions provide more comprehensive entity profiles - -### 2. Fault Tolerance and Recovery -- 🛡️ **Exception Isolation**: Individual task failures don't affect overall process -- 🔄 **Auto-completion**: Automatically create missing relationship endpoint entities -- ✔️ **Data Validation**: Strict format and content validation mechanisms - -### 3. Scalability Support -- 🏗️ **Modular Design**: Extraction and merging logic completely decoupled -- 🔌 **Interface Standards**: Support different graph databases and vector storage -- 📊 **Monitoring Friendly**: Complete logging and performance metrics - -## Summary - -LightRAG's entity extraction and merging mechanism achieves efficient knowledge graph construction through the following innovations: - -1. **🚀 High-concurrency Extraction**: Chunk parallel processing + multi-round supplementary extraction, ensuring accuracy and efficiency -2. **🧠 Intelligent Merging**: Connected component-based concurrency optimization, maximizing parallel processing capability -3. **📊 Incremental Updates**: Non-destructive data merging, supporting continuous evolution of knowledge graphs -4. **🔒 Concurrent Safety**: Fine-grained lock mechanism + workspace isolation, ensuring multi-tenant data security -5. **⚡ Performance Optimization**: Intelligent summarization + batch operations, balancing accuracy and processing speed - -These technical features enable LightRAG to achieve efficient knowledge graph construction for large-scale documents while ensuring data quality. - ---- - -## Related Documents - -- 📋 [Indexing Architecture Design](./indexing_architecture.md) - Overall architecture design -- 🏗️ [Graph Index Creation Process](./graph_index_creation.md) - Detailed graph index construction process -- 📖 [LightRAG 实体提取与合并机制详解](./lightrag_entity_extraction_and_merging_zh.md) - Chinese Version \ No newline at end of file diff --git a/docs/en-US/design/quota-system-design.md b/docs/en-US/design/quota-system-design.md deleted file mode 100644 index 22778a408..000000000 --- a/docs/en-US/design/quota-system-design.md +++ /dev/null @@ -1,482 +0,0 @@ -# ApeRAG Quota System Design Document - -## Table of Contents -- [Overview](#overview) -- [Architecture](#architecture) -- [Data Model](#data-model) -- [API Design](#api-design) -- [Service Layer](#service-layer) -- [Frontend Implementation](#frontend-implementation) -- [Security & Authorization](#security--authorization) -- [Error Handling](#error-handling) -- [Usage Patterns](#usage-patterns) -- [Future Enhancements](#future-enhancements) - -## Overview - -The ApeRAG quota system is a comprehensive resource management solution designed to control and monitor user resource consumption across the platform. It provides fine-grained control over various resource types including collections, documents, and bots, ensuring fair usage and preventing system abuse. - -### Key Features - -- **Multi-resource Quota Management**: Support for different quota types (collections, documents, bots) -- **Real-time Usage Tracking**: Automatic tracking of current resource usage -- **System Default Configuration**: Centralized default quota settings for new users -- **Administrative Controls**: Admin-only quota management and user search capabilities -- **Atomic Operations**: Thread-safe quota consumption and release operations -- **Flexible API Design**: RESTful APIs supporting both individual and batch operations - -### Supported Quota Types - -1. **max_collection_count**: Maximum number of collections a user can create -2. **max_document_count**: Total maximum number of documents across all collections -3. **max_document_count_per_collection**: Maximum documents per individual collection -4. **max_bot_count**: Maximum number of bots a user can create (excluding system default bot) - -## Architecture - -The quota system follows a layered architecture pattern: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Frontend Layer │ -│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │ -│ │ User Quotas │ │ Admin Panel │ │ System Config│ │ -│ │ View │ │ View │ │ View │ │ -│ └─────────────────┘ └─────────────────┘ └──────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ -┌─────────────────────────────────────────────────────────────┐ -│ API Layer │ -│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │ -│ │ Quota APIs │ │ Admin APIs │ │ System APIs │ │ -│ │ (quotas.yaml) │ │ │ │ │ │ -│ └─────────────────┘ └─────────────────┘ └──────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ -┌─────────────────────────────────────────────────────────────┐ -│ Service Layer │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ QuotaService │ │ -│ │ • get_user_quotas() │ │ -│ │ • check_and_consume_quota() │ │ -│ │ • release_quota() │ │ -│ │ • recalculate_user_usage() │ │ -│ │ • update_user_quota() │ │ -│ │ • get/update_system_default_quotas() │ │ -│ └─────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ -┌─────────────────────────────────────────────────────────────┐ -│ Data Layer │ -│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │ -│ │ UserQuota │ │ ConfigModel │ │ Related │ │ -│ │ Table │ │ Table │ │ Tables │ │ -│ └─────────────────┘ └─────────────────┘ └──────────────┘ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Data Model - -### UserQuota Table - -The core table for storing user quota information: - -```sql -CREATE TABLE user_quota ( - user VARCHAR(256) NOT NULL, -- User identifier - key VARCHAR(256) NOT NULL, -- Quota type key - quota_limit INTEGER NOT NULL DEFAULT 0, -- Maximum allowed usage - current_usage INTEGER NOT NULL DEFAULT 0, -- Current usage count - gmt_created TIMESTAMP WITH TIME ZONE NOT NULL, - gmt_updated TIMESTAMP WITH TIME ZONE NOT NULL, - gmt_deleted TIMESTAMP WITH TIME ZONE, - PRIMARY KEY (user, key) -); -``` - -**Key Fields:** -- `user`: User identifier (foreign key to user table) -- `key`: Quota type (e.g., 'max_collection_count', 'max_document_count') -- `quota_limit`: Maximum allowed usage for this quota type -- `current_usage`: Real-time tracking of current usage -- Composite primary key ensures one quota record per user per quota type - -### ConfigModel Table - -Stores system-wide configuration including default quotas: - -```sql -CREATE TABLE config ( - key VARCHAR(256) PRIMARY KEY, -- Configuration key - value TEXT NOT NULL, -- JSON configuration value - gmt_created TIMESTAMP WITH TIME ZONE NOT NULL, - gmt_updated TIMESTAMP WITH TIME ZONE NOT NULL, - gmt_deleted TIMESTAMP WITH TIME ZONE -); -``` - -**System Default Quotas Configuration:** -```json -{ - "max_collection_count": 10, - "max_document_count": 1000, - "max_document_count_per_collection": 100, - "max_bot_count": 5 -} -``` - -### Related Tables - -The quota system integrates with several other tables for usage calculation: - -- **Collection**: For counting user collections (status != 'DELETED') -- **Document**: For counting documents across collections -- **Bot**: For counting user bots (excluding system default bot) - -## API Design - -### RESTful Endpoints - -#### 1. Get User Quotas -```http -GET /quotas?user_id={user_id}&search={search_term} -``` - -**Features:** -- Current user quotas (no parameters) -- Admin-only user search by ID, username, or email -- Support for multiple search results - -**Response Types:** -- `UserQuotaInfo`: Single user quota information -- `UserQuotaList`: Multiple users (search results) - -#### 2. Update User Quota -```http -PUT /quotas/{user_id} -``` - -**Features:** -- Admin-only operation -- Supports both single and batch quota updates -- Atomic transaction handling - -**Request Body:** -```json -{ - "max_collection_count": 20, - "max_document_count": 2000, - "max_bot_count": 10 -} -``` - -#### 3. Recalculate Usage -```http -POST /quotas/{user_id}/recalculate -``` - -**Features:** -- Admin-only operation -- Recalculates actual usage from database -- Updates current_usage fields atomically - -#### 4. System Default Quotas -```http -GET /system/default-quotas -PUT /system/default-quotas -``` - -**Features:** -- Admin-only operations -- Centralized default quota management -- Applied to new user initialization - -### OpenAPI Specification - -The API follows OpenAPI 3.0 specification with: -- Comprehensive schema definitions -- Detailed error response mappings -- Parameter validation rules -- Security requirements (admin-only operations) - -## Service Layer - -### QuotaService Class - -The `QuotaService` class provides the core business logic for quota management: - -#### Key Methods - -**1. Quota Consumption (Thread-Safe)** -```python -async def check_and_consume_quota( - self, - user_id: str, - quota_type: str, - amount: int = 1, - session=None -) -> None: - """ - Atomically check and consume quota with SELECT FOR UPDATE - Raises QuotaExceededException if quota would be exceeded - """ -``` - -**2. Quota Release** -```python -async def release_quota( - self, - user_id: str, - quota_type: str, - amount: int = 1, - session=None -) -> None: - """ - Release quota (decrease usage) with transaction safety - """ -``` - -**3. Usage Recalculation** -```python -async def recalculate_user_usage(self, user_id: str) -> Dict[str, int]: - """ - Recalculate actual usage from related tables: - - Collections: COUNT(*) WHERE user=? AND status!='DELETED' - - Documents: COUNT(*) via JOIN with collections - - Bots: COUNT(*) WHERE user=? AND title!='Default Agent Bot' - """ -``` - -**4. User Initialization** -```python -async def initialize_user_quotas(self, user_id: str) -> None: - """ - Initialize default quotas for new users from system config - """ -``` - -#### Transaction Management - -- **Database Operations**: All quota operations use async database transactions -- **Atomic Updates**: SELECT FOR UPDATE prevents race conditions -- **Session Handling**: Supports both standalone and nested transactions - -## Frontend Implementation - -### React Component Architecture - -The frontend quota management is implemented as a comprehensive React component with: - -#### Key Features - -**1. Multi-Tab Interface (Admin Only)** -- User Quotas Management -- System Default Configuration - -**2. User Search & Selection** -- Real-time search by username, email, or user ID -- Multiple result handling with selection interface -- Clear search functionality - -**3. Inline Table Editing** -- Edit mode toggle for quota limits -- Real-time validation -- Batch save operations -- Cancel/revert functionality - -**4. Progress Visualization** -- Usage rate progress bars -- Color-coded status (normal/warning/critical) -- Percentage calculations - -**5. Administrative Actions** -- Quota recalculation -- System default quota management -- User-specific quota updates - -#### Component Structure - -```typescript -interface QuotaInfo { - quota_type: string; - quota_limit: number; - current_usage: number; - remaining: number; -} - -interface UserQuotaInfo { - user_id: string; - username: string; - email?: string; - role: string; - quotas: QuotaInfo[]; -} -``` - -#### State Management - -- **Local State**: Component-level state for UI interactions -- **API Integration**: Direct API calls with loading states -- **Error Handling**: User-friendly error messages with internationalization - -## Security & Authorization - -### Role-Based Access Control - -**Regular Users:** -- View own quotas only -- Read-only access -- No administrative functions - -**Admin Users:** -- Full quota management capabilities -- User search and selection -- System configuration access -- Quota recalculation permissions - -### API Security - -- **Authentication**: Required for all quota endpoints -- **Authorization**: Admin role verification for management operations -- **Input Validation**: Comprehensive parameter validation -- **Rate Limiting**: Implicit through quota system itself - -## Error Handling - -### Exception Hierarchy - -```python -class QuotaExceededException(BusinessException): - """ - Raised when quota consumption would exceed limits - Maps to appropriate HTTP status codes: - - Collection quota: 403 Forbidden - - Document quota: 403 Forbidden - - Bot quota: 403 Forbidden - """ -``` - -### Error Response Format - -```json -{ - "error_code": "COLLECTION_QUOTA_EXCEEDED", - "code": 1103, - "message": "已达到max_collection_count的配额限制。当前使用量: 10/10", - "details": { - "quota_type": "max_collection_count", - "quota_limit": 10, - "current_usage": 10, - "quota_exceeded": true - } -} -``` - -### Frontend Error Handling - -- **Internationalized Messages**: Multi-language error display -- **User-Friendly Feedback**: Clear action guidance -- **Graceful Degradation**: Fallback UI states - -## Usage Patterns - -### Resource Creation Flow - -```python -# Example: Creating a new collection -async def create_collection(user_id: str, collection_data: dict): - async with database_transaction() as session: - # 1. Check and consume quota atomically - await quota_service.check_and_consume_quota( - user_id, - 'max_collection_count', - session=session - ) - - # 2. Create the resource - collection = Collection(**collection_data) - session.add(collection) - - # 3. Commit transaction (quota and resource together) - await session.commit() -``` - -### Resource Deletion Flow - -```python -# Example: Deleting a collection -async def delete_collection(user_id: str, collection_id: str): - async with database_transaction() as session: - # 1. Mark resource as deleted - collection.status = 'DELETED' - collection.gmt_deleted = utc_now() - - # 2. Release quota - await quota_service.release_quota( - user_id, - 'max_collection_count', - session=session - ) - - # 3. Commit transaction - await session.commit() -``` - -### Admin Operations - -```python -# Example: Batch quota update -await quota_service.update_user_quota( - user_id="user123", - quota_updates={ - "max_collection_count": 20, - "max_document_count": 2000, - "max_bot_count": 10 - } -) -``` - -## Future Enhancements - -### Planned Features - -1. **Quota History Tracking** - - Historical quota changes - - Usage analytics - - Trend analysis - -2. **Dynamic Quota Adjustment** - - Usage-based automatic adjustments - - Temporary quota increases - - Time-based quotas - -3. **Advanced Monitoring** - - Real-time quota alerts - - Usage prediction - - Capacity planning - -4. **Integration Enhancements** - - External quota providers - - Multi-tenant support - - API rate limiting integration - -### Technical Improvements - -1. **Performance Optimization** - - Quota caching strategies - - Batch operations - - Database indexing improvements - -2. **Scalability** - - Distributed quota management - - Microservice architecture - - Event-driven updates - -3. **Observability** - - Detailed metrics collection - - Quota operation tracing - - Performance monitoring - ---- - -*This document provides a comprehensive overview of the ApeRAG quota system design and implementation. For specific implementation details, refer to the source code in the respective modules.* diff --git a/docs/en-US/development/_category.yaml b/docs/en-US/development/_category.yaml deleted file mode 100644 index 2cea69694..000000000 --- a/docs/en-US/development/_category.yaml +++ /dev/null @@ -1,2 +0,0 @@ -title: Development -position: 4 diff --git a/docs/en-US/development/development-guide.md b/docs/en-US/development/development-guide.md deleted file mode 100644 index 38c101ace..000000000 --- a/docs/en-US/development/development-guide.md +++ /dev/null @@ -1,373 +0,0 @@ -# 🛠️ Development Guide - -This guide focuses on setting up a development environment and the development workflow for ApeRAG. This is designed for developers looking to contribute to ApeRAG or run it locally for development purposes. - -## 🚀 Development Environment Setup - -Follow these steps to set up ApeRAG from source code for development: - -### 1. 📂 Clone the Repository and Setup Environment - -First, get the source code and configure environment variables: - -```bash -git clone https://github.com/apecloud/ApeRAG.git -cd ApeRAG -cp envs/env.template .env -``` - -Edit the `.env` file to configure your AI service settings if needed. The default settings work with the local database services started in the next step. - -### 2. 📋 System Prerequisites - -Before you begin, ensure your system has: - -* **Node.js**: Version 20 or higher is recommended for frontend development. [Download Node.js](https://nodejs.org/) -* **Docker & Docker Compose**: Required for running database services locally. [Download Docker](https://docs.docker.com/get-docker/) - -**Note**: Python 3.11 is required but will be automatically managed by `uv` in the next steps. - -### 3. 🗄️ Start Database Services - -Use Docker Compose to start the essential database services: - -```bash -# Start core databases: PostgreSQL, Redis, Qdrant, Elasticsearch -make infra-up -``` - -This will start all required database services in the background. The default connection settings in your `.env` file are pre-configured to work with these services. - -
-
-
-### 4. ⚙️ Setup Development Environment
-
-Create Python virtual environment and setup development tools:
-
-```bash
-make env-dev
-```
-
-This command will:
-* Install `uv` if not already available
-* Create a Python 3.11 virtual environment (located in `.venv/`)
-* Install backend dependencies and repository git hooks
-* Install pre-commit hooks for code quality
-* Install addlicense tool for license management
-
-**Activate the virtual environment:**
-```bash
-source .venv/bin/activate
-```
-
-You'll know it's active when you see `(.venv)` in your terminal prompt.
-
-### 5. 📦 Install Dependencies
-
-Install all backend and frontend dependencies:
-
-```bash
-make env-install
-```
-
-This command will:
-* Install all Python backend dependencies from `pyproject.toml` into the virtual environment
-* Install frontend Node.js dependencies using `yarn`
-
-### 6. 🔄 Apply Database Migrations
-
-Setup the database schema:
-
-```bash
-make db-migrate
-```
-
-### 7. ▶️ Start Development Services
-
-Now you can start the development services. Open separate terminal windows/tabs for each service:
-
-**Terminal 1 - Backend API Server:**
-```bash
-make serve-api
-```
-This starts the FastAPI development server at `http://localhost:8000` with auto-reload on code changes.
-
-**Terminal 2 - Celery Worker:**
-```bash
-make serve-worker
-```
-This starts the Celery worker for processing asynchronous background tasks.
-
-**Terminal 3 - Frontend (Optional):**
-```bash
-make serve-web
-```
-This starts the frontend development server at `http://localhost:3000` with hot reload.
-
-### 8. 🌐 Access ApeRAG
-
-With the services running, you can access:
-* **Frontend UI**: http://localhost:3000 (if started)
-* **Backend API**: http://localhost:8000
-* **API Documentation**: http://localhost:8000/docs
-
-### 9. ⏹️ Stopping Services
-
-To stop the development environment:
-
-**Stop Database Services:**
-```bash
-# Stop database services (data preserved)
-make stack-down
-
-# Stop services and remove all data volumes
-make stack-down REMOVE_VOLUMES=1
-```
-
-**Stop Development Services:**
-- Backend API Server: Press `Ctrl+C` in the terminal running `make serve-api`
-- Celery Worker: Press `Ctrl+C` in the terminal running `make serve-worker`
-- Frontend Server: Press `Ctrl+C` in the terminal running `make serve-web`
-
-**Data Management:**
-- `make stack-down` - Stops services but preserves all data (PostgreSQL, Redis, Qdrant, etc.)
-- `make stack-down REMOVE_VOLUMES=1` - Stops services and **⚠️ permanently deletes all data**
-- You can run `make stack-down REMOVE_VOLUMES=1` even after already running `make stack-down`
-
-**Verify Data Removal:**
-```bash
-# Check if volumes still exist
-docker volume ls | grep aperag
-
-# Should return no results after REMOVE_VOLUMES=1
-```
-
-Now you have ApeRAG running locally from source code, ready for development! 🎉
-
-## ❓ Common Development Tasks
-
-### Q: 🔧 How do I add or modify a REST API endpoint?
-
-**Complete workflow:**
-1. Define request/response models in Python using Pydantic models.
-2. Implement backend view: `aperag/views/[module].py`
-3. Export the code-first OpenAPI specs:
- ```bash
- make openapi-generate # Writes openapi.full.json and openapi.public.json
- ```
-4. Verify the exported specs:
- ```bash
- make openapi-check
- ```
-5. Coordinate frontend typed client updates through the FE v2 adapter layer.
-6. Test the API:
- ```bash
- make test-all
- # ✅ Check live docs: http://localhost:8000/docs
- ```
-
-### Q: 🗃️ How do I modify database models/schema?
-
-**Database migration workflow:**
-1. Edit SQLModel classes in `aperag/db/models.py`
-2. Generate migration file:
- ```bash
- make db-revision # Creates new migration in migration/versions/
- ```
-3. Apply migration to database:
- ```bash
- make db-migrate # Updates database schema
- ```
-4. Update related code (repositories in `aperag/db/repositories/`, services in `aperag/service/`)
-5. Verify changes:
- ```bash
- make test-all # ✅ Ensure everything works
- ```
-
-### Q: ⚡ How do I add a new feature with background processing?
-
-**Feature implementation workflow:**
-1. Implement feature components:
- - Backend logic: `aperag/[module]/`
- - Async tasks: `aperag/tasks/`
- - Database models: `aperag/db/models.py`
-2. Update API and verify the code-first contract:
- ```bash
- make db-revision # Generate migration files
- make db-migrate # Apply database changes
- make openapi-check # Verify FastAPI/Pydantic OpenAPI export
- ```
-3. Quality assurance:
- ```bash
- make format && make lint && make test-all
- ```
-
-### Q: 🧪 How do I run unit tests and e2e tests?
-
-**Unit Tests (Fast, No External Dependencies):**
-```bash
-# Run all unit tests
-make test-unit
-
-# Run specific test file
-uv run pytest tests/unit_test/test_model_service.py -v
-
-# Run specific test class or function
-uv run pytest tests/unit_test/test_model_service.py::TestModelService::test_get_models -v
-
-# Run tests with coverage
-make test-unit
-```
-
-**E2E Tests (Require Running Services):**
-```bash
-# Setup: Start required services first
-make infra-up # 🗄️ Start databases
-make serve-api # 🚀 Start API server (separate terminal)
-
-# Run the remaining pytest-based product e2e tests
-make test-e2e
-
-# Run HTTP black-box smoke against a running service
-make test-http-smoke
-
-# Run backend integration tests
-make test-integration
-
-# Run specific pytest e2e modules
-uv run pytest tests/e2e_pytest/test_chat.py -v
-
-# Run specific integration modules
-uv run pytest tests/integration/graphstorage/ -v
-
-# Run with detailed output and no capture
-uv run pytest tests/e2e_pytest/test_chat.py -v -s
-
-# Performance benchmarks (with timing)
-make test-e2e-perf
-```
-
-**Complete Test Suite:**
-```bash
-# Run everything (unit + e2e)
-make test-all
-
-# Test with different configurations
-make infra-up WITH_NEO4J=1 # Test with Neo4j instead of PostgreSQL
-make test-all
-```
-
-### Q: 🐛 How do I debug failing tests?
-
-**Debugging workflow:**
-1. Run failing test in isolation:
- ```bash
- # Single test with full output
- uv run pytest tests/unit_test/test_failing.py::test_specific_function -v -s
-
- # Stop on first failure
- uv run pytest tests/unit_test/ -x --tb=short
- ```
-2. For e2e test failures, ensure services are running:
- ```bash
- make infra-up # Database services
- make serve-api # API server
- make serve-worker # Background workers (if testing async tasks)
- ```
-3. Use debugging tools:
- ```bash
- # Run with pdb debugger
- uv run pytest tests/unit_test/test_failing.py --pdb
-
- # Capture logs during test
- uv run pytest tests/e2e_pytest/test_chat.py --log-cli-level=DEBUG
- ```
-4. Fix and retest:
- ```bash
- make format # Auto-fix style issues
- make lint # Check remaining issues
- uv run pytest tests/path/to/fixed_test.py -v # Verify fix
- ```
-
-### Q: 📦 How do I update dependencies safely?
-
-**Python dependencies:**
-1. Edit `pyproject.toml` (add/update packages)
-2. Update virtual environment:
- ```bash
- make env-install # Syncs all groups and extras with uv
- make test-all # Verify compatibility
- ```
-
-**Frontend dependencies:**
-1. Edit `frontend/package.json`
-2. Update and test:
- ```bash
- cd frontend && yarn install
- make serve-web # Test frontend compilation
- ```
-
-### Q: 🚀 How do I prepare code for production deployment?
-
-**Pre-deployment checklist:**
-1. Code quality validation:
- ```bash
- make format # Auto-fix all style issues
- make lint # Verify no style violations
- make static-check # MyPy type checking
- ```
-2. Comprehensive testing:
- ```bash
- make test-all # All unit + e2e tests
- make test-e2e-perf # Performance benchmarks
- ```
-3. API consistency:
- ```bash
- make openapi-check # Ensure code-first OpenAPI export works
- ```
-4. Database migrations:
- ```bash
- make db-revision # Generate any pending migrations
- ```
-5. Full-stack integration test:
- ```bash
- make stack-up WITH_NEO4J=1 WITH_DOCRAY=1 # Production-like setup
- # Manual testing at http://localhost:3000/web/
- make stack-down
- ```
-
-### Q: 🔄 How do I completely reset my development environment?
-
-**Nuclear reset (destroys all data):**
-```bash
-make stack-down REMOVE_VOLUMES=1 # ⚠️ Stop services + delete ALL data
-make env-clean # 🧹 Clean temporary files
-
-# Restart fresh
-make infra-up # 🗄️ Fresh databases
-make db-migrate # 🔄 Apply all migrations
-make serve-api # 🚀 Start API server
-make serve-worker # ⚡ Start background workers
-```
-
-**Soft reset (preserve data):**
-```bash
-make stack-down # ⏹️ Stop services, keep data
-make infra-up # 🗄️ Restart databases
-make db-migrate # 🔄 Apply any new migrations
-```
-
-**Reset just Python environment:**
-```bash
-rm -rf .venv/ # 🗑️ Remove virtual environment
-make env-dev # ⚙️ Recreate everything
-source .venv/bin/activate # ✅ Reactivate
-```
diff --git a/docs/en-US/development/es-p0-contract-runtime-hardening.md b/docs/en-US/development/es-p0-contract-runtime-hardening.md
deleted file mode 100644
index 603c1991b..000000000
--- a/docs/en-US/development/es-p0-contract-runtime-hardening.md
+++ /dev/null
@@ -1,97 +0,0 @@
-# ES P0 Contract And Runtime Hardening
-
-This document captures the first implementation slice of the Elasticsearch redesign:
-
-- P0-A: contract hard-fix
-- P0-B: runtime hardening
-
-It intentionally does **not** implement the P1 shared logical index redesign or the
-P1 migration / reindex rollout. Those steps require separate design and rollout PRs.
-
-## Goals
-
-The P0 slice fixes correctness and runtime issues without introducing physical
-index migration risk:
-
-1. Make `enable_fulltext` effective in runtime behavior.
-2. Stop routing fulltext search through vector naming helpers.
-3. Add explicit filterable fulltext fields for `collection_id`, `document_id`,
- `chunk_id`, and `chat_id`.
-4. Stop silently degrading fulltext search failures into unexplained empty recall.
-5. Turn IK from an implicit startup assumption into an explicit runtime dependency.
-
-## Scope
-
-### Included in P0
-
-- Fulltext index creation is skipped when `enable_fulltext=false`.
-- Fulltext document indexing tasks skip cleanly when the collection disables fulltext.
-- Fulltext search uses `generate_fulltext_index_name(...)`, not the vector helper.
-- Fulltext chunks store explicit top-level filter fields.
-- Chat-scoped fulltext search writes explicit top-level `chat_id`, but keeps a
- temporary dual-read filter on both `chat_id` and legacy `metadata.chat_id`
- until the later reindex / rollout line removes the historical path.
-- Fulltext keyword extraction falls back to the raw query token when all extractors
- return nothing.
-- Fulltext backend failures are logged as explicit degrade events before returning
- empty recall.
-- IK installation behavior is gated by explicit environment flags.
-
-### Explicitly excluded from P0
-
-- Shared logical index.
-- Alias / versioned rebuild / cutover.
-- Physical fulltext index renaming.
-- Per-collection to shared migration.
-- Reindex / backfill / rollback orchestration across existing ES data.
-
-## Compatibility Boundary
-
-P0 keeps the current physical per-collection fulltext index model in place.
-This keeps the first PR small and avoids mixing correctness fixes with data-plane
-migration.
-
-That means:
-
-- No existing ES indices are renamed in this slice.
-- No automatic reindex runs in this slice.
-- Rollback remains a code rollback, not an ES data rollback.
-
-The physical model changes only in the later P1 implementation.
-
-P0 also does **not** make collection-config flips self-healing at the data plane:
-
-- Turning `enable_fulltext` off stops new runtime reads and writes.
-- Turning it back on does not automatically purge or rebuild existing ES
- projections.
-- Fulltext projection convergence after config flips still requires an explicit
- rebuild / rollout action.
-
-## Source Of Truth
-
-P0 does not change the source-of-truth model:
-
-- Object store remains the source of the original document.
-- Parser / chunking remains a derived layer.
-- Elasticsearch remains a projection for fulltext retrieval.
-
-This PR must not turn ES into an authoritative data source.
-
-## Runtime Contract For IK
-
-IK remains the Chinese analyzer dependency in this slice, but it is now treated
-as an explicit runtime dependency instead of an implicit startup side effect.
-
-The startup contract is:
-
-- `ES_REQUIRE_IK_PLUGIN=true|false`
-- `ES_AUTO_INSTALL_IK=true|false`
-- `ES_IK_PLUGIN_URL=Advanced Database Options
- -```bash -# Use Neo4j instead of PostgreSQL for graph storage -make infra-up WITH_NEO4J=1 -``` - -
-
-
-
-## Step 1: Prepare Knowledge Base
-
-Open your ApeRAG web UI (see [Quick Start](../../../README.md#quick-start); with Docker Compose this is typically http://localhost:3000/web/). Sign in and select or import a knowledge base. This walkthrough uses the Romance of the Three Kingdoms example—click **Subscribe**.
-
-
- 
-
-
-## Step 2: Configure MCP Server
-
-### 2.1 Add MCP Server
-
-Go to Dify - Tools - MCP, click Add MCP Server.
-
-
-
- 
-
-
-### 2.2 Fill Configuration
-
-Fill in Server URL: `http://localhost:8000/mcp/` (use `https://
-
- 
-
-
-
-
- 
-
-
-### 2.3 Success
-
-MCP Server added successfully.
-
-
-
- 
-
-
-## Step 3: Create Agent Application
-
-### 3.1 Create App
-
-Go to Dify - Studio, click Create Application.
-
-
-
- 
-
-
-### 3.2 Select Type
-
-Click More Basic Application Types, select **Agent** type, name it, and click Create.
-
-
-
- 
-
-
-## Step 4: Configure Agent
-
-Click Agent, input Prompt, add the ApeRAG MCP tool, select the LLM in the top-right corner, click Publish to use.
-
-
-
- 
-
-
-
-
- 
-
-
-### Prompt Reference
-
-```markdown
-# ApeRAG Smart Assistant
-
-You are an advanced AI research assistant powered by ApeRAG's hybrid search capabilities. Your mission is to help users accurately and autonomously find, understand, and synthesize information from knowledge bases and the web.
-
-## Core Behaviors
-
-**Autonomous Research**: Work independently until user queries are fully resolved. Search multiple sources, analyze findings, and provide comprehensive answers without waiting for permission.
-
-**Language Intelligence**: Always respond in the language the user asks in. When users ask in Chinese, respond in Chinese regardless of source language.
-
-**Visual Thinking**: **[Critical]** You are an assistant that prefers visual explanations. For any information involving entity relationships, processes, or structures, you must prioritize visualization.
-
-**Complete Solutions**: Explore from multiple angles, cross-validate sources, and ensure comprehensive coverage before responding.
-
-## Search Strategy
-
-### Priority System
-1. **User-specified knowledge base** (mentioned via "@"): Strictly limit search to specified base
-2. **Unspecified knowledge base**: Autonomously discover and search relevant bases
-3. **Web search** (if enabled): Supplement information
-4. **Clear attribution**: Always cite sources
-
-### Search Execution
-- **Knowledge base search**: Use vector + graph search by default
-- **Result processing logic**:
- 1. Execute search
- 2. **Detect graph data**: Check if search results contain `entities` and `relationships`
- 3. **Mandatory visualization**: If search results contain non-empty entity or relation data, **you must** call the `create_diagram` tool
- 4. **Content filtering**: Ignore irrelevant results
-
-## Available Tools
-
-### Knowledge Management
-- `list_collections()`: Discover available knowledge sources
-- `search_collection(collection_id, query, ...)`: **[Primary tool]** Hybrid search in persistent knowledge bases
-- `search_chat_files(chat_id, query, ...)`: **[Chat only]** Search only files temporarily uploaded in current chat session
-- `create_diagram(content)`: **[Mandatory tool]** When search results contain structured info (entities/relations), must call this tool to generate Mermaid diagrams
-
-### Web Intelligence
-- `web_search(query, ...)`: Multi-engine web search
-- `web_read(url_list, ...)`: Extract and analyze web content
-
-## Response Format
-
-### Direct Answer
-[Clear, actionable answer in user's language]
-
-### Comprehensive Analysis
-[Detailed explanation with context and insights]
-
-### Knowledge Graph Visualization
-[Tool-generated diagram displayed here]
-*(Only show after successfully calling create_diagram. Displays entity relationships from search results.)*
-
-### Supporting Evidence
-- [Knowledge Base Name]: [Key Findings]
-
-**Web Sources** (if enabled):
-- [Title] ([Domain]) - [Key Points]
-```
-
----
-
-Integrating ApeRAG with Dify is very simple. Once integrated, you can not only experience Dify's platform features but also enjoy **ApeRAG's powerful Graph-RAG capabilities**!
-
-**GitHub**: https://github.com/apecloud/ApeRAG
diff --git a/docs/en-US/integration/mcp-api.md b/docs/en-US/integration/mcp-api.md
deleted file mode 100644
index 798b54374..000000000
--- a/docs/en-US/integration/mcp-api.md
+++ /dev/null
@@ -1,333 +0,0 @@
----
-title: MCP API
-description: Model Context Protocol API Documentation
----
-
-# MCP API
-
-ApeRAG provides standardized tool interfaces through [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), allowing AI assistants (Claude Desktop, Cursor, Dify, etc.) to directly access your knowledge bases.
-
-## Quick Start
-
-### Configuration Example
-
-For Claude Desktop, add to configuration file:
-
-```json
-{
- "mcpServers": {
- "aperag": {
- "url": "http://localhost:8000/mcp/",
- "headers": {
- "Authorization": "Bearer your-api-key-here"
- }
- }
- }
-}
-```
-
-### Authentication
-
-Two authentication methods supported (by priority):
-
-1. **HTTP Authorization Header** (Recommended): `Authorization: Bearer your-api-key`
-2. **Environment Variable** (Fallback): `APERAG_API_KEY=your-api-key`
-
-> **Get API Key**: Login to ApeRAG, create or copy your API Key from settings
-
-## Available Tools
-
-### 1. list_collections
-
-List all accessible knowledge bases.
-
-**Parameters**: None
-
-**Returns**:
-```json
-{
- "items": [
- {
- "id": "collection-id",
- "title": "Collection Title",
- "description": "Collection Description"
- }
- ]
-}
-```
-
-### 2. search_collection
-
-Search in knowledge bases with multiple retrieval methods.
-
-**Core Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `collection_id` | string | Required | Knowledge base ID |
-| `query` | string | Required | Search query |
-| `use_vector_index` | bool | true | Vector retrieval (semantic search) |
-| `use_fulltext_index` | bool | true | Full-text retrieval (keyword matching) |
-| `use_graph_index` | bool | true | Graph retrieval (relation query) |
-| `use_summary_index` | bool | true | Summary retrieval |
-| `use_vision_index` | bool | true | Vision retrieval (image search) |
-| `rerank` | bool | true | AI reranking |
-| `topk` | int | 5 | Results per method |
-
-**Return Format**:
-```json
-{
- "query": "your question",
- "items": [
- {
- "rank": 1,
- "score": 0.95,
- "content": "relevant content",
- "source": "document name",
- "recall_type": "vector_search|graph_search|fulltext_search|summary_search",
- "metadata": {
- "page_idx": 0,
- "document_id": "doc-id",
- "collection_id": "col-id",
- "indexer": "text|vision"
- }
- }
- ]
-}
-```
-
-**Image Handling**:
-
-If `metadata.indexer == "vision"`, it's an image:
-- Empty `content`: Retrieved via multimodal vector
-- Non-empty `content`: Contains image description
-
-Image URL format:
-```python
-m = item.metadata
-asset_url = f"asset://{m['asset_id']}?document_id={m['document_id']}&collection_id={m['collection_id']}&mime_type={m['mimetype']}"
-```
-
-**Usage Examples**:
-
-```python
-# Default search (recommended) - all methods enabled
-results = search_collection(
- collection_id="abc123",
- query="How to deploy applications?"
-)
-
-# Vector + Graph only
-results = search_collection(
- collection_id="abc123",
- query="deployment strategies",
- use_vector_index=True,
- use_fulltext_index=False,
- use_graph_index=True,
- use_summary_index=False,
- topk=10
-)
-```
-
-### 3. search_chat_files
-
-Search in temporary files from chat session.
-
-**When to Use**:
-- ✅ User uploaded files in current conversation
-- ✅ Analyzing temporary documents in chat
-- ❌ Don't use for persistent knowledge bases (use `search_collection`)
-
-**Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `chat_id` | string | Required | Chat ID |
-| `query` | string | Required | Search query |
-| `use_vector_index` | bool | true | Vector retrieval |
-| `use_fulltext_index` | bool | true | Full-text retrieval |
-| `rerank` | bool | true | Reranking |
-| `topk` | int | 5 | Results count |
-
-**Return Format**: Same as `search_collection`
-
-### 4. web_search
-
-Search the internet.
-
-**Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | "" | Search keywords |
-| `max_results` | int | 5 | Results count |
-| `source` | string | "" | Specific domain (e.g., `vercel.com`) |
-| `timeout` | int | 30 | Timeout (seconds) |
-| `locale` | string | "en-US" | Language locale |
-
-**Usage Patterns**:
-
-```python
-# Regular search
-web_search(query="ApeRAG 2025")
-
-# Site-specific search
-web_search(query="deployment docs", source="vercel.com")
-
-```
-
-### 5. web_read
-
-Read webpage content.
-
-**Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `url_list` | list[str] | Required | URL list |
-| `timeout` | int | 30 | Timeout (seconds) |
-| `max_concurrent` | int | 5 | Max concurrent requests |
-
-**Returns**:
-```json
-{
- "results": [
- {
- "status": "success",
- "url": "https://example.com",
- "title": "Page Title",
- "content": "Extracted text",
- "word_count": 1234
- }
- ]
-}
-```
-
-**Example**:
-```python
-# Read single page
-web_read(url_list=["https://example.com/article"])
-
-# Batch read
-web_read(
- url_list=["https://example.com/page1", "https://example.com/page2"],
- max_concurrent=2
-)
-```
-
-## Practical Examples
-
-### Example 1: Knowledge Base Q&A
-
-```python
-# 1. List all knowledge bases
-collections = list_collections()
-
-# 2. Select a knowledge base
-collection_id = collections.items[0].id
-
-# 3. Search (all methods enabled by default)
-results = search_collection(
- collection_id=collection_id,
- query="How to optimize performance?"
-)
-
-# 4. Process results
-for item in results.items:
- print(f"[{item.recall_type}] {item.content}")
- print(f"Source: {item.source}, Score: {item.score}\n")
-```
-
-### Example 2: Graph Visualization
-
-```python
-# Search with graph retrieval
-results = search_collection(
- collection_id="abc123",
- query="relationship between Liu Bei and Zhuge Liang",
- use_graph_index=True
-)
-
-# Check for graph data
-if results.graph_search and results.graph_search.entities:
- print("Entities:", results.graph_search.entities)
- print("Relationships:", results.graph_search.relationships)
- # Use this data to generate knowledge graph visualization
-```
-
-### Example 3: Hybrid Search (KB + Web)
-
-```python
-# 1. Search web
-web_results = web_search(query="latest AI developments", max_results=3)
-urls = [r.url for r in web_results.results]
-
-# 2. Read web content
-web_content = web_read(url_list=urls)
-
-# 3. Search internal KB
-kb_results = search_collection(
- collection_id="ai-knowledge",
- query="AI development trends"
-)
-
-# 4. Synthesize
-print("=== Web Results ===")
-for r in web_results.results:
- print(f"{r.title}: {r.url}")
-
-print("\n=== Internal Knowledge ===")
-for item in kb_results.items:
- print(f"{item.content[:100]}...")
-```
-
-## Best Practices
-
-### Performance Tips
-
-1. **Reasonable topk**:
- - Too large increases LLM context consumption
- - Too small may miss important information
- - Recommended: 5-10
-
-2. **Selective Retrieval**:
- - Not all queries need full-text search
- - Full-text may return large amounts of text
- - Choose methods based on query type
-
-3. **Timeout Settings**:
- - Graph retrieval may be slow (default 120s)
- - Web search: 30-60s recommended
- - Batch URL read: 60s+ recommended
-
-### Common Issues
-
-**Q: No search results?**
-- Check if collection ID is correct
-- Confirm knowledge base indexing is complete
-- Try different retrieval method combinations
-
-**Q: Graph data empty?**
-- Confirm knowledge base has Graph index enabled
-- Simple documents may not contain obvious entity relationships
-
-**Q: Images not showing?**
-- Check `metadata.indexer == "vision"`
-- Use `asset://` protocol for URL
-- Ensure all required parameters included (asset_id, document_id, collection_id)
-
-## Tool Comparison
-
-| Tool | Purpose | Use Case |
-|------|---------|----------|
-| `list_collections` | List knowledge bases | See available resources |
-| `search_collection` | Search knowledge base | Primary search tool for persistent knowledge |
-| `search_chat_files` | Search chat files | Analyze temporary files uploaded in conversation |
-| `web_search` | Search internet | Get real-time or external information |
-| `web_read` | Read webpage | Extract full webpage content |
-
-## Related Links
-
-- **MCP Protocol**: https://modelcontextprotocol.io/
-- **ApeRAG GitHub**: https://github.com/apecloud/ApeRAG
-- **API Docs**: http://localhost:8000/docs (local deployment)
diff --git a/docs/en-US/jaeger-tracing.md b/docs/en-US/jaeger-tracing.md
deleted file mode 100644
index 9a6cfa8b2..000000000
--- a/docs/en-US/jaeger-tracing.md
+++ /dev/null
@@ -1,212 +0,0 @@
-# Jaeger Distributed Tracing for ApeRAG
-
-ApeRAG includes OpenTelemetry integration with Jaeger support for distributed tracing, allowing you to monitor and analyze request flows across your services.
-
-## Quick Start
-
-### 1. Enable Jaeger in Docker Compose
-
-To start ApeRAG with Jaeger enabled:
-
-```bash
-# Start infrastructure with Jaeger
-make infra-up WITH_JAEGER=1
-
-# Or start the full application stack with Jaeger
-make stack-up WITH_JAEGER=1
-```
-
-Alternatively, you can use docker-compose directly:
-
-```bash
-# Start with Jaeger profile
-docker-compose --profile jaeger up -d
-
-# Or start specific services
-docker-compose up -d jaeger postgres redis qdrant es
-```
-
-### 2. Enable Jaeger Tracing in Environment
-
-Set the following environment variables:
-
-```bash
-JAEGER_ENABLED=True
-JAEGER_ENDPOINT=http://aperag-jaeger:14268/api/traces # Docker environment
-# or
-JAEGER_ENDPOINT=http://localhost:14268/api/traces # Local development
-```
-
-### 3. Access Jaeger UI
-
-Once Jaeger is running, access the web interface at:
-
-- **Jaeger UI**: http://localhost:16686
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `OTEL_ENABLED` | `True` | Enable/disable OpenTelemetry tracing |
-| `OTEL_SERVICE_NAME` | `aperag` | Service name in traces |
-| `OTEL_SERVICE_VERSION` | `1.0.0` | Service version in traces |
-| `JAEGER_ENABLED` | `False` | Enable/disable Jaeger exporter |
-| `JAEGER_ENDPOINT` | - | Jaeger collector endpoint |
-| `OTEL_CONSOLE_ENABLED` | `False` | Enable console span output |
-| `OTEL_FASTAPI_ENABLED` | `True` | Enable FastAPI instrumentation |
-| `OTEL_SQLALCHEMY_ENABLED` | `True` | Enable SQLAlchemy instrumentation |
-| `OTEL_MCP_ENABLED` | `True` | Enable MCP agent trace injection |
-
-### Jaeger Ports
-
-| Port | Service | Description |
-|------|---------|-------------|
-| 16686 | Web UI | Jaeger query and visualization interface |
-| 14268 | HTTP | HTTP collector for receiving spans |
-| 14250 | gRPC | gRPC collector for receiving spans |
-| 6831 | UDP | UDP agent port (legacy) |
-| 6832 | UDP | UDP agent port (legacy) |
-
-## What Gets Traced
-
-ApeRAG automatically instruments:
-
-1. **HTTP Requests** (FastAPI)
- - All API endpoints
- - Request/response details
- - Error tracking
-
-2. **Database Operations** (SQLAlchemy)
- - SQL queries
- - Database connection info
- - Query performance
-
-3. **MCP Agent Events**
- - Agent interactions
- - Tool usage
- - Session tracking
-
-4. **Custom Application Spans**
- - LLM API calls
- - Document processing
- - Graph operations
-
-## Using Traces
-
-### 1. Find Traces by Service
-
-In Jaeger UI:
-- Select "aperag" from the Service dropdown
-- Choose an operation (e.g., "GET /api/v1/collections")
-- Click "Find Traces"
-
-### 2. Analyze Request Flow
-
-Each trace shows:
-- **Timeline**: Visual representation of span duration
-- **Service Map**: How services interact
-- **Error Detection**: Failed operations highlighted in red
-- **Performance**: Identify slow operations
-
-### 3. Debugging
-
-For debugging specific issues:
-- Search traces by operation name
-- Filter by tags (user_id, collection_id, etc.)
-- Look for error spans (marked in red)
-- Examine span logs for detailed error information
-
-## Development Usage
-
-### Local Development
-
-For local development without Docker:
-
-```bash
-# Start Jaeger locally
-docker run -d --name jaeger \
- -p 16686:16686 \
- -p 14268:14268 \
- jaegertracing/all-in-one:1.60
-
-# Set environment variables
-export JAEGER_ENABLED=True
-export JAEGER_ENDPOINT=http://localhost:14268/api/traces
-
-# Start your application
-make serve-api
-```
-
-### Custom Tracing
-
-Add custom spans to your code:
-
-```python
-from aperag.trace import get_tracer, create_span
-
-tracer = get_tracer(__name__)
-
-# Synchronous function
-with create_span(tracer, "my_operation", custom_attr="value"):
- # Your code here
- pass
-
-# Or use decorators
-from aperag.trace import trace_async_function
-
-@trace_async_function("custom_operation")
-async def my_async_function():
- # This function will be automatically traced
- pass
-```
-
-## Production Considerations
-
-### Performance Impact
-
-- Tracing adds minimal overhead (~1-5ms per request)
-- Sampling can be configured to reduce load
-- Consider disabling console output in production
-
-### Data Retention
-
-- Jaeger stores traces in memory by default
-- For production, consider using persistent storage:
- - Elasticsearch backend
- - Cassandra backend
- - Kafka for high-throughput scenarios
-
-### Security
-
-- Jaeger UI has no built-in authentication
-- Consider putting it behind a reverse proxy with auth
-- Traces may contain sensitive data - review what gets traced
-
-## Troubleshooting
-
-### Jaeger Not Receiving Traces
-
-1. Check if Jaeger is running: `docker ps | grep jaeger`
-2. Verify endpoint configuration: `echo $JAEGER_ENDPOINT`
-3. Check application logs for tracing errors
-4. Ensure `OTEL_ENABLED=True` and `JAEGER_ENABLED=True`
-
-### Missing Spans
-
-1. Verify instrumentation is enabled for the component
-2. Check if the operation is being captured
-3. Look for exceptions in span creation
-
-### Performance Issues
-
-1. Disable console output: `OTEL_CONSOLE_ENABLED=False`
-2. Configure sampling rates if needed
-3. Monitor Jaeger resource usage
-
-## Resources
-
-- [Jaeger Documentation](https://www.jaegertracing.io/docs/)
-- [OpenTelemetry Python](https://opentelemetry.io/docs/instrumentation/python/)
-- [FastAPI OpenTelemetry](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/fastapi/fastapi.html)
\ No newline at end of file
diff --git a/docs/en-US/reference/HOW-TO-DEBUG.md b/docs/en-US/reference/HOW-TO-DEBUG.md
deleted file mode 100644
index 8e831744b..000000000
--- a/docs/en-US/reference/HOW-TO-DEBUG.md
+++ /dev/null
@@ -1,51 +0,0 @@
-# ApeRAG Project PyCharm Debugging Guide
-
-This document details how to debug two core components of the ApeRAG project in PyCharm: the Celery asynchronous task service and the web backend service.
-
----
-
-## 1. Debugging Celery Tasks
-
-### 1.1 Create Debug Configuration
-1. Go to PyCharm's top menu: **Run > Edit Configurations**
-2. Click **+** and select **Python**
-3. Configure with the following parameters:
-
-**Core Settings:**
-```python
-Name: celery
-Python interpreter: [uv virtual environment path]
- # Get via: `readlink .venv/bin/python`
-Script path: [Celery executable path]
- # Get via: `which celery` in terminal
-Parameters: -A config.celery worker -l INFO --pool=solo
- # Critical parameter: --pool=solo (enables single-process mode for debugging)
-Environment variables:
- PYTHONUNBUFFERED=1;PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-```
-
-
-
----
-
-## 2. Debugging Web Backend
-
-### 2.1 Create Debug Configuration
-1. Go to **Run > Edit Configurations**
-2. Click **+** and select **Python**
-3. Configure with the following parameters:
-
-**Core Settings:**
-```python
-Name: backend
-Python interpreter: [Same uv virtual environment as Celery]
-Script path: [uvicorn executable path]
- # Get via: `which uvicorn`
-Parameters: aperag.app:app --host 0.0.0.0 --log-config scripts/uvicorn-log-config.yaml
-Environment variables:
- PYTHONUNBUFFERED=1;
- DJANGO_SETTINGS_MODULE=config.settings;
-```
-
-
-
diff --git a/docs/en-US/reference/how-to-configure-ollama.md b/docs/en-US/reference/how-to-configure-ollama.md
deleted file mode 100644
index 491b48084..000000000
--- a/docs/en-US/reference/how-to-configure-ollama.md
+++ /dev/null
@@ -1,70 +0,0 @@
-# How to Configure Local Ollama with ApeRAG
-
-This guide shows how to configure local Ollama models in your ApeRAG deployment.
-
-## Prerequisites
-
-- ApeRAG running locally
-- [Ollama](https://ollama.ai/) installed and running locally
-- Ollama models downloaded
-
-## Step 1: Add Ollama Provider
-
-Navigate to **Settings > Models** in your ApeRAG interface and click **"Add Provider"**.
-
-Enter a provider name (e.g., "local-ollama") and set the **Base URL** to: `http://localhost:11434/v1`
-
-Click **Save**.
-
-
-
-## Step 2: Add Ollama Models
-
-Click the **three dots** on the right side of your newly created provider and select **"Models"** to enter the model management page.
-
-Click **"Add Model"** and configure:
-- **Model Name**: Enter your model name (e.g., `gpt-oss:20b`)
-- **Model Type**: Select `Completion`
-- **LLM Provider**: Select `openai` (Ollama is OpenAI-compatible)
-
-Click **Save**.
-
-
-
-
-
-## Step 3: Enable Model Usage
-
-You'll notice each model has two toggle switches: **Agent** and **Collection**. You can enable both:
-
-- **Agent**: Allows the model to be used for answering questions
-- **Collection**: Allows the model to be used when building Collection indexes
-
-
-
-
-## Step 4: Enable Ollama Provider
-
-Return to the Providers page and click the **three dots** on the right side of your Ollama provider, then select **"Enable"**.
-
-When prompted for an API key, enter any random string since Ollama is self-hosted and doesn't require actual authentication.
-
-
-
-
-Your Ollama models should now appear in the models list, ready for use.
-
-
-
-## Usage
-
-Once configured, your local Ollama models will be available:
-
-- **For Collections**: Select Ollama models in LLM settings when creating or configuring collections
-- **For Chat**: Choose Ollama models in the chat interface for conversations
-
-
-
-
-
-Your local Ollama models are now ready to use with ApeRAG!
diff --git a/docs/zh-CN/design/agent-backend.md b/docs/zh-CN/design/agent-backend.md
deleted file mode 100644
index 15162d3d6..000000000
--- a/docs/zh-CN/design/agent-backend.md
+++ /dev/null
@@ -1,529 +0,0 @@
-# ApeRAG Agent 后端接口设计方案
-
-## 1. 设计概述
-
-基于现有ApeRAG项目架构，为Agent功能设计一套完整的后端接口系统。Agent作为一个独立的智能对话助手，需要支持Web搜索、模型切换等功能，并提供流畅的对话体验。集成现有的MCP接口进行collection搜索和管理。
-
-## 2. 接口架构设计
-
-### 2.1 接口路径规划
-
-根据现有API设计模式，分为两大模块：
-
-```
-/api/v1/agent/
-└── chats/ # Agent对话管理
-
-/api/v1/web/
-├── search # Web搜索
-└── read # Web内容读取
-```
-
-### 2.2 数据流架构
-
-```
-Frontend → Agent API → Agent Service → [
- MCP Service (collection搜索，由Agent后端调用)
- Web Service (网络搜索和内容读取)
- LLM Service (模型调用)
- Chat Service (对话历史)
-]
-```
-
-## 3. 接口详细设计
-
-### 3.1 Agent对话管理接口
-
-#### 3.1.1 创建Agent对话
-```
-POST /api/v1/agent/chats
-```
-
-**请求体**：
-```json
-{
- "title": "新对话" // 可选，默认自动生成
-}
-```
-
-**响应**：
-```json
-{
- "id": "chat_12345",
- "title": "新对话",
- "created": "2025-01-07T10:00:00Z",
- "updated": "2025-01-07T10:00:00Z"
-}
-```
-
-#### 3.1.2 获取对话列表
-```
-GET /api/v1/agent/chats
-```
-
-**响应**：
-```json
-{
- "items": [
- {
- "id": "chat_12345",
- "title": "技术问题讨论",
- "created": "2025-01-07T10:00:00Z",
- "updated": "2025-01-07T10:00:00Z"
- }
- ]
-}
-```
-
-#### 3.1.3 获取对话详情
-```
-GET /api/v1/agent/chats/{chat_id}
-```
-
-**响应**：
-```json
-{
- "id": "chat_12345",
- "title": "技术问题讨论",
- "created": "2025-01-07T10:00:00Z",
- "updated": "2025-01-07T10:00:00Z",
- "messages": [
- {
- "id": "msg_67890",
- "type": "user",
- "content": "请介绍一下ApeRAG的架构",
- "web_search_enabled": false,
- "model_used": "claude-3-5-sonnet",
- "timestamp": "2025-01-07T10:01:00Z"
- },
- {
- "id": "msg_67891",
- "type": "assistant",
- "content": "ApeRAG是一个...",
- "sources": [
- {
- "collection_id": "col_1",
- "collection_name": "技术文档",
- "score": 0.95,
- "text": "相关文档内容...",
- "metadata": {"source": "doc1.pdf"}
- }
- ],
- "web_search_results": null,
- "model_used": "claude-3-5-sonnet",
- "timestamp": "2025-01-07T10:01:05Z"
- }
- ]
-}
-```
-
-#### 3.1.4 发送消息
-```
-POST /api/v1/agent/chats/{chat_id}/messages
-```
-
-**请求体**：
-```json
-{
- "content": "请介绍一下ApeRAG的架构",
- "model_id": "claude-3-5-sonnet", // 可选，使用默认模型
- "web_search_enabled": false, // 可选，默认false
- "stream": true // 可选，是否流式响应
-}
-```
-
-**流式响应**（Server-Sent Events）：
-```
-data: {"type": "start", "message_id": "msg_67890"}
-
-data: {"type": "content", "content": "ApeRAG是一个"}
-
-data: {"type": "content", "content": "强大的"}
-
-data: {"type": "sources", "sources": [...]}
-
-data: {"type": "end", "message_id": "msg_67890"}
-```
-
-**非流式响应**：
-```json
-{
- "id": "msg_67890",
- "content": "ApeRAG是一个强大的RAG系统...",
- "sources": [
- {
- "collection_id": "col_1",
- "collection_name": "技术文档",
- "score": 0.95,
- "text": "相关文档内容...",
- "metadata": {"source": "doc1.pdf"}
- }
- ],
- "web_search_results": null,
- "model_used": "claude-3-5-sonnet",
- "created": "2025-01-07T10:01:05Z"
-}
-```
-
-### 3.2 Web服务接口
-
-#### 3.2.1 Web搜索接口
-
-**HTTP接口**：
-```
-POST /api/v1/web/search
-```
-
-**MCP工具**：
-```
-web_search(query, max_results, search_engine, ...)
-```
-
-参考[JINA Reader API](https://jina.ai/reader)的`s.jina.ai`设计。
-
-**请求体**：
-```json
-{
- "query": "ApeRAG 2025年最新发展",
- "max_results": 5, // 可选，默认5
- "search_engine": "google", // 可选，默认google
- "timeout": 30, // 可选，超时时间（秒）
- "locale": "zh-CN" // 可选，浏览器语言
-}
-```
-
-**响应**：
-```json
-{
- "query": "ApeRAG 2025年最新发展",
- "results": [
- {
- "rank": 1,
- "title": "ApeRAG 2025年技术路线图",
- "url": "https://example.com/aperag-2025-roadmap",
- "snippet": "ApeRAG在2025年将重点发展...",
- "domain": "example.com",
- "timestamp": "2025-01-01T00:00:00Z"
- }
- ],
- "search_engine": "google",
- "total_results": 1250,
- "search_time": 1.2
-}
-```
-
-#### 3.2.2 Web内容读取接口
-
-**HTTP接口**：
-```
-POST /api/v1/web/read
-```
-
-**MCP工具**：
-```
-web_read(urls, timeout, ...)
-```
-
-参考[JINA Reader API](https://jina.ai/reader)的`r.jina.ai`设计。
-
-**请求体**：
-```json
-{
- "urls": [ // 支持单个URL字符串或URL数组
- "https://example.com/aperag-2025-roadmap",
- "https://example.com/another-page"
- ],
- "timeout": 30, // 可选，超时时间
- "css_selector": null, // 可选，CSS选择器，提取特定内容
- "wait_for_selector": null, // 可选，等待选择器，适用于SPA页面
- "exclude_selector": null, // 可选，排除选择器，去掉广告等无用内容
- "bypass_cache": false, // 可选，绕过缓存，获取最新内容
- "locale": "zh-CN", // 可选，浏览器语言
- "max_concurrent": 3 // 可选，最大并发数（仅对多个URL有效）
-}
-```
-
-**响应**：
-```json
-{
- "results": [
- {
- "url": "https://example.com/aperag-2025-roadmap",
- "status": "success",
- "title": "ApeRAG 2025年技术路线图",
- "content": "# ApeRAG 2025年技术路线图\n\nApeRAG在2025年将...",
- "extracted_at": "2025-01-07T10:01:00Z",
- "word_count": 1250,
- "token_count": 3200
- },
- {
- "url": "https://example.com/another-page",
- "status": "error",
- "error": "页面无法访问",
- "error_code": "TIMEOUT"
- }
- ],
- "total_urls": 2,
- "successful": 1,
- "failed": 1,
- "processing_time": 5.2
-}
-```
-
-## 4. 前后端交互流程
-
-### 4.1 对话创建流程
-
-```mermaid
-sequenceDiagram
- participant F as Frontend
- participant A as Agent API
- participant S as Agent Service
- participant D as Database
-
- F->>A: POST /api/v1/agent/chats
- A->>S: agent_service.create_chat()
- S->>D: 创建对话记录
- D-->>S: 返回chat_id
- S-->>A: 返回chat对象
- A-->>F: 返回JSON响应
-```
-
-### 4.2 消息发送流程
-
-```mermaid
-sequenceDiagram
- participant F as Frontend
- participant A as Agent API
- participant S as Agent Service
- participant MCP as MCP Service
- participant WS as Web Service
- participant LLM as LLM Service
-
- F->>A: POST /api/v1/agent/chats/{chat_id}/messages
- A->>S: agent_service.send_message()
-
- Note over S: Agent后端智能选择collections
- S->>MCP: 调用MCP搜索接口
- MCP-->>S: 返回搜索结果
-
- alt 如果启用web搜索
- S->>WS: 调用Web搜索/读取接口
- WS-->>S: 返回web结果
- end
-
- S->>LLM: 调用LLM生成回答
- LLM-->>S: 返回流式响应
- S-->>A: 返回流式响应
- A-->>F: SSE流式响应
-```
-
-### 4.3 Web服务调用流程
-
-```mermaid
-sequenceDiagram
- participant Client as 客户端
- participant HTTP as HTTP API
- participant MCP as MCP Tools
- participant WS as Web Service
- participant JINA as JINA API (初期)
-
- alt HTTP调用
- Client->>HTTP: POST /api/v1/web/search
- HTTP->>WS: 执行搜索
- WS->>JINA: 调用JINA API
- JINA-->>WS: 返回结果
- WS-->>HTTP: 格式化响应
- HTTP-->>Client: 返回标准格式
- else MCP调用
- Client->>MCP: web_search(query, ...)
- MCP->>WS: 执行搜索
- WS->>JINA: 调用JINA API
- JINA-->>WS: 返回结果
- WS-->>MCP: 格式化响应
- MCP-->>Client: 返回MCP格式
- end
-```
-
-## 5. 数据模型设计
-
-### 5.1 Agent对话模型
-
-```python
-class AgentChat:
- id: str
- title: str
- created: datetime
- updated: datetime
-
-class AgentMessage:
- id: str
- chat_id: str
- type: Literal["user", "assistant"]
- content: str
- model_used: str
- web_search_enabled: bool
- sources: List[SearchSource]
- web_search_results: List[WebSearchResult]
- created: datetime
-```
-
-### 5.2 Web服务模型
-
-```python
-class WebSearchRequest:
- query: str
- max_results: Optional[int] = 5
- search_engine: Optional[str] = "google"
- include_content: Optional[bool] = False
- timeout: Optional[int] = 30
- format: Optional[str] = "markdown"
- # ... 其他JINA参数
-
-class WebReadRequest:
- urls: Union[str, List[str]]
- format: Optional[str] = "markdown"
- timeout: Optional[int] = 30
- max_concurrent: Optional[int] = 3
- # ... 其他JINA参数
-
-class WebResult:
- url: str
- status: Literal["success", "error"]
- title: Optional[str]
- content: Optional[str]
- format: str
- word_count: Optional[int]
- token_count: Optional[int]
- images: Optional[List[WebImage]]
- links: Optional[List[WebLink]]
- error: Optional[str]
- error_code: Optional[str]
-```
-
-## 6. 实现策略
-
-### 6.1 分阶段实现
-
-**第一阶段：JINA集成**
-- 直接集成[JINA Reader API](https://jina.ai/reader)作为后端
-- 提供标准化的HTTP和MCP接口
-- 支持JINA的主要参数和功能
-
-**第二阶段：自研实现**
-- 实现自己的web搜索引擎集成
-- 实现自己的网页内容提取
-- 逐步替换JINA依赖
-
-**第三阶段：优化增强**
-- 添加缓存机制
-- 实现智能内容摘要
-- 支持更多搜索引擎
-
-### 6.2 接口兼容性
-
-无论使用JINA还是自研实现，都保持相同的接口格式，确保：
-- HTTP API接口不变
-- MCP工具接口不变
-- 响应格式保持一致
-- 参数含义保持兼容
-
-## 7. 错误处理设计
-
-### 7.1 标准错误响应格式
-
-```json
-{
- "error": "WEB_SEARCH_FAILED",
- "message": "网络搜索服务暂时不可用",
- "details": {
- "search_engine": "google",
- "retry_after": 30
- }
-}
-```
-
-### 7.2 常见错误码
-
-| 错误码 | HTTP状态码 | 描述 |
-|--------|-----------|------|
-| CHAT_NOT_FOUND | 404 | 对话不存在 |
-| MODEL_NOT_AVAILABLE | 400 | 模型不可用 |
-| WEB_SEARCH_FAILED | 500 | Web搜索失败 |
-| WEB_READ_FAILED | 500 | 网页读取失败 |
-| URL_NOT_ACCESSIBLE | 400 | URL无法访问 |
-| QUOTA_EXCEEDED | 429 | 超过配额限制 |
-| INVALID_URL_FORMAT | 400 | URL格式错误 |
-| TIMEOUT_ERROR | 408 | 请求超时 |
-
-## 8. 性能优化设计
-
-### 8.1 缓存策略
-
-- **Web搜索结果缓存**：相同查询的搜索结果缓存1小时
-- **Web页面内容缓存**：页面内容缓存6小时，支持ETags
-- **模型配置缓存**：缓存用户可用的模型列表
-
-### 8.2 异步处理
-
-- **流式响应**：使用异步生成器实现流式输出
-- **并发Web读取**：支持批量并发读取多个页面
-- **超时控制**：所有外部请求都有超时限制
-
-### 8.3 限流和配额
-
-- **用户级限流**：每个用户每分钟最多10次web搜索
-- **IP级限流**：防止滥用
-- **内容大小限制**：单个页面内容最大5MB
-
-## 9. 安全性设计
-
-### 9.1 认证授权
-
-- **API认证**：支持Bearer Token和Cookie认证
-- **权限控制**：用户只能访问自己的对话
-- **API限流**：防止接口滥用
-
-### 9.2 数据安全
-
-- **URL验证**：验证URL格式和域名白名单
-- **内容过滤**：过滤恶意内容和敏感信息
-- **XSS防护**：对Web内容进行sanitization
-
-## 10. 实现优先级
-
-### 10.1 第一阶段（核心功能）
-
-1. Agent对话管理接口
-2. 基础消息发送（非流式）
-3. MCP接口集成（collection搜索）
-4. Web服务接口（HTTP + MCP）
-5. JINA Reader API集成
-
-### 10.2 第二阶段（高级功能）
-
-1. 流式响应支持
-2. 缓存优化
-3. 错误处理完善
-4. 自研Web服务开发
-
-### 10.3 第三阶段（优化功能）
-
-1. 高级错误处理
-2. 性能监控
-3. 审计日志集成
-4. 智能内容摘要
-
-## 11. 总结
-
-这个设计方案基于现有ApeRAG架构，通过新增Agent专用接口和独立的Web服务来支持智能对话功能。主要特点：
-
-1. **架构兼容**：复用现有的service层和数据模型
-2. **MCP集成**：通过MCP接口进行collection搜索，Agent后端智能选择
-3. **Web服务独立**：参考[JINA Reader API](https://jina.ai/reader)设计，提供HTTP和MCP双接口
-4. **参数兼容**：完整支持JINA的参数体系，便于初期集成
-5. **渐进替换**：先用JINA实现，后续自研替换，接口保持兼容
-6. **扩展性强**：接口设计支持未来功能扩展
-7. **性能优化**：考虑缓存、异步处理等性能优化
-
-前端可以通过这些接口实现类似Cursor的对话体验，用户可以轻松切换模型，并获得智能的搜索和问答服务。Agent后端会智能调用MCP接口进行collection搜索，同时Web服务提供强大的网络信息获取能力。
\ No newline at end of file
diff --git a/docs/zh-CN/design/agent_runtime_v3.md b/docs/zh-CN/design/agent_runtime_v3.md
deleted file mode 100644
index c841f42e5..000000000
--- a/docs/zh-CN/design/agent_runtime_v3.md
+++ /dev/null
@@ -1,568 +0,0 @@
----
-title: Agent Runtime V3 设计方案
-description: ApeRAG Agent Runtime V3 的详细设计文档，定义新的 Turn、TimelineEvent、Artifact、SSE 协议与迁移边界
-keywords: Agent Runtime, SSE, Turn, TimelineEvent, Artifact, PydanticAI, MCP
-position: 4
----
-
-# ApeRAG Agent Runtime V3 设计方案
-
-## 1. 背景与目标
-
-当前 ApeRAG 的 agent chat 主链路基于 `mcp-agent` 运行。它已经证明“能工作”，但对 ApeRAG 的长期产品目标并不合适：
-
-- 当前最脆弱的部分不是业务 API，而是 runtime 胶水层
-- 事件分发、流式输出、前后端消息格式、会话缓存、工具结果回推都与第三方 runtime 的内部语义耦合过深
-- 这类耦合会直接转化成私有化交付后的维护成本、排障成本和答疑成本
-
-因此，这次设计的目标不是“换一个更强的 agent 框架”，而是：
-
-1. 重做 ApeRAG 的 `agent 产品层`
-2. 保持 ApeRAG 的 `业务能力面` 稳定
-3. 建立一个更适合 `私有化部署`、`简单可靠`、`低维护成本`、`低答疑成本` 的单 agent runtime
-
-本设计的约束前提如下：
-
-- 优先面向 `私有化交付`
-- 假设客户普遍缺乏强运维能力
-- 功能可以保守，但默认行为必须稳健
-- agent 层允许整体切换，不为旧 WebSocket grammar、旧 Redis message shape、旧 message part 语义做长期兼容
-
-## 2. 设计结论
-
-Agent Runtime V3 的正式结论如下：
-
-1. `mcp-agent` 不再作为 ApeRAG 的长期核心 runtime
-2. `FastAPI + FastMCP + 现有业务 API/provider` 继续作为稳定业务面保留
-3. 主传输协议统一为 `SSE`
-4. 核心产品契约统一为 `Turn + TimelineEvent + Artifact`
-5. 第一阶段 runtime 实现使用 `PydanticAI adapter`
-6. 长期保留进一步收敛到 `自研薄编排层` 的空间
-7. 不把核心 runtime 迁移到 `Vercel AI SDK`、`OpenAI Agents SDK` 或 `LangGraph`
-
-这意味着：外部库只负责帮助实现，不再定义 ApeRAG 的产品语义。
-
-## 3. 设计原则
-
-### 3.1 私有化优先
-
-所有设计优先满足以下诉求：
-
-- 默认配置可运行
-- 失败行为可诊断
-- 升级和回退边界清晰
-- 少依赖隐式前提
-- 少引入双栈和兼容包袱
-
-### 3.2 产品契约归 ApeRAG 自己拥有
-
-新的对外契约由 ApeRAG 自己定义，包括：
-
-- API 入口
-- SSE 事件流
-- 前端可见状态词表
-- TimelineEvent schema
-- Artifact schema
-- History commit policy
-
-第三方 runtime 只能适配这套契约，不能反向决定它。
-
-### 3.3 最终回答与过程事件彻底分离
-
-最终 answer、运行过程、references、tool result 不再混塞进一条 assistant message。
-
-分层原则如下：
-
-- `answer` 是 answer artifact
-- `timeline` 是过程事件流
-- `references` 是独立 artifact
-- `tool result` 通过摘要事件 + artifact 引用暴露
-
-### 3.4 明确缩小 Phase 1 能力边界
-
-Phase 1 只支持：
-
-- 单 agent
-- 串行 tool loop
-- 单 MCP server 视图
-- 单个 turn 内多轮 internal loop
-
-Phase 1 不支持：
-
-- 多 agent
-- 并发 tool fan-out
-- workflow/graph orchestration
-- 长任务编排
-
-## 4. 核心对象模型
-
-## 4.1 Turn
-
-`Turn` 表示一次用户 query 对应的一次完整 agent execution。
-
-需要特别澄清的是：
-
-- 一个 turn 不是“一步回答”
-- 一个 turn 内允许多次 thinking、多次 web search、多次 tool call、多次读取结果和多轮内部推理
-- turn 是外层执行边界，不是内部 loop 次数的限制器
-
-### 4.1.1 设计目标
-
-Turn 统一承担以下职责：
-
-- 幂等边界
-- 取消边界
-- 超时边界
-- 恢复边界
-- 最终历史提交边界
-- 回放与评测边界
-
-### 4.1.2 建议字段
-
-```text
-schema_version
-turn_id
-chat_id
-user_id
-request_id
-client_idempotency_key
-status
-input_text
-model_profile
-started_at
-finished_at
-error_code
-error_message
-answer_artifact_id
-reference_bundle_artifact_id
-timeline_cursor
-```
-
-### 4.1.3 状态机
-
-```text
-queued -> running -> completed
-queued -> running -> failed
-queued -> running -> cancelled
-```
-
-### 4.1.4 硬约束
-
-1. 一个 turn 只允许一个最终 `answer_artifact_id`
-2. 同一个 turn 不允许被执行两次
-3. 同一个 `chat_id + client_idempotency_key` 只能创建一个有效 turn
-
-## 4.2 TimelineEvent
-
-`TimelineEvent` 表示 turn 执行过程中的标准化事件流。
-
-它既是：
-
-- 前端时间线展示模型
-- SSE 传输模型
-- 诊断与回放模型
-
-但它不是：
-
-- runtime 原始内部日志 dump
-- debug event 任意透出层
-
-### 4.2.1 必备字段
-
-```text
-schema_version
-event_id
-turn_id
-sequence
-timestamp
-type
-label
-status
-actor
-data
-```
-
-### 4.2.2 硬约束
-
-1. `sequence` 在 turn 内必须严格单调递增
-2. 不允许前端按时间戳猜顺序
-3. `actor` 只允许：`agent | tool | system`
-4. `data` 只携带最小必要 payload
-5. timeline 必须支持重放
-
-### 4.2.3 事件类型
-
-Phase 1 标准事件类型定义如下：
-
-- `turn.started`
-- `agent.state.changed`
-- `tool.started`
-- `tool.progress`
-- `tool.finished`
-- `external_action.started`
-- `external_action.finished`
-- `text.delta`
-- `artifact.created`
-- `turn.completed`
-- `turn.failed`
-- `turn.cancelled`
-- `heartbeat`
-
-### 4.2.4 事件分层约束
-
-- `tool.*` 用于标准 tool loop
-- `external_action.*` 只用于少数用户可感知外部动作，例如 `web_search`
-- 不允许把所有内部小步骤都提升到 timeline 层
-
-## 4.3 Artifact
-
-`Artifact` 表示需要持久化、可重读、可复用、可排障的大对象。
-
-### 4.3.1 建议类型
-
-- `answer`
-- `reference_bundle`
-- `tool_result_summary`
-- `search_result_summary`
-- `error_summary`
-
-### 4.3.2 建议字段
-
-```text
-schema_version
-artifact_id
-turn_id
-artifact_type
-created_at
-summary
-storage_ref | payload
-```
-
-### 4.3.3 硬约束
-
-1. stream 中不直接推送大正文
-2. stream 只推送摘要、artifact id 和必要元数据
-3. references 必须 materialize 成独立 artifact
-
-## 5. 用户可见状态词表
-
-前端不直接显示 runtime 原始事件名，而是统一映射成稳定的用户可见状态词表。
-
-Phase 1 固定为：
-
-- `Thinking`
-- `Searching`
-- `Calling Tool`
-- `Reading Result`
-- `Streaming Answer`
-- `Completed`
-- `Failed`
-
-这样做的目的有两个：
-
-1. 避免后端内部状态演进反复影响前端展示
-2. 降低用户理解成本与答疑成本
-
-## 6. 前后端协议设计
-
-## 6.1 主传输协议
-
-主链路只保留 `SSE`。
-
-不长期保留 `WebSocket + SSE` 双栈共存。
-
-## 6.2 API 设计
-
-### 6.2.1 创建 turn
-
-```text
-POST /api/v2/agent/chats/{chat_id}/turns
-```
-
-请求体建议包含：
-
-- `query`
-- `context`
-- `model_profile`
-- `client_idempotency_key`
-
-响应建议包含：
-
-- `turn_id`
-- `status`
-- `stream_url`
-
-### 6.2.2 订阅 turn 事件流
-
-```text
-GET /api/v2/agent/chats/{chat_id}/turns/{turn_id}/events
-```
-
-返回类型：
-
-```text
-Content-Type: text/event-stream
-```
-
-### 6.2.3 获取 turn snapshot
-
-```text
-GET /api/v2/agent/chats/{chat_id}/turns/{turn_id}
-```
-
-用于：
-
-- 刷新页面恢复
-- SSE 重连失败时兜底
-- 调试和诊断
-
-### 6.2.4 取消 turn
-
-```text
-POST /api/v2/agent/chats/{chat_id}/turns/{turn_id}/cancel
-```
-
-### 6.2.5 获取 artifact
-
-```text
-GET /api/v2/agent/artifacts/{artifact_id}
-```
-
-### 6.2.6 OpenAI-compatible adapter
-
-```text
-POST /v1/chat/completions
-```
-
-这个接口是给 OpenAI 形状客户端使用的兼容 adapter，不是前端主 UI
-contract。实现必须把每个请求转换成 Agent Runtime V3 turn，再按
-OpenAI 形状格式化输出：
-
-- `stream=false` 时返回 `chat.completion` JSON
-- `stream=true` 时返回 `text/event-stream` 的 `chat.completion.chunk`
- 帧
-
-adapter contract 固定为：
-
-- `bot_id` 是必填 query 参数
-- `chat_id` 可选；不传时后端创建并在请求结束后删除 ephemeral chat
-- `language` 可选，默认 `en-US`
-- `Idempotency-Key` / `X-Idempotency-Key` 映射为
- `client_idempotency_key`
-
-## 6.3 幂等与重连
-
-### 6.3.1 幂等策略
-
-- `POST turn` 必须支持客户端幂等键
-- 同一 `chat_id + client_idempotency_key` 下重复请求不得创建多个 turn
-- 同一个 turn 一旦创建成功，就不允许重复执行
-
-### 6.3.2 SSE 重连策略
-
-- 默认支持 `Last-Event-ID` 或 offset 续传
-- 如果服务端 event buffer 已过期，则：
- 1. 客户端先拉 `turn snapshot`
- 2. 再从当前最新游标继续订阅
-
-## 6.4 心跳、背压与超时
-
-SSE 层必须定义以下行为：
-
-- heartbeat 事件
-- event buffer 上限
-- delta 合并策略
-- 过载时摘要/截断策略
-
-同时区分以下超时：
-
-- 单 tool timeout
-- 单轮 total runtime timeout
-- stream idle timeout
-
-## 7. 权限与安全边界
-
-新 runtime 入口必须重新做完整鉴权，不能依赖旧 WebSocket 路径中的隐式前提。
-
-每个 turn 创建时必须重新校验：
-
-- `chat_id` ownership
-- collection/file context visibility
-- tool 可见范围
-
-artifact 读取接口也必须重新做权限校验，防止通过 artifact id 越权读取。
-
-## 8. 存储设计
-
-## 8.1 Redis 职责
-
-Redis 只负责短期运行态与流式恢复：
-
-- `turn runtime state`
-- `stream cursor`
-- `transient event buffer`
-- `in-flight text buffer`
-
-Redis 不再承担：
-
-- 旧 message grammar 兼容
-- 最终产品层消息协议
-- 长期历史表达
-
-## 8.2 DB / Persistent Store 职责
-
-持久化层负责保存：
-
-- `conversation_turn`
-- `timeline_event`（至少关键事件）
-- `artifact`
-- `reference_bundle`
-- `error_summary`
-
-Timeline 必须可重放，不能只存在于流式阶段。
-
-## 8.3 History Commit Policy
-
-最终 history 不按 token 流实时写入。
-
-策略如下：
-
-1. stream 期间只写运行态/缓存态
-2. 只有在 `done` 或明确 `error` 后，才一次性提交标准化 turn 记录
-
-这样可以避免：
-
-- 半截输出污染 history
-- 取消后残留脏记录
-- 重连或回退留下不可解释状态
-
-## 9. 前端体验模型
-
-新的前端展示不再以“一个 assistant bubble 包含一切”为核心。
-
-建议拆成五个视图层：
-
-1. `Turn Header`
-2. `Timeline`
-3. `Final Answer Panel`
-4. `References Panel`
-5. `Diagnostics Drawer`
-
-其中：
-
-- Timeline 只展示过程
-- Final Answer Panel 只展示最终 answer
-- References Panel 只展示引用和来源
-- Diagnostics Drawer 只在需要时展开
-
-## 10. Runtime 路线
-
-## 10.1 Phase 1：PydanticAI Adapter
-
-第一阶段 runtime 实现采用 `PydanticAI`，原因不是它定义契约，而是它能降低第一阶段实现成本。
-
-它适合用来实现：
-
-- 单 turn 内部 loop
-- tool 调用
-- provider 调用
-- 状态映射
-
-但不负责定义：
-
-- 对外 API
-- TimelineEvent schema
-- Artifact schema
-- History commit policy
-
-## 10.2 长期路线
-
-如果 `PydanticAI adapter` 运行稳定、维护成本可接受，可以继续保留。
-
-如果后续仍发现第三方 runtime 对行为边界约束太多，则继续把底层收成完全自研薄编排层。
-
-由于契约层已经独立，届时只替换 runtime 实现，不需要再次重写前后端协议。
-
-## 11. 替换边界
-
-本次重写：
-
-- 保留：`FastAPI`、`FastMCP`、业务 API、provider 接入、业务数据实体
-- 替换：`mcp-agent runtime glue`、旧 WebSocket grammar、旧 Redis message shape、旧前端消息渲染模型、旧事件回推机制
-
-这意味着：
-
-- 业务价值复用
-- 产品层和运行时耦合重做
-
-## 12. 迁移与回退原则
-
-### 12.1 迁移原则
-
-- 可以保留短期 feature flag 灰度
-- 不允许长期双栈共存
-- 一旦新链路稳定，旧 WebSocket grammar 和旧 runtime glue 直接下线
-
-### 12.2 回退条件
-
-以下情况允许回退：
-
-- SSE 在企业代理环境中明显不稳定
-- timeline 重放和恢复不可靠
-- turn/history/artifact 兼容性不成立
-- tool/provider 错误不可诊断
-
-### 12.3 回退要求
-
-回退后必须保证：
-
-- 历史记录可读
-- turn 记录不变成孤儿
-- artifact 不变成不可追踪残留
-
-## 13. Phase 1 实施清单
-
-Phase 1 的实施目标是跑通新的最小主链路，而不是一次性追求最终形态。
-
-建议的 Phase 1 任务：
-
-1. 新建 `aperag/agent_runtime/` 模块
-2. 定义 `Turn / TimelineEvent / Artifact` schema
-3. 实现 `TurnService`
-4. 实现 `EventService`
-5. 实现 `ArtifactService`
-6. 实现 `HistoryWriter`
-7. 定义 `AgentRuntime` 抽象
-8. 实现 `PydanticAIRuntime`
-9. 实现 MCP client adapter
-10. 实现 `SSE StreamEmitter`
-11. 新增 v2 agent API
-12. 新增前端 timeline 组件
-13. 新增 answer/references/diagnostics 分层面板
-14. 实现 snapshot 恢复与 cancel
-15. 补充契约级 E2E 覆盖
-
-## 14. 验收标准
-
-Phase 1 完成时，至少应满足：
-
-1. 新 API 可以创建 turn、订阅 SSE、拉 snapshot、读取 artifact、取消 turn
-2. 单 turn 内部可以完成多轮 search/tool/thinking loop
-3. Timeline 可重连、可重放
-4. 最终 answer 与过程事件完全分离
-5. 历史提交策略不产生半截脏记录
-6. `mcp-agent` 已退出主 chat 运行路径
-
-## 15. 正式拍板
-
-Agent Runtime V3 的正式拍板如下：
-
-- 不再继续修补 `mcp-agent`
-- 不再继续修补旧 WebSocket grammar
-- 新 runtime 契约统一为 `Turn + TimelineEvent + Artifact + SSE`
-- 第一阶段采用 `PydanticAI adapter`
-- 后续由实现同学按本设计文档推进，架构侧负责监督契约边界与长期方向
-
-一句话总结：
-
-这次不是“把某个 agent 库换掉”，而是为 ApeRAG 重建一个更适合私有化交付的 agent runtime 产品层。
diff --git a/docs/zh-CN/design/architecture.md b/docs/zh-CN/design/architecture.md
deleted file mode 100644
index 572e2ed72..000000000
--- a/docs/zh-CN/design/architecture.md
+++ /dev/null
@@ -1,850 +0,0 @@
----
-title: 系统架构
-description: ApeRAG 架构设计与核心组件详解
-keywords: ApeRAG, 架构, RAG, 知识图谱, LightRAG
-position: 1
----
-
-# ApeRAG 系统架构
-
-## 1. 什么是 ApeRAG
-
-ApeRAG 是一个**开放的、Agentic 的 Graph RAG 平台**。它不仅仅是一个简单的向量检索系统，而是将知识图谱、多模态检索和智能 Agent 深度融合的生产级解决方案。
-
-传统的 RAG 系统主要依赖向量相似度检索，虽然能找到语义相关的内容，但往往缺乏对知识之间关系的理解。ApeRAG 的核心创新在于：
-
-- **Graph RAG**：从文档中自动提取实体（人物、地点、概念）和关系，构建知识图谱，理解知识之间的关联
-- **Agentic**：内置智能 Agent，能够自主规划、调用工具、多轮对话，提供更智能的问答体验
-- **开放集成**：通过 **RESTful API** 和 **MCP 协议**对外暴露能力，可以轻松集成到 Dify、Claude、Cursor 等外部系统
-
-### 核心优势
-
-与传统 RAG 方案相比，ApeRAG 提供了：
-
-- **更强的文档处理能力**：支持 PDF、Word、Excel 等多种格式，能处理复杂的表格、公式、图片
-- **多种检索方式**：向量检索、全文检索、图谱检索，三者互补，各取所长
-- **知识关联理解**：通过知识图谱理解概念之间的关系，而不仅仅是文本相似度
-- **开放的集成能力**：RESTful API + MCP 协议，可以作为 Dify、Claude Desktop、Cursor 的知识后端
-- **生产级架构**：异步处理、多存储、高并发，可以直接用于生产环境
-
-### 整体架构一览
-
-```mermaid
-graph TB
- User[用户] --> Frontend[Web 前端]
- User --> External[外部系统
-Dify/Claude/Cursor] - - Frontend --> API[RESTful API] - External --> MCP[MCP 协议] - - API --> DocProcess[文档处理] - API --> Search[检索服务] - API --> Agent[Agent 对话] - MCP --> Search - MCP --> Agent - - DocProcess --> Tasks[异步任务层] - Tasks --> Storage[存储层] - - Search --> Storage - Agent --> Search - - Storage --> PG[(PostgreSQL)] - Storage --> Qdrant[(Qdrant
向量库)] - Storage --> ES[(Elasticsearch
全文搜索)] - Storage --> Neo4j[(Neo4j
图数据库)] - Storage --> MinIO[(MinIO
文件存储)] - - style User fill:#e1f5ff - style Frontend fill:#bbdefb - style External fill:#bbdefb - style API fill:#90caf9 - style MCP fill:#90caf9 - style DocProcess fill:#fff59d - style Search fill:#fff59d - style Agent fill:#fff59d - style Tasks fill:#c5e1a5 - style Storage fill:#ffccbc -``` - -## 2. 系统分层架构 - -ApeRAG 采用清晰的分层设计，每一层各司其职： - -```mermaid -graph TB - subgraph Layer1[客户端层] - Web[Web 前端
Next.js] - Dify[Dify] - Cursor[Cursor] - Claude[Claude Desktop] - end - - subgraph Layer2[接口层] - API[RESTful API
FastAPI] - MCP[MCP Server
Model Context Protocol] - end - - subgraph Layer3[服务层] - CollSvc[Collection 服务] - DocSvc[文档服务] - SearchSvc[检索服务] - GraphSvc[图谱服务] - AgentSvc[Agent 服务] - end - - subgraph Layer4[任务层] - Celery[Celery Worker
异步任务] - MinerU[MinerU
文档解析] - end - - subgraph Layer5[存储层] - PG[(PostgreSQL)] - Qdrant[(Qdrant)] - ES[(Elasticsearch)] - Neo4j[(Neo4j)] - Redis[(Redis)] - MinIO[(MinIO)] - end - - Web --> API - Dify --> MCP - Cursor --> MCP - Claude --> MCP - - API --> CollSvc - API --> DocSvc - API --> SearchSvc - API --> GraphSvc - API --> AgentSvc - - MCP --> SearchSvc - MCP --> AgentSvc - - CollSvc --> Celery - DocSvc --> Celery - GraphSvc --> Celery - - Celery --> MinerU - Celery --> PG - Celery --> Qdrant - Celery --> ES - Celery --> Neo4j - Celery --> MinIO - - SearchSvc --> PG - SearchSvc --> Qdrant - SearchSvc --> ES - SearchSvc --> Neo4j - - style Layer1 fill:#e3f2fd - style Layer2 fill:#f3e5f5 - style Layer3 fill:#fff3e0 - style Layer4 fill:#e8f5e9 - style Layer5 fill:#fce4ec -``` - -**各层职责说明**： - -- **客户端层**：多种接入方式，Web UI 用于管理，MCP 客户端（Dify、Cursor、Claude 等）用于集成 -- **接口层**：RESTful API（传统 HTTP 接口）和 MCP Server（AI 工具协议）并行提供服务 -- **服务层**：核心业务逻辑，协调各种资源完成具体功能 -- **任务层**：处理耗时操作（文档解析、索引构建），保证 API 快速响应 -- **存储层**：多种存储系统，针对不同数据类型选择最优方案 - -## 3. 文档处理全流程 - -这是 ApeRAG 的核心能力之一。从一个 PDF 文件上传，到最终可以被检索，经历了一系列精心设计的处理步骤。 - -### 3.1 文档上传与解析 - -当你上传一个文档时，ApeRAG 会自动识别格式并选择合适的解析器： - -```mermaid -flowchart TD - Upload[用户上传文档] --> Detect[格式检测] - - Detect --> |PDF| MinerU[MinerU 解析器] - Detect --> |Word/Excel| MarkItDown[MarkItDown 解析器] - Detect --> |Markdown| DirectParse[直接解析] - Detect --> |图片| OCR[OCR 识别] - - MinerU --> Extract[内容提取] - MarkItDown --> Extract - DirectParse --> Extract - OCR --> Extract - - Extract --> Parts[文档片段
Parts 对象] - - style Upload fill:#e1f5ff - style Extract fill:#c5e1a5 - style Parts fill:#fff59d -``` - -**MinerU 的强大之处**： - -- 能准确识别复杂 PDF 的表格结构，保留表格内容的完整性 -- 提取 LaTeX 数学公式，保持公式的可读性 -- 对扫描版 PDF 进行 OCR，支持中英文混排 -- 识别文档中的图片区域，支持图片内容理解 - -### 3.2 智能分块策略 - -文档解析后，需要切分成合适大小的块（chunk）。这个步骤很关键，分块太大会影响检索精度，太小会丢失上下文。 - -```mermaid -flowchart TD - Parts[文档片段] --> Rechunk[智能重分块] - - Rechunk --> Analysis[分析文档结构] - Analysis --> Hierarchy[识别标题层级] - Hierarchy --> Group[按标题分组] - - Group --> Check{块大小检查} - Check --> |过大| Split[语义分割] - Check --> |合适| Chunks[最终块] - Split --> Chunks - - Chunks --> AddContext[添加上下文] - AddContext --> FinalChunks[带上下文的文档块] - - style Rechunk fill:#bbdefb - style Split fill:#ffccbc - style FinalChunks fill:#c5e1a5 -``` - -**分块策略的特点**： - -- **保持语义完整性**：尽量不在句子中间切断 -- **保留标题上下文**：每个块都知道自己属于哪个章节 -- **层级化分割**：先按段落分，不行再按句子分，最后才按字符分 -- **智能合并**：相邻的小标题块会被合并，避免信息碎片化 - -分块参数配置： -- 默认块大小：1200 tokens（约 800-1000 个中文字符） -- 重叠大小：100 tokens（保证上下文连续性） - -### 3.3 多索引并行构建 - -文档分块后，会同时创建多种索引。每种索引有不同的用途，互相补充： - -| 索引类型 | 适用场景 | 存储位置 | 检索方式 | -|---------|---------|---------|---------| -| **向量索引** | 语义相似问题，比如"如何优化性能" | Qdrant | 余弦相似度 | -| **全文索引** | 精确关键词搜索，比如"PostgreSQL 配置" | Elasticsearch | BM25 算法 | -| **图谱索引** | 关系型问题，比如"A 和 B 有什么联系" | PostgreSQL/Neo4j | 图遍历 | -| **摘要索引** | 快速了解文档概要 | PostgreSQL | 向量匹配 | -| **视觉索引** | 图片内容搜索 | Qdrant | 多模态向量 | - -```mermaid -flowchart LR - Chunks[文档块] --> IndexMgr[索引管理器] - - IndexMgr --> VectorIdx[向量索引创建] - IndexMgr --> FulltextIdx[全文索引创建] - IndexMgr --> GraphIdx[图谱索引创建] - IndexMgr --> VisionIdx[视觉索引创建] - - VectorIdx --> Qdrant1[(Qdrant)] - FulltextIdx --> ES[(Elasticsearch)] - GraphIdx --> Graph[(Neo4j/PG)] - VisionIdx --> Qdrant2[(Qdrant)] - - style IndexMgr fill:#fff59d - style VectorIdx fill:#bbdefb - style FulltextIdx fill:#c5e1a5 - style GraphIdx fill:#ffccbc - style VisionIdx fill:#e1bee7 -``` - -**并行构建的优势**： -- 不同索引可以同时构建，提高速度 -- 某个索引失败不影响其他索引 -- 可以按需启用特定类型的索引 - -### 3.4 知识图谱构建 - -图谱索引是 ApeRAG 的核心特色，它能从文档中提取结构化的知识。 - -```mermaid -flowchart TD - Chunks[文档块] --> EntityExtract[实体提取] - - EntityExtract --> LLM1[调用 LLM
识别实体] - LLM1 --> Entities[实体列表
人物、地点、概念] - - Entities --> RelationExtract[关系提取] - RelationExtract --> LLM2[调用 LLM
识别关系] - LLM2 --> Relations[关系列表
谁与谁有什么关系] - - Entities --> Merge[实体合并] - Relations --> Merge - - Merge --> Components[连通分量分析] - Components --> Parallel[并行处理各分量] - Parallel --> Graph[(知识图谱)] - - style EntityExtract fill:#bbdefb - style RelationExtract fill:#c5e1a5 - style Merge fill:#ffccbc - style Components fill:#fff59d -``` - -**图谱构建的关键步骤**： - -1. **实体提取**：LLM 从文档块中识别出有意义的实体 - - 示例：从"张三在北京的清华大学学习人工智能"中提取 - - 实体：张三（人物）、北京（地点）、清华大学（组织）、人工智能（概念） - -2. **关系提取**：识别实体之间的关系 - - 示例：张三 --学习--> 人工智能，张三 --就读于--> 清华大学 - -3. **实体合并**：同一实体可能有不同的表述，需要归一化 - - 示例："LightRAG"、"light rag"、"Light-RAG" → 合并为统一实体 - -4. **连通分量优化**：把图谱分成独立的子图，并行处理 - - 性能提升：2-3 倍吞吐量 - -**为什么需要连通分量优化？** - -假设你有 100 篇文档，它们讨论不同的主题。关于"数据库"的实体和关于"机器学习"的实体之间没有连接，可以独立处理。连通分量算法会找出这些独立的"知识岛"，然后并行处理，大大提高速度。 - -### 3.5 异步任务系统 - -文档处理是一个耗时的操作，ApeRAG 采用"双链路架构"来保证用户体验： - -```mermaid -graph TB - subgraph Frontend["🚀 前端链路 - 快速响应"] - direction TB - A1["📤 用户上传文档"] --> A2["🔌 API 接收请求"] - A2 --> A3["📋 Index Manager"] - A3 --> A4["💾 写入数据库
status = PENDING
version = 1"] - A4 --> A5["✅ 立即返回成功
< 100ms"] - end - - subgraph Backend["⚙️ 后端链路 - 异步处理"] - direction TB - B1["⏰ Celery Beat
每 30 秒检查"] --> B2["🔍 Reconciler 检测
version ≠ observed_version"] - B2 --> B3{"🎯 发现待处理任务?"} - B3 -->|是| B4["🚀 调度 Worker"] - B3 -->|否| B1 - B4 --> B5["📄 解析文档"] - B5 --> B6["🔀 并行创建索引
Vector + Fulltext
+ Graph + Vision"] - B6 --> B7["✨ 更新状态
status = ACTIVE
observed_version = 1"] - B7 --> B1 - end - - A4 -.-|"数据库状态变化"| B2 - - style Frontend fill:#e3f2fd,stroke:#1976d2,stroke-width:3px - style Backend fill:#fff3e0,stroke:#f57c00,stroke-width:3px - style A5 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px - style B7 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px - style B3 fill:#fff9c4,stroke:#fbc02d,stroke-width:2px -``` - -**双链路的好处**： - -- **前端快速响应**：用户上传文档后，API 在 100ms 内返回，不需要等待处理完成 -- **后端异步处理**：真正的处理工作在后台慢慢做，不阻塞用户操作 -- **自动重试**：如果处理失败，系统会自动重试，保证最终成功 -- **状态可查**：用户可以随时查看文档处理进度 - -**索引状态机**： - -```mermaid -stateDiagram-v2 - [*] --> PENDING: 📤 文档上传 - - PENDING --> CREATING: 🚀 Reconciler 检测到
开始处理 - - CREATING --> ACTIVE: ✅ 所有索引创建成功 - CREATING --> FAILED: ❌ 处理失败 - - FAILED --> CREATING: 🔄 自动重试
(最多 3 次) - FAILED --> [*]: 💔 超过重试次数
标记为失败 - - ACTIVE --> CREATING: 🔄 文档更新
重建索引 - ACTIVE --> [*]: 🗑️ 删除文档 - - note right of PENDING - version = 1 - observed_version = 0 - end note - - note right of CREATING - 正在处理中 - 可能需要几分钟 - end note - - note right of ACTIVE - version = 1 - observed_version = 1 - 可以被检索 - end note -``` - -## 4. 检索问答全流程 - -有了索引之后,用户就可以提问了。ApeRAG 的检索系统会智能地选择合适的检索策略。 - -### 4.1 混合检索系统 - -不同类型的问题适合用不同的检索方式。ApeRAG 会同时使用多种检索策略，然后融合结果： - -```mermaid -flowchart TB - Query[用户问题] --> Router[检索路由] - - Router --> |并行检索| Vector[向量检索] - Router --> |并行检索| Fulltext[全文检索] - Router --> |并行检索| Graph[图谱检索] - - Vector --> Embed[生成问题向量] - Embed --> QdrantSearch[Qdrant 相似度搜索] - QdrantSearch --> R1[结果1] - - Fulltext --> ESSearch[Elasticsearch BM25] - ESSearch --> R2[结果2] - - Graph --> GraphQuery[图谱查询
local/global/hybrid] - GraphQuery --> R3[结果3] - - R1 --> Merge[结果融合] - R2 --> Merge - R3 --> Merge - - Merge --> Rerank[Rerank 重排序] - Rerank --> Final[最终结果] - - style Query fill:#e1f5ff - style Vector fill:#bbdefb - style Fulltext fill:#c5e1a5 - style Graph fill:#ffccbc - style Rerank fill:#fff59d - style Final fill:#c5e1a5 -``` - -**检索策略说明**： - -- **向量检索**：用于语义相似的问题 - - 问："如何提升系统性能？" - - 能找到："优化数据库查询"、"使用缓存"等相关内容 - -- **全文检索**：用于精确关键词匹配 - - 问："PostgreSQL 的配置文件在哪？" - - 能找到包含"PostgreSQL"和"配置文件"的精确段落 - -- **图谱检索**：用于关系型问题 - - 问："LightRAG 和 Neo4j 有什么关系？" - - 会查询图谱中这两个实体的连接路径 - -**结果融合策略**： - -不同检索方式的结果需要合并。ApeRAG 使用 Rerank 模型对所有候选结果重新打分： - -1. 收集所有检索结果（可能有重复） -2. 去重，保留最相关的片段 -3. 使用 Rerank 模型评估每个片段与问题的相关性 -4. 按新的分数重新排序 -5. 返回 Top-K 结果 - -### 4.2 知识图谱查询 - -图谱检索有三种模式，适用于不同类型的问题： - -| 模式 | 适用场景 | 查询方式 | 示例问题 | -|------|---------|---------|---------| -| **local** | 查询某个实体的局部信息 | 向量匹配相似实体 → 获取邻居节点 | "张三的个人信息" | -| **global** | 查询整体关系和模式 | 向量匹配相似关系 → 获取关联路径 | "公司的组织架构是怎样的" | -| **hybrid** | 综合性问题 | local + global 结合 | "张三在公司的角色和职责" | - -```mermaid -flowchart TD - Question[用户问题] --> Analyze[问题分析] - - Analyze --> Local[Local 模式
实体中心] - Analyze --> Global[Global 模式
关系中心] - Analyze --> Hybrid[Hybrid 模式
综合查询] - - Local --> FindEntity[找到相关实体] - FindEntity --> GetNeighbors[获取邻居和关系] - - Global --> FindRelations[找到相关关系] - FindRelations --> GetContext[获取关系上下文] - - Hybrid --> Local - Hybrid --> Global - - GetNeighbors --> Context[生成上下文] - GetContext --> Context - - Context --> Return[返回给 LLM] - - style Local fill:#bbdefb - style Global fill:#c5e1a5 - style Hybrid fill:#fff59d -``` - -**实际例子**： - -假设知识图谱中有： -- 实体：张三（人物）、数据库团队（组织）、PostgreSQL（技术） -- 关系：张三 --属于--> 数据库团队，张三 --擅长--> PostgreSQL - -问题："张三负责什么？" - -1. **Local 模式**： - - 找到"张三"实体 - - 获取所有直接相连的节点 - - 返回："张三属于数据库团队，擅长 PostgreSQL" - -2. **Global 模式**： - - 找到相关的关系模式："负责"、"属于" - - 返回整个团队的结构和职责分工 - -3. **Hybrid 模式**： - - 同时使用上述两种方式 - - 给出更全面的答案 - -### 4.3 Agent 对话系统 - -Agent 是 ApeRAG 的智能助手，它能调用各种工具来回答问题。 - -```mermaid -sequenceDiagram - participant User as 用户 - participant API as API Server - participant Agent as Agent 服务 - participant LLM as LLM 服务 - participant MCP as MCP 工具 - participant Search as 检索服务 - - User->>API: 发送问题 - API->>Agent: 转发问题 - - Agent->>LLM: 调用 LLM
携带工具列表 - LLM-->>Agent: 决定调用 search_collection 工具 - - Agent->>MCP: 执行工具调用 - MCP->>Search: 混合检索 - Search-->>MCP: 返回相关文档片段 - MCP-->>Agent: 工具执行结果 - - Agent->>LLM: 再次调用 LLM
携带检索到的上下文 - LLM-->>Agent: 生成最终答案 - - Agent-->>API: 流式返回 - API-->>User: SSE 推送答案 -``` - -**Agent 的工作流程**： - -1. **接收问题**：用户发送一个问题 - -2. **工具决策**：LLM 分析问题，决定需要调用哪些工具 - - 可能的工具：search_collection（检索知识库）、web_search（搜索网络）、web_read（读取网页）等 - -3. **执行工具**：Agent 调用对应的工具 - - 示例：search_collection 会触发混合检索，返回相关文档 - -4. **生成答案**：LLM 基于检索到的上下文生成答案 - -5. **流式返回**：答案通过 SSE（Server-Sent Events）实时推送给用户，不用等待全部生成完毕 - -**MCP 协议的作用**： - -MCP（Model Context Protocol）是一个标准化的工具协议，让 AI 助手（如 Claude Desktop、Cursor）能够方便地调用 ApeRAG 的能力。通过 MCP，外部 AI 工具可以： -- 列出你的知识库 -- 搜索知识库内容 -- 读取网页内容 -- 搜索互联网 - -**对话示例**： - -``` -用户：ApeRAG 的图谱索引是怎么工作的？ - -Agent 思考：需要检索知识库 -↓ -调用工具：search_collection(query="图谱索引工作原理", collection_id="aperag-docs") -↓ -检索结果：返回关于图谱构建、实体提取、关系抽取的文档片段 -↓ -Agent 回答：ApeRAG 的图谱索引通过以下步骤工作...（基于检索到的内容生成） -``` - -## 5. 存储架构 - -ApeRAG 采用多存储架构，为不同类型的数据选择最合适的存储方案。 - -### 5.1 存储选型决策 - -```mermaid -flowchart TD - Data["🎯 数据类型分类"] --> Choice{"📊 什么数据?"} - - Choice --> |"📋 结构化数据
用户、配置等"| PG["PostgreSQL"] - Choice --> |"🔢 向量数据
embeddings"| Qdrant["Qdrant"] - Choice --> |"📝 文本数据
全文搜索"| ES["Elasticsearch"] - Choice --> |"📁 文件数据
原始文档"| MinIO["MinIO/S3"] - Choice --> |"🕸️ 图数据
知识图谱"| GraphChoice{"图规模?"} - Choice --> |"⚡ 缓存数据
临时数据"| Redis["Redis"] - - GraphChoice -->|"小规模
< 10万实体
💰 推荐"| PG2["PostgreSQL
内置图存储"] - GraphChoice -->|"大规模
> 100万实体"| Neo4j["Neo4j
专业图数据库"] - - PG --> PGUse["✅ 事务支持
✅ 关系查询
✅ 小规模图存储
✅ 成熟稳定"] - PG2 --> PG2Use["✅ 无需额外组件
✅ 降低运维成本
✅ 足够应对大多数场景"] - Qdrant --> QdrantUse["✅ 向量相似度搜索
✅ 高维数据检索
✅ 支持过滤条件"] - ES --> ESUse["✅ 全文检索 BM25
✅ 关键词搜索
✅ 中文分词 IK"] - MinIO --> MinIOUse["✅ 大文件存储
✅ S3 协议兼容
✅ 成本低"] - Neo4j --> Neo4jUse["✅ 大规模图查询
✅ 复杂关系遍历
✅ 图算法支持"] - Redis --> RedisUse["✅ Celery 任务队列
✅ LLM 调用缓存
✅ 毫秒级访问"] - - style Data fill:#e3f2fd,stroke:#1976d2,stroke-width:3px - style Choice fill:#fff59d,stroke:#fbc02d,stroke-width:3px - style GraphChoice fill:#fff59d,stroke:#fbc02d,stroke-width:2px - style PG fill:#bbdefb,stroke:#1976d2,stroke-width:2px - style PG2 fill:#c8e6c9,stroke:#388e3c,stroke-width:3px - style Qdrant fill:#c5e1a5,stroke:#689f38,stroke-width:2px - style ES fill:#ffccbc,stroke:#e64a19,stroke-width:2px - style MinIO fill:#e1bee7,stroke:#8e24aa,stroke-width:2px - style Neo4j fill:#f8bbd0,stroke:#c2185b,stroke-width:2px - style Redis fill:#ffecb3,stroke:#ffa000,stroke-width:2px -``` - -### 5.2 数据流向 - -不同的数据在系统中流转到不同的存储： - -```mermaid -flowchart LR - Doc[上传文档] --> Parser[解析器] - Parser --> |原始文件| MinIO[(MinIO)] - Parser --> |文档元数据| PG1[(PostgreSQL)] - Parser --> |文档块| Chunks[分块] - - Chunks --> |生成向量| Embed[Embedding] - Embed --> Qdrant[(Qdrant)] - - Chunks --> |文本内容| ES[(Elasticsearch)] - - Chunks --> |实体关系提取| Graph[图谱构建] - Graph --> |小规模| PG2[(PostgreSQL)] - Graph --> |大规模| Neo4j[(Neo4j)] - - PG1 -.元数据.-> Cache - Cache -.缓存.-> Redis[(Redis)] - - style Doc fill:#e1f5ff - style MinIO fill:#e1bee7 - style PG1 fill:#bbdefb - style PG2 fill:#bbdefb - style Qdrant fill:#c5e1a5 - style ES fill:#ffccbc - style Neo4j fill:#f8bbd0 - style Redis fill:#ffecb3 -``` - -### 5.3 核心存储系统 - -**PostgreSQL**（主数据库） - -存储内容： -- 用户信息、权限、配置 -- Collection（知识库）元数据 -- 文档元数据和索引状态 -- 对话历史 -- 小规模知识图谱（< 10 万实体） - -为什么选择： -- 强大的事务支持，保证数据一致性 -- 成熟稳定，运维成本低 -- pgvector 扩展，支持向量存储 -- 可以承载小规模图数据，不需要额外的图数据库 - -**Qdrant**（向量数据库） - -存储内容： -- 文档块的 embedding 向量 -- 实体和关系的向量表示 -- 图片的多模态向量 - -为什么选择： -- 专门为向量检索优化，速度快 -- 支持过滤条件，可以结合元数据筛选 -- 支持集群部署，可以水平扩展 - -**Elasticsearch**（全文搜索） - -存储内容： -- 文档块的文本内容 -- 支持中文分词（IK Analyzer） - -为什么选择： -- BM25 算法对关键词搜索效果好 -- 支持复杂的查询和聚合 -- 自带高亮显示 - -**MinIO**（对象存储） - -存储内容： -- 原始文档文件（PDF、Word 等） -- 解析后的中间结果 -- 上传的临时文件 - -为什么选择： -- S3 协议兼容，可以替换为云存储 -- 存储成本低 -- 支持大文件 - -**图数据库选择：PostgreSQL vs Neo4j** - -ApeRAG 支持两种图数据库方案： - -**PostgreSQL**（默认，推荐用于小规模） - -存储内容： -- 知识图谱（< 10 万实体） -- 图节点和边的关系数据 - -推荐理由： -- 无需额外部署，降低运维成本 -- 性能足够应对大多数场景 -- 事务支持完善，数据一致性有保障 -- 可以和其他业务数据共用一个数据库 - -**Neo4j**（可选，用于大规模） - -存储内容： -- 大规模知识图谱（> 100 万实体） - -什么时候需要： -- 实体数量超过 10 万，PostgreSQL 查询性能下降 -- 需要复杂的图遍历查询（多跳关系） -- 需要使用图算法（PageRank、社区发现等） - -**总结**：对于大多数企业应用，PostgreSQL 完全够用。只有在知识图谱规模非常大时，才需要考虑 Neo4j。 - -**Redis**（缓存和队列） - -存储内容： -- Celery 任务队列 -- LLM 调用缓存 -- 用户会话缓存 - -为什么选择： -- 速度极快，适合高频访问 -- 支持多种数据结构 -- 可以做任务队列的 Broker - -## 6. 技术亮点 - -### 6.1 无状态 LightRAG 重构 - -**背景问题**： - -原版 LightRAG 使用全局状态，所有任务共享一个实例。这在多用户、多 Collection 的场景下会导致数据混乱和并发冲突。 - -**ApeRAG 的解决方案**： - -- 每个任务创建独立的 LightRAG 实例 -- 通过 `workspace` 参数隔离不同 Collection 的数据 -- 实体命名规范：`entity:{name}:{workspace}` -- 关系命名规范：`relationship:{src}:{tgt}:{workspace}` - -这样，不同用户的图谱数据不会互相干扰，真正实现了多租户隔离。 - -### 6.2 双链异步架构 - -**传统做法的问题**： - -用户上传文档后，API 需要等待解析、索引构建全部完成才能返回，可能要等几分钟甚至更久。 - -**双链架构的优势**： - -- **前端链路**：API 只负责写状态到数据库，100ms 内返回 -- **后端链路**：Reconciler 定时检测状态变化，调度异步任务 -- **版本控制**：通过 version 和 observed_version 实现幂等性 -- **自动重试**：任务失败后自动重试，保证最终一致性 - -这个设计灵感来自 Kubernetes 的 Reconciler 模式，非常适合处理长时间运行的任务。 - -### 6.3 连通分量并发优化 - -**问题**： - -知识图谱构建时,需要合并相似的实体。如果串行处理,速度很慢。如果全部并行,又会有锁竞争问题。 - -**解决方案**： - -使用连通分量算法,把图谱分成多个独立的子图： - -1. 构建实体关系邻接表 -2. BFS 遍历找出所有连通分量 -3. 不同分量之间没有连接，可以完全并行处理 -4. 同一分量内部串行处理（避免冲突） - -**效果**： - -- 性能提升 2-3 倍 -- 零锁竞争 -- 对于多样化的文档集合效果最好 - -### 6.4 Provider 抽象模式 - -ApeRAG 支持 100+ 种 LLM 提供商（OpenAI、Claude、Gemini、国产大模型等）。如何统一管理？ - -**设计思路**： - -- 定义统一的 Provider 接口 -- 每个提供商实现自己的 Provider -- 通过 LiteLLM 库做适配 - -这样，切换模型只需要改配置，不需要改代码。同样的模式也应用在： -- Embedding Service（支持多种向量模型） -- Rerank Service（支持多种重排序模型） -- Web Search Service（DuckDuckGo、JINA 等） - -### 6.5 多模态索引支持 - -除了文本，ApeRAG 也能处理图片： - -**Vision Index 的两条路径**： - -1. **纯视觉向量**：使用多模态模型（如 CLIP）直接生成图片向量 -2. **视觉转文本**：使用 VLM 生成图片描述 + OCR 识别文字 → 文本向量化 - -**融合策略**： - -- 文本检索结果和视觉检索结果分开排序 -- 通过 Rerank 模型统一打分 -- 最终合并展示 - -## 7. 总结 - -ApeRAG 通过以下设计实现了生产级的 RAG 能力： - -**核心优势**： -- **强大的文档处理**：支持多格式、复杂布局、表格公式 -- **知识图谱融合**：不仅是向量匹配，还能理解知识关联 -- **多种检索方式**：向量、全文、图谱三管齐下 -- **异步架构**：快速响应，后台处理，用户体验好 -- **生产级设计**：多存储、高并发、易扩展 - -**技术创新**： -- 无状态 LightRAG，真正的多租户支持 -- 双链异步架构，API 响应 < 100ms -- 连通分量并发优化，图谱构建快 2-3 倍 -- Provider 抽象，支持 100+ LLM - -**适用场景**： -- 企业知识库搜索 -- 技术文档问答 -- 客服机器人 -- 研究论文分析 -- 任何需要理解文档并提供智能问答的场景 - -整个系统的设计理念是：**让复杂的事情变简单，让简单的事情变自动**。用户只需要上传文档，剩下的一切都由 ApeRAG 自动完成。 diff --git a/docs/zh-CN/design/chat_history_design.md b/docs/zh-CN/design/chat_history_design.md deleted file mode 100644 index 87f95d6b5..000000000 --- a/docs/zh-CN/design/chat_history_design.md +++ /dev/null @@ -1,584 +0,0 @@ -# ApeRAG 聊天历史消息数据流程 - -## 概述 - -本文档详细说明ApeRAG项目中聊天历史消息的完整数据流程，从前端API调用到后端存储的全链路实现。 - -**核心接口**: `GET /api/v1/bots/{bot_id}/chats/{chat_id}` - -## 数据流图 - -``` -┌─────────────────┐ -│ Frontend │ -│ (Next.js) │ -└────────┬────────┘ - │ GET /api/v1/bots/{bot_id}/chats/{chat_id} - ▼ -┌─────────────────────────────────────────────┐ -│ View Layer │ -│ aperag/views/chat.py │ -│ - get_chat_view() │ -│ - JWT身份验证 │ -│ - 参数验证 │ -└────────┬────────────────────────────────────┘ - │ chat_service_global.get_chat() - ▼ -┌─────────────────────────────────────────────┐ -│ Service Layer │ -│ aperag/service/chat_service.py │ -│ - get_chat() │ -│ - 业务逻辑编排 │ -└────────┬────────────────────────────────────┘ - │ - ├──────────────┬─────────────┐ - │ │ │ - ▼ ▼ ▼ -┌────────────┐ ┌───────────┐ ┌──────────────┐ -│ PostgreSQL │ │ Redis │ │ PostgreSQL │ -│ chat表 │ │ 消息历史 │ │ feedback表 │ -│(基本信息) │ │(会话内容) │ │(用户反馈) │ -└────────────┘ └───────────┘ └──────────────┘ - │ │ │ - └──────────────┴──────────────────┘ - │ - ▼ - ┌──────────────┐ - │ ChatDetails │ - │ (组装响应) │ - └──────────────┘ -``` - -## 完整流程详解 - -### 1. View层 - HTTP请求处理 - -**文件**: `aperag/views/chat.py` - -```python -@router.get("/bots/{bot_id}/chats/{chat_id}") -async def get_chat_view( - request: Request, - bot_id: str, - chat_id: str, - user: User = Depends(required_user) -) -> view_models.ChatDetails: - return await chat_service_global.get_chat(str(user.id), bot_id, chat_id) -``` - -**职责**: -- 接收HTTP GET请求 -- JWT Token身份验证 -- 提取路径参数 (bot_id, chat_id) -- 调用Service层 -- 返回`ChatDetails`响应 - -### 2. Service层 - 业务逻辑编排 - -**文件**: `aperag/service/chat_service.py` - -```python -async def get_chat(self, user: str, bot_id: str, chat_id: str) -> view_models.ChatDetails: - from aperag.utils.history import query_chat_messages - - # Step 1: 从PostgreSQL查询Chat基本信息 - chat = await self.db_ops.query_chat(user, bot_id, chat_id) - if chat is None: - raise ChatNotFoundException(chat_id) - - # Step 2: 从Redis查询聊天消息历史 - messages = await query_chat_messages(user, chat_id) - - # Step 3: 构建响应对象（消息中已包含feedback信息） - chat_obj = self.build_chat_response(chat) - return ChatDetails(**chat_obj.model_dump(), history=messages) -``` - -**核心逻辑**: - -1. **查询Chat元数据** (PostgreSQL) -2. **查询消息历史** (Redis + PostgreSQL反馈信息) -3. **组装完整响应** - -### 3. 数据存储层 - -#### 3.1 PostgreSQL - Chat基本信息 - -**表**: `chat` - -**文件**: `aperag/db/models.py` - -```python -class Chat(Base): - __tablename__ = "chat" - - id = Column(String(24), primary_key=True) # chat_xxxx - user = Column(String(256), nullable=False) # 用户ID - bot_id = Column(String(24), nullable=False) # Bot ID - title = Column(String(256)) # 会话标题 - peer_type = Column(EnumColumn(ChatPeerType)) # 对话类型 - peer_id = Column(String(256)) # 对话ID - status = Column(EnumColumn(ChatStatus)) # 状态 - gmt_created = Column(DateTime(timezone=True)) # 创建时间 - gmt_updated = Column(DateTime(timezone=True)) # 更新时间 - gmt_deleted = Column(DateTime(timezone=True)) # 删除时间（软删除） -``` - -**用途**: 存储Chat会话的元数据，不包含具体消息内容 - -#### 3.2 Redis - 聊天消息历史 - -**文件**: `aperag/utils/history.py` - -**Key格式**: `message_store:{chat_id}` - -**数据结构**: Redis List (使用LPUSH，最新消息在前) - -**核心类**: - -```python -class RedisChatMessageHistory: - def __init__(self, session_id: str, key_prefix: str = "message_store:"): - self.session_id = session_id - self.key_prefix = key_prefix - - @property - def key(self) -> str: - return self.key_prefix + self.session_id # message_store:chat_abc123 - - @property - async def messages(self) -> List[StoredChatMessage]: - # 从Redis读取所有消息 - _items = await self.redis_client.lrange(self.key, 0, -1) - # 反转为时间顺序（因为LPUSH导致最新在前） - items = [json.loads(m.decode("utf-8")) for m in _items[::-1]] - return [storage_dict_to_message(item) for item in items] -``` - -**消息查询函数**: - -```python -async def query_chat_messages(user: str, chat_id: str): - """查询聊天消息并转换为前端格式""" - - # 1. 从Redis获取消息历史 - chat_history = RedisChatMessageHistory(chat_id, redis_client=get_async_redis_client()) - stored_messages = await chat_history.messages - - if not stored_messages: - return [] - - # 2. 从PostgreSQL获取反馈信息 - feedbacks = await async_db_ops.query_chat_feedbacks(user, chat_id) - feedback_map = {feedback.message_id: feedback for feedback in feedbacks} - - # 3. 转换为前端格式并附加反馈信息 - result = [] - for stored_message in stored_messages: - # 转换为前端格式 - chat_message_list = stored_message.to_frontend_format() - - # 为AI消息添加反馈数据 - for chat_msg in chat_message_list: - feedback = feedback_map.get(chat_msg.id) - if feedback and chat_msg.role == "ai": - chat_msg.feedback = Feedback( - type=feedback.type, - tag=feedback.tag, - message=feedback.message - ) - - result.append(chat_message_list) - - return result # [[message1_parts], [message2_parts], [message3_parts], ...] -``` - -#### 3.3 PostgreSQL - 用户反馈信息 - -**表**: `message_feedback` - -```python -class MessageFeedback(Base): - __tablename__ = "message_feedback" - - user = Column(String(256), nullable=False) # 用户ID - chat_id = Column(String(24), primary_key=True) # 会话ID - message_id = Column(String(256), primary_key=True) # 消息ID - type = Column(EnumColumn(MessageFeedbackType)) # like/dislike - tag = Column(EnumColumn(MessageFeedbackTag)) # 反馈标签 - message = Column(Text) # 反馈内容 - question = Column(Text) # 原始问题 - original_answer = Column(Text) # 原始回答 - status = Column(EnumColumn(MessageFeedbackStatus)) # 状态 - gmt_created = Column(DateTime(timezone=True)) - gmt_updated = Column(DateTime(timezone=True)) -``` - -**用途**: 存储用户对AI回复的反馈（点赞/点踩），用于质量监控和模型优化 - -## 数据格式详解 - -### 存储格式 (Redis) - -消息在Redis中以JSON格式存储，采用**Part-Based设计**： - -#### StoredChatMessage - 一条完整消息 - -```python -class StoredChatMessage(BaseModel): - """一条完整消息（用户的一条消息 或 AI的一条消息）""" - parts: List[StoredChatMessagePart] # 消息的多个部分 - files: List[Dict[str, Any]] # 关联的上传文件 -``` - -#### StoredChatMessagePart - 消息的一个部分 - -```python -class StoredChatMessagePart(BaseModel): - """消息的单个部分（原子单元）""" - - # 标识信息 - chat_id: str # 所属会话 - message_id: str # 所属消息（同一条消息的多个part共享） - part_id: str # 部分的唯一ID - timestamp: float # 生成时间戳 - - # 内容分类 - type: Literal["message", "tool_call_result", "thinking", "references"] - role: Literal["human", "ai", "system"] - content: str - - # 扩展字段 - references: List[Dict] # 文档引用 - urls: List[str] # URL引用 - metadata: Optional[Dict] # 额外元数据 -``` - -#### Part类型说明 - -| Type | 说明 | 包含在LLM上下文 | -|------|------|---------------| -| `message` | 主要对话内容 | ✅ 是 | -| `tool_call_result` | 工具调用过程 | ❌ 否（仅展示） | -| `thinking` | AI思考过程 | ❌ 否（仅展示） | -| `references` | 文档引用和链接 | ❌ 否（仅展示） | - -**设计原因**: AI的一条回复包含多个阶段（工具调用、思考、回答、引用），这些内容按时序产生且互相穿插，单一字段无法表达。用户的消息通常只有1个part（type="message"），但也支持多个part以保持结构一致性。 - -#### Redis存储示例 - -**用户消息**: -```json -{ - "parts": [ - { - "chat_id": "chat_abc123", - "message_id": "uuid-1", - "part_id": "uuid-part-1", - "timestamp": 1699999999.0, - "type": "message", - "role": "human", - "content": "什么是LightRAG？", - "references": [], - "urls": [], - "metadata": null - } - ], - "files": [] -} -``` - -**AI回复（包含多个part）**: -```json -{ - "parts": [ - { - "message_id": "uuid-2", - "part_id": "uuid-part-2", - "type": "tool_call_result", - "role": "ai", - "content": "正在检索知识库...", - "timestamp": 1699999999.1 - }, - { - "message_id": "uuid-2", - "part_id": "uuid-part-3", - "type": "message", - "role": "ai", - "content": "LightRAG是一个轻量级的RAG框架，由ApeCloud团队深度改造...", - "timestamp": 1699999999.5 - }, - { - "message_id": "uuid-2", - "part_id": "uuid-part-4", - "type": "references", - "role": "ai", - "content": "", - "references": [ - { - "score": 0.95, - "text": "LightRAG架构说明...", - "metadata": {"source": "lightrag_doc.pdf", "page": 3} - } - ], - "urls": ["https://github.com/HKUDS/LightRAG"], - "timestamp": 1699999999.6 - } - ], - "files": [] -} -``` - -### API响应格式 - -**ChatDetails Schema**（FastAPI/Pydantic code-first schema）: - -```yaml -chatDetails: - type: object - properties: - id: string # chat_abc123 - title: string # 会话标题 - bot_id: string # bot_xyz - peer_id: string - peer_type: string # system/feishu/weixin/web - status: string # active/archived - created: string # ISO 8601 - updated: string # ISO 8601 - history: # 二维数组 - type: array - description: 对话历史，每个元素是一条消息 - items: - type: array - description: 一条消息包含多个parts（工具调用、思考、回答、引用等） - items: - $ref: '#/chatMessage' -``` - -**ChatMessage Schema**: - -```yaml -chatMessage: - type: object - properties: - id: string # message_id（同一轮次相同） - part_id: string # part_id（每个part唯一） - type: string # message/tool_call_result/thinking/references - timestamp: number # Unix时间戳 - role: string # human/ai - data: string # 消息内容 - references: # 文档引用（可选） - type: array - items: - type: object - properties: - score: number - text: string - metadata: object - urls: # URL引用（可选） - type: array - items: - type: string - feedback: # 用户反馈（可选） - type: object - properties: - type: string # like/dislike - tag: string - message: string - files: # 关联文件（可选） - type: array -``` - -### 前端接收示例 - -```json -{ - "id": "chat_abc123", - "title": "关于LightRAG的讨论", - "bot_id": "bot_xyz", - "status": "active", - "created": "2025-01-01T00:00:00Z", - "updated": "2025-01-01T01:00:00Z", - "history": [ - [ - { - "id": "uuid-1", - "part_id": "uuid-part-1", - "type": "message", - "timestamp": 1699999999.0, - "role": "human", - "data": "什么是LightRAG？", - "files": [] - } - ], - [ - { - "id": "uuid-2", - "part_id": "uuid-part-2", - "type": "tool_call_result", - "timestamp": 1699999999.1, - "role": "ai", - "data": "正在检索知识库...", - "files": [] - }, - { - "id": "uuid-2", - "part_id": "uuid-part-3", - "type": "message", - "timestamp": 1699999999.5, - "role": "ai", - "data": "LightRAG是一个轻量级的RAG框架...", - "files": [] - }, - { - "id": "uuid-2", - "part_id": "uuid-part-4", - "type": "references", - "timestamp": 1699999999.6, - "role": "ai", - "data": "", - "references": [ - { - "score": 0.95, - "text": "LightRAG架构说明...", - "metadata": {"source": "lightrag_doc.pdf"} - } - ], - "urls": ["https://github.com/HKUDS/LightRAG"], - "files": [] - } - ] - ] -} -``` - -**注意**: `history`是二维数组，第一维是消息序列（按时间顺序），第二维是该条消息的多个part。例如： -- `history[0]` = 用户的第1条消息的parts（通常只有1个part） -- `history[1]` = AI的第1条回复的parts（可能有多个part：工具调用、思考、回答、引用） -- `history[2]` = 用户的第2条消息的parts -- `history[3]` = AI的第2条回复的parts -- ... - -## 消息写入流程 - -### Agent Runtime 写入路径 - -旧的 WebSocket 聊天接口 `WS /api/v1/bots/{bot_id}/chats/{chat_id}/connect` 已经退休。 -当前 Agent 聊天写入走的是 v2 turn/timeline API 加 SSE 事件流。上面的 history schema 仍可作为背景说明， -但不应再依据这份文档实现新的 WebSocket chat client。 - -## 设计特点 - -### 1. 混合存储架构 - -| 存储 | 内容 | 原因 | -|------|------|------| -| PostgreSQL | Chat元数据 | 持久化、支持复杂查询 | -| Redis | 消息历史 | 高性能读写、支持TTL | -| PostgreSQL | 用户反馈 | 持久化、用于分析 | - -**优势**: -- 性能优化：消息历史使用Redis快速读写 -- 数据持久化：重要元数据存储在PostgreSQL -- 灵活性：可独立配置TTL、备份策略 - -### 2. Part-Based消息设计 - -**核心价值**: -- ✅ 支持复杂的AI回复流程（工具调用→思考→回答→引用） -- ✅ 前端可差异化渲染不同类型的内容 -- ✅ 完整记录时序关系（通过timestamp） -- ✅ 灵活扩展（新增type无需改表结构） - -**为什么一条消息需要多个part**: - -AI的一条回复过程是时序产生、互相穿插的，例如： -1. 🔍 Part1 (tool_call_result): "正在查询数据库..." -2. 💭 Part2 (thinking): "找到了327条记录..." -3. 🔍 Part3 (tool_call_result): "正在计算增长率..." -4. 💭 Part4 (thinking): "环比增长15%..." -5. 💬 Part5 (message): "根据数据分析，Q4表现优秀..." -6. 📚 Part6 (references): [文档1, 文档2] - -这6个part属于AI的**一条消息**（共享同一个message_id），单一字段无法表达这种复杂的时序关系。 - -### 3. 格式转换解耦 - -提供三种格式转换： - -```python -class StoredChatMessage: - def to_frontend_format(self) -> List[ChatMessage]: - """转换为前端展示格式""" - # 包含所有types的parts - - def to_openai_format(self) -> List[Dict]: - """转换为LLM调用格式""" - # 只包含type="message"的parts - - def get_main_content(self) -> str: - """获取主要回答内容""" - # 第一个type="message"的content -``` - -**优势**: -- 内部存储格式与外部接口解耦 -- 支持不同的消费场景 -- LLM上下文只包含实际对话内容，不包含工具调用和思考过程 - -### 4. 三级ID设计 - -```python -chat_id = "chat_abc123" # 会话级别 -message_id = "uuid-msg-1" # 消息级别（同一条消息的多个part共享） -part_id = "uuid-part-1" # 部分级别（每个part独立） -``` - -**作用**: -- `chat_id`: 标识一个聊天会话 -- `message_id`: 将同一条消息的多个part分组（用于前端展示和反馈关联） -- `part_id`: 每个part独立标识（用于单独操作，如复制、引用） - -## 性能考虑 - -### Redis优化 -- **List数据结构**: LPUSH O(1), LRANGE O(N) -- **可选TTL**: 自动过期历史消息 -- **连接池复用**: 全局Redis客户端 - -### PostgreSQL优化 -- **索引**: user, bot_id, chat_id, status字段 -- **软删除**: 使用gmt_deleted -- **分页查询**: list_chats支持分页 - -### 传输优化 -- **WebSocket流式**: 边生成边发送 -- **增量更新**: 只传输新的part -- **按需加载**: 懒加载历史消息 - -## 相关文件 - -### 核心实现 -- `aperag/views/chat.py` - View层接口 -- `aperag/service/chat_service.py` - Service层业务逻辑 -- `aperag/utils/history.py` - Redis消息历史管理 -- `aperag/chat/history/message.py` - 消息数据结构 -- `aperag/db/models.py` - 数据库模型 -- `aperag/db/repositories/chat.py` - Chat数据库操作 -- `aperag/schema/view_models.py` - Pydantic OpenAPI schema models - -### 前端实现 -- `web/src/app/workspace/bots/[botId]/chats/[chatId]/page.tsx` - 聊天详情页面 -- `web/src/components/chat/chat-messages.tsx` - 消息展示组件 - -## 总结 - -ApeRAG的聊天历史消息系统采用**混合存储 + Part-Based消息设计**： - -1. **PostgreSQL**存储Chat元数据和反馈（持久化、可查询） -2. **Redis**存储消息历史（高性能、支持过期） -3. **Part-Based设计**支持复杂的AI回复流程（工具调用、思考、回答、引用） -4. **三级ID设计**支持消息分组和独立操作 -5. **清晰的分层架构**（View → Service → Repository → Storage） - -这种设计既保证了性能，又支持复杂的AI交互场景，同时具有良好的可扩展性。 diff --git a/docs/zh-CN/design/collection_knowledge_export_design.md b/docs/zh-CN/design/collection_knowledge_export_design.md deleted file mode 100644 index 24b728da2..000000000 --- a/docs/zh-CN/design/collection_knowledge_export_design.md +++ /dev/null @@ -1,322 +0,0 @@ -# 知识库导出功能设计 - -**状态**: 已实现（MVP） - ---- - -## 1. 背景与目标 - -ApeRAG 文档处理管线会将用户上传的原始文件（PDF、Word 等）解析为结构化知识内容，并将所有产物存储在对象存储中。这些内容有独立的使用价值： - -- 将解析结果迁移到其他 RAG 框架（如 LlamaIndex、Dify） -- 审查文档解析质量，发现截断/格式错误 -- 离线分析分块策略效果 -- 数据备份与合规存档 - -本功能在知识库操作菜单中新增「**导出知识库**」按钮，允许知识库 Owner 将对象存储中该知识库目录的全部内容打包为 ZIP 文件下载。 - -**MVP 范围**： -- ✅ 触发导出（异步后台打包） -- ✅ 实时进度展示（前端轮询） -- ✅ 完成后一键下载 ZIP -- ❌ 后台运行（对话框保持打开直到完成或失败） -- ❌ 导出历史页面 - ---- - -## 2. 导出内容 - -对象存储中每个知识库的目录结构如下： - -``` -.objects/ -└── user-{user_id}/ - └── {collection_id}/ ← 导出此前缀下的全部内容 - ├── {document_id_1}/ - │ ├── original.pdf - │ ├── converted.pdf - │ ├── processed_content.md - │ ├── chunks/ - │ │ ├── chunk_0.json - │ │ └── chunk_1.json - │ └── images/ - │ ├── page_0.png - │ └── page_1.png - └── {document_id_2}/ - └── ... -``` - -**导出策略**：对 `user-{user_id}/{collection_id}/` 前缀下的所有对象做全量导出，无任何过滤。 - -生成的 ZIP 包结构： - -``` -{collection_title}_export_{YYYY-MM-DD}.zip -├── manifest.json ← 元数据（id → 标题映射） -├── {document_id_1}/ -│ └── ...（与对象存储目录结构一致） -└── {document_id_2}/ - └── ... -``` - -`manifest.json` 格式： - -```json -{ - "schema_version": "1.0", - "collection": { - "id": "colff4f33902752abee", - "title": "医学文献库", - "exported_at": "2026-03-04T10:00:00Z" - }, - "documents": [ - { "id": "doc_xyz789", "title": "高血压诊疗指南", "status": "COMPLETE" } - ] -} -``` - -> `manifest.json` 仅作信息记录，不影响哪些文件被导出。导出的 ZIP 保存在对象存储的 `exports/user-{user_id}/export_{task_id}.zip`，7 天后由定时任务清理。 - ---- - -## 3. 权限 - -| 角色 | 能否导出 | -|------|---------| -| Collection Owner | ✅ | -| 订阅用户（Marketplace） | ❌ | -| 未登录用户 | ❌ | - -- 「导出知识库」按钮**仅对 Owner 渲染**（在 `collection-header.tsx` 中用 `collection.user === currentUser.id` 判断） -- 后端通过查询 `Collection.user == user_id` 做权限校验，非 Owner 返回 `403` - ---- - -## 4. 系统架构 - -### 整体流程 - -``` -用户点击「导出知识库」 - │ - ▼ -前端弹出确认对话框 - │ 点击「开始导出」 - ▼ -POST /api/v1/collections/{id}/export - │ - ├─► 校验 Owner 权限（非 Owner → 403） - ├─► 检查并发任务数（同一用户 > 3 → 429） - ├─► 创建 ExportTask（status=PENDING） - └─► 触发 Celery 任务 export_collection_task.delay(task_id) - │ - ▼ 202 Accepted: { export_task_id, status: "PENDING" } - │ -前端切换为进度对话框，每 2 秒轮询 -GET /api/v1/export-tasks/{task_id} - │ - ▼ status=COMPLETED -前端显示「下载 ZIP」按钮 - │ - ▼ -GET /api/v1/export-tasks/{task_id}/download -（后端从对象存储读取 ZIP，StreamingResponse 流式返回） -``` - -### Celery Worker 处理流程（`config/export_tasks.py`） - -``` -1. 更新 ExportTask.status → PROCESSING -2. 列举对象存储 user-{user_id}/{collection_id}/ 下所有对象（list_objects_by_prefix） -3. 创建本地临时目录 /tmp/export_{task_id}/ -4. 并发下载所有文件（最多 5 个并发，ThreadPoolExecutor） - progress = downloaded / total * 85 -5. 从数据库查文档列表，生成 manifest.json → /tmp/export_{task_id}/ - progress → 90% -6. 打包 ZIP（ZIP_DEFLATED）→ /tmp/export_{task_id}.zip - progress → 95% -7. 上传 ZIP 到对象存储 exports/user-{user_id}/export_{task_id}.zip - progress → 98% -8. 清理本地临时文件 -9. 更新 ExportTask： - status=COMPLETED, progress=100, file_size, gmt_completed, gmt_expires=now()+7d - （失败时：status=FAILED, error_message=
Dify/Claude/Cursor] - - Frontend --> API[RESTful API] - External --> MCP[MCP Protocol] - - API --> DocProcess[Document Processing] - API --> Search[Search Service] - API --> Agent[Agent Dialogue] - MCP --> Search - MCP --> Agent - - DocProcess --> Tasks[Async Task Layer] - Tasks --> Storage[Storage Layer] - - Search --> Storage - Agent --> Search - - Storage --> PG[(PostgreSQL)] - Storage --> Qdrant[(Qdrant
Vector DB)] - Storage --> ES[(Elasticsearch
Full-text Search)] - Storage --> Neo4j[(Neo4j
Graph DB)] - Storage --> MinIO[(MinIO
Object Storage)] - - style User fill:#e1f5ff - style Frontend fill:#bbdefb - style External fill:#bbdefb - style API fill:#90caf9 - style MCP fill:#90caf9 - style DocProcess fill:#fff59d - style Search fill:#fff59d - style Agent fill:#fff59d - style Tasks fill:#c5e1a5 - style Storage fill:#ffccbc -``` - -## 2. Layered Architecture - -ApeRAG adopts a clear layered design, with each layer serving its specific purpose: - -```mermaid -graph TB - subgraph Layer1[Client Layer] - Web[Web Frontend
Next.js] - Dify[Dify] - Cursor[Cursor] - Claude[Claude Desktop] - end - - subgraph Layer2[Interface Layer] - API[RESTful API
FastAPI] - MCP[MCP Server
Model Context Protocol] - end - - subgraph Layer3[Service Layer] - CollSvc[Collection Service] - DocSvc[Document Service] - SearchSvc[Search Service] - GraphSvc[Graph Service] - AgentSvc[Agent Service] - end - - subgraph Layer4[Task Layer] - Celery[Celery Worker
Async Tasks] - MinerU[MinerU
Document Parser] - end - - subgraph Layer5[Storage Layer] - PG[(PostgreSQL)] - Qdrant[(Qdrant)] - ES[(Elasticsearch)] - Neo4j[(Neo4j)] - Redis[(Redis)] - MinIO[(MinIO)] - end - - Web --> API - Dify --> MCP - Cursor --> MCP - Claude --> MCP - - API --> CollSvc - API --> DocSvc - API --> SearchSvc - API --> GraphSvc - API --> AgentSvc - - MCP --> SearchSvc - MCP --> AgentSvc - - CollSvc --> Celery - DocSvc --> Celery - GraphSvc --> Celery - - Celery --> MinerU - Celery --> PG - Celery --> Qdrant - Celery --> ES - Celery --> Neo4j - Celery --> MinIO - - SearchSvc --> PG - SearchSvc --> Qdrant - SearchSvc --> ES - SearchSvc --> Neo4j - - style Layer1 fill:#e3f2fd - style Layer2 fill:#f3e5f5 - style Layer3 fill:#fff3e0 - style Layer4 fill:#e8f5e9 - style Layer5 fill:#fce4ec -``` - -**Layer Responsibilities**: - -- **Client Layer**: Multiple access methods - Web UI for management, MCP clients (Dify, Cursor, Claude, etc.) for integration -- **Interface Layer**: RESTful API (traditional HTTP interface) and MCP Server (AI tool protocol) provide services in parallel -- **Service Layer**: Core business logic, coordinating resources to complete specific functions -- **Task Layer**: Handles time-consuming operations (document parsing, index building) to ensure fast API responses -- **Storage Layer**: Multiple storage systems, selecting optimal solutions for different data types - -## 3. Document Processing Flow - -This is one of ApeRAG's core capabilities. From uploading a PDF file to making it searchable involves a series of carefully designed processing steps. - -### 3.1 Document Upload and Parsing - -When you upload a document, ApeRAG automatically identifies the format and selects the appropriate parser: - -```mermaid -flowchart TD - Upload[User Upload Document] --> Detect[Format Detection] - - Detect --> |PDF| MinerU[MinerU Parser] - Detect --> |Word/Excel| MarkItDown[MarkItDown Parser] - Detect --> |Markdown| DirectParse[Direct Parse] - Detect --> |Image| OCR[OCR Recognition] - - MinerU --> Extract[Content Extraction] - MarkItDown --> Extract - DirectParse --> Extract - OCR --> Extract - - Extract --> Parts[Document Parts
Parts Objects] - - style Upload fill:#e1f5ff - style Extract fill:#c5e1a5 - style Parts fill:#fff59d -``` - -**MinerU's Power**: - -- Accurately recognizes complex PDF table structures, preserving table content integrity -- Extracts LaTeX mathematical formulas, maintaining formula readability -- Performs OCR on scanned PDFs, supporting mixed Chinese-English text -- Identifies image regions in documents, supporting image content understanding - -### 3.2 Intelligent Chunking Strategy - -After document parsing, content needs to be split into appropriately sized chunks. This step is critical - chunks too large affect retrieval precision, too small lose context. - -```mermaid -flowchart TD - Parts[Document Parts] --> Rechunk[Smart Re-chunking] - - Rechunk --> Analysis[Analyze Document Structure] - Analysis --> Hierarchy[Identify Title Hierarchy] - Hierarchy --> Group[Group by Titles] - - Group --> Check{Check Chunk Size} - Check --> |Too Large| Split[Semantic Splitting] - Check --> |Appropriate| Chunks[Final Chunks] - Split --> Chunks - - Chunks --> AddContext[Add Context] - AddContext --> FinalChunks[Chunks with Context] - - style Rechunk fill:#bbdefb - style Split fill:#ffccbc - style FinalChunks fill:#c5e1a5 -``` - -**Chunking Strategy Features**: - -- **Maintain Semantic Integrity**: Avoid breaking sentences in the middle -- **Preserve Title Context**: Each chunk knows which section it belongs to -- **Hierarchical Splitting**: Split by paragraphs first, then sentences, finally characters -- **Smart Merging**: Adjacent small title chunks are merged to avoid information fragmentation - -Chunking Parameters: -- Default chunk size: 1200 tokens (approximately 800-1000 Chinese characters) -- Overlap size: 100 tokens (ensures context continuity) - -### 3.3 Parallel Multi-Index Building - -After chunking, multiple indexes are created simultaneously. Each index serves different purposes and complements the others: - -| Index Type | Use Case | Storage | Retrieval Method | -|-----------|----------|---------|------------------| -| **Vector Index** | Semantic similarity questions, e.g., "how to optimize performance" | Qdrant | Cosine Similarity | -| **Full-text Index** | Exact keyword search, e.g., "PostgreSQL configuration" | Elasticsearch | BM25 Algorithm | -| **Graph Index** | Relationship questions, e.g., "what's the connection between A and B" | PostgreSQL/Neo4j | Graph Traversal | -| **Summary Index** | Quick document overview | PostgreSQL | Vector Matching | -| **Vision Index** | Image content search | Qdrant | Multimodal Vector | - -```mermaid -flowchart LR - Chunks[Document Chunks] --> IndexMgr[Index Manager] - - IndexMgr --> VectorIdx[Vector Index Creation] - IndexMgr --> FulltextIdx[Full-text Index Creation] - IndexMgr --> GraphIdx[Graph Index Creation] - IndexMgr --> VisionIdx[Vision Index Creation] - - VectorIdx --> Qdrant1[(Qdrant)] - FulltextIdx --> ES[(Elasticsearch)] - GraphIdx --> Graph[(Neo4j/PG)] - VisionIdx --> Qdrant2[(Qdrant)] - - style IndexMgr fill:#fff59d - style VectorIdx fill:#bbdefb - style FulltextIdx fill:#c5e1a5 - style GraphIdx fill:#ffccbc - style VisionIdx fill:#e1bee7 -``` - -**Advantages of Parallel Building**: -- Different indexes can be built simultaneously, improving speed -- Failure of one index doesn't affect others -- Can enable specific index types on demand - -### 3.4 Knowledge Graph Construction - -Graph indexing is ApeRAG's signature feature, extracting structured knowledge from documents. - -```mermaid -flowchart TD - Chunks[Document Chunks] --> EntityExtract[Entity Extraction] - - EntityExtract --> LLM1[Call LLM
Identify Entities] - LLM1 --> Entities[Entity List
People, Places, Concepts] - - Entities --> RelationExtract[Relation Extraction] - RelationExtract --> LLM2[Call LLM
Identify Relations] - LLM2 --> Relations[Relation List
Who relates to whom and how] - - Entities --> Merge[Entity Merging] - Relations --> Merge - - Merge --> Components[Connected Components Analysis] - Components --> Parallel[Parallel Processing of Components] - Parallel --> Graph[(Knowledge Graph)] - - style EntityExtract fill:#bbdefb - style RelationExtract fill:#c5e1a5 - style Merge fill:#ffccbc - style Components fill:#fff59d -``` - -**Key Steps in Graph Construction**: - -1. **Entity Extraction**: LLM identifies meaningful entities from document chunks - - Example: From "Zhang San studies AI at Tsinghua University in Beijing" - - Entities: Zhang San (person), Beijing (location), Tsinghua University (organization), AI (concept) - -2. **Relation Extraction**: Identifies relationships between entities - - Example: Zhang San --studies--> AI, Zhang San --attends--> Tsinghua University - -3. **Entity Merging**: Same entity may have different expressions, needs normalization - - Example: "LightRAG", "light rag", "Light-RAG" → merged into unified entity - -4. **Connected Components Optimization**: Divides graph into independent subgraphs for parallel processing - - Performance improvement: 2-3x throughput - -**Why Connected Components Optimization?** - -Suppose you have 100 documents discussing different topics. Entities about "databases" and entities about "machine learning" have no connections and can be processed independently. The connected components algorithm identifies these independent "knowledge islands" and processes them in parallel, greatly improving speed. - -### 3.5 Async Task System - -Document processing is time-consuming, so ApeRAG uses a "dual-chain architecture" to ensure good user experience: - -```mermaid -graph TB - subgraph Frontend["🚀 Frontend Chain - Fast Response"] - direction TB - A1["📤 User Upload Document"] --> A2["🔌 API Receives Request"] - A2 --> A3["📋 Index Manager"] - A3 --> A4["💾 Write to Database
status = PENDING
version = 1"] - A4 --> A5["✅ Return Success Immediately
< 100ms"] - end - - subgraph Backend["⚙️ Backend Chain - Async Processing"] - direction TB - B1["⏰ Celery Beat
Check every 30s"] --> B2["🔍 Reconciler Detects
version ≠ observed_version"] - B2 --> B3{"🎯 Found Pending Tasks?"} - B3 -->|Yes| B4["🚀 Schedule Worker"] - B3 -->|No| B1 - B4 --> B5["📄 Parse Document"] - B5 --> B6["🔀 Parallel Index Creation
Vector + Fulltext
+ Graph + Vision"] - B6 --> B7["✨ Update Status
status = ACTIVE
observed_version = 1"] - B7 --> B1 - end - - A4 -.-|"Database State Change"| B2 - - style Frontend fill:#e3f2fd,stroke:#1976d2,stroke-width:3px - style Backend fill:#fff3e0,stroke:#f57c00,stroke-width:3px - style A5 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px - style B7 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px - style B3 fill:#fff9c4,stroke:#fbc02d,stroke-width:2px -``` - -**Benefits of Dual-Chain**: - -- **Fast Frontend Response**: API returns within 100ms after user uploads, no need to wait for processing -- **Async Backend Processing**: Real processing work happens in background without blocking user operations -- **Auto Retry**: System automatically retries if processing fails, ensuring eventual success -- **Status Tracking**: Users can check document processing progress anytime - -**Index State Machine**: - -```mermaid -stateDiagram-v2 - [*] --> PENDING: 📤 Document Upload - - PENDING --> CREATING: 🚀 Reconciler Detected
Start Processing - - CREATING --> ACTIVE: ✅ All Indexes Created Successfully - CREATING --> FAILED: ❌ Processing Failed - - FAILED --> CREATING: 🔄 Auto Retry
(max 3 times) - FAILED --> [*]: 💔 Exceeded Retry Limit
Mark as Failed - - ACTIVE --> CREATING: 🔄 Document Updated
Rebuild Index - ACTIVE --> [*]: 🗑️ Delete Document - - note right of PENDING - version = 1 - observed_version = 0 - end note - - note right of CREATING - Processing in progress - May take several minutes - end note - - note right of ACTIVE - version = 1 - observed_version = 1 - Ready for search - end note -``` - -## 4. Retrieval and Q&A Flow - -Once indexed, users can ask questions. ApeRAG's retrieval system intelligently selects appropriate retrieval strategies. - -### 4.1 Hybrid Retrieval System - -Different types of questions suit different retrieval methods. ApeRAG uses multiple retrieval strategies simultaneously and fuses results: - -```mermaid -flowchart TB - Query[User Query] --> Router[Retrieval Router] - - Router --> |Parallel| Vector[Vector Retrieval] - Router --> |Parallel| Fulltext[Full-text Retrieval] - Router --> |Parallel| Graph[Graph Retrieval] - - Vector --> Embed[Generate Query Vector] - Embed --> QdrantSearch[Qdrant Similarity Search] - QdrantSearch --> R1[Results 1] - - Fulltext --> ESSearch[Elasticsearch BM25] - ESSearch --> R2[Results 2] - - Graph --> GraphQuery[Graph Query
local/global/hybrid] - GraphQuery --> R3[Results 3] - - R1 --> Merge[Result Fusion] - R2 --> Merge - R3 --> Merge - - Merge --> Rerank[Rerank Re-scoring] - Rerank --> Final[Final Results] - - style Query fill:#e1f5ff - style Vector fill:#bbdefb - style Fulltext fill:#c5e1a5 - style Graph fill:#ffccbc - style Rerank fill:#fff59d - style Final fill:#c5e1a5 -``` - -**Retrieval Strategy Explanation**: - -- **Vector Retrieval**: For semantically similar questions - - Q: "How to improve system performance?" - - Finds: "Optimize database queries", "Use caching", etc. - -- **Full-text Retrieval**: For exact keyword matching - - Q: "Where is PostgreSQL configuration file?" - - Finds paragraphs containing exactly "PostgreSQL" and "configuration file" - -- **Graph Retrieval**: For relationship questions - - Q: "What's the relationship between LightRAG and Neo4j?" - - Queries connection paths between these two entities in the graph - -**Result Fusion Strategy**: - -Results from different retrieval methods need merging. ApeRAG uses a Rerank model to re-score all candidate results: - -1. Collect all retrieval results (may have duplicates) -2. Deduplicate, keep most relevant segments -3. Use Rerank model to evaluate relevance of each segment to the question -4. Re-sort by new scores -5. Return Top-K results - -### 4.2 Knowledge Graph Query - -Graph retrieval has three modes for different types of questions: - -| Mode | Use Case | Query Method | Example Question | -|------|----------|--------------|------------------| -| **local** | Query local info about an entity | Vector match similar entities → Get neighbor nodes | "Zhang San's personal info" | -| **global** | Query overall relationships and patterns | Vector match similar relations → Get connection paths | "What's the company's organizational structure" | -| **hybrid** | Comprehensive questions | local + global combined | "Zhang San's role and responsibilities in the company" | - -```mermaid -flowchart TD - Question[User Question] --> Analyze[Question Analysis] - - Analyze --> Local[Local Mode
Entity-centric] - Analyze --> Global[Global Mode
Relation-centric] - Analyze --> Hybrid[Hybrid Mode
Comprehensive Query] - - Local --> FindEntity[Find Related Entities] - FindEntity --> GetNeighbors[Get Neighbors and Relations] - - Global --> FindRelations[Find Related Relations] - FindRelations --> GetContext[Get Relation Context] - - Hybrid --> Local - Hybrid --> Global - - GetNeighbors --> Context[Generate Context] - GetContext --> Context - - Context --> Return[Return to LLM] - - style Local fill:#bbdefb - style Global fill:#c5e1a5 - style Hybrid fill:#fff59d -``` - -**Real Example**: - -Suppose the knowledge graph contains: -- Entities: Zhang San (person), Database Team (organization), PostgreSQL (technology) -- Relations: Zhang San --belongs to--> Database Team, Zhang San --excels at--> PostgreSQL - -Question: "What is Zhang San responsible for?" - -1. **Local Mode**: - - Finds "Zhang San" entity - - Gets all directly connected nodes - - Returns: "Zhang San belongs to Database Team, excels at PostgreSQL" - -2. **Global Mode**: - - Finds related relation patterns: "responsible for", "belongs to" - - Returns entire team structure and responsibility division - -3. **Hybrid Mode**: - - Uses both methods above - - Provides more comprehensive answer - -### 4.3 Agent Dialogue System - -Agent is ApeRAG's intelligent assistant that can invoke various tools to answer questions. - -```mermaid -sequenceDiagram - participant User as User - participant API as API Server - participant Agent as Agent Service - participant LLM as LLM Service - participant MCP as MCP Tools - participant Search as Search Service - - User->>API: Send Question - API->>Agent: Forward Question - - Agent->>LLM: Call LLM
with Tool List - LLM-->>Agent: Decide to call search_collection tool - - Agent->>MCP: Execute Tool Call - MCP->>Search: Hybrid Retrieval - Search-->>MCP: Return Relevant Document Segments - MCP-->>Agent: Tool Execution Result - - Agent->>LLM: Call LLM Again
with Retrieved Context - LLM-->>Agent: Generate Final Answer - - Agent-->>API: Stream Response - API-->>User: SSE Push Answer -``` - -**Agent Workflow**: - -1. **Receive Question**: User sends a question - -2. **Tool Decision**: LLM analyzes question and decides which tools to call - - Possible tools: search_collection (search knowledge base), web_search (search internet), web_read (read web page), etc. - -3. **Execute Tools**: Agent calls corresponding tools - - Example: search_collection triggers hybrid retrieval, returns relevant documents - -4. **Generate Answer**: LLM generates answer based on retrieved context - -5. **Stream Response**: Answer pushed to user in real-time via SSE (Server-Sent Events), no need to wait for complete generation - -**Role of MCP Protocol**: - -MCP (Model Context Protocol) is a standardized tool protocol that allows AI assistants (like Claude Desktop, Cursor) to easily invoke ApeRAG's capabilities. Through MCP, external AI tools can: -- List your knowledge bases -- Search knowledge base content -- Read web page content -- Search the internet - -**Dialogue Example**: - -``` -User: How does ApeRAG's graph indexing work? - -Agent thinks: Need to search knowledge base -↓ -Call tool: search_collection(query="graph indexing principles", collection_id="aperag-docs") -↓ -Retrieval results: Returns document segments about graph construction, entity extraction, relation extraction -↓ -Agent answers: ApeRAG's graph indexing works through the following steps... (generated based on retrieved content) -``` - -## 5. Storage Architecture - -ApeRAG adopts a multi-storage architecture, selecting the most appropriate storage solution for different data types. - -### 5.1 Storage Selection Decision - -```mermaid -flowchart TD - Data["🎯 Data Type Classification"] --> Choice{"📊 What Data?"} - - Choice --> |"📋 Structured Data
Users, Configs, etc."| PG["PostgreSQL"] - Choice --> |"🔢 Vector Data
embeddings"| Qdrant["Qdrant"] - Choice --> |"📝 Text Data
Full-text Search"| ES["Elasticsearch"] - Choice --> |"📁 File Data
Raw Documents"| MinIO["MinIO/S3"] - Choice --> |"🕸️ Graph Data
Knowledge Graph"| GraphChoice{"Graph Scale?"} - Choice --> |"⚡ Cache Data
Temporary Data"| Redis["Redis"] - - GraphChoice -->|"Small Scale
< 100K entities
💰 Recommended"| PG2["PostgreSQL
Built-in Graph Storage"] - GraphChoice -->|"Large Scale
> 1M entities"| Neo4j["Neo4j
Professional Graph DB"] - - PG --> PGUse["✅ Transaction Support
✅ Relational Queries
✅ Small-scale Graph Storage
✅ Mature & Stable"] - PG2 --> PG2Use["✅ No Extra Components
✅ Lower Ops Cost
✅ Sufficient for Most Cases"] - Qdrant --> QdrantUse["✅ Vector Similarity Search
✅ High-dimensional Data Retrieval
✅ Filter Support"] - ES --> ESUse["✅ Full-text Search BM25
✅ Keyword Search
✅ Chinese Tokenization IK"] - MinIO --> MinIOUse["✅ Large File Storage
✅ S3 Protocol Compatible
✅ Low Cost"] - Neo4j --> Neo4jUse["✅ Large-scale Graph Query
✅ Complex Relation Traversal
✅ Graph Algorithm Support"] - Redis --> RedisUse["✅ Celery Task Queue
✅ LLM Call Cache
✅ Millisecond Access"] - - style Data fill:#e3f2fd,stroke:#1976d2,stroke-width:3px - style Choice fill:#fff59d,stroke:#fbc02d,stroke-width:3px - style GraphChoice fill:#fff59d,stroke:#fbc02d,stroke-width:2px - style PG fill:#bbdefb,stroke:#1976d2,stroke-width:2px - style PG2 fill:#c8e6c9,stroke:#388e3c,stroke-width:3px - style Qdrant fill:#c5e1a5,stroke:#689f38,stroke-width:2px - style ES fill:#ffccbc,stroke:#e64a19,stroke-width:2px - style MinIO fill:#e1bee7,stroke:#8e24aa,stroke-width:2px - style Neo4j fill:#f8bbd0,stroke:#c2185b,stroke-width:2px - style Redis fill:#ffecb3,stroke:#ffa000,stroke-width:2px -``` - -### 5.2 Data Flow - -Different data flows to different storage systems: - -```mermaid -flowchart LR - Doc[Upload Document] --> Parser[Parser] - Parser --> |Raw Files| MinIO[(MinIO)] - Parser --> |Document Metadata| PG1[(PostgreSQL)] - Parser --> |Document Chunks| Chunks[Chunking] - - Chunks --> |Generate Vectors| Embed[Embedding] - Embed --> Qdrant[(Qdrant)] - - Chunks --> |Text Content| ES[(Elasticsearch)] - - Chunks --> |Extract Entity Relations| Graph[Graph Construction] - Graph --> |Small Scale| PG2[(PostgreSQL)] - Graph --> |Large Scale| Neo4j[(Neo4j)] - - PG1 -.Metadata.-> Cache - Cache -.Cache.-> Redis[(Redis)] - - style Doc fill:#e1f5ff - style MinIO fill:#e1bee7 - style PG1 fill:#bbdefb - style PG2 fill:#bbdefb - style Qdrant fill:#c5e1a5 - style ES fill:#ffccbc - style Neo4j fill:#f8bbd0 - style Redis fill:#ffecb3 -``` - -### 5.3 Core Storage Systems - -**PostgreSQL** (Primary Database) - -Storage Content: -- User info, permissions, configurations -- Collection (knowledge base) metadata -- Document metadata and index status -- Conversation history -- Small-scale knowledge graphs (< 100K entities) - -Why Choose: -- Strong transaction support, ensures data consistency -- Mature and stable, low operational cost -- pgvector extension, supports vector storage -- Can handle small-scale graph data without extra graph database - -**Qdrant** (Vector Database) - -Storage Content: -- Document chunk embedding vectors -- Entity and relation vector representations -- Multimodal vectors for images - -Why Choose: -- Optimized specifically for vector retrieval, fast -- Supports filter conditions, can combine with metadata filtering -- Supports cluster deployment, horizontally scalable - -**Elasticsearch** (Full-text Search) - -Storage Content: -- Document chunk text content -- Supports Chinese tokenization (IK Analyzer) - -Why Choose: -- BM25 algorithm works well for keyword search -- Supports complex queries and aggregations -- Built-in highlighting - -**MinIO** (Object Storage) - -Storage Content: -- Raw document files (PDF, Word, etc.) -- Intermediate results after parsing -- Temporary uploaded files - -Why Choose: -- S3 protocol compatible, can replace with cloud storage -- Low storage cost -- Supports large files - -**Graph Database Choice: PostgreSQL vs Neo4j** - -ApeRAG supports two graph database solutions: - -**PostgreSQL** (Default, Recommended for Small Scale) - -Storage Content: -- Knowledge graphs (< 100K entities) -- Graph node and edge relationship data - -Recommendation Reasons: -- No extra deployment, lower operational cost -- Performance sufficient for most scenarios -- Complete transaction support, data consistency guaranteed -- Can share database with other business data - -**Neo4j** (Optional, for Large Scale) - -Storage Content: -- Large-scale knowledge graphs (> 1M entities) - -When Needed: -- Entity count exceeds 100K, PostgreSQL query performance degrades -- Need complex graph traversal queries (multi-hop relations) -- Need to use graph algorithms (PageRank, community detection, etc.) - -**Summary**: For most enterprise applications, PostgreSQL is completely sufficient. Only consider Neo4j when knowledge graph scale is very large. - -**Redis** (Cache and Queue) - -Storage Content: -- Celery task queue -- LLM call cache -- User session cache - -Why Choose: -- Extremely fast, suitable for high-frequency access -- Supports multiple data structures -- Can serve as task queue Broker - -## 6. Technical Highlights - -### 6.1 Stateless LightRAG Refactoring - -**Background Problem**: - -Original LightRAG uses global state, all tasks share one instance. This causes data confusion and concurrency conflicts in multi-user, multi-Collection scenarios. - -**ApeRAG's Solution**: - -- Each task creates independent LightRAG instance -- Isolates different Collection data through `workspace` parameter -- Entity naming convention: `entity:{name}:{workspace}` -- Relation naming convention: `relationship:{src}:{tgt}:{workspace}` - -This way, different users' graph data won't interfere with each other, truly achieving multi-tenant isolation. - -### 6.2 Dual-Chain Async Architecture - -**Traditional Approach Problem**: - -After user uploads document, API needs to wait for parsing and index building to complete before returning, possibly taking several minutes or longer. - -**Dual-Chain Architecture Advantages**: - -- **Frontend Chain**: API only writes state to database, returns within 100ms -- **Backend Chain**: Reconciler periodically detects state changes, schedules async tasks -- **Version Control**: Implements idempotency through version and observed_version -- **Auto Retry**: Automatically retries failed tasks, ensures eventual consistency - -This design is inspired by Kubernetes' Reconciler pattern, very suitable for handling long-running tasks. - -### 6.3 Connected Components Concurrency Optimization - -**Problem**: - -During knowledge graph construction, similar entities need merging. Serial processing is slow. Full parallelization has lock contention issues. - -**Solution**: - -Use connected components algorithm to divide graph into multiple independent subgraphs: - -1. Build entity-relation adjacency list -2. BFS traversal to find all connected components -3. Different components have no connections, can be fully processed in parallel -4. Same component processed serially internally (avoid conflicts) - -**Results**: - -- 2-3x performance improvement -- Zero lock contention -- Best results for diverse document collections - -### 6.4 Provider Abstraction Pattern - -ApeRAG supports 100+ LLM providers (OpenAI, Claude, Gemini, domestic LLMs, etc.). How to manage uniformly? - -**Design Approach**: - -- Define unified Provider interface -- Each provider implements its own Provider -- Adapt through LiteLLM library - -This way, switching models only requires config change, no code change. Same pattern also applies to: -- Embedding Service (supports multiple vector models) -- Rerank Service (supports multiple reranking models) -- Web Search Service (DuckDuckGo, JINA, etc.) - -### 6.5 Multimodal Index Support - -Besides text, ApeRAG can also handle images: - -**Vision Index's Two Paths**: - -1. **Pure Visual Vectors**: Use multimodal models (like CLIP) to directly generate image vectors -2. **Vision to Text**: Use VLM to generate image descriptions + OCR to recognize text → text vectorization - -**Fusion Strategy**: - -- Text and visual retrieval results sorted separately -- Unified scoring through Rerank model -- Final merged display - -## 7. Summary - -ApeRAG achieves production-grade RAG capabilities through the following design: - -**Core Advantages**: -- **Powerful Document Processing**: Supports multiple formats, complex layouts, tables and formulas -- **Knowledge Graph Fusion**: Not just vector matching, but understanding knowledge relationships -- **Multiple Retrieval Methods**: Vector, full-text, and graph working together -- **Async Architecture**: Fast response, background processing, good user experience -- **Production-Grade Design**: Multi-storage, high concurrency, easy to scale - -**Technical Innovations**: -- Stateless LightRAG, true multi-tenant support -- Dual-chain async architecture, API response < 100ms -- Connected components concurrency optimization, 2-3x faster graph construction -- Provider abstraction, supports 100+ LLMs - -**Use Cases**: -- Enterprise knowledge base search -- Technical documentation Q&A -- Customer service bots -- Research paper analysis -- Any scenario requiring document understanding and intelligent Q&A - -The system's design philosophy is: **Make complex things simple, make simple things automatic**. Users just need to upload documents, everything else is handled automatically by ApeRAG. diff --git a/web/docs/en-US/design/chat_history_design.md b/web/docs/en-US/design/chat_history_design.md deleted file mode 100644 index f2d729f1d..000000000 --- a/web/docs/en-US/design/chat_history_design.md +++ /dev/null @@ -1,590 +0,0 @@ ---- -title: Chat History Message Data Flow -description: Complete data flow of chat history messages in ApeRAG, from frontend API calls to backend storage -keywords: [chat, history, message, redis, postgresql, websocket, part-based design] ---- - -# ApeRAG Chat History Message Data Flow - -## Overview - -This document details the complete data flow of chat history messages in the ApeRAG project, covering the full-stack implementation from frontend API calls to backend storage. - -**Core API**: `GET /api/v1/bots/{bot_id}/chats/{chat_id}` - -## Data Flow Diagram - -``` -┌─────────────────┐ -│ Frontend │ -│ (Next.js) │ -└────────┬────────┘ - │ GET /api/v1/bots/{bot_id}/chats/{chat_id} - ▼ -┌─────────────────────────────────────────────┐ -│ View Layer │ -│ aperag/views/chat.py │ -│ - get_chat_view() │ -│ - JWT Authentication │ -│ - Parameter Validation │ -└────────┬────────────────────────────────────┘ - │ chat_service_global.get_chat() - ▼ -┌─────────────────────────────────────────────┐ -│ Service Layer │ -│ aperag/service/chat_service.py │ -│ - get_chat() │ -│ - Business Logic Orchestration │ -└────────┬────────────────────────────────────┘ - │ - ├──────────────┬─────────────┐ - │ │ │ - ▼ ▼ ▼ -┌────────────┐ ┌───────────┐ ┌──────────────┐ -│ PostgreSQL │ │ Redis │ │ PostgreSQL │ -│ chat table │ │ Message │ │feedback table│ -│ (Metadata) │ │ History │ │(User Feedback)│ -└────────────┘ └───────────┘ └──────────────┘ - │ │ │ - └──────────────┴──────────────────┘ - │ - ▼ - ┌──────────────┐ - │ ChatDetails │ - │ (Assemble) │ - └──────────────┘ -``` - -## Detailed Flow - -### 1. View Layer - HTTP Request Handling - -**File**: `aperag/views/chat.py` - -```python -@router.get("/bots/{bot_id}/chats/{chat_id}") -async def get_chat_view( - request: Request, - bot_id: str, - chat_id: str, - user: User = Depends(required_user) -) -> view_models.ChatDetails: - return await chat_service_global.get_chat(str(user.id), bot_id, chat_id) -``` - -**Responsibilities**: -- Receive HTTP GET requests -- JWT Token authentication -- Extract path parameters (bot_id, chat_id) -- Call Service layer -- Return `ChatDetails` response - -### 2. Service Layer - Business Logic Orchestration - -**File**: `aperag/service/chat_service.py` - -```python -async def get_chat(self, user: str, bot_id: str, chat_id: str) -> view_models.ChatDetails: - from aperag.utils.history import query_chat_messages - - # Step 1: Query Chat metadata from PostgreSQL - chat = await self.db_ops.query_chat(user, bot_id, chat_id) - if chat is None: - raise ChatNotFoundException(chat_id) - - # Step 2: Query message history from Redis - messages = await query_chat_messages(user, chat_id) - - # Step 3: Build response object (messages already include feedback info) - chat_obj = self.build_chat_response(chat) - return ChatDetails(**chat_obj.model_dump(), history=messages) -``` - -**Core Logic**: - -1. **Query Chat Metadata** (PostgreSQL) -2. **Query Message History** (Redis + PostgreSQL feedback) -3. **Assemble Complete Response** - -### 3. Data Storage Layer - -#### 3.1 PostgreSQL - Chat Metadata - -**Table**: `chat` - -**File**: `aperag/db/models.py` - -```python -class Chat(Base): - __tablename__ = "chat" - - id = Column(String(24), primary_key=True) # chat_xxxx - user = Column(String(256), nullable=False) # User ID - bot_id = Column(String(24), nullable=False) # Bot ID - title = Column(String(256)) # Chat title - peer_type = Column(EnumColumn(ChatPeerType)) # Peer type - peer_id = Column(String(256)) # Peer ID - status = Column(EnumColumn(ChatStatus)) # Status - gmt_created = Column(DateTime(timezone=True)) # Created time - gmt_updated = Column(DateTime(timezone=True)) # Updated time - gmt_deleted = Column(DateTime(timezone=True)) # Deleted time (soft delete) -``` - -**Purpose**: Store Chat session metadata without actual message content - -#### 3.2 Redis - Message History - -**File**: `aperag/utils/history.py` - -**Key Format**: `message_store:{chat_id}` - -**Data Structure**: Redis List (using LPUSH, newest messages first) - -**Core Class**: - -```python -class RedisChatMessageHistory: - def __init__(self, session_id: str, key_prefix: str = "message_store:"): - self.session_id = session_id - self.key_prefix = key_prefix - - @property - def key(self) -> str: - return self.key_prefix + self.session_id # message_store:chat_abc123 - - @property - async def messages(self) -> List[StoredChatMessage]: - # Read all messages from Redis - _items = await self.redis_client.lrange(self.key, 0, -1) - # Reverse to chronological order (LPUSH puts newest first) - items = [json.loads(m.decode("utf-8")) for m in _items[::-1]] - return [storage_dict_to_message(item) for item in items] -``` - -**Message Query Function**: - -```python -async def query_chat_messages(user: str, chat_id: str): - """Query chat messages and convert to frontend format""" - - # 1. Get message history from Redis - chat_history = RedisChatMessageHistory(chat_id, redis_client=get_async_redis_client()) - stored_messages = await chat_history.messages - - if not stored_messages: - return [] - - # 2. Get feedback info from PostgreSQL - feedbacks = await async_db_ops.query_chat_feedbacks(user, chat_id) - feedback_map = {feedback.message_id: feedback for feedback in feedbacks} - - # 3. Convert to frontend format and attach feedback info - result = [] - for stored_message in stored_messages: - # Convert to frontend format - chat_message_list = stored_message.to_frontend_format() - - # Add feedback data for AI messages - for chat_msg in chat_message_list: - feedback = feedback_map.get(chat_msg.id) - if feedback and chat_msg.role == "ai": - chat_msg.feedback = Feedback( - type=feedback.type, - tag=feedback.tag, - message=feedback.message - ) - - result.append(chat_message_list) - - return result # [[message1_parts], [message2_parts], [message3_parts], ...] -``` - -#### 3.3 PostgreSQL - User Feedback - -**Table**: `message_feedback` - -```python -class MessageFeedback(Base): - __tablename__ = "message_feedback" - - user = Column(String(256), nullable=False) # User ID - chat_id = Column(String(24), primary_key=True) # Chat ID - message_id = Column(String(256), primary_key=True) # Message ID - type = Column(EnumColumn(MessageFeedbackType)) # like/dislike - tag = Column(EnumColumn(MessageFeedbackTag)) # Feedback tag - message = Column(Text) # Feedback content - question = Column(Text) # Original question - original_answer = Column(Text) # Original answer - status = Column(EnumColumn(MessageFeedbackStatus)) # Status - gmt_created = Column(DateTime(timezone=True)) - gmt_updated = Column(DateTime(timezone=True)) -``` - -**Purpose**: Store user feedback on AI responses (like/dislike) for quality monitoring and model optimization - -## Data Format Specification - -### Storage Format (Redis) - -Messages in Redis are stored in JSON format using **Part-Based Design**: - -#### StoredChatMessage - A Complete Message - -```python -class StoredChatMessage(BaseModel): - """A complete message (either a user message or an AI message)""" - parts: List[StoredChatMessagePart] # Multiple parts of the message - files: List[Dict[str, Any]] # Associated uploaded files -``` - -#### StoredChatMessagePart - A Message Part - -```python -class StoredChatMessagePart(BaseModel): - """A single part of a message (atomic unit)""" - - # Identification - chat_id: str # Chat session ID - message_id: str # Message ID (shared by multiple parts of the same message) - part_id: str # Unique part ID - timestamp: float # Generation timestamp - - # Content Classification - type: Literal["message", "tool_call_result", "thinking", "references"] - role: Literal["human", "ai", "system"] - content: str - - # Extended Fields - references: List[Dict] # Document references - urls: List[str] # URL references - metadata: Optional[Dict] # Additional metadata -``` - -#### Part Type Descriptions - -| Type | Description | Included in LLM Context | -|------|-------------|------------------------| -| `message` | Main conversation content | ✅ Yes | -| `tool_call_result` | Tool calling process | ❌ No (display only) | -| `thinking` | AI thinking process | ❌ No (display only) | -| `references` | Document references and links | ❌ No (display only) | - -**Design Rationale**: A single AI response contains multiple stages (tool calling, thinking, answering, references), and these contents are generated sequentially and interleaved. A single field cannot express this. User messages typically have only 1 part (type="message"), but also support multiple parts to maintain structural consistency. - -#### Redis Storage Example - -**User Message**: -```json -{ - "parts": [ - { - "chat_id": "chat_abc123", - "message_id": "uuid-1", - "part_id": "uuid-part-1", - "timestamp": 1699999999.0, - "type": "message", - "role": "human", - "content": "What is LightRAG?", - "references": [], - "urls": [], - "metadata": null - } - ], - "files": [] -} -``` - -**AI Response (with multiple parts)**: -```json -{ - "parts": [ - { - "message_id": "uuid-2", - "part_id": "uuid-part-2", - "type": "tool_call_result", - "role": "ai", - "content": "Searching knowledge base...", - "timestamp": 1699999999.1 - }, - { - "message_id": "uuid-2", - "part_id": "uuid-part-3", - "type": "message", - "role": "ai", - "content": "LightRAG is a lightweight RAG framework, deeply modified by the ApeCloud team...", - "timestamp": 1699999999.5 - }, - { - "message_id": "uuid-2", - "part_id": "uuid-part-4", - "type": "references", - "role": "ai", - "content": "", - "references": [ - { - "score": 0.95, - "text": "LightRAG architecture description...", - "metadata": {"source": "lightrag_doc.pdf", "page": 3} - } - ], - "urls": ["https://github.com/HKUDS/LightRAG"], - "timestamp": 1699999999.6 - } - ], - "files": [] -} -``` - -### API Response Format - -**ChatDetails Schema** (`aperag/api/components/schemas/chat.yaml`): - -```yaml -chatDetails: - type: object - properties: - id: string # chat_abc123 - title: string # Chat title - bot_id: string # bot_xyz - peer_id: string - peer_type: string # system/feishu/weixin/web - status: string # active/archived - created: string # ISO 8601 - updated: string # ISO 8601 - history: # 2D array - type: array - description: Conversation history, each element is a message - items: - type: array - description: A message contains multiple parts (tool calls, thinking, answer, references, etc.) - items: - $ref: '#/chatMessage' -``` - -**ChatMessage Schema**: - -```yaml -chatMessage: - type: object - properties: - id: string # message_id (same for one turn) - part_id: string # part_id (unique for each part) - type: string # message/tool_call_result/thinking/references - timestamp: number # Unix timestamp - role: string # human/ai - data: string # Message content - references: # Document references (optional) - type: array - items: - type: object - properties: - score: number - text: string - metadata: object - urls: # URL references (optional) - type: array - items: - type: string - feedback: # User feedback (optional) - type: object - properties: - type: string # like/dislike - tag: string - message: string - files: # Associated files (optional) - type: array -``` - -### Frontend Response Example - -```json -{ - "id": "chat_abc123", - "title": "Discussion about LightRAG", - "bot_id": "bot_xyz", - "status": "active", - "created": "2025-01-01T00:00:00Z", - "updated": "2025-01-01T01:00:00Z", - "history": [ - [ - { - "id": "uuid-1", - "part_id": "uuid-part-1", - "type": "message", - "timestamp": 1699999999.0, - "role": "human", - "data": "What is LightRAG?", - "files": [] - } - ], - [ - { - "id": "uuid-2", - "part_id": "uuid-part-2", - "type": "tool_call_result", - "timestamp": 1699999999.1, - "role": "ai", - "data": "Searching knowledge base...", - "files": [] - }, - { - "id": "uuid-2", - "part_id": "uuid-part-3", - "type": "message", - "timestamp": 1699999999.5, - "role": "ai", - "data": "LightRAG is a lightweight RAG framework...", - "files": [] - }, - { - "id": "uuid-2", - "part_id": "uuid-part-4", - "type": "references", - "timestamp": 1699999999.6, - "role": "ai", - "data": "", - "references": [ - { - "score": 0.95, - "text": "LightRAG architecture description...", - "metadata": {"source": "lightrag_doc.pdf"} - } - ], - "urls": ["https://github.com/HKUDS/LightRAG"], - "files": [] - } - ] - ] -} -``` - -**Note**: `history` is a 2D array. The first dimension is the message sequence (in chronological order), and the second dimension is the multiple parts of that message. For example: -- `history[0]` = Parts of user's 1st message (usually only 1 part) -- `history[1]` = Parts of AI's 1st response (may have multiple parts: tool calls, thinking, answer, references) -- `history[2]` = Parts of user's 2nd message -- `history[3]` = Parts of AI's 2nd response -- ... - -## Message Write Flow - -### Agent Runtime Write Path - -The legacy WebSocket chat endpoint `WS /api/v1/bots/{bot_id}/chats/{chat_id}/connect` has been retired. -Current agent chat writes now go through the v2 turn/timeline APIs and SSE event stream. Keep the history -schema above as background, but do not use this document to implement new WebSocket chat clients. - -## Design Features - -### 1. Hybrid Storage Architecture - -| Storage | Content | Reason | -|---------|---------|--------| -| PostgreSQL | Chat metadata | Persistence, complex queries | -| Redis | Message history | High-performance read/write, TTL support | -| PostgreSQL | User feedback | Persistence, for analysis | - -**Advantages**: -- Performance optimization: Message history uses Redis for fast read/write -- Data persistence: Important metadata stored in PostgreSQL -- Flexibility: Independent TTL and backup strategy configuration - -### 2. Part-Based Message Design - -**Core Value**: -- ✅ Support complex AI response flow (tool calling → thinking → answer → references) -- ✅ Frontend can render different types of content differently -- ✅ Complete temporal relationship recording (via timestamp) -- ✅ Flexible extension (adding new types doesn't require schema changes) - -**Why does a single message need multiple parts?** - -A single AI response is generated sequentially and interleaved, for example: -1. 🔍 Part1 (tool_call_result): "Querying database..." -2. 💭 Part2 (thinking): "Found 327 records..." -3. 🔍 Part3 (tool_call_result): "Calculating growth rate..." -4. 💭 Part4 (thinking): "15% QoQ growth..." -5. 💬 Part5 (message): "Based on data analysis, Q4 performance is excellent..." -6. 📚 Part6 (references): [Document 1, Document 2] - -These 6 parts belong to **one AI message** (sharing the same message_id). A single field cannot express such complex temporal relationships. - -### 3. Format Conversion Decoupling - -Three format conversions are provided: - -```python -class StoredChatMessage: - def to_frontend_format(self) -> List[ChatMessage]: - """Convert to frontend display format""" - # Include all types of parts - - def to_openai_format(self) -> List[Dict]: - """Convert to LLM call format""" - # Only include type="message" parts - - def get_main_content(self) -> str: - """Get main answer content""" - # Content of the first type="message" part -``` - -**Advantages**: -- Internal storage format decoupled from external interfaces -- Support different consumption scenarios -- LLM context only includes actual conversation content, not tool calls and thinking processes - -### 4. Three-Level ID Design - -```python -chat_id = "chat_abc123" # Session level -message_id = "uuid-msg-1" # Message level (shared by parts of the same message) -part_id = "uuid-part-1" # Part level (each part is independent) -``` - -**Purpose**: -- `chat_id`: Identifies a chat session -- `message_id`: Groups parts of the same message (for frontend display and feedback association) -- `part_id`: Uniquely identifies each part (for individual operations like copy, reference) - -## Performance Considerations - -### Redis Optimization -- **List Data Structure**: LPUSH O(1), LRANGE O(N) -- **Optional TTL**: Automatic expiration of historical messages -- **Connection Pool Reuse**: Global Redis client - -### PostgreSQL Optimization -- **Indexes**: user, bot_id, chat_id, status fields -- **Soft Delete**: Using gmt_deleted -- **Paginated Queries**: list_chats supports pagination - -### Transmission Optimization -- **WebSocket Streaming**: Generate and send simultaneously -- **Incremental Updates**: Only transmit new parts -- **Lazy Loading**: Load historical messages on demand - -## Related Files - -### Core Implementation -- `aperag/views/chat.py` - View layer interface -- `aperag/service/chat_service.py` - Service layer business logic -- `aperag/utils/history.py` - Redis message history management -- `aperag/chat/history/message.py` - Message data structures -- `aperag/db/models.py` - Database models -- `aperag/db/repositories/chat.py` - Chat database operations -- `aperag/api/components/schemas/chat.yaml` - OpenAPI Schema - -### Frontend Implementation -- `web/src/app/workspace/bots/[botId]/chats/[chatId]/page.tsx` - Chat detail page -- `web/src/components/chat/chat-messages.tsx` - Message display component - -## Summary - -ApeRAG's chat history message system adopts **Hybrid Storage + Part-Based Message Design**: - -1. **PostgreSQL** stores Chat metadata and feedback (persistence, queryable) -2. **Redis** stores message history (high performance, expiration support) -3. **Part-Based Design** supports complex AI response flows (tool calling, thinking, answering, references) -4. **Three-Level ID Design** supports message grouping and independent operations -5. **Clear Layered Architecture** (View → Service → Repository → Storage) - -This design ensures both performance and support for complex AI interaction scenarios, while maintaining good scalability. diff --git a/web/docs/en-US/design/document_upload_design.md b/web/docs/en-US/design/document_upload_design.md deleted file mode 100644 index b31d76ba7..000000000 --- a/web/docs/en-US/design/document_upload_design.md +++ /dev/null @@ -1,1076 +0,0 @@ ---- -title: Document Upload Design ---- - -# ApeRAG Document Upload Architecture Design - -## Overview - -This document details the complete architecture design of the document upload module in the ApeRAG project, covering the full pipeline from file upload, temporary storage, document parsing, format conversion to final index construction. - -**Core Design Philosophy**: Adopts a **two-phase commit** pattern, separating file upload (temporary storage) from document confirmation (formal addition), providing better user experience and resource management capabilities. - -## System Architecture - -### Overall Architecture - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Frontend │ -│ (Next.js) │ -└────────┬───────────────────────────────────┬────────────────┘ - │ │ - │ Step 1: Upload │ Step 2: Confirm - │ POST /documents/upload │ POST /documents/confirm - ▼ ▼ -┌─────────────────────────────────────────────────────────────┐ -│ View Layer: aperag/views/collections.py │ -│ - HTTP request handling │ -│ - JWT authentication │ -│ - Parameter validation │ -└────────┬───────────────────────────────────┬────────────────┘ - │ │ - │ document_service.upload_document() │ document_service.confirm_documents() - ▼ ▼ -┌─────────────────────────────────────────────────────────────┐ -│ Service Layer: aperag/service/document_service.py │ -│ - Business logic orchestration │ -│ - File validation (type, size) │ -│ - SHA-256 hash deduplication │ -│ - Quota checking │ -│ - Transaction management │ -└────────┬───────────────────────────────────┬────────────────┘ - │ │ - │ Step 1 │ Step 2 - ▼ ▼ -┌────────────────────────┐ ┌────────────────────────────┐ -│ 1. Create Document │ │ 1. Update Document status │ -│ status=UPLOADED │ │ UPLOADED → PENDING │ -│ 2. Save to ObjectStore│ │ 2. Create DocumentIndex │ -│ 3. Calculate hash │ │ 3. Trigger indexing tasks │ -└────────┬───────────────┘ └────────┬───────────────────┘ - │ │ - ▼ ▼ -┌─────────────────────────────────────────────────────────────┐ -│ Storage Layer │ -│ │ -│ ┌───────────────┐ ┌──────────────────┐ ┌─────────────┐ │ -│ │ PostgreSQL │ │ Object Store │ │ Vector DB │ │ -│ │ │ │ │ │ │ │ -│ │ - document │ │ - Local/S3 │ │ - Qdrant │ │ -│ │ - document_ │ │ - Original files │ │ - Vectors │ │ -│ │ index │ │ - Converted files│ │ │ │ -│ └───────────────┘ └──────────────────┘ └─────────────┘ │ -│ │ -│ ┌───────────────┐ ┌──────────────────┐ │ -│ │ Elasticsearch │ │ Neo4j/PG │ │ -│ │ │ │ │ │ -│ │ - Full-text │ │ - Knowledge Graph│ │ -│ └───────────────┘ └──────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌───────────────────┐ - │ Celery Workers │ - │ │ - │ - Doc parsing │ - │ - Format convert │ - │ - Content extract│ - │ - Doc chunking │ - │ - Index building │ - └───────────────────┘ -``` - -### Layered Architecture - -``` -┌─────────────────────────────────────────────┐ -│ View Layer (views/collections.py) │ HTTP handling, auth, validation -└─────────────────┬───────────────────────────┘ - │ calls -┌─────────────────▼───────────────────────────┐ -│ Service Layer (service/document_service.py)│ Business logic, transaction, permission -└─────────────────┬───────────────────────────┘ - │ calls -┌─────────────────▼───────────────────────────┐ -│ Repository Layer (db/ops.py, objectstore/) │ Data access abstraction -└─────────────────┬───────────────────────────┘ - │ accesses -┌─────────────────▼───────────────────────────┐ -│ Storage Layer (PG, S3, Qdrant, ES, Neo4j) │ Data persistence -└─────────────────────────────────────────────┘ -``` - -## Core Process Details - -### Phase 0: API Interface Definition - -The system provides three main interfaces: - -1. **Upload File** (Two-phase mode - Step 1) - - Endpoint: `POST /api/v1/collections/{collection_id}/documents/upload` - - Function: Upload file to temporary storage, status `UPLOADED` - - Returns: `document_id`, `filename`, `size`, `status` - -2. **Confirm Documents** (Two-phase mode - Step 2) - - Endpoint: `POST /api/v1/collections/{collection_id}/documents/confirm` - - Function: Confirm uploaded documents, trigger index building - - Parameters: `document_ids` array - - Returns: `confirmed_count`, `failed_count`, `failed_documents` - -3. **One-step Upload** (Legacy mode, backward compatible) - - Endpoint: `POST /api/v1/collections/{collection_id}/documents` - - Function: Upload and directly add to knowledge base, status directly to `PENDING` - - Supports batch upload - -### Phase 1: File Upload and Temporary Storage - -#### 1.1 Upload Flow - -``` -User selects files - │ - ▼ -Frontend calls upload API - │ - ▼ -View layer validates identity and params - │ - ▼ -Service layer processes business logic: - │ - ├─► Verify collection exists and active - │ - ├─► Validate file type and size - │ - ├─► Read file content - │ - ├─► Calculate SHA-256 hash - │ - └─► Transaction processing: - │ - ├─► Duplicate detection (by filename + hash) - │ ├─ Exact match: Return existing doc (idempotent) - │ ├─ Same name, different content: Throw conflict error - │ └─ New document: Continue creation - │ - ├─► Create Document record (status=UPLOADED) - │ - ├─► Upload to object store - │ └─ Path: user-{user_id}/{collection_id}/{document_id}/original{suffix} - │ - └─► Update document metadata (object_path) -``` - -#### 1.2 File Validation - -**Supported File Types**: -- Documents: `.pdf`, `.doc`, `.docx`, `.ppt`, `.pptx`, `.xls`, `.xlsx` -- Text: `.txt`, `.md`, `.html`, `.json`, `.xml`, `.yaml`, `.yml`, `.csv` -- Images: `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tiff`, `.tif` -- Audio: `.mp3`, `.wav`, `.m4a` -- Archives: `.zip`, `.tar`, `.gz`, `.tgz` - -**Size Limits**: -- Default: 100 MB (configurable via `MAX_DOCUMENT_SIZE` environment variable) -- Extracted total size: 5 GB (`MAX_EXTRACTED_SIZE`) - -#### 1.3 Duplicate Detection Mechanism - -Uses **filename + SHA-256 hash** dual detection: - -| Scenario | Filename | Hash | System Behavior | -|----------|----------|------|-----------------| -| Exact match | Same | Same | Return existing document (idempotent) | -| Name conflict | Same | Different | Throw `DocumentNameConflictException` | -| New document | Different | - | Create new document record | - -**Advantages**: -- ✅ Supports idempotent upload: Network retries won't create duplicates -- ✅ Prevents content conflicts: Same name with different content prompts user -- ✅ Saves storage space: Same content stored only once - -### Phase 2: Temporary Storage Configuration - -#### 2.1 Object Storage Types - -System supports two object storage backends, switchable via environment variables: - -**1. Local Storage (Local filesystem)** - -Use cases: -- Development and testing environments -- Small-scale deployments -- Single-machine deployments - -Configuration: -```bash -# Development environment -OBJECT_STORE_TYPE=local -OBJECT_STORE_LOCAL_ROOT_DIR=.objects - -# Docker environment -OBJECT_STORE_TYPE=local -OBJECT_STORE_LOCAL_ROOT_DIR=/shared/objects -``` - -Storage path example: -``` -.objects/ -└── user-google-oauth2-123456/ - └── col_abc123/ - └── doc_xyz789/ - ├── original.pdf # Original file - ├── converted.pdf # Converted PDF - ├── processed_content.md # Parsed Markdown - ├── chunks/ # Chunked data - │ ├── chunk_0.json - │ └── chunk_1.json - └── images/ # Extracted images - ├── page_0.png - └── page_1.png -``` - -**2. S3 Storage (Compatible with AWS S3/MinIO/OSS, etc.)** - -Use cases: -- Production environments -- Large-scale deployments -- Distributed deployments -- High availability and disaster recovery needs - -Configuration: -```bash -OBJECT_STORE_TYPE=s3 -OBJECT_STORE_S3_ENDPOINT=http://127.0.0.1:9000 # MinIO/S3 address -OBJECT_STORE_S3_REGION=us-east-1 # AWS Region -OBJECT_STORE_S3_ACCESS_KEY=minioadmin # Access Key -OBJECT_STORE_S3_SECRET_KEY=minioadmin # Secret Key -OBJECT_STORE_S3_BUCKET=aperag # Bucket name -OBJECT_STORE_S3_PREFIX_PATH=dev/ # Optional path prefix -OBJECT_STORE_S3_USE_PATH_STYLE=true # Set to true for MinIO -``` - -#### 2.2 Object Storage Path Rules - -**Path Format**: -``` -{prefix}/user-{user_id}/{collection_id}/{document_id}/{filename} -``` - -**Components**: -- `prefix`: Optional global prefix (S3 only) -- `user_id`: User ID (`|` replaced with `-`) -- `collection_id`: Collection ID -- `document_id`: Document ID -- `filename`: Filename (e.g., `original.pdf`, `page_0.png`) - -**Multi-tenancy Isolation**: -- Each user has an independent namespace -- Each collection has an independent storage directory -- Each document has an independent folder - -### Phase 3: Document Confirmation and Index Building - -#### 3.1 Confirmation Flow - -``` -User clicks "Save to Collection" - │ - ▼ -Frontend calls confirm API - │ - ▼ -Service layer processes: - │ - ├─► Validate collection configuration - │ - ├─► Check Quota (deduct quota at confirmation stage) - │ - └─► For each document_id: - │ - ├─► Verify document status is UPLOADED - │ - ├─► Update document status: UPLOADED → PENDING - │ - ├─► Create index records based on collection config: - │ ├─ VECTOR (Vector index, required) - │ ├─ FULLTEXT (Full-text index, required) - │ ├─ GRAPH (Knowledge graph, optional) - │ ├─ SUMMARY (Document summary, optional) - │ └─ VISION (Vision index, optional) - │ - └─► Return confirmation result - │ - ▼ -Trigger Celery task: reconcile_document_indexes - │ - ▼ -Background async index building -``` - -#### 3.2 Quota Management - -**Check Timing**: -- ❌ Not checked during upload phase (temporary storage doesn't consume quota) -- ✅ Checked during confirmation phase (formal addition consumes quota) - -**Quota Types**: - -1. **User Global Quota** - - `max_document_count`: Total document count limit per user - - Default: 1000 (configurable via `MAX_DOCUMENT_COUNT`) - -2. **Per-Collection Quota** - - `max_document_count_per_collection`: Document count limit per collection - - Excludes `UPLOADED` and `DELETED` status documents - -**Quota Exceeded Handling**: -- Throws `QuotaExceededException` -- Returns HTTP 400 error -- Includes current usage and quota limit information - -### Phase 4: Document Parsing and Format Conversion - -#### 4.1 Parser Architecture - -System uses a **multi-parser chain invocation** architecture, where each parser handles specific file types: - -``` -DocParser (Main Controller) - │ - ├─► MinerUParser - │ └─ Function: High-precision PDF parsing (commercial API) - │ └─ Supports: .pdf - │ - ├─► ImageParser - │ └─ Function: Image content recognition (OCR + vision understanding) - │ └─ Supports: .jpg, .png, .gif, .bmp, .tiff - │ - ├─► AudioParser - │ └─ Function: Audio transcription (Speech-to-Text) - │ └─ Supports: .mp3, .wav, .m4a - │ - └─► MarkItDownParser (Fallback) - └─ Function: Universal document to Markdown conversion - └─ Supports: Almost all common formats -``` - -#### 4.2 Parser Configuration - -**Configuration Method**: Dynamically controlled via Collection Config - -```json -{ - "parser_config": { - "use_mineru": false, // Enable MinerU (requires API Token) - "use_markitdown": true, // Enable MarkItDown (default) - "mineru_api_token": "xxx" // MinerU API Token (optional) - } -} -``` - -**Environment Variable Configuration**: -```bash -USE_MINERU_API=false # Globally enable MinerU -MINERU_API_TOKEN=your_token # MinerU API Token -``` - -#### 4.3 Parsing Flow - -``` -Celery Worker receives indexing task - │ - ▼ -1. Download original file from object store - │ - ▼ -2. Select Parser based on file extension - │ - ├─► Try first matching Parser - │ ├─ Success: Return parsing result - │ └─ Failure: FallbackError → Try next Parser - │ - └─► Final fallback: MarkItDownParser - │ - ▼ -3. Parsing result (Parts): - │ - ├─► MarkdownPart: Text content - │ └─ Contains: headings, paragraphs, lists, tables, etc. - │ - ├─► PdfPart: PDF file - │ └─ For: linearization, page rendering - │ - └─► AssetBinPart: Binary resources - └─ Contains: images, embedded files, etc. - │ - ▼ -4. Post-processing: - │ - ├─► PDF pages to images (required for Vision index) - │ └─ Each page rendered as PNG image - │ └─ Saved to {document_path}/images/page_N.png - │ - ├─► PDF linearization (speed up browser loading) - │ └─ Use pikepdf to optimize PDF structure - │ └─ Saved to {document_path}/converted.pdf - │ - └─► Extract text content (plain text) - └─ Merge all MarkdownPart content - └─ Saved to {document_path}/processed_content.md - │ - ▼ -5. Save to object store -``` - -#### 4.4 Format Conversion Examples - -**Example 1: PDF Document** -``` -Input: user_manual.pdf (5 MB) - │ - ▼ -Parser selection: MinerUParser / MarkItDownParser - │ - ▼ -Output Parts: - ├─ MarkdownPart: "# User Manual\n\n## Chapter 1\n..." - └─ PdfPart:
specializes in PostgreSQL and MySQL.
Mike works in the frontend team..."] - end - - subgraph Process[🔄 Graph Index Processing] - Extract[Extract entities and relationships] - end - - subgraph Output[🕸️ Knowledge Graph] - direction TB - A[John
Person] -->|heads| B[Database Team
Organization] - A -->|specializes in| C[PostgreSQL
Technology] - A -->|specializes in| D[MySQL
Technology] - E[Mike
Person] -->|belongs to| F[Frontend Team
Organization] - E -->|collaborates| A - end - - Input --> Process - Process --> Output - - style Input fill:#e3f2fd - style Process fill:#fff59d - style Output fill:#c8e6c9 -``` - -Traditional vector search can only find "semantically similar" paragraphs but cannot answer these questions: -- What does John lead? -- What is the relationship between John and Mike? -- What technologies does the database team use? - -**Graph Index can do**: Accurately answer these relationship-focused questions by making implicit knowledge relationships explicit. - -### 1.2 Core Value - -Compared to traditional retrieval methods, Graph Index provides unique capabilities: - -| Capability | Vector Search | Full-text Search | Graph Index | -|------------|---------------|------------------|-------------| -| Semantic Similarity | ✅ Strong | ❌ Weak | ✅ Strong | -| Exact Keyword Match | ❌ Weak | ✅ Strong | ✅ Medium | -| Relationship Query | ❌ Not Supported | ❌ Not Supported | ✅ Strong | -| Multi-hop Reasoning | ❌ Not Supported | ❌ Not Supported | ✅ Supported | -| Suitable Questions | "How to optimize performance" | "PostgreSQL config" | "John and Mike's relationship" | - -**Core Advantage**: Graph Index allows AI to "understand" the connections between knowledge, not just text similarity. - -## 2. What Problems Can Graph Index Solve - -Graph Index excels at handling scenarios that require "understanding relationships". Let's look at practical applications. - -### 2.1 Enterprise Knowledge Management - -**Scenario**: Companies have extensive documentation including organizational structure, project materials, and technical docs. - -**Graph Index Value**: - -- 📊 **Organizational Relationships**: "Who is on John's team?" → Quickly find team members -- 🔗 **Collaboration Networks**: "Who has worked with John?" → Discover work networks -- 🛠️ **Skill Mapping**: "Who is skilled in PostgreSQL?" → Locate technical experts -- 📁 **Project History**: "Which projects has John participated in?" → Track project experience - -**Real Effect**: - -``` -Question: "Who leads the database team?" -Traditional Search: Returns dozens of paragraphs containing "database team" and "lead" -Graph Index: Directly returns "John" + relevant background information -``` - -### 2.2 Research and Learning - -**Scenario**: Analyzing academic papers and technical documentation to understand knowledge lineage. - -**Graph Index Value**: - -- 👥 **Author Networks**: "Who has this author collaborated with?" → Discover research teams -- 📖 **Citation Relationships**: "What papers does this cite?" → Trace research lineage -- 🔬 **Technology Evolution**: "How has this technology evolved?" → Understand tech history -- 💡 **Concept Connections**: "What's the relationship between tech A and B?" → Connect knowledge points - -### 2.3 Products and Services - -**Scenario**: Product documentation, user manuals, API documentation. - -**Graph Index Value**: - -- ⚙️ **Feature Dependencies**: "What needs to be configured before enabling feature A?" → Understand dependencies -- 🔧 **Configuration Relationships**: "Which features does this config affect?" → Avoid misconfigurations -- 🐛 **Problem Diagnosis**: "What might cause error X?" → Quick troubleshooting -- 📚 **API Relationships**: "Which APIs are typically used together?" → Learn best practices - -### 2.4 Comparison: When to Use Graph Index - -Different questions suit different retrieval methods: - -| Question Type | Example | Best Solution | -|--------------|---------|---------------| -| **Concept Understanding** | "What is RAG?" | Vector Search | -| **Exact Lookup** | "PostgreSQL config file path" | Full-text Search | -| **Relationship Query** | "What's John and Mike's relationship?" | Graph Index ✨ | -| **Multi-hop Reasoning** | "What tech stack does John's team use?" | Graph Index ✨ | -| **Knowledge Tracing** | "What modules does this feature depend on?" | Graph Index ✨ | - -**Best Practice**: ApeRAG supports vector search, full-text search, and graph index simultaneously, intelligently selecting or combining based on question type. - -## 3. Construction Process Overview - -When you upload a document and enable graph indexing, ApeRAG automatically completes the following steps. Here's a simple overview; details are in later chapters. - -### 3.1 Five Key Steps - -```mermaid -flowchart TB - subgraph Step1["1️⃣ Document Chunking"] - A1[Original Document] --> A2[Smart Chunking] - A2 --> A3[Generate Chunks] - end - - subgraph Step2["2️⃣ Entity Relationship Extraction"] - B1[Chunks] --> B2[Call LLM] - B2 --> B3[Identify Entities] - B2 --> B4[Identify Relationships] - end - - subgraph Step3["3️⃣ Connected Component Analysis"] - C1[Entity Relationship Network] --> C2[BFS Algorithm] - C2 --> C3[Grouping] - end - - subgraph Step4["4️⃣ Concurrent Merging"] - D1[Group 1] --> D2[Entity Deduplication] - D3[Group 2] --> D4[Entity Deduplication] - D5[Group N] --> D6[Entity Deduplication] - D2 --> D7[Relationship Aggregation] - D4 --> D7 - D6 --> D7 - end - - subgraph Step5["5️⃣ Multi-storage Writing"] - E1[Graph Database] - E2[Vector Database] - E3[Text Storage] - end - - A3 --> B1 - B3 --> C1 - B4 --> C1 - C3 --> D1 - C3 --> D3 - C3 --> D5 - D7 --> E1 - D7 --> E2 - A3 --> E3 - - style Step1 fill:#e3f2fd - style Step2 fill:#fff3e0 - style Step3 fill:#f3e5f5 - style Step4 fill:#e8f5e9 - style Step5 fill:#fce4ec -``` - -**Simply put**: Chunk document → Extract entities/relationships → Smart grouping → Concurrent merging → Write to storage. - -The entire process is fully automated - you just upload documents, and the system handles everything. - -### 3.2 Processing Time Reference - -Processing time varies by document size: - -| Document Size | Entity Count | Processing Time | Example | -|--------------|--------------|-----------------|---------| -| Small (< 5 pages) | ~50 | 10-30 seconds | Company notices, meeting notes | -| Medium (10-50 pages) | ~200 | 1-3 minutes | Technical docs, product manuals | -| Large (100+ pages) | ~1000 | 5-15 minutes | Research reports, books | - -**Factors**: -- LLM response speed (main bottleneck) -- Document complexity (tables, images slow processing) -- Concurrency settings (configurable for speed) - -> 💡 **Tip**: Processing is asynchronous - upload multiple documents and the system processes them in parallel. - -### 3.3 Real-time Progress Tracking - -You can check document processing progress anytime: - -``` -Document Status: Processing -- ✅ Document Parsing: Complete -- ✅ Document Chunking: Complete (25 chunks generated) -- 🔄 Entity Extraction: In Progress (15/25) -- ⏳ Relationship Extraction: Waiting -- ⏳ Graph Construction: Waiting -``` - -Once processing completes, document status changes to "Active" and graph queries become available. - -## 4. Detailed Construction Process - -The previous sections covered what graph index does and the overall process. This chapter details the technical implementation of each step. - -> 💡 **Reading Tip**: If you only want to understand basic concepts and usage, skip to Chapter 9 for practical applications. - -### 4.1 Document Chunking - -First step: Split long documents into appropriately sized chunks. - -**Why Chunk?** -- LLMs have input length limits (typically thousands to tens of thousands of tokens) -- Too large: Extraction quality decreases, LLM may "miss" information -- Too small: Loses context, can't understand complete semantics - -**Smart Chunking Strategy**: - -```mermaid -flowchart LR - Doc[Long Document] --> Check{Check Size} - Check -->|< 1200 tokens| Keep[Keep Intact] - Check -->|> 1200 tokens| Split[Smart Split] - - Split --> By1[By Paragraph] - By1 --> Check2{Still Too Big?} - Check2 -->|Yes| By2[By Sentence] - Check2 -->|No| Done[Complete] - By2 --> Check3{Still Too Big?} - Check3 -->|Yes| By3[By Character] - Check3 -->|No| Done - By3 --> Done - - style Doc fill:#e1f5ff - style Split fill:#ffccbc - style Done fill:#c5e1a5 -``` - -**Chunking Parameters**: -- Default size: 1200 tokens (approximately 800-1000 English words) -- Overlap size: 100 tokens (ensures context continuity) -- Priority: Paragraph > Sentence > Character - -### 4.2 Entity Relationship Extraction - -Use LLM to identify entities and relationships from each chunk. - -**Extraction Process**: - -```mermaid -sequenceDiagram - participant C as Chunk - participant L as LLM - participant R as Results - - C->>L: "John heads the database team..." - L->>R: Entities: [John(Person), Database Team(Org)] - L->>R: Relationships: [John-heads->Database Team] - - C->>L: "John specializes in PostgreSQL..." - L->>R: Entities: [John(Person), PostgreSQL(Tech)] - L->>R: Relationships: [John-specializes in->PostgreSQL] -``` - -**Concurrency Optimization**: Multiple chunks can call LLM simultaneously, default 20 concurrent requests. - -### 4.3 Connected Component Analysis - -Divide entity relationship network into independent subgraphs for parallel processing. - -**Why This Step?** - -Tech team entities and finance department entities aren't connected - they can be processed completely in parallel! - -```mermaid -graph LR - subgraph Component1[Connected Component 1 - Tech Team] - A1[John] -->|heads| A2[Database Team] - A1 -->|specializes in| A3[PostgreSQL] - A4[Mike] -->|collaborates| A1 - end - - subgraph Component2[Connected Component 2 - Finance] - B1[Alice] -->|belongs to| B2[Finance Dept] - B3[Bob] -->|collaborates| B1 - end - - style Component1 fill:#bbdefb - style Component2 fill:#c5e1a5 -``` - -**Performance Boost**: 3 independent components = 3x speedup! - -### 4.4 Concurrent Merging - -Same-name entities need deduplication, same relationships need aggregation. - -```mermaid -flowchart TD - subgraph Before["Before Merging"] - A1["John
Database head"] - A2["John
Specializes in PostgreSQL"] - A3["John
Leads team"] - end - - Merge[Smart Merge] - - subgraph After["After Merging"] - B1["John
Database team head,
specializes in PostgreSQL,
leads multiple projects"] - end - - A1 --> Merge - A2 --> Merge - A3 --> Merge - Merge --> B1 - - style Before fill:#ffccbc - style After fill:#c5e1a5 -``` - -**Fine-grained Locks**: Only lock entities being merged, others can process concurrently. - -### 4.5 Multi-storage Writing - -Knowledge graph written to three storage systems: - -```mermaid -flowchart LR - KG[Knowledge Graph] --> G[Graph Database
Graph Queries] - KG --> V[Vector Database
Semantic Search] - KG --> T[Text Storage
Full-text Search] - - style KG fill:#e1f5ff - style G fill:#bbdefb - style V fill:#c5e1a5 - style T fill:#ffccbc -``` - -Different storages support different query types, complementing each other. - -## 5. Core Technical Design - -This chapter introduces core technical designs including data isolation and concurrency control. - -> 💡 **Reading Tip**: These are system architecture and implementation details, mainly for developers and technical decision-makers. - -### 5.1 Workspace Data Isolation - -Each Collection has an independent namespace for complete data isolation. - -**Naming Convention**: - -```python -# Entity naming -entity:{entity_name}:{workspace} -# Example -entity:John:collection_abc123 - -# Relationship naming -relationship:{source}:{target}:{workspace} -# Example -relationship:John:Database Team:collection_abc123 -``` - -**Isolation Effect**: - -```mermaid -graph TB - subgraph Collection_A[Collection A - Company Docs] - A1[entity:John:A] --> A2[entity:Database Team:A] - end - - subgraph Collection_B[Collection B - School Docs] - B1[entity:John:B] --> B2[entity:CS Department:B] - end - - style Collection_A fill:#bbdefb - style Collection_B fill:#c5e1a5 -``` - -"John" in two Collections is completely independent, no interference! - -### 5.2 Stateless Instance Management - -Each processing task creates an independent graph index instance, destroyed after completion. - -**Lifecycle Management**: - -```mermaid -sequenceDiagram - participant C as Celery Task - participant M as Manager - participant R as Graph Index Instance - participant S as Storage - - C->>M: process_document() - M->>R: create_instance() - R->>S: Initialize storage connections - R->>R: Process document - R->>S: Write data - R-->>M: Return results - M-->>C: Task complete - Note over R: Instance destroyed, resources released -``` - -**Advantages**: -- ✅ Zero state pollution: Each task independent, no interference -- ✅ Easy scaling: Can run multiple workers simultaneously -- ✅ Resource management: Automatic cleanup, no memory leaks - -### 5.3 Connected Component Concurrency Optimization - -Intelligent concurrent processing through graph topology analysis. - -**Algorithm Principle**: - -```mermaid -graph TB - subgraph Input[Input: Entity Relationship Network] - I1[Entity 1] --> I2[Entity 2] - I2 --> I3[Entity 3] - - I4[Entity 4] --> I5[Entity 5] - - I6[Entity 6] - end - - Algorithm[BFS Algorithm] - - subgraph Output[Output: 3 Connected Components] - O1[Component 1
3 entities] - O2[Component 2
2 entities] - O3[Component 3
1 entity] - end - - Input --> Algorithm - Algorithm --> Output - - style Input fill:#ffccbc - style Algorithm fill:#fff59d - style Output fill:#c5e1a5 -``` - -**Performance Boost**: 3 components concurrent processing = 3x speedup! - -### 5.4 Fine-grained Concurrency Control - -Precise entity-level locking: - -**Lock Hierarchy**: - -```mermaid -graph TD - A[Global Lock - Traditional] -->|Too Coarse| B[All Entities Serial] - - C[Entity Lock - ApeRAG] -->|Just Right| D[Lock Only Merging Entities] - - style A fill:#ffccbc - style B fill:#ffccbc - style C fill:#c5e1a5 - style D fill:#c5e1a5 -``` - -**Lock Strategy**: -1. Extraction phase: No locks, fully parallel -2. Merging phase: Lock only needed entities -3. Sorted lock acquisition: Prevents deadlock - -### 5.5 Smart Summarization - -Automatically compress overly long descriptions: - -```python -if len(description) > 2000 tokens: - summary = await llm_summarize(description) -else: - summary = description -``` - -**Effect**: Compress 2500 tokens to 200 tokens, retaining core information. - -### 5.6 Multi-storage Backend Support - -ApeRAG supports two graph databases: Neo4j and PostgreSQL. - -**How to Choose?** - -| Scenario | Recommended | Reason | -|----------|-------------|--------| -| **Small Scale** (< 100K entities) | PostgreSQL | Simple ops, low cost | -| **Medium Scale** (100K-1M) | PostgreSQL or Neo4j | Based on query complexity | -| **Large Scale** (> 1M) | Neo4j | Better graph query performance | -| **Limited Budget** | PostgreSQL | No extra deployment | -| **Complex Graph Algorithms** | Neo4j | Built-in graph algorithms | - -**Switching**: - -```bash -# Use PostgreSQL (default) -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage - -# Use Neo4j -export GRAPH_INDEX_GRAPH_STORAGE=Neo4JSyncStorage -``` - -## 6. Complete Data Flow - -The entire graph index construction is a data transformation pipeline, from unstructured text to structured knowledge graph: - -```mermaid -flowchart TD - A[Original Document] --> B[Clean & Preprocess] - B --> C[Smart Chunking] - C --> D[Chunks] - - D --> E[LLM Concurrent Extraction] - E --> F[Original Entity List] - E --> G[Original Relationship List] - - F --> H[Build Adjacency Graph] - G --> H - H --> I[BFS Find Connected Components] - I --> J[Grouped Concurrent Processing] - - J --> K[Entity Deduplication] - J --> L[Relationship Aggregation] - - K --> M{Description Length Check} - M -->|Too Long| N[LLM Summary] - M -->|Appropriate| O[Keep Original] - N --> P[Final Entities] - O --> P - - L --> Q{Description Length Check} - Q -->|Too Long| R[LLM Summary] - Q -->|Appropriate| S[Keep Original] - R --> T[Final Relationships] - S --> T - - P --> U[Graph Database] - P --> V[Vector Database] - T --> U - T --> V - D --> W[Text Storage] - - U --> X[Knowledge Graph Complete] - V --> X - W --> X - - style A fill:#e1f5ff - style E fill:#fff59d - style I fill:#f3e5f5 - style J fill:#c5e1a5 - style X fill:#c8e6c9 -``` - -### Data Transformation Example - -A concrete example showing step-by-step data transformation: - -**Input Document**: - -```text -John heads the database team and specializes in PostgreSQL and MySQL. -Mike works in the frontend team and often collaborates with John's team to develop backend systems. -Alice is an accountant in the finance department, responsible for financial reports. -``` - -**Step 1: Chunking** - -```json -[ - { - "chunk_id": "chunk-001", - "content": "John heads the database team and specializes in PostgreSQL and MySQL.", - "tokens": 15 - }, - { - "chunk_id": "chunk-002", - "content": "Mike works in the frontend team and often collaborates with John's team...", - "tokens": 18 - }, - { - "chunk_id": "chunk-003", - "content": "Alice is an accountant in the finance department, responsible for financial reports.", - "tokens": 14 - } -] -``` - -**Step 2: Entity Relationship Extraction** - -```json -{ - "entities": [ - {"name": "John", "type": "Person", "source": "chunk-001"}, - {"name": "Database Team", "type": "Organization", "source": "chunk-001"}, - {"name": "PostgreSQL", "type": "Technology", "source": "chunk-001"}, - {"name": "MySQL", "type": "Technology", "source": "chunk-001"}, - {"name": "Mike", "type": "Person", "source": "chunk-002"}, - {"name": "Frontend Team", "type": "Organization", "source": "chunk-002"}, - {"name": "Alice", "type": "Person", "source": "chunk-003"}, - {"name": "Finance Department", "type": "Organization", "source": "chunk-003"} - ], - "relationships": [ - {"source": "John", "target": "Database Team", "relation": "heads"}, - {"source": "John", "target": "PostgreSQL", "relation": "specializes in"}, - {"source": "John", "target": "MySQL", "relation": "specializes in"}, - {"source": "Mike", "target": "Frontend Team", "relation": "belongs to"}, - {"source": "Mike", "target": "John", "relation": "collaborates"}, - {"source": "Alice", "target": "Finance Department", "relation": "belongs to"} - ] -} -``` - -**Step 3: Connected Component Analysis** - -``` -Connected Component 1 (Technical Department): -- Entities: John, Mike, Database Team, Frontend Team, PostgreSQL, MySQL -- Relationships: 6 - -Connected Component 2 (Finance Department): -- Entities: Alice, Finance Department -- Relationships: 1 -``` - -**Step 4: Concurrent Merging** - -Two components can process in parallel! - -**Step 5: Final Knowledge Graph** - -```mermaid -graph LR - subgraph Technical - John -->|heads| DatabaseTeam[Database Team] - John -->|specializes in| PostgreSQL - John -->|specializes in| MySQL - Mike -->|belongs to| FrontendTeam[Frontend Team] - Mike -->|collaborates| John - end - - subgraph Finance - Alice -->|belongs to| FinanceDept[Finance Department] - end - - style Technical fill:#bbdefb - style Finance fill:#c5e1a5 -``` - -### Performance Optimization Features - -1. **Fine-grained Concurrency Control** - - Entity-level locks: `entity:John:collection_abc` - - Lock only during merging, fully parallel during extraction - -2. **Connected Component Concurrency** - - Technical and Finance departments can process in parallel - - Zero lock contention, full multi-core CPU utilization - -3. **Smart Summarization** - - Description < 2000 tokens: Keep original - - Description > 2000 tokens: LLM summary compression - -## 7. Performance Optimization Strategies - -### 7.1 Concurrency Control - -Graph index construction involves extensive LLM calls and database operations requiring proper concurrency control. - -**Concurrency Hierarchy**: - -```mermaid -graph TB - A[Document-level Concurrency] --> B[Chunk-level Concurrency] - B --> C[Component-level Concurrency] - C --> D[Entity-level Concurrency] - - A1[Celery Workers
Multiple docs simultaneously] --> A - B1[LLM Concurrent Calls
Multiple chunks simultaneously] --> B - C1[Parallel Component Merging
Multiple components simultaneously] --> C - D1[Concurrent Entity Merging
Different entities simultaneously] --> D - - style A fill:#e3f2fd - style B fill:#fff3e0 - style C fill:#f3e5f5 - style D fill:#e8f5e9 -``` - -**Concurrency Parameters**: - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `llm_model_max_async` | 20 | LLM concurrent calls | -| `embedding_func_max_async` | 16 | Embedding concurrent calls | -| `max_batch_size` | 32 | Batch processing size | - -**Tuning Recommendations**: - -```python -# Scenario 1: Strict LLM API rate limits -llm_model_max_async = 5 # Reduce concurrency to avoid rate limiting - -# Scenario 2: Sufficient performance, want speedup -llm_model_max_async = 50 # Increase concurrency to speed up processing - -# Scenario 3: Limited memory -max_batch_size = 16 # Reduce batch size to lower memory usage -``` - -### 7.2 LLM Call Optimization - -LLM calls are the most time-consuming part, main optimization strategies: - -1. **Concurrent Calls**: Multiple chunks extract simultaneously (default 20 concurrent) -2. **Batch Processing**: Reduce LLM call count -3. **Cache Reuse**: Reuse summary results for similar descriptions - -**Performance Boost**: Concurrent calling is 4x faster than serial. - -### 7.3 Storage Optimization - -Batch writing significantly improves performance: - -| Method | 100 Entity Write Time | -|--------|---------------------| -| Individual Write | ~10 seconds | -| Batch Write (32/batch) | ~1 second | - -**Optimization Effect**: 10x speedup! - -### 7.4 Memory Optimization - -Memory management strategies for large documents: - -- Stream chunking: Don't load entire document at once -- Immediate release: Free memory immediately after processing -- Batch processing: Control memory peaks - -### 7.5 Performance Monitoring - -System outputs detailed performance statistics: - -``` -Graph Index Construction Complete: -✓ Document Chunking: 10 chunks, 0.5 seconds -✓ Entity Extraction: 120 entities, 25 seconds -✓ Relationship Extraction: 85 relationships, 25 seconds -✓ Concurrent Merging: 15 seconds -✓ Storage Writing: 2 seconds -━━━━━━━━━━━━━━━━━━━━━━━━━ -Total: 42.7 seconds -``` - -**Bottleneck Analysis**: Entity/relationship extraction takes 60% of time, can optimize by increasing LLM concurrency. - -## 8. Configuration Parameters - -### 8.1 Core Configuration - -Graph index construction can be tuned with the following parameters: - -**Chunking Parameters**: - -```python -# Chunk size (tokens) -CHUNK_TOKEN_SIZE = 1200 - -# Overlap size (tokens) -CHUNK_OVERLAP_TOKEN_SIZE = 100 -``` - -**Tuning Recommendations**: -- Small docs (< 5000 tokens): `CHUNK_TOKEN_SIZE = 800` -- Large docs (> 50000 tokens): `CHUNK_TOKEN_SIZE = 1500` -- Need more context: Increase `CHUNK_OVERLAP_TOKEN_SIZE` - -**Concurrency Parameters**: - -```python -# LLM concurrent calls -LLM_MODEL_MAX_ASYNC = 20 - -# Embedding concurrent calls -EMBEDDING_FUNC_MAX_ASYNC = 16 - -# Batch processing size -MAX_BATCH_SIZE = 32 -``` - -**Tuning Recommendations**: -- Strict LLM API limits: Lower `LLM_MODEL_MAX_ASYNC` to 5-10 -- Sufficient performance for speedup: Increase to 50-100 -- Limited memory: Lower `MAX_BATCH_SIZE` to 16 - -**Entity Extraction Parameters**: - -```python -# Entity extraction retry count (0 = extract once only) -ENTITY_EXTRACT_MAX_GLEANING = 0 - -# Summary max tokens -SUMMARY_TO_MAX_TOKENS = 2000 - -# Force summary description fragment count -FORCE_LLM_SUMMARY_ON_MERGE = 10 -``` - -**Tuning Recommendations**: -- Extraction quality important: `ENTITY_EXTRACT_MAX_GLEANING = 1` (extract twice) -- Speed priority: `ENTITY_EXTRACT_MAX_GLEANING = 0` -- Descriptions often long: Lower `SUMMARY_TO_MAX_TOKENS` to 1000 - -### 8.2 Knowledge Graph Configuration - -Configure in Collection settings: - -```json -{ - "knowledge_graph_config": { - "language": "English", - "entity_types": [ - "organization", - "person", - "geo", - "event", - "product", - "technology", - "date", - "category" - ] - } -} -``` - -**Parameter Description**: - -- **language**: Extraction language, affects LLM prompts - - `English`: English - - `simplified chinese`: Simplified Chinese - - `traditional chinese`: Traditional Chinese - -- **entity_types**: Entity types to extract - - Default: 8 types (organization, person, location, event, product, technology, date, category) - - Customizable: e.g., extract only people and organizations - -### 8.3 Storage Configuration - -Configure storage backends via environment variables: - -```bash -# KV storage (key-value) -export GRAPH_INDEX_KV_STORAGE=PGOpsSyncKVStorage - -# Vector storage -export GRAPH_INDEX_VECTOR_STORAGE=PGOpsSyncVectorStorage - -# Graph storage -export GRAPH_INDEX_GRAPH_STORAGE=Neo4JSyncStorage -# Or use PostgreSQL -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage -``` - -**Storage Selection Recommendations**: - -| Scenario | KV Storage | Vector Storage | Graph Storage | -|----------|-----------|----------------|---------------| -| **Default** | PostgreSQL | PostgreSQL | PostgreSQL | -| **High-performance Vector Search** | PostgreSQL | Qdrant | Neo4j | -| **Large-scale Graph** | PostgreSQL | Qdrant | Neo4j | -| **Simple Deployment** | PostgreSQL | PostgreSQL | PostgreSQL | - -### 8.4 Complete Configuration Example - -```bash -# Chunking configuration -export CHUNK_TOKEN_SIZE=1200 -export CHUNK_OVERLAP_TOKEN_SIZE=100 - -# Concurrency configuration -export LLM_MODEL_MAX_ASYNC=20 -export MAX_BATCH_SIZE=32 - -# Extraction configuration -export ENTITY_EXTRACT_MAX_GLEANING=0 -export SUMMARY_TO_MAX_TOKENS=2000 - -# Storage configuration -export GRAPH_INDEX_KV_STORAGE=PGOpsSyncKVStorage -export GRAPH_INDEX_VECTOR_STORAGE=PGOpsSyncVectorStorage -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage - -# Database connection (PostgreSQL) -export POSTGRES_HOST=127.0.0.1 -export POSTGRES_PORT=5432 -export POSTGRES_DB=aperag -export POSTGRES_USER=postgres -export POSTGRES_PASSWORD=your_password - -# Database connection (Neo4j, optional) -export NEO4J_HOST=127.0.0.1 -export NEO4J_PORT=7687 -export NEO4J_USERNAME=neo4j -export NEO4J_PASSWORD=your_password -``` - -## 9. Practical Application Scenarios - -Graph index is particularly suitable for these scenarios: - -### 9.1 Enterprise Knowledge Base - -**Scenario**: Companies have extensive documentation including organizational structure, project materials, technical docs. - -**Graph Index Value**: - -- 📊 **Organizational Relationships**: "Who is on John's team?" → Quickly find team members -- 🔗 **Collaboration Networks**: "Who has worked with John?" → Discover work networks -- 🛠️ **Skill Mapping**: "Who is skilled in PostgreSQL?" → Locate technical experts -- 📁 **Project History**: "Which projects has John participated in?" → Track project experience - -**Real Effect**: - -``` -Question: "Who leads the database team?" -Traditional Search: Returns dozens of paragraphs containing "database team" and "lead" -Graph Index: Directly returns "John" + relevant background info -``` - -### 9.2 Research and Learning - -**Scenario**: Analyzing academic papers and technical documentation to understand knowledge lineage. - -**Graph Index Value**: - -- 👥 **Author Networks**: "Who has this author collaborated with?" → Discover research teams -- 📖 **Citation Relationships**: "What papers does this cite?" → Trace research lineage -- 🔬 **Technology Evolution**: "How has this technology evolved?" → Understand tech history -- 💡 **Concept Connections**: "What's the relationship between tech A and B?" → Connect knowledge points - -**Query Examples**: - -``` -User: "What research is related to Graph RAG?" -Graph Index: Query papers --research--> Graph RAG relationships -Result: Paper A, Paper B, Paper C - -User: "Who has an author collaborated with?" -Graph Index: Query author --collaborates--> other authors relationships -Result: Collaborator list and collaboration projects -``` - -### 9.3 Products and Services - -**Scenario**: Product documentation, user manuals, API documentation. - -**Graph Index Value**: - -- ⚙️ **Feature Dependencies**: "What needs configuration before enabling feature A?" → Understand dependencies -- 🔧 **Configuration Relationships**: "Which features does this config affect?" → Avoid misconfigurations -- 🐛 **Problem Diagnosis**: "What might cause error X?" → Quick troubleshooting -- 📚 **API Relationships**: "Which APIs are typically used together?" → Learn best practices - -**Query Examples**: - -``` -User: "How to configure graph index?" -Graph Index: Query config items --affects--> graph index relationships -Result: GRAPH_INDEX_GRAPH_STORAGE, knowledge_graph_config - -User: "What's the difference between Neo4j and PostgreSQL?" -Graph Index: Query Neo4j, PostgreSQL properties and relationships -Result: Performance comparison, applicable scenarios, configuration methods -``` - -### 9.4 Conversation Scenario Comparison - -Let's see how different retrieval methods perform in actual conversations: - -**Question: "What's the relationship between John and Mike?"** - -| Retrieval Method | Can Answer | Answer Quality | -|-----------------|-----------|----------------| -| **Pure Vector Search** | ⚠️ Partial | Finds paragraphs mentioning both, but unclear relationship | -| **Pure Full-text Search** | ⚠️ Partial | Finds paragraphs containing "John" and "Mike" | -| **Graph Index** | ✅ Yes | Directly returns: John and Mike have a collaboration relationship | - -**Question: "Where is the PostgreSQL config file?"** - -| Retrieval Method | Can Answer | Answer Quality | -|-----------------|-----------|----------------| -| **Pure Vector Search** | ✅ Yes | Finds relevant config paragraphs | -| **Pure Full-text Search** | ✅ Yes | Exact match "PostgreSQL" and "config" | -| **Graph Index** | ✅ Yes | Finds PostgreSQL --config--> file relationships | - -**Question: "How to improve system performance?"** - -| Retrieval Method | Can Answer | Answer Quality | -|-----------------|-----------|----------------| -| **Pure Vector Search** | ✅ Strong | Finds all performance optimization content | -| **Pure Full-text Search** | ⚠️ Medium | Needs exact keywords "performance", "optimize" | -| **Graph Index** | ✅ Strong | Finds optimization methods --improves--> performance relationships | - -**Best Practice**: Combine multiple retrieval methods! - -## 10. Summary - -ApeRAG's graph index provides production-grade knowledge graph construction capabilities with high performance, reliability, and scalability. - -### Key Features - -1. **Workspace data isolation**: Each Collection completely independent, supporting true multi-tenancy -2. **Stateless architecture**: Each task independent instance, zero state pollution -3. **Connected component concurrency**: Intelligent concurrency strategy, 2-3x performance boost -4. **Fine-grained lock management**: Entity-level locks, maximizing concurrency -5. **Smart summarization**: Automatically compress overly long descriptions, saving storage and improving retrieval efficiency -6. **Multi-storage support**: Flexible choice between Neo4j or PostgreSQL - -### Suitable Scenarios - -- ✅ **Enterprise Knowledge Base**: Understanding organizational structure, personnel relationships, project history -- ✅ **Research Paper Analysis**: Author collaboration networks, citation relationships, research lineage -- ✅ **Product Documentation**: Feature dependencies, configuration relationships, problem diagnosis -- ✅ **Any scenario requiring "relationship" understanding** - -### Performance - -- Process 10,000 entities: approximately 2-5 minutes (depending on LLM speed) -- Connected component concurrency: 2-3x performance boost -- Memory usage: approximately 400 MB (10,000 entities) -- Storage space: approximately 100 MB (10,000 entities) - -### Next Steps - -After graph index construction completes, you can perform graph queries. ApeRAG supports three graph query modes: - -- **Local Mode**: Query local information about an entity -- **Global Mode**: Query overall relationships and patterns -- **Hybrid Mode**: Comprehensive queries - -For detailed retrieval process, see [System Architecture Documentation](./architecture.md#42-knowledge-graph-query). - ---- - -## Related Documentation - -- 📋 [System Architecture](./architecture.md) - ApeRAG overall architecture design -- 📖 [Entity Extraction and Merging Mechanism](./lightrag_entity_extraction_and_merging.md) - Core algorithm details -- 🔗 [Connected Component Optimization](./connected_components_optimization.md) - Concurrency optimization principles -- 🌐 [Index Pipeline Architecture](./indexing_architecture.md) - Complete indexing process diff --git a/web/docs/en-US/development/_category.yaml b/web/docs/en-US/development/_category.yaml deleted file mode 100644 index 2cea69694..000000000 --- a/web/docs/en-US/development/_category.yaml +++ /dev/null @@ -1,2 +0,0 @@ -title: Development -position: 4 diff --git a/web/docs/en-US/development/development-guide.md b/web/docs/en-US/development/development-guide.md deleted file mode 100644 index 54bbe4277..000000000 --- a/web/docs/en-US/development/development-guide.md +++ /dev/null @@ -1,382 +0,0 @@ -# 🛠️ Development Guide - -This guide focuses on setting up a development environment and the development workflow for ApeRAG. This is designed for developers looking to contribute to ApeRAG or run it locally for development purposes. - -## 🚀 Development Environment Setup - -Follow these steps to set up ApeRAG from source code for development: - -### 1. 📂 Clone the Repository and Setup Environment - -First, get the source code and configure environment variables: - -```bash -git clone https://github.com/apecloud/ApeRAG.git -cd ApeRAG -cp envs/env.template .env -``` - -Edit the `.env` file to configure your AI service settings if needed. The default settings work with the local database services started in the next step. - -### 2. 📋 System Prerequisites - -Before you begin, ensure your system has: - -* **Node.js**: Version 20 or higher is recommended for frontend development. [Download Node.js](https://nodejs.org/) -* **Docker & Docker Compose**: Required for running database services locally. [Download Docker](https://docs.docker.com/get-docker/) - -**Note**: Python 3.11 is required but will be automatically managed by `uv` in the next steps. - -### 3. 🗄️ Start Database Services - -Use Docker Compose to start the essential database services: - -```bash -# Start core databases: PostgreSQL, Redis, Qdrant, Elasticsearch -make infra-up -``` - -This will start all required database services in the background. The default connection settings in your `.env` file are pre-configured to work with these services. - -
-
-
-### 4. ⚙️ Setup Development Environment
-
-Create Python virtual environment and setup development tools:
-
-```bash
-make env-dev
-```
-
-This command will:
-* Install `uv` if not already available
-* Create a Python 3.11 virtual environment (located in `.venv/`)
-* Install development tools (redocly, openapi-generator-cli, etc.)
-* Install pre-commit hooks for code quality
-* Install addlicense tool for license management
-
-**Activate the virtual environment:**
-```bash
-source .venv/bin/activate
-```
-
-You'll know it's active when you see `(.venv)` in your terminal prompt.
-
-### 5. 📦 Install Dependencies
-
-Install all backend and frontend dependencies:
-
-```bash
-make env-install
-```
-
-This command will:
-* Install all Python backend dependencies from `pyproject.toml` into the virtual environment
-* Install frontend Node.js dependencies using `yarn`
-
-### 6. 🔄 Apply Database Migrations
-
-Setup the database schema:
-
-```bash
-make db-migrate
-```
-
-### 7. ▶️ Start Development Services
-
-Now you can start the development services. Open separate terminal windows/tabs for each service:
-
-**Terminal 1 - Backend API Server:**
-```bash
-make serve-api
-```
-This starts the FastAPI development server at `http://localhost:8000` with auto-reload on code changes.
-
-**Terminal 2 - Celery Worker:**
-```bash
-make serve-worker
-```
-This starts the Celery worker for processing asynchronous background tasks.
-
-**Terminal 3 - Frontend (Optional):**
-```bash
-make serve-web
-```
-This starts the frontend development server at `http://localhost:3000` with hot reload.
-
-### 8. 🌐 Access ApeRAG
-
-With the services running, you can access:
-* **Frontend UI**: http://localhost:3000 (if started)
-* **Backend API**: http://localhost:8000
-* **API Documentation**: http://localhost:8000/docs
-
-### 9. ⏹️ Stopping Services
-
-To stop the development environment:
-
-**Stop Database Services:**
-```bash
-# Stop database services (data preserved)
-make stack-down
-
-# Stop services and remove all data volumes
-make stack-down REMOVE_VOLUMES=1
-```
-
-**Stop Development Services:**
-- Backend API Server: Press `Ctrl+C` in the terminal running `make serve-api`
-- Celery Worker: Press `Ctrl+C` in the terminal running `make serve-worker`
-- Frontend Server: Press `Ctrl+C` in the terminal running `make serve-web`
-
-**Data Management:**
-- `make stack-down` - Stops services but preserves all data (PostgreSQL, Redis, Qdrant, etc.)
-- `make stack-down REMOVE_VOLUMES=1` - Stops services and **⚠️ permanently deletes all data**
-- You can run `make stack-down REMOVE_VOLUMES=1` even after already running `make stack-down`
-
-**Verify Data Removal:**
-```bash
-# Check if volumes still exist
-docker volume ls | grep aperag
-
-# Should return no results after REMOVE_VOLUMES=1
-```
-
-Now you have ApeRAG running locally from source code, ready for development! 🎉
-
-## ❓ Common Development Tasks
-
-### Q: 🔧 How do I add or modify a REST API endpoint?
-
-**Complete workflow:**
-1. Edit OpenAPI specification: `aperag/api/paths/[endpoint-name].yaml`
-2. Regenerate backend models:
- ```bash
- make api-generate-models # This runs merge-openapi internally
- ```
-3. Implement backend view: `aperag/views/[module].py`
-4. Generate frontend TypeScript client:
- ```bash
- make api-generate-sdk # Updates frontend/src/api/
- ```
-5. Test the API:
- ```bash
- make test-all
- # ✅ Check live docs: http://localhost:8000/docs
- ```
-
-### Q: 🗃️ How do I modify database models/schema?
-
-**Database migration workflow:**
-1. Edit SQLModel classes in `aperag/db/models.py`
-2. Generate migration file:
- ```bash
- make db-revision # Creates new migration in migration/versions/
- ```
-3. Apply migration to database:
- ```bash
- make db-migrate # Updates database schema
- ```
-4. Update related code (repositories in `aperag/db/repositories/`, services in `aperag/service/`)
-5. Verify changes:
- ```bash
- make test-all # ✅ Ensure everything works
- ```
-
-### Q: ⚡ How do I add a new feature with background processing?
-
-**Feature implementation workflow:**
-1. Implement feature components:
- - Backend logic: `aperag/[module]/`
- - Async tasks: `aperag/tasks/`
- - Database models: `aperag/db/models.py`
-2. Update API and generate code:
- ```bash
- make db-revision # Generate migration files
- make db-migrate # Apply database changes
- make api-generate-models # Update Pydantic models
- make api-generate-sdk # Update TypeScript client
- ```
-3. Quality assurance:
- ```bash
- make format && make lint && make test-all
- ```
-
-### Q: 🧪 How do I run unit tests and e2e tests?
-
-**Unit Tests (Fast, No External Dependencies):**
-```bash
-# Run all unit tests
-make test-unit
-
-# Run specific test file
-uv run pytest tests/unit_test/test_model_service.py -v
-
-# Run specific test class or function
-uv run pytest tests/unit_test/test_model_service.py::TestModelService::test_get_models -v
-
-# Run tests with coverage
-uv run pytest tests/unit_test/ --cov=aperag --cov-report=html
-```
-
-**E2E Tests (Require Running Services):**
-```bash
-# Setup: Start required services first
-make infra-up # 🗄️ Start databases
-make serve-api # 🚀 Start API server (separate terminal)
-
-# Run all e2e tests
-make test-e2e
-
-# Run specific e2e test modules
-uv run pytest tests/e2e_test/test_chat/ -v
-uv run pytest tests/e2e_test/graphstorage/ -v
-
-# Run with detailed output and no capture
-uv run pytest tests/e2e_test/test_specific.py -v -s
-
-# Performance benchmarks (with timing)
-make test-e2e-perf
-```
-
-**Complete Test Suite:**
-```bash
-# Run everything (unit + e2e)
-make test-all
-
-# Test with different configurations
-make infra-up WITH_NEO4J=1 # Test with Neo4j instead of PostgreSQL
-make test-all
-```
-
-### Q: 🐛 How do I debug failing tests?
-
-**Debugging workflow:**
-1. Run failing test in isolation:
- ```bash
- # Single test with full output
- uv run pytest tests/unit_test/test_failing.py::test_specific_function -v -s
-
- # Stop on first failure
- uv run pytest tests/unit_test/ -x --tb=short
- ```
-2. For e2e test failures, ensure services are running:
- ```bash
- make infra-up # Database services
- make serve-api # API server
- make serve-worker # Background workers (if testing async tasks)
- ```
-3. Use debugging tools:
- ```bash
- # Run with pdb debugger
- uv run pytest tests/unit_test/test_failing.py --pdb
-
- # Capture logs during test
- uv run pytest tests/e2e_test/test_failing.py --log-cli-level=DEBUG
- ```
-4. Fix and retest:
- ```bash
- make format # Auto-fix style issues
- make lint # Check remaining issues
- uv run pytest tests/path/to/fixed_test.py -v # Verify fix
- ```
-
-### Q: 📊 How do I run RAG evaluation and analysis?
-
-**Evaluation workflow:**
-```bash
-# Ensure environment is ready
-make infra-up WITH_NEO4J=1 # Use Neo4j for better graph performance
-make serve-api
-make serve-worker
-
-# Run comprehensive RAG evaluation
-make evaluate # 📊 Runs aperag.evaluation.run module
-
-# 📈 Check evaluation reports in tests/report/
-```
-
-### Q: 📦 How do I update dependencies safely?
-
-**Python dependencies:**
-1. Edit `pyproject.toml` (add/update packages)
-2. Update virtual environment:
- ```bash
- make env-install # Syncs all groups and extras with uv
- make test-all # Verify compatibility
- ```
-
-**Frontend dependencies:**
-1. Edit `frontend/package.json`
-2. Update and test:
- ```bash
- cd frontend && yarn install
- make serve-web # Test frontend compilation
- make api-generate-sdk # Ensure API client still works
- ```
-
-### Q: 🚀 How do I prepare code for production deployment?
-
-**Pre-deployment checklist:**
-1. Code quality validation:
- ```bash
- make format # Auto-fix all style issues
- make lint # Verify no style violations
- make static-check # MyPy type checking
- ```
-2. Comprehensive testing:
- ```bash
- make test-all # All unit + e2e tests
- make test-e2e-perf # Performance benchmarks
- ```
-3. API consistency:
- ```bash
- make api-generate-models # Ensure models match OpenAPI spec
- make api-generate-sdk # Update frontend client
- ```
-4. Database migrations:
- ```bash
- make db-revision # Generate any pending migrations
- ```
-5. Full-stack integration test:
- ```bash
- make stack-up WITH_NEO4J=1 WITH_DOCRAY=1 # Production-like setup
- # Manual testing at http://localhost:3000/web/
- make stack-down
- ```
-
-### Q: 🔄 How do I completely reset my development environment?
-
-**Nuclear reset (destroys all data):**
-```bash
-make stack-down REMOVE_VOLUMES=1 # ⚠️ Stop services + delete ALL data
-make env-clean # 🧹 Clean temporary files
-
-# Restart fresh
-make infra-up # 🗄️ Fresh databases
-make db-migrate # 🔄 Apply all migrations
-make serve-api # 🚀 Start API server
-make serve-worker # ⚡ Start background workers
-```
-
-**Soft reset (preserve data):**
-```bash
-make stack-down # ⏹️ Stop services, keep data
-make infra-up # 🗄️ Restart databases
-make db-migrate # 🔄 Apply any new migrations
-```
-
-**Reset just Python environment:**
-```bash
-rm -rf .venv/ # 🗑️ Remove virtual environment
-make env-dev # ⚙️ Recreate everything
-source .venv/bin/activate # ✅ Reactivate
-```
diff --git a/web/docs/en-US/images/dify/aperag-banner.png b/web/docs/en-US/images/dify/aperag-banner.png
deleted file mode 100644
index 338290248..000000000
Binary files a/web/docs/en-US/images/dify/aperag-banner.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step1-subscribe-collection.png b/web/docs/en-US/images/dify/step1-subscribe-collection.png
deleted file mode 100644
index 3a9ede8ad..000000000
Binary files a/web/docs/en-US/images/dify/step1-subscribe-collection.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step2-add-mcp.png b/web/docs/en-US/images/dify/step2-add-mcp.png
deleted file mode 100644
index 92eb2154d..000000000
Binary files a/web/docs/en-US/images/dify/step2-add-mcp.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step2-api-key.png b/web/docs/en-US/images/dify/step2-api-key.png
deleted file mode 100644
index 555b0b7de..000000000
Binary files a/web/docs/en-US/images/dify/step2-api-key.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step2-configure-mcp.png b/web/docs/en-US/images/dify/step2-configure-mcp.png
deleted file mode 100644
index 149787ef9..000000000
Binary files a/web/docs/en-US/images/dify/step2-configure-mcp.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step2-mcp-success.png b/web/docs/en-US/images/dify/step2-mcp-success.png
deleted file mode 100644
index 7f91bc07e..000000000
Binary files a/web/docs/en-US/images/dify/step2-mcp-success.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step3-create-app.png b/web/docs/en-US/images/dify/step3-create-app.png
deleted file mode 100644
index a41dc7cdf..000000000
Binary files a/web/docs/en-US/images/dify/step3-create-app.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step3-select-agent.png b/web/docs/en-US/images/dify/step3-select-agent.png
deleted file mode 100644
index ed3b0c00a..000000000
Binary files a/web/docs/en-US/images/dify/step3-select-agent.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step4-configure-agent.png b/web/docs/en-US/images/dify/step4-configure-agent.png
deleted file mode 100644
index ac1ea12af..000000000
Binary files a/web/docs/en-US/images/dify/step4-configure-agent.png and /dev/null differ
diff --git a/web/docs/en-US/images/dify/step4-test-agent.png b/web/docs/en-US/images/dify/step4-test-agent.png
deleted file mode 100644
index 92f90d101..000000000
Binary files a/web/docs/en-US/images/dify/step4-test-agent.png and /dev/null differ
diff --git a/web/docs/en-US/integration/_category.yaml b/web/docs/en-US/integration/_category.yaml
deleted file mode 100644
index a3e10338d..000000000
--- a/web/docs/en-US/integration/_category.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-title: Integration
-position: 2
diff --git a/web/docs/en-US/integration/dify.md b/web/docs/en-US/integration/dify.md
deleted file mode 100644
index f6595da66..000000000
--- a/web/docs/en-US/integration/dify.md
+++ /dev/null
@@ -1,168 +0,0 @@
----
-title: Integrating ApeRAG with Dify
-description: Quick integration of ApeRAG's Graph RAG capabilities via MCP protocol
-keywords: Dify, ApeRAG, MCP, Graph RAG
----
-
-# Integrating ApeRAG with Dify
-
-ApeRAG is a production-grade RAG platform with multimodal indexing, AI agents, MCP support, and scalable K8s deployment capabilities. It helps users build complex AI applications with **hybrid retrieval**, **multimodal document processing**, and **enterprise-grade management**.
-
-**Core Features**:
-- Unlike "standard" RAG, ApeRAG implements **Graph-RAG**, building knowledge graphs to understand deep relationships between data elements
-- Integrates **MinerU**, designed for complex documents, scientific papers, and financial reports, accurately extracting tables, formulas, and engineering diagrams
-- Full Kubernetes support with built-in **high availability**, **scalability**, and **enterprise-grade management**
-
-## Video Demo
-
-Advanced Database Options
- -```bash -# Use Neo4j instead of PostgreSQL for graph storage -make infra-up WITH_NEO4J=1 -``` - -
-
-
-
-## Step 1: Prepare Knowledge Base
-
-Open your ApeRAG web UI (see [Quick Start](../../../../README.md#quick-start); with Docker Compose this is typically http://localhost:3000/web/). Sign in and select or import a knowledge base. This walkthrough uses the Romance of the Three Kingdoms example—click **Subscribe**.
-
-
-
-
-
-## Step 2: Configure MCP Server
-
-### 2.1 Add MCP Server
-
-Go to Dify - Tools - MCP, click Add MCP Server.
-
-
-
-
-
-
-### 2.2 Fill Configuration
-
-Fill in Server URL: `http://localhost:8000/mcp/` (use `https://
-
-
-
-
-
-
-
-
-
-### 2.3 Success
-
-MCP Server added successfully.
-
-
-
-
-
-
-## Step 3: Create Agent Application
-
-### 3.1 Create App
-
-Go to Dify - Studio, click Create Application.
-
-
-
-
-
-
-### 3.2 Select Type
-
-Click More Basic Application Types, select **Agent** type, name it, and click Create.
-
-
-
-
-
-
-## Step 4: Configure Agent
-
-Click Agent, input Prompt, add the ApeRAG MCP tool, select the LLM in the top-right corner, click Publish to use.
-
-
-
-
-
-
-
-
-
-
-
-### Prompt Reference
-
-```markdown
-# ApeRAG Smart Assistant
-
-You are an advanced AI research assistant powered by ApeRAG's hybrid search capabilities. Your mission is to help users accurately and autonomously find, understand, and synthesize information from knowledge bases and the web.
-
-## Core Behaviors
-
-**Autonomous Research**: Work independently until user queries are fully resolved. Search multiple sources, analyze findings, and provide comprehensive answers without waiting for permission.
-
-**Language Intelligence**: Always respond in the language the user asks in. When users ask in Chinese, respond in Chinese regardless of source language.
-
-**Visual Thinking**: **[Critical]** You are an assistant that prefers visual explanations. For any information involving entity relationships, processes, or structures, you must prioritize visualization.
-
-**Complete Solutions**: Explore from multiple angles, cross-validate sources, and ensure comprehensive coverage before responding.
-
-## Search Strategy
-
-### Priority System
-1. **User-specified knowledge base** (mentioned via "@"): Strictly limit search to specified base
-2. **Unspecified knowledge base**: Autonomously discover and search relevant bases
-3. **Web search** (if enabled): Supplement information
-4. **Clear attribution**: Always cite sources
-
-### Search Execution
-- **Knowledge base search**: Use vector + graph search by default
-- **Result processing logic**:
- 1. Execute search
- 2. **Detect graph data**: Check if search results contain `entities` and `relationships`
- 3. **Mandatory visualization**: If search results contain non-empty entity or relation data, **you must** call the `create_diagram` tool
- 4. **Content filtering**: Ignore irrelevant results
-
-## Available Tools
-
-### Knowledge Management
-- `list_collections()`: Discover available knowledge sources
-- `search_collection(collection_id, query, ...)`: **[Primary tool]** Hybrid search in persistent knowledge bases
-- `search_chat_files(chat_id, query, ...)`: **[Chat only]** Search only files temporarily uploaded in current chat session
-- `create_diagram(content)`: **[Mandatory tool]** When search results contain structured info (entities/relations), must call this tool to generate Mermaid diagrams
-
-### Web Intelligence
-- `web_search(query, ...)`: Multi-engine web search
-- `web_read(url_list, ...)`: Extract and analyze web content
-
-## Response Format
-
-### Direct Answer
-[Clear, actionable answer in user's language]
-
-### Comprehensive Analysis
-[Detailed explanation with context and insights]
-
-### Knowledge Graph Visualization
-[Tool-generated diagram displayed here]
-*(Only show after successfully calling create_diagram. Displays entity relationships from search results.)*
-
-### Supporting Evidence
-- [Knowledge Base Name]: [Key Findings]
-
-**Web Sources** (if enabled):
-- [Title] ([Domain]) - [Key Points]
-```
-
----
-
-Integrating ApeRAG with Dify is very simple. Once integrated, you can not only experience Dify's platform features but also enjoy **ApeRAG's powerful Graph-RAG capabilities**!
-
-**GitHub**: https://github.com/apecloud/ApeRAG
diff --git a/web/docs/en-US/integration/mcp-api.md b/web/docs/en-US/integration/mcp-api.md
deleted file mode 100644
index 798b54374..000000000
--- a/web/docs/en-US/integration/mcp-api.md
+++ /dev/null
@@ -1,333 +0,0 @@
----
-title: MCP API
-description: Model Context Protocol API Documentation
----
-
-# MCP API
-
-ApeRAG provides standardized tool interfaces through [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), allowing AI assistants (Claude Desktop, Cursor, Dify, etc.) to directly access your knowledge bases.
-
-## Quick Start
-
-### Configuration Example
-
-For Claude Desktop, add to configuration file:
-
-```json
-{
- "mcpServers": {
- "aperag": {
- "url": "http://localhost:8000/mcp/",
- "headers": {
- "Authorization": "Bearer your-api-key-here"
- }
- }
- }
-}
-```
-
-### Authentication
-
-Two authentication methods supported (by priority):
-
-1. **HTTP Authorization Header** (Recommended): `Authorization: Bearer your-api-key`
-2. **Environment Variable** (Fallback): `APERAG_API_KEY=your-api-key`
-
-> **Get API Key**: Login to ApeRAG, create or copy your API Key from settings
-
-## Available Tools
-
-### 1. list_collections
-
-List all accessible knowledge bases.
-
-**Parameters**: None
-
-**Returns**:
-```json
-{
- "items": [
- {
- "id": "collection-id",
- "title": "Collection Title",
- "description": "Collection Description"
- }
- ]
-}
-```
-
-### 2. search_collection
-
-Search in knowledge bases with multiple retrieval methods.
-
-**Core Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `collection_id` | string | Required | Knowledge base ID |
-| `query` | string | Required | Search query |
-| `use_vector_index` | bool | true | Vector retrieval (semantic search) |
-| `use_fulltext_index` | bool | true | Full-text retrieval (keyword matching) |
-| `use_graph_index` | bool | true | Graph retrieval (relation query) |
-| `use_summary_index` | bool | true | Summary retrieval |
-| `use_vision_index` | bool | true | Vision retrieval (image search) |
-| `rerank` | bool | true | AI reranking |
-| `topk` | int | 5 | Results per method |
-
-**Return Format**:
-```json
-{
- "query": "your question",
- "items": [
- {
- "rank": 1,
- "score": 0.95,
- "content": "relevant content",
- "source": "document name",
- "recall_type": "vector_search|graph_search|fulltext_search|summary_search",
- "metadata": {
- "page_idx": 0,
- "document_id": "doc-id",
- "collection_id": "col-id",
- "indexer": "text|vision"
- }
- }
- ]
-}
-```
-
-**Image Handling**:
-
-If `metadata.indexer == "vision"`, it's an image:
-- Empty `content`: Retrieved via multimodal vector
-- Non-empty `content`: Contains image description
-
-Image URL format:
-```python
-m = item.metadata
-asset_url = f"asset://{m['asset_id']}?document_id={m['document_id']}&collection_id={m['collection_id']}&mime_type={m['mimetype']}"
-```
-
-**Usage Examples**:
-
-```python
-# Default search (recommended) - all methods enabled
-results = search_collection(
- collection_id="abc123",
- query="How to deploy applications?"
-)
-
-# Vector + Graph only
-results = search_collection(
- collection_id="abc123",
- query="deployment strategies",
- use_vector_index=True,
- use_fulltext_index=False,
- use_graph_index=True,
- use_summary_index=False,
- topk=10
-)
-```
-
-### 3. search_chat_files
-
-Search in temporary files from chat session.
-
-**When to Use**:
-- ✅ User uploaded files in current conversation
-- ✅ Analyzing temporary documents in chat
-- ❌ Don't use for persistent knowledge bases (use `search_collection`)
-
-**Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `chat_id` | string | Required | Chat ID |
-| `query` | string | Required | Search query |
-| `use_vector_index` | bool | true | Vector retrieval |
-| `use_fulltext_index` | bool | true | Full-text retrieval |
-| `rerank` | bool | true | Reranking |
-| `topk` | int | 5 | Results count |
-
-**Return Format**: Same as `search_collection`
-
-### 4. web_search
-
-Search the internet.
-
-**Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | "" | Search keywords |
-| `max_results` | int | 5 | Results count |
-| `source` | string | "" | Specific domain (e.g., `vercel.com`) |
-| `timeout` | int | 30 | Timeout (seconds) |
-| `locale` | string | "en-US" | Language locale |
-
-**Usage Patterns**:
-
-```python
-# Regular search
-web_search(query="ApeRAG 2025")
-
-# Site-specific search
-web_search(query="deployment docs", source="vercel.com")
-
-```
-
-### 5. web_read
-
-Read webpage content.
-
-**Parameters**:
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `url_list` | list[str] | Required | URL list |
-| `timeout` | int | 30 | Timeout (seconds) |
-| `max_concurrent` | int | 5 | Max concurrent requests |
-
-**Returns**:
-```json
-{
- "results": [
- {
- "status": "success",
- "url": "https://example.com",
- "title": "Page Title",
- "content": "Extracted text",
- "word_count": 1234
- }
- ]
-}
-```
-
-**Example**:
-```python
-# Read single page
-web_read(url_list=["https://example.com/article"])
-
-# Batch read
-web_read(
- url_list=["https://example.com/page1", "https://example.com/page2"],
- max_concurrent=2
-)
-```
-
-## Practical Examples
-
-### Example 1: Knowledge Base Q&A
-
-```python
-# 1. List all knowledge bases
-collections = list_collections()
-
-# 2. Select a knowledge base
-collection_id = collections.items[0].id
-
-# 3. Search (all methods enabled by default)
-results = search_collection(
- collection_id=collection_id,
- query="How to optimize performance?"
-)
-
-# 4. Process results
-for item in results.items:
- print(f"[{item.recall_type}] {item.content}")
- print(f"Source: {item.source}, Score: {item.score}\n")
-```
-
-### Example 2: Graph Visualization
-
-```python
-# Search with graph retrieval
-results = search_collection(
- collection_id="abc123",
- query="relationship between Liu Bei and Zhuge Liang",
- use_graph_index=True
-)
-
-# Check for graph data
-if results.graph_search and results.graph_search.entities:
- print("Entities:", results.graph_search.entities)
- print("Relationships:", results.graph_search.relationships)
- # Use this data to generate knowledge graph visualization
-```
-
-### Example 3: Hybrid Search (KB + Web)
-
-```python
-# 1. Search web
-web_results = web_search(query="latest AI developments", max_results=3)
-urls = [r.url for r in web_results.results]
-
-# 2. Read web content
-web_content = web_read(url_list=urls)
-
-# 3. Search internal KB
-kb_results = search_collection(
- collection_id="ai-knowledge",
- query="AI development trends"
-)
-
-# 4. Synthesize
-print("=== Web Results ===")
-for r in web_results.results:
- print(f"{r.title}: {r.url}")
-
-print("\n=== Internal Knowledge ===")
-for item in kb_results.items:
- print(f"{item.content[:100]}...")
-```
-
-## Best Practices
-
-### Performance Tips
-
-1. **Reasonable topk**:
- - Too large increases LLM context consumption
- - Too small may miss important information
- - Recommended: 5-10
-
-2. **Selective Retrieval**:
- - Not all queries need full-text search
- - Full-text may return large amounts of text
- - Choose methods based on query type
-
-3. **Timeout Settings**:
- - Graph retrieval may be slow (default 120s)
- - Web search: 30-60s recommended
- - Batch URL read: 60s+ recommended
-
-### Common Issues
-
-**Q: No search results?**
-- Check if collection ID is correct
-- Confirm knowledge base indexing is complete
-- Try different retrieval method combinations
-
-**Q: Graph data empty?**
-- Confirm knowledge base has Graph index enabled
-- Simple documents may not contain obvious entity relationships
-
-**Q: Images not showing?**
-- Check `metadata.indexer == "vision"`
-- Use `asset://` protocol for URL
-- Ensure all required parameters included (asset_id, document_id, collection_id)
-
-## Tool Comparison
-
-| Tool | Purpose | Use Case |
-|------|---------|----------|
-| `list_collections` | List knowledge bases | See available resources |
-| `search_collection` | Search knowledge base | Primary search tool for persistent knowledge |
-| `search_chat_files` | Search chat files | Analyze temporary files uploaded in conversation |
-| `web_search` | Search internet | Get real-time or external information |
-| `web_read` | Read webpage | Extract full webpage content |
-
-## Related Links
-
-- **MCP Protocol**: https://modelcontextprotocol.io/
-- **ApeRAG GitHub**: https://github.com/apecloud/ApeRAG
-- **API Docs**: http://localhost:8000/docs (local deployment)
diff --git a/web/docs/zh-CN/deployment/_category.yaml b/web/docs/zh-CN/deployment/_category.yaml
deleted file mode 100644
index eba379f30..000000000
--- a/web/docs/zh-CN/deployment/_category.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-title: 部署
-position: 3
diff --git a/web/docs/zh-CN/deployment/build-docker-image.md b/web/docs/zh-CN/deployment/build-docker-image.md
deleted file mode 100644
index a1cd89c5c..000000000
--- a/web/docs/zh-CN/deployment/build-docker-image.md
+++ /dev/null
@@ -1,50 +0,0 @@
----
-title: 构建 Docker 镜像
-description: 如何构建 ApeRAG 容器镜像
----
-
-# 构建指南
-
-本节介绍如何构建 ApeRAG 容器镜像。这主要适用于需要创建自己的构建或部署到"快速开始"中未涵盖的环境的用户。
-
-## 构建容器镜像
-
-项目使用 Docker 和 `make` 命令来构建容器镜像。
-
-* **本地平台构建**：
- 这些命令为您当前机器的架构构建镜像。
- ```bash
- # 为本地平台构建所有必要的镜像
- make build-local
-
- # 仅为本地平台构建后端镜像
- make build-aperag-local
-
- # 仅为本地平台构建前端镜像
- make build-aperag-frontend-local
- ```
-
-* **多平台构建**：
- 这些命令为多种架构（例如 amd64、arm64）构建镜像。这需要设置和配置 Docker Buildx。
- ```bash
- # 为多个平台构建所有必要的镜像
- make build
-
- # 仅为多个平台构建后端镜像
- make build-aperag
-
- # 仅为多个平台构建前端镜像
- make build-aperag-frontend
- ```
- 您可以使用 `PLATFORMS` 变量指定目标平台，例如：
- ```bash
- make build PLATFORMS=linux/amd64,linux/arm64
- ```
-
-## 部署
-
-有关常见的部署方法，请参考主 README 中的"快速开始"部分：
-* [Kubernetes 快速开始](../README-zh.md#kubernetes-部署推荐生产环境)
-* [Docker Compose 快速开始](../README-zh.md#快速开始)
-
-对于自定义部署，您需要调整这些方法或使用构建的容器镜像与您选择的编排平台配合使用。确保所有必需的服务（数据库、后端、前端、Celery worker）都正确配置并能够相互通信。
\ No newline at end of file
diff --git a/web/docs/zh-CN/design/_category.yaml b/web/docs/zh-CN/design/_category.yaml
deleted file mode 100644
index 1211298a0..000000000
--- a/web/docs/zh-CN/design/_category.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-title: 设计
-position: 1
diff --git a/web/docs/zh-CN/design/architecture.md b/web/docs/zh-CN/design/architecture.md
deleted file mode 100644
index 572e2ed72..000000000
--- a/web/docs/zh-CN/design/architecture.md
+++ /dev/null
@@ -1,850 +0,0 @@
----
-title: 系统架构
-description: ApeRAG 架构设计与核心组件详解
-keywords: ApeRAG, 架构, RAG, 知识图谱, LightRAG
-position: 1
----
-
-# ApeRAG 系统架构
-
-## 1. 什么是 ApeRAG
-
-ApeRAG 是一个**开放的、Agentic 的 Graph RAG 平台**。它不仅仅是一个简单的向量检索系统，而是将知识图谱、多模态检索和智能 Agent 深度融合的生产级解决方案。
-
-传统的 RAG 系统主要依赖向量相似度检索，虽然能找到语义相关的内容，但往往缺乏对知识之间关系的理解。ApeRAG 的核心创新在于：
-
-- **Graph RAG**：从文档中自动提取实体（人物、地点、概念）和关系，构建知识图谱，理解知识之间的关联
-- **Agentic**：内置智能 Agent，能够自主规划、调用工具、多轮对话，提供更智能的问答体验
-- **开放集成**：通过 **RESTful API** 和 **MCP 协议**对外暴露能力，可以轻松集成到 Dify、Claude、Cursor 等外部系统
-
-### 核心优势
-
-与传统 RAG 方案相比，ApeRAG 提供了：
-
-- **更强的文档处理能力**：支持 PDF、Word、Excel 等多种格式，能处理复杂的表格、公式、图片
-- **多种检索方式**：向量检索、全文检索、图谱检索，三者互补，各取所长
-- **知识关联理解**：通过知识图谱理解概念之间的关系，而不仅仅是文本相似度
-- **开放的集成能力**：RESTful API + MCP 协议，可以作为 Dify、Claude Desktop、Cursor 的知识后端
-- **生产级架构**：异步处理、多存储、高并发，可以直接用于生产环境
-
-### 整体架构一览
-
-```mermaid
-graph TB
- User[用户] --> Frontend[Web 前端]
- User --> External[外部系统
-Dify/Claude/Cursor] - - Frontend --> API[RESTful API] - External --> MCP[MCP 协议] - - API --> DocProcess[文档处理] - API --> Search[检索服务] - API --> Agent[Agent 对话] - MCP --> Search - MCP --> Agent - - DocProcess --> Tasks[异步任务层] - Tasks --> Storage[存储层] - - Search --> Storage - Agent --> Search - - Storage --> PG[(PostgreSQL)] - Storage --> Qdrant[(Qdrant
向量库)] - Storage --> ES[(Elasticsearch
全文搜索)] - Storage --> Neo4j[(Neo4j
图数据库)] - Storage --> MinIO[(MinIO
文件存储)] - - style User fill:#e1f5ff - style Frontend fill:#bbdefb - style External fill:#bbdefb - style API fill:#90caf9 - style MCP fill:#90caf9 - style DocProcess fill:#fff59d - style Search fill:#fff59d - style Agent fill:#fff59d - style Tasks fill:#c5e1a5 - style Storage fill:#ffccbc -``` - -## 2. 系统分层架构 - -ApeRAG 采用清晰的分层设计，每一层各司其职： - -```mermaid -graph TB - subgraph Layer1[客户端层] - Web[Web 前端
Next.js] - Dify[Dify] - Cursor[Cursor] - Claude[Claude Desktop] - end - - subgraph Layer2[接口层] - API[RESTful API
FastAPI] - MCP[MCP Server
Model Context Protocol] - end - - subgraph Layer3[服务层] - CollSvc[Collection 服务] - DocSvc[文档服务] - SearchSvc[检索服务] - GraphSvc[图谱服务] - AgentSvc[Agent 服务] - end - - subgraph Layer4[任务层] - Celery[Celery Worker
异步任务] - MinerU[MinerU
文档解析] - end - - subgraph Layer5[存储层] - PG[(PostgreSQL)] - Qdrant[(Qdrant)] - ES[(Elasticsearch)] - Neo4j[(Neo4j)] - Redis[(Redis)] - MinIO[(MinIO)] - end - - Web --> API - Dify --> MCP - Cursor --> MCP - Claude --> MCP - - API --> CollSvc - API --> DocSvc - API --> SearchSvc - API --> GraphSvc - API --> AgentSvc - - MCP --> SearchSvc - MCP --> AgentSvc - - CollSvc --> Celery - DocSvc --> Celery - GraphSvc --> Celery - - Celery --> MinerU - Celery --> PG - Celery --> Qdrant - Celery --> ES - Celery --> Neo4j - Celery --> MinIO - - SearchSvc --> PG - SearchSvc --> Qdrant - SearchSvc --> ES - SearchSvc --> Neo4j - - style Layer1 fill:#e3f2fd - style Layer2 fill:#f3e5f5 - style Layer3 fill:#fff3e0 - style Layer4 fill:#e8f5e9 - style Layer5 fill:#fce4ec -``` - -**各层职责说明**： - -- **客户端层**：多种接入方式，Web UI 用于管理，MCP 客户端（Dify、Cursor、Claude 等）用于集成 -- **接口层**：RESTful API（传统 HTTP 接口）和 MCP Server（AI 工具协议）并行提供服务 -- **服务层**：核心业务逻辑，协调各种资源完成具体功能 -- **任务层**：处理耗时操作（文档解析、索引构建），保证 API 快速响应 -- **存储层**：多种存储系统，针对不同数据类型选择最优方案 - -## 3. 文档处理全流程 - -这是 ApeRAG 的核心能力之一。从一个 PDF 文件上传，到最终可以被检索，经历了一系列精心设计的处理步骤。 - -### 3.1 文档上传与解析 - -当你上传一个文档时，ApeRAG 会自动识别格式并选择合适的解析器： - -```mermaid -flowchart TD - Upload[用户上传文档] --> Detect[格式检测] - - Detect --> |PDF| MinerU[MinerU 解析器] - Detect --> |Word/Excel| MarkItDown[MarkItDown 解析器] - Detect --> |Markdown| DirectParse[直接解析] - Detect --> |图片| OCR[OCR 识别] - - MinerU --> Extract[内容提取] - MarkItDown --> Extract - DirectParse --> Extract - OCR --> Extract - - Extract --> Parts[文档片段
Parts 对象] - - style Upload fill:#e1f5ff - style Extract fill:#c5e1a5 - style Parts fill:#fff59d -``` - -**MinerU 的强大之处**： - -- 能准确识别复杂 PDF 的表格结构，保留表格内容的完整性 -- 提取 LaTeX 数学公式，保持公式的可读性 -- 对扫描版 PDF 进行 OCR，支持中英文混排 -- 识别文档中的图片区域，支持图片内容理解 - -### 3.2 智能分块策略 - -文档解析后，需要切分成合适大小的块（chunk）。这个步骤很关键，分块太大会影响检索精度，太小会丢失上下文。 - -```mermaid -flowchart TD - Parts[文档片段] --> Rechunk[智能重分块] - - Rechunk --> Analysis[分析文档结构] - Analysis --> Hierarchy[识别标题层级] - Hierarchy --> Group[按标题分组] - - Group --> Check{块大小检查} - Check --> |过大| Split[语义分割] - Check --> |合适| Chunks[最终块] - Split --> Chunks - - Chunks --> AddContext[添加上下文] - AddContext --> FinalChunks[带上下文的文档块] - - style Rechunk fill:#bbdefb - style Split fill:#ffccbc - style FinalChunks fill:#c5e1a5 -``` - -**分块策略的特点**： - -- **保持语义完整性**：尽量不在句子中间切断 -- **保留标题上下文**：每个块都知道自己属于哪个章节 -- **层级化分割**：先按段落分，不行再按句子分，最后才按字符分 -- **智能合并**：相邻的小标题块会被合并，避免信息碎片化 - -分块参数配置： -- 默认块大小：1200 tokens（约 800-1000 个中文字符） -- 重叠大小：100 tokens（保证上下文连续性） - -### 3.3 多索引并行构建 - -文档分块后，会同时创建多种索引。每种索引有不同的用途，互相补充： - -| 索引类型 | 适用场景 | 存储位置 | 检索方式 | -|---------|---------|---------|---------| -| **向量索引** | 语义相似问题，比如"如何优化性能" | Qdrant | 余弦相似度 | -| **全文索引** | 精确关键词搜索，比如"PostgreSQL 配置" | Elasticsearch | BM25 算法 | -| **图谱索引** | 关系型问题，比如"A 和 B 有什么联系" | PostgreSQL/Neo4j | 图遍历 | -| **摘要索引** | 快速了解文档概要 | PostgreSQL | 向量匹配 | -| **视觉索引** | 图片内容搜索 | Qdrant | 多模态向量 | - -```mermaid -flowchart LR - Chunks[文档块] --> IndexMgr[索引管理器] - - IndexMgr --> VectorIdx[向量索引创建] - IndexMgr --> FulltextIdx[全文索引创建] - IndexMgr --> GraphIdx[图谱索引创建] - IndexMgr --> VisionIdx[视觉索引创建] - - VectorIdx --> Qdrant1[(Qdrant)] - FulltextIdx --> ES[(Elasticsearch)] - GraphIdx --> Graph[(Neo4j/PG)] - VisionIdx --> Qdrant2[(Qdrant)] - - style IndexMgr fill:#fff59d - style VectorIdx fill:#bbdefb - style FulltextIdx fill:#c5e1a5 - style GraphIdx fill:#ffccbc - style VisionIdx fill:#e1bee7 -``` - -**并行构建的优势**： -- 不同索引可以同时构建，提高速度 -- 某个索引失败不影响其他索引 -- 可以按需启用特定类型的索引 - -### 3.4 知识图谱构建 - -图谱索引是 ApeRAG 的核心特色，它能从文档中提取结构化的知识。 - -```mermaid -flowchart TD - Chunks[文档块] --> EntityExtract[实体提取] - - EntityExtract --> LLM1[调用 LLM
识别实体] - LLM1 --> Entities[实体列表
人物、地点、概念] - - Entities --> RelationExtract[关系提取] - RelationExtract --> LLM2[调用 LLM
识别关系] - LLM2 --> Relations[关系列表
谁与谁有什么关系] - - Entities --> Merge[实体合并] - Relations --> Merge - - Merge --> Components[连通分量分析] - Components --> Parallel[并行处理各分量] - Parallel --> Graph[(知识图谱)] - - style EntityExtract fill:#bbdefb - style RelationExtract fill:#c5e1a5 - style Merge fill:#ffccbc - style Components fill:#fff59d -``` - -**图谱构建的关键步骤**： - -1. **实体提取**：LLM 从文档块中识别出有意义的实体 - - 示例：从"张三在北京的清华大学学习人工智能"中提取 - - 实体：张三（人物）、北京（地点）、清华大学（组织）、人工智能（概念） - -2. **关系提取**：识别实体之间的关系 - - 示例：张三 --学习--> 人工智能，张三 --就读于--> 清华大学 - -3. **实体合并**：同一实体可能有不同的表述，需要归一化 - - 示例："LightRAG"、"light rag"、"Light-RAG" → 合并为统一实体 - -4. **连通分量优化**：把图谱分成独立的子图，并行处理 - - 性能提升：2-3 倍吞吐量 - -**为什么需要连通分量优化？** - -假设你有 100 篇文档，它们讨论不同的主题。关于"数据库"的实体和关于"机器学习"的实体之间没有连接，可以独立处理。连通分量算法会找出这些独立的"知识岛"，然后并行处理，大大提高速度。 - -### 3.5 异步任务系统 - -文档处理是一个耗时的操作，ApeRAG 采用"双链路架构"来保证用户体验： - -```mermaid -graph TB - subgraph Frontend["🚀 前端链路 - 快速响应"] - direction TB - A1["📤 用户上传文档"] --> A2["🔌 API 接收请求"] - A2 --> A3["📋 Index Manager"] - A3 --> A4["💾 写入数据库
status = PENDING
version = 1"] - A4 --> A5["✅ 立即返回成功
< 100ms"] - end - - subgraph Backend["⚙️ 后端链路 - 异步处理"] - direction TB - B1["⏰ Celery Beat
每 30 秒检查"] --> B2["🔍 Reconciler 检测
version ≠ observed_version"] - B2 --> B3{"🎯 发现待处理任务?"} - B3 -->|是| B4["🚀 调度 Worker"] - B3 -->|否| B1 - B4 --> B5["📄 解析文档"] - B5 --> B6["🔀 并行创建索引
Vector + Fulltext
+ Graph + Vision"] - B6 --> B7["✨ 更新状态
status = ACTIVE
observed_version = 1"] - B7 --> B1 - end - - A4 -.-|"数据库状态变化"| B2 - - style Frontend fill:#e3f2fd,stroke:#1976d2,stroke-width:3px - style Backend fill:#fff3e0,stroke:#f57c00,stroke-width:3px - style A5 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px - style B7 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px - style B3 fill:#fff9c4,stroke:#fbc02d,stroke-width:2px -``` - -**双链路的好处**： - -- **前端快速响应**：用户上传文档后，API 在 100ms 内返回，不需要等待处理完成 -- **后端异步处理**：真正的处理工作在后台慢慢做，不阻塞用户操作 -- **自动重试**：如果处理失败，系统会自动重试，保证最终成功 -- **状态可查**：用户可以随时查看文档处理进度 - -**索引状态机**： - -```mermaid -stateDiagram-v2 - [*] --> PENDING: 📤 文档上传 - - PENDING --> CREATING: 🚀 Reconciler 检测到
开始处理 - - CREATING --> ACTIVE: ✅ 所有索引创建成功 - CREATING --> FAILED: ❌ 处理失败 - - FAILED --> CREATING: 🔄 自动重试
(最多 3 次) - FAILED --> [*]: 💔 超过重试次数
标记为失败 - - ACTIVE --> CREATING: 🔄 文档更新
重建索引 - ACTIVE --> [*]: 🗑️ 删除文档 - - note right of PENDING - version = 1 - observed_version = 0 - end note - - note right of CREATING - 正在处理中 - 可能需要几分钟 - end note - - note right of ACTIVE - version = 1 - observed_version = 1 - 可以被检索 - end note -``` - -## 4. 检索问答全流程 - -有了索引之后,用户就可以提问了。ApeRAG 的检索系统会智能地选择合适的检索策略。 - -### 4.1 混合检索系统 - -不同类型的问题适合用不同的检索方式。ApeRAG 会同时使用多种检索策略，然后融合结果： - -```mermaid -flowchart TB - Query[用户问题] --> Router[检索路由] - - Router --> |并行检索| Vector[向量检索] - Router --> |并行检索| Fulltext[全文检索] - Router --> |并行检索| Graph[图谱检索] - - Vector --> Embed[生成问题向量] - Embed --> QdrantSearch[Qdrant 相似度搜索] - QdrantSearch --> R1[结果1] - - Fulltext --> ESSearch[Elasticsearch BM25] - ESSearch --> R2[结果2] - - Graph --> GraphQuery[图谱查询
local/global/hybrid] - GraphQuery --> R3[结果3] - - R1 --> Merge[结果融合] - R2 --> Merge - R3 --> Merge - - Merge --> Rerank[Rerank 重排序] - Rerank --> Final[最终结果] - - style Query fill:#e1f5ff - style Vector fill:#bbdefb - style Fulltext fill:#c5e1a5 - style Graph fill:#ffccbc - style Rerank fill:#fff59d - style Final fill:#c5e1a5 -``` - -**检索策略说明**： - -- **向量检索**：用于语义相似的问题 - - 问："如何提升系统性能？" - - 能找到："优化数据库查询"、"使用缓存"等相关内容 - -- **全文检索**：用于精确关键词匹配 - - 问："PostgreSQL 的配置文件在哪？" - - 能找到包含"PostgreSQL"和"配置文件"的精确段落 - -- **图谱检索**：用于关系型问题 - - 问："LightRAG 和 Neo4j 有什么关系？" - - 会查询图谱中这两个实体的连接路径 - -**结果融合策略**： - -不同检索方式的结果需要合并。ApeRAG 使用 Rerank 模型对所有候选结果重新打分： - -1. 收集所有检索结果（可能有重复） -2. 去重，保留最相关的片段 -3. 使用 Rerank 模型评估每个片段与问题的相关性 -4. 按新的分数重新排序 -5. 返回 Top-K 结果 - -### 4.2 知识图谱查询 - -图谱检索有三种模式，适用于不同类型的问题： - -| 模式 | 适用场景 | 查询方式 | 示例问题 | -|------|---------|---------|---------| -| **local** | 查询某个实体的局部信息 | 向量匹配相似实体 → 获取邻居节点 | "张三的个人信息" | -| **global** | 查询整体关系和模式 | 向量匹配相似关系 → 获取关联路径 | "公司的组织架构是怎样的" | -| **hybrid** | 综合性问题 | local + global 结合 | "张三在公司的角色和职责" | - -```mermaid -flowchart TD - Question[用户问题] --> Analyze[问题分析] - - Analyze --> Local[Local 模式
实体中心] - Analyze --> Global[Global 模式
关系中心] - Analyze --> Hybrid[Hybrid 模式
综合查询] - - Local --> FindEntity[找到相关实体] - FindEntity --> GetNeighbors[获取邻居和关系] - - Global --> FindRelations[找到相关关系] - FindRelations --> GetContext[获取关系上下文] - - Hybrid --> Local - Hybrid --> Global - - GetNeighbors --> Context[生成上下文] - GetContext --> Context - - Context --> Return[返回给 LLM] - - style Local fill:#bbdefb - style Global fill:#c5e1a5 - style Hybrid fill:#fff59d -``` - -**实际例子**： - -假设知识图谱中有： -- 实体：张三（人物）、数据库团队（组织）、PostgreSQL（技术） -- 关系：张三 --属于--> 数据库团队，张三 --擅长--> PostgreSQL - -问题："张三负责什么？" - -1. **Local 模式**： - - 找到"张三"实体 - - 获取所有直接相连的节点 - - 返回："张三属于数据库团队，擅长 PostgreSQL" - -2. **Global 模式**： - - 找到相关的关系模式："负责"、"属于" - - 返回整个团队的结构和职责分工 - -3. **Hybrid 模式**： - - 同时使用上述两种方式 - - 给出更全面的答案 - -### 4.3 Agent 对话系统 - -Agent 是 ApeRAG 的智能助手，它能调用各种工具来回答问题。 - -```mermaid -sequenceDiagram - participant User as 用户 - participant API as API Server - participant Agent as Agent 服务 - participant LLM as LLM 服务 - participant MCP as MCP 工具 - participant Search as 检索服务 - - User->>API: 发送问题 - API->>Agent: 转发问题 - - Agent->>LLM: 调用 LLM
携带工具列表 - LLM-->>Agent: 决定调用 search_collection 工具 - - Agent->>MCP: 执行工具调用 - MCP->>Search: 混合检索 - Search-->>MCP: 返回相关文档片段 - MCP-->>Agent: 工具执行结果 - - Agent->>LLM: 再次调用 LLM
携带检索到的上下文 - LLM-->>Agent: 生成最终答案 - - Agent-->>API: 流式返回 - API-->>User: SSE 推送答案 -``` - -**Agent 的工作流程**： - -1. **接收问题**：用户发送一个问题 - -2. **工具决策**：LLM 分析问题，决定需要调用哪些工具 - - 可能的工具：search_collection（检索知识库）、web_search（搜索网络）、web_read（读取网页）等 - -3. **执行工具**：Agent 调用对应的工具 - - 示例：search_collection 会触发混合检索，返回相关文档 - -4. **生成答案**：LLM 基于检索到的上下文生成答案 - -5. **流式返回**：答案通过 SSE（Server-Sent Events）实时推送给用户，不用等待全部生成完毕 - -**MCP 协议的作用**： - -MCP（Model Context Protocol）是一个标准化的工具协议，让 AI 助手（如 Claude Desktop、Cursor）能够方便地调用 ApeRAG 的能力。通过 MCP，外部 AI 工具可以： -- 列出你的知识库 -- 搜索知识库内容 -- 读取网页内容 -- 搜索互联网 - -**对话示例**： - -``` -用户：ApeRAG 的图谱索引是怎么工作的？ - -Agent 思考：需要检索知识库 -↓ -调用工具：search_collection(query="图谱索引工作原理", collection_id="aperag-docs") -↓ -检索结果：返回关于图谱构建、实体提取、关系抽取的文档片段 -↓ -Agent 回答：ApeRAG 的图谱索引通过以下步骤工作...（基于检索到的内容生成） -``` - -## 5. 存储架构 - -ApeRAG 采用多存储架构，为不同类型的数据选择最合适的存储方案。 - -### 5.1 存储选型决策 - -```mermaid -flowchart TD - Data["🎯 数据类型分类"] --> Choice{"📊 什么数据?"} - - Choice --> |"📋 结构化数据
用户、配置等"| PG["PostgreSQL"] - Choice --> |"🔢 向量数据
embeddings"| Qdrant["Qdrant"] - Choice --> |"📝 文本数据
全文搜索"| ES["Elasticsearch"] - Choice --> |"📁 文件数据
原始文档"| MinIO["MinIO/S3"] - Choice --> |"🕸️ 图数据
知识图谱"| GraphChoice{"图规模?"} - Choice --> |"⚡ 缓存数据
临时数据"| Redis["Redis"] - - GraphChoice -->|"小规模
< 10万实体
💰 推荐"| PG2["PostgreSQL
内置图存储"] - GraphChoice -->|"大规模
> 100万实体"| Neo4j["Neo4j
专业图数据库"] - - PG --> PGUse["✅ 事务支持
✅ 关系查询
✅ 小规模图存储
✅ 成熟稳定"] - PG2 --> PG2Use["✅ 无需额外组件
✅ 降低运维成本
✅ 足够应对大多数场景"] - Qdrant --> QdrantUse["✅ 向量相似度搜索
✅ 高维数据检索
✅ 支持过滤条件"] - ES --> ESUse["✅ 全文检索 BM25
✅ 关键词搜索
✅ 中文分词 IK"] - MinIO --> MinIOUse["✅ 大文件存储
✅ S3 协议兼容
✅ 成本低"] - Neo4j --> Neo4jUse["✅ 大规模图查询
✅ 复杂关系遍历
✅ 图算法支持"] - Redis --> RedisUse["✅ Celery 任务队列
✅ LLM 调用缓存
✅ 毫秒级访问"] - - style Data fill:#e3f2fd,stroke:#1976d2,stroke-width:3px - style Choice fill:#fff59d,stroke:#fbc02d,stroke-width:3px - style GraphChoice fill:#fff59d,stroke:#fbc02d,stroke-width:2px - style PG fill:#bbdefb,stroke:#1976d2,stroke-width:2px - style PG2 fill:#c8e6c9,stroke:#388e3c,stroke-width:3px - style Qdrant fill:#c5e1a5,stroke:#689f38,stroke-width:2px - style ES fill:#ffccbc,stroke:#e64a19,stroke-width:2px - style MinIO fill:#e1bee7,stroke:#8e24aa,stroke-width:2px - style Neo4j fill:#f8bbd0,stroke:#c2185b,stroke-width:2px - style Redis fill:#ffecb3,stroke:#ffa000,stroke-width:2px -``` - -### 5.2 数据流向 - -不同的数据在系统中流转到不同的存储： - -```mermaid -flowchart LR - Doc[上传文档] --> Parser[解析器] - Parser --> |原始文件| MinIO[(MinIO)] - Parser --> |文档元数据| PG1[(PostgreSQL)] - Parser --> |文档块| Chunks[分块] - - Chunks --> |生成向量| Embed[Embedding] - Embed --> Qdrant[(Qdrant)] - - Chunks --> |文本内容| ES[(Elasticsearch)] - - Chunks --> |实体关系提取| Graph[图谱构建] - Graph --> |小规模| PG2[(PostgreSQL)] - Graph --> |大规模| Neo4j[(Neo4j)] - - PG1 -.元数据.-> Cache - Cache -.缓存.-> Redis[(Redis)] - - style Doc fill:#e1f5ff - style MinIO fill:#e1bee7 - style PG1 fill:#bbdefb - style PG2 fill:#bbdefb - style Qdrant fill:#c5e1a5 - style ES fill:#ffccbc - style Neo4j fill:#f8bbd0 - style Redis fill:#ffecb3 -``` - -### 5.3 核心存储系统 - -**PostgreSQL**（主数据库） - -存储内容： -- 用户信息、权限、配置 -- Collection（知识库）元数据 -- 文档元数据和索引状态 -- 对话历史 -- 小规模知识图谱（< 10 万实体） - -为什么选择： -- 强大的事务支持，保证数据一致性 -- 成熟稳定，运维成本低 -- pgvector 扩展，支持向量存储 -- 可以承载小规模图数据，不需要额外的图数据库 - -**Qdrant**（向量数据库） - -存储内容： -- 文档块的 embedding 向量 -- 实体和关系的向量表示 -- 图片的多模态向量 - -为什么选择： -- 专门为向量检索优化，速度快 -- 支持过滤条件，可以结合元数据筛选 -- 支持集群部署，可以水平扩展 - -**Elasticsearch**（全文搜索） - -存储内容： -- 文档块的文本内容 -- 支持中文分词（IK Analyzer） - -为什么选择： -- BM25 算法对关键词搜索效果好 -- 支持复杂的查询和聚合 -- 自带高亮显示 - -**MinIO**（对象存储） - -存储内容： -- 原始文档文件（PDF、Word 等） -- 解析后的中间结果 -- 上传的临时文件 - -为什么选择： -- S3 协议兼容，可以替换为云存储 -- 存储成本低 -- 支持大文件 - -**图数据库选择：PostgreSQL vs Neo4j** - -ApeRAG 支持两种图数据库方案： - -**PostgreSQL**（默认，推荐用于小规模） - -存储内容： -- 知识图谱（< 10 万实体） -- 图节点和边的关系数据 - -推荐理由： -- 无需额外部署，降低运维成本 -- 性能足够应对大多数场景 -- 事务支持完善，数据一致性有保障 -- 可以和其他业务数据共用一个数据库 - -**Neo4j**（可选，用于大规模） - -存储内容： -- 大规模知识图谱（> 100 万实体） - -什么时候需要： -- 实体数量超过 10 万，PostgreSQL 查询性能下降 -- 需要复杂的图遍历查询（多跳关系） -- 需要使用图算法（PageRank、社区发现等） - -**总结**：对于大多数企业应用，PostgreSQL 完全够用。只有在知识图谱规模非常大时，才需要考虑 Neo4j。 - -**Redis**（缓存和队列） - -存储内容： -- Celery 任务队列 -- LLM 调用缓存 -- 用户会话缓存 - -为什么选择： -- 速度极快，适合高频访问 -- 支持多种数据结构 -- 可以做任务队列的 Broker - -## 6. 技术亮点 - -### 6.1 无状态 LightRAG 重构 - -**背景问题**： - -原版 LightRAG 使用全局状态，所有任务共享一个实例。这在多用户、多 Collection 的场景下会导致数据混乱和并发冲突。 - -**ApeRAG 的解决方案**： - -- 每个任务创建独立的 LightRAG 实例 -- 通过 `workspace` 参数隔离不同 Collection 的数据 -- 实体命名规范：`entity:{name}:{workspace}` -- 关系命名规范：`relationship:{src}:{tgt}:{workspace}` - -这样，不同用户的图谱数据不会互相干扰，真正实现了多租户隔离。 - -### 6.2 双链异步架构 - -**传统做法的问题**： - -用户上传文档后，API 需要等待解析、索引构建全部完成才能返回，可能要等几分钟甚至更久。 - -**双链架构的优势**： - -- **前端链路**：API 只负责写状态到数据库，100ms 内返回 -- **后端链路**：Reconciler 定时检测状态变化，调度异步任务 -- **版本控制**：通过 version 和 observed_version 实现幂等性 -- **自动重试**：任务失败后自动重试，保证最终一致性 - -这个设计灵感来自 Kubernetes 的 Reconciler 模式，非常适合处理长时间运行的任务。 - -### 6.3 连通分量并发优化 - -**问题**： - -知识图谱构建时,需要合并相似的实体。如果串行处理,速度很慢。如果全部并行,又会有锁竞争问题。 - -**解决方案**： - -使用连通分量算法,把图谱分成多个独立的子图： - -1. 构建实体关系邻接表 -2. BFS 遍历找出所有连通分量 -3. 不同分量之间没有连接，可以完全并行处理 -4. 同一分量内部串行处理（避免冲突） - -**效果**： - -- 性能提升 2-3 倍 -- 零锁竞争 -- 对于多样化的文档集合效果最好 - -### 6.4 Provider 抽象模式 - -ApeRAG 支持 100+ 种 LLM 提供商（OpenAI、Claude、Gemini、国产大模型等）。如何统一管理？ - -**设计思路**： - -- 定义统一的 Provider 接口 -- 每个提供商实现自己的 Provider -- 通过 LiteLLM 库做适配 - -这样，切换模型只需要改配置，不需要改代码。同样的模式也应用在： -- Embedding Service（支持多种向量模型） -- Rerank Service（支持多种重排序模型） -- Web Search Service（DuckDuckGo、JINA 等） - -### 6.5 多模态索引支持 - -除了文本，ApeRAG 也能处理图片： - -**Vision Index 的两条路径**： - -1. **纯视觉向量**：使用多模态模型（如 CLIP）直接生成图片向量 -2. **视觉转文本**：使用 VLM 生成图片描述 + OCR 识别文字 → 文本向量化 - -**融合策略**： - -- 文本检索结果和视觉检索结果分开排序 -- 通过 Rerank 模型统一打分 -- 最终合并展示 - -## 7. 总结 - -ApeRAG 通过以下设计实现了生产级的 RAG 能力： - -**核心优势**： -- **强大的文档处理**：支持多格式、复杂布局、表格公式 -- **知识图谱融合**：不仅是向量匹配，还能理解知识关联 -- **多种检索方式**：向量、全文、图谱三管齐下 -- **异步架构**：快速响应，后台处理，用户体验好 -- **生产级设计**：多存储、高并发、易扩展 - -**技术创新**： -- 无状态 LightRAG，真正的多租户支持 -- 双链异步架构，API 响应 < 100ms -- 连通分量并发优化，图谱构建快 2-3 倍 -- Provider 抽象，支持 100+ LLM - -**适用场景**： -- 企业知识库搜索 -- 技术文档问答 -- 客服机器人 -- 研究论文分析 -- 任何需要理解文档并提供智能问答的场景 - -整个系统的设计理念是：**让复杂的事情变简单，让简单的事情变自动**。用户只需要上传文档，剩下的一切都由 ApeRAG 自动完成。 diff --git a/web/docs/zh-CN/design/chat_history_design.md b/web/docs/zh-CN/design/chat_history_design.md deleted file mode 100644 index b92611e8e..000000000 --- a/web/docs/zh-CN/design/chat_history_design.md +++ /dev/null @@ -1,590 +0,0 @@ ---- -title: 聊天历史消息数据流程 -description: 详细说明ApeRAG项目中聊天历史消息的完整数据流程，从前端API调用到后端存储的全链路实现 -keywords: [chat, history, message, redis, postgresql, websocket, part-based design] ---- - -# 聊天历史消息数据流程 - -## 概述 - -本文档详细说明ApeRAG项目中聊天历史消息的完整数据流程，从前端API调用到后端存储的全链路实现。 - -**核心接口**: `GET /api/v1/bots/{bot_id}/chats/{chat_id}` - -## 数据流图 - -``` -┌─────────────────┐ -│ Frontend │ -│ (Next.js) │ -└────────┬────────┘ - │ GET /api/v1/bots/{bot_id}/chats/{chat_id} - ▼ -┌─────────────────────────────────────────────┐ -│ View Layer │ -│ aperag/views/chat.py │ -│ - get_chat_view() │ -│ - JWT身份验证 │ -│ - 参数验证 │ -└────────┬────────────────────────────────────┘ - │ chat_service_global.get_chat() - ▼ -┌─────────────────────────────────────────────┐ -│ Service Layer │ -│ aperag/service/chat_service.py │ -│ - get_chat() │ -│ - 业务逻辑编排 │ -└────────┬────────────────────────────────────┘ - │ - ├──────────────┬─────────────┐ - │ │ │ - ▼ ▼ ▼ -┌────────────┐ ┌───────────┐ ┌──────────────┐ -│ PostgreSQL │ │ Redis │ │ PostgreSQL │ -│ chat表 │ │ 消息历史 │ │ feedback表 │ -│(基本信息) │ │(会话内容) │ │(用户反馈) │ -└────────────┘ └───────────┘ └──────────────┘ - │ │ │ - └──────────────┴──────────────────┘ - │ - ▼ - ┌──────────────┐ - │ ChatDetails │ - │ (组装响应) │ - └──────────────┘ -``` - -## 完整流程详解 - -### 1. View层 - HTTP请求处理 - -**文件**: `aperag/views/chat.py` - -```python -@router.get("/bots/{bot_id}/chats/{chat_id}") -async def get_chat_view( - request: Request, - bot_id: str, - chat_id: str, - user: User = Depends(required_user) -) -> view_models.ChatDetails: - return await chat_service_global.get_chat(str(user.id), bot_id, chat_id) -``` - -**职责**: -- 接收HTTP GET请求 -- JWT Token身份验证 -- 提取路径参数 (bot_id, chat_id) -- 调用Service层 -- 返回`ChatDetails`响应 - -### 2. Service层 - 业务逻辑编排 - -**文件**: `aperag/service/chat_service.py` - -```python -async def get_chat(self, user: str, bot_id: str, chat_id: str) -> view_models.ChatDetails: - from aperag.utils.history import query_chat_messages - - # Step 1: 从PostgreSQL查询Chat基本信息 - chat = await self.db_ops.query_chat(user, bot_id, chat_id) - if chat is None: - raise ChatNotFoundException(chat_id) - - # Step 2: 从Redis查询聊天消息历史 - messages = await query_chat_messages(user, chat_id) - - # Step 3: 构建响应对象（消息中已包含feedback信息） - chat_obj = self.build_chat_response(chat) - return ChatDetails(**chat_obj.model_dump(), history=messages) -``` - -**核心逻辑**: - -1. **查询Chat元数据** (PostgreSQL) -2. **查询消息历史** (Redis + PostgreSQL反馈信息) -3. **组装完整响应** - -### 3. 数据存储层 - -#### 3.1 PostgreSQL - Chat基本信息 - -**表**: `chat` - -**文件**: `aperag/db/models.py` - -```python -class Chat(Base): - __tablename__ = "chat" - - id = Column(String(24), primary_key=True) # chat_xxxx - user = Column(String(256), nullable=False) # 用户ID - bot_id = Column(String(24), nullable=False) # Bot ID - title = Column(String(256)) # 会话标题 - peer_type = Column(EnumColumn(ChatPeerType)) # 对话类型 - peer_id = Column(String(256)) # 对话ID - status = Column(EnumColumn(ChatStatus)) # 状态 - gmt_created = Column(DateTime(timezone=True)) # 创建时间 - gmt_updated = Column(DateTime(timezone=True)) # 更新时间 - gmt_deleted = Column(DateTime(timezone=True)) # 删除时间（软删除） -``` - -**用途**: 存储Chat会话的元数据，不包含具体消息内容 - -#### 3.2 Redis - 聊天消息历史 - -**文件**: `aperag/utils/history.py` - -**Key格式**: `message_store:{chat_id}` - -**数据结构**: Redis List (使用LPUSH，最新消息在前) - -**核心类**: - -```python -class RedisChatMessageHistory: - def __init__(self, session_id: str, key_prefix: str = "message_store:"): - self.session_id = session_id - self.key_prefix = key_prefix - - @property - def key(self) -> str: - return self.key_prefix + self.session_id # message_store:chat_abc123 - - @property - async def messages(self) -> List[StoredChatMessage]: - # 从Redis读取所有消息 - _items = await self.redis_client.lrange(self.key, 0, -1) - # 反转为时间顺序（因为LPUSH导致最新在前） - items = [json.loads(m.decode("utf-8")) for m in _items[::-1]] - return [storage_dict_to_message(item) for item in items] -``` - -**消息查询函数**: - -```python -async def query_chat_messages(user: str, chat_id: str): - """查询聊天消息并转换为前端格式""" - - # 1. 从Redis获取消息历史 - chat_history = RedisChatMessageHistory(chat_id, redis_client=get_async_redis_client()) - stored_messages = await chat_history.messages - - if not stored_messages: - return [] - - # 2. 从PostgreSQL获取反馈信息 - feedbacks = await async_db_ops.query_chat_feedbacks(user, chat_id) - feedback_map = {feedback.message_id: feedback for feedback in feedbacks} - - # 3. 转换为前端格式并附加反馈信息 - result = [] - for stored_message in stored_messages: - # 转换为前端格式 - chat_message_list = stored_message.to_frontend_format() - - # 为AI消息添加反馈数据 - for chat_msg in chat_message_list: - feedback = feedback_map.get(chat_msg.id) - if feedback and chat_msg.role == "ai": - chat_msg.feedback = Feedback( - type=feedback.type, - tag=feedback.tag, - message=feedback.message - ) - - result.append(chat_message_list) - - return result # [[message1_parts], [message2_parts], [message3_parts], ...] -``` - -#### 3.3 PostgreSQL - 用户反馈信息 - -**表**: `message_feedback` - -```python -class MessageFeedback(Base): - __tablename__ = "message_feedback" - - user = Column(String(256), nullable=False) # 用户ID - chat_id = Column(String(24), primary_key=True) # 会话ID - message_id = Column(String(256), primary_key=True) # 消息ID - type = Column(EnumColumn(MessageFeedbackType)) # like/dislike - tag = Column(EnumColumn(MessageFeedbackTag)) # 反馈标签 - message = Column(Text) # 反馈内容 - question = Column(Text) # 原始问题 - original_answer = Column(Text) # 原始回答 - status = Column(EnumColumn(MessageFeedbackStatus)) # 状态 - gmt_created = Column(DateTime(timezone=True)) - gmt_updated = Column(DateTime(timezone=True)) -``` - -**用途**: 存储用户对AI回复的反馈（点赞/点踩），用于质量监控和模型优化 - -## 数据格式详解 - -### 存储格式 (Redis) - -消息在Redis中以JSON格式存储，采用**Part-Based设计**： - -#### StoredChatMessage - 一条完整消息 - -```python -class StoredChatMessage(BaseModel): - """一条完整消息（用户的一条消息 或 AI的一条消息）""" - parts: List[StoredChatMessagePart] # 消息的多个部分 - files: List[Dict[str, Any]] # 关联的上传文件 -``` - -#### StoredChatMessagePart - 消息的一个部分 - -```python -class StoredChatMessagePart(BaseModel): - """消息的单个部分（原子单元）""" - - # 标识信息 - chat_id: str # 所属会话 - message_id: str # 所属消息（同一条消息的多个part共享） - part_id: str # 部分的唯一ID - timestamp: float # 生成时间戳 - - # 内容分类 - type: Literal["message", "tool_call_result", "thinking", "references"] - role: Literal["human", "ai", "system"] - content: str - - # 扩展字段 - references: List[Dict] # 文档引用 - urls: List[str] # URL引用 - metadata: Optional[Dict] # 额外元数据 -``` - -#### Part类型说明 - -| Type | 说明 | 包含在LLM上下文 | -|------|------|---------------| -| `message` | 主要对话内容 | ✅ 是 | -| `tool_call_result` | 工具调用过程 | ❌ 否（仅展示） | -| `thinking` | AI思考过程 | ❌ 否（仅展示） | -| `references` | 文档引用和链接 | ❌ 否（仅展示） | - -**设计原因**: AI的一条回复包含多个阶段（工具调用、思考、回答、引用），这些内容按时序产生且互相穿插，单一字段无法表达。用户的消息通常只有1个part（type="message"），但也支持多个part以保持结构一致性。 - -#### Redis存储示例 - -**用户消息**: -```json -{ - "parts": [ - { - "chat_id": "chat_abc123", - "message_id": "uuid-1", - "part_id": "uuid-part-1", - "timestamp": 1699999999.0, - "type": "message", - "role": "human", - "content": "什么是LightRAG？", - "references": [], - "urls": [], - "metadata": null - } - ], - "files": [] -} -``` - -**AI回复（包含多个part）**: -```json -{ - "parts": [ - { - "message_id": "uuid-2", - "part_id": "uuid-part-2", - "type": "tool_call_result", - "role": "ai", - "content": "正在检索知识库...", - "timestamp": 1699999999.1 - }, - { - "message_id": "uuid-2", - "part_id": "uuid-part-3", - "type": "message", - "role": "ai", - "content": "LightRAG是一个轻量级的RAG框架，由ApeCloud团队深度改造...", - "timestamp": 1699999999.5 - }, - { - "message_id": "uuid-2", - "part_id": "uuid-part-4", - "type": "references", - "role": "ai", - "content": "", - "references": [ - { - "score": 0.95, - "text": "LightRAG架构说明...", - "metadata": {"source": "lightrag_doc.pdf", "page": 3} - } - ], - "urls": ["https://github.com/HKUDS/LightRAG"], - "timestamp": 1699999999.6 - } - ], - "files": [] -} -``` - -### API响应格式 - -**ChatDetails Schema** (`aperag/api/components/schemas/chat.yaml`): - -```yaml -chatDetails: - type: object - properties: - id: string # chat_abc123 - title: string # 会话标题 - bot_id: string # bot_xyz - peer_id: string - peer_type: string # system/feishu/weixin/web - status: string # active/archived - created: string # ISO 8601 - updated: string # ISO 8601 - history: # 二维数组 - type: array - description: 对话历史，每个元素是一条消息 - items: - type: array - description: 一条消息包含多个parts（工具调用、思考、回答、引用等） - items: - $ref: '#/chatMessage' -``` - -**ChatMessage Schema**: - -```yaml -chatMessage: - type: object - properties: - id: string # message_id（同一轮次相同） - part_id: string # part_id（每个part唯一） - type: string # message/tool_call_result/thinking/references - timestamp: number # Unix时间戳 - role: string # human/ai - data: string # 消息内容 - references: # 文档引用（可选） - type: array - items: - type: object - properties: - score: number - text: string - metadata: object - urls: # URL引用（可选） - type: array - items: - type: string - feedback: # 用户反馈（可选） - type: object - properties: - type: string # like/dislike - tag: string - message: string - files: # 关联文件（可选） - type: array -``` - -### 前端接收示例 - -```json -{ - "id": "chat_abc123", - "title": "关于LightRAG的讨论", - "bot_id": "bot_xyz", - "status": "active", - "created": "2025-01-01T00:00:00Z", - "updated": "2025-01-01T01:00:00Z", - "history": [ - [ - { - "id": "uuid-1", - "part_id": "uuid-part-1", - "type": "message", - "timestamp": 1699999999.0, - "role": "human", - "data": "什么是LightRAG？", - "files": [] - } - ], - [ - { - "id": "uuid-2", - "part_id": "uuid-part-2", - "type": "tool_call_result", - "timestamp": 1699999999.1, - "role": "ai", - "data": "正在检索知识库...", - "files": [] - }, - { - "id": "uuid-2", - "part_id": "uuid-part-3", - "type": "message", - "timestamp": 1699999999.5, - "role": "ai", - "data": "LightRAG是一个轻量级的RAG框架...", - "files": [] - }, - { - "id": "uuid-2", - "part_id": "uuid-part-4", - "type": "references", - "timestamp": 1699999999.6, - "role": "ai", - "data": "", - "references": [ - { - "score": 0.95, - "text": "LightRAG架构说明...", - "metadata": {"source": "lightrag_doc.pdf"} - } - ], - "urls": ["https://github.com/HKUDS/LightRAG"], - "files": [] - } - ] - ] -} -``` - -**注意**: `history`是二维数组，第一维是消息序列（按时间顺序），第二维是该条消息的多个part。例如： -- `history[0]` = 用户的第1条消息的parts（通常只有1个part） -- `history[1]` = AI的第1条回复的parts（可能有多个part：工具调用、思考、回答、引用） -- `history[2]` = 用户的第2条消息的parts -- `history[3]` = AI的第2条回复的parts -- ... - -## 消息写入流程 - -### Agent Runtime 写入路径 - -旧的 WebSocket 聊天接口 `WS /api/v1/bots/{bot_id}/chats/{chat_id}/connect` 已经退休。 -当前 Agent 聊天写入走的是 v2 turn/timeline API 加 SSE 事件流。上面的 history schema 仍可作为背景说明， -但不应再依据这份文档实现新的 WebSocket chat client。 - -## 设计特点 - -### 1. 混合存储架构 - -| 存储 | 内容 | 原因 | -|------|------|------| -| PostgreSQL | Chat元数据 | 持久化、支持复杂查询 | -| Redis | 消息历史 | 高性能读写、支持TTL | -| PostgreSQL | 用户反馈 | 持久化、用于分析 | - -**优势**: -- 性能优化：消息历史使用Redis快速读写 -- 数据持久化：重要元数据存储在PostgreSQL -- 灵活性：可独立配置TTL、备份策略 - -### 2. Part-Based消息设计 - -**核心价值**: -- ✅ 支持复杂的AI回复流程（工具调用→思考→回答→引用） -- ✅ 前端可差异化渲染不同类型的内容 -- ✅ 完整记录时序关系（通过timestamp） -- ✅ 灵活扩展（新增type无需改表结构） - -**为什么一条消息需要多个part**: - -AI的一条回复过程是时序产生、互相穿插的，例如： -1. 🔍 Part1 (tool_call_result): "正在查询数据库..." -2. 💭 Part2 (thinking): "找到了327条记录..." -3. 🔍 Part3 (tool_call_result): "正在计算增长率..." -4. 💭 Part4 (thinking): "环比增长15%..." -5. 💬 Part5 (message): "根据数据分析，Q4表现优秀..." -6. 📚 Part6 (references): [文档1, 文档2] - -这6个part属于AI的**一条消息**（共享同一个message_id），单一字段无法表达这种复杂的时序关系。 - -### 3. 格式转换解耦 - -提供三种格式转换： - -```python -class StoredChatMessage: - def to_frontend_format(self) -> List[ChatMessage]: - """转换为前端展示格式""" - # 包含所有types的parts - - def to_openai_format(self) -> List[Dict]: - """转换为LLM调用格式""" - # 只包含type="message"的parts - - def get_main_content(self) -> str: - """获取主要回答内容""" - # 第一个type="message"的content -``` - -**优势**: -- 内部存储格式与外部接口解耦 -- 支持不同的消费场景 -- LLM上下文只包含实际对话内容，不包含工具调用和思考过程 - -### 4. 三级ID设计 - -```python -chat_id = "chat_abc123" # 会话级别 -message_id = "uuid-msg-1" # 消息级别（同一条消息的多个part共享） -part_id = "uuid-part-1" # 部分级别（每个part独立） -``` - -**作用**: -- `chat_id`: 标识一个聊天会话 -- `message_id`: 将同一条消息的多个part分组（用于前端展示和反馈关联） -- `part_id`: 每个part独立标识（用于单独操作，如复制、引用） - -## 性能考虑 - -### Redis优化 -- **List数据结构**: LPUSH O(1), LRANGE O(N) -- **可选TTL**: 自动过期历史消息 -- **连接池复用**: 全局Redis客户端 - -### PostgreSQL优化 -- **索引**: user, bot_id, chat_id, status字段 -- **软删除**: 使用gmt_deleted -- **分页查询**: list_chats支持分页 - -### 传输优化 -- **WebSocket流式**: 边生成边发送 -- **增量更新**: 只传输新的part -- **按需加载**: 懒加载历史消息 - -## 相关文件 - -### 核心实现 -- `aperag/views/chat.py` - View层接口 -- `aperag/service/chat_service.py` - Service层业务逻辑 -- `aperag/utils/history.py` - Redis消息历史管理 -- `aperag/chat/history/message.py` - 消息数据结构 -- `aperag/db/models.py` - 数据库模型 -- `aperag/db/repositories/chat.py` - Chat数据库操作 -- `aperag/api/components/schemas/chat.yaml` - OpenAPI Schema - -### 前端实现 -- `web/src/app/workspace/bots/[botId]/chats/[chatId]/page.tsx` - 聊天详情页面 -- `web/src/components/chat/chat-messages.tsx` - 消息展示组件 - -## 总结 - -ApeRAG的聊天历史消息系统采用**混合存储 + Part-Based消息设计**： - -1. **PostgreSQL**存储Chat元数据和反馈（持久化、可查询） -2. **Redis**存储消息历史（高性能、支持过期） -3. **Part-Based设计**支持复杂的AI回复流程（工具调用、思考、回答、引用） -4. **三级ID设计**支持消息分组和独立操作 -5. **清晰的分层架构**（View → Service → Repository → Storage） - -这种设计既保证了性能，又支持复杂的AI交互场景，同时具有良好的可扩展性。 diff --git a/web/docs/zh-CN/design/document_upload_design.md b/web/docs/zh-CN/design/document_upload_design.md deleted file mode 100644 index 0587be9c9..000000000 --- a/web/docs/zh-CN/design/document_upload_design.md +++ /dev/null @@ -1,1077 +0,0 @@ ---- -title: 文档上传设计 -position: 3 ---- - -# ApeRAG 文档上传架构设计 - -## 概述 - -本文档详细说明 ApeRAG 项目中文档上传模块的完整架构设计，涵盖从文件上传、临时存储、文档解析、格式转换到最终索引构建的全链路流程。 - -**核心设计理念**：采用**两阶段提交**模式，将文件上传（临时存储）和文档确认（正式添加）分离，提供更好的用户体验和资源管理能力。 - -## 系统架构 - -### 整体架构图 - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Frontend │ -│ (Next.js) │ -└────────┬───────────────────────────────────┬────────────────┘ - │ │ - │ Step 1: Upload │ Step 2: Confirm - │ POST /documents/upload │ POST /documents/confirm - ▼ ▼ -┌─────────────────────────────────────────────────────────────┐ -│ View Layer: aperag/views/collections.py │ -│ - HTTP请求处理 │ -│ - JWT身份验证 │ -│ - 参数验证 │ -└────────┬───────────────────────────────────┬────────────────┘ - │ │ - │ document_service.upload_document() │ document_service.confirm_documents() - ▼ ▼ -┌─────────────────────────────────────────────────────────────┐ -│ Service Layer: aperag/service/document_service.py │ -│ - 业务逻辑编排 │ -│ - 文件验证（类型、大小） │ -│ - SHA-256 哈希去重 │ -│ - Quota 检查 │ -│ - 事务管理 │ -└────────┬───────────────────────────────────┬────────────────┘ - │ │ - │ Step 1 │ Step 2 - ▼ ▼ -┌────────────────────────┐ ┌────────────────────────────┐ -│ 1. 创建 Document 记录 │ │ 1. 更新 Document 状态 │ -│ status=UPLOADED │ │ UPLOADED → PENDING │ -│ 2. 保存到 ObjectStore │ │ 2. 创建 DocumentIndex 记录│ -│ 3. 计算 content_hash │ │ 3. 触发索引构建任务 │ -└────────┬───────────────┘ └────────┬───────────────────┘ - │ │ - ▼ ▼ -┌─────────────────────────────────────────────────────────────┐ -│ Storage Layer │ -│ │ -│ ┌───────────────┐ ┌──────────────────┐ ┌─────────────┐ │ -│ │ PostgreSQL │ │ Object Store │ │ Vector DB │ │ -│ │ │ │ │ │ │ │ -│ │ - document │ │ - Local/S3 │ │ - Qdrant │ │ -│ │ - document_ │ │ - 原始文件 │ │ - 向量索引 │ │ -│ │ index │ │ - 转换后的文件 │ │ │ │ -│ └───────────────┘ └──────────────────┘ └─────────────┘ │ -│ │ -│ ┌───────────────┐ ┌──────────────────┐ │ -│ │ Elasticsearch │ │ Neo4j/PG │ │ -│ │ │ │ │ │ -│ │ - 全文索引 │ │ - 知识图谱 │ │ -│ └───────────────┘ └──────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌───────────────────┐ - │ Celery Workers │ - │ │ - │ - 文档解析 │ - │ - 格式转换 │ - │ - 内容提取 │ - │ - 文档分块 │ - │ - 索引构建 │ - └───────────────────┘ -``` - -### 分层架构 - -``` -┌─────────────────────────────────────────────┐ -│ View Layer (views/collections.py) │ HTTP 处理、认证、参数验证 -└─────────────────┬───────────────────────────┘ - │ 调用 -┌─────────────────▼───────────────────────────┐ -│ Service Layer (service/document_service.py)│ 业务逻辑、事务编排、权限控制 -└─────────────────┬───────────────────────────┘ - │ 调用 -┌─────────────────▼───────────────────────────┐ -│ Repository Layer (db/ops.py, objectstore/) │ 数据访问抽象、对象存储接口 -└─────────────────┬───────────────────────────┘ - │ 访问 -┌─────────────────▼───────────────────────────┐ -│ Storage Layer (PG, S3, Qdrant, ES, Neo4j) │ 数据持久化 -└─────────────────────────────────────────────┘ -``` - -## 核心流程详解 - -### 阶段 0: API 接口定义 - -系统提供三个主要接口： - -1. **上传文件**（两阶段模式 - 第一步） - - 接口：`POST /api/v1/collections/{collection_id}/documents/upload` - - 功能：上传文件到临时存储，状态为 `UPLOADED` - - 返回：`document_id`、`filename`、`size`、`status` - -2. **确认文档**（两阶段模式 - 第二步） - - 接口：`POST /api/v1/collections/{collection_id}/documents/confirm` - - 功能：确认已上传的文档，触发索引构建 - - 参数：`document_ids` 数组 - - 返回：`confirmed_count`、`failed_count`、`failed_documents` - -3. **一步上传**（传统模式，兼容旧版） - - 接口：`POST /api/v1/collections/{collection_id}/documents` - - 功能：上传并直接添加到知识库，状态直接为 `PENDING` - - 支持批量上传 - -### 阶段 1: 文件上传与临时存储 - -#### 1.1 上传流程 - -``` -用户选择文件 - │ - ▼ -前端调用 upload API - │ - ▼ -View 层验证身份和参数 - │ - ▼ -Service 层处理业务逻辑： - │ - ├─► 验证集合存在且激活 - │ - ├─► 验证文件类型和大小 - │ - ├─► 读取文件内容 - │ - ├─► 计算 SHA-256 哈希 - │ - └─► 事务处理： - │ - ├─► 重复检测（按文件名+哈希） - │ ├─ 完全相同：返回已存在文档（幂等） - │ ├─ 同名不同内容：抛出冲突异常 - │ └─ 新文档：继续创建 - │ - ├─► 创建 Document 记录（status=UPLOADED） - │ - ├─► 上传到对象存储 - │ └─ 路径：user-{user_id}/{collection_id}/{document_id}/original{suffix} - │ - └─► 更新文档元数据（object_path） -``` - -#### 1.2 文件验证 - -**支持的文件类型**： -- 文档：`.pdf`, `.doc`, `.docx`, `.ppt`, `.pptx`, `.xls`, `.xlsx` -- 文本：`.txt`, `.md`, `.html`, `.json`, `.xml`, `.yaml`, `.yml`, `.csv` -- 图片：`.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tiff`, `.tif` -- 音频：`.mp3`, `.wav`, `.m4a` -- 压缩包：`.zip`, `.tar`, `.gz`, `.tgz` - -**大小限制**： -- 默认：100 MB（可通过 `MAX_DOCUMENT_SIZE` 环境变量配置） -- 解压后总大小：5 GB（`MAX_EXTRACTED_SIZE`） - -#### 1.3 重复检测机制 - -采用**文件名 + SHA-256 哈希**双重检测： - -| 场景 | 文件名 | 哈希值 | 系统行为 | -|------|--------|--------|----------| -| 完全相同 | 相同 | 相同 | 返回已存在文档（幂等操作） | -| 文件名冲突 | 相同 | 不同 | 抛出 `DocumentNameConflictException` | -| 新文档 | 不同 | - | 创建新文档记录 | - -**优势**： -- ✅ 支持幂等上传：网络重传不会创建重复文档 -- ✅ 避免内容冲突：同名不同内容会提示用户 -- ✅ 节省存储空间：相同内容只存储一次 - -### 阶段 2: 临时存储配置 - -#### 2.1 对象存储类型 - -系统支持两种对象存储后端，可通过环境变量切换： - -**1. Local 存储（本地文件系统）** - -适用场景： -- 开发测试环境 -- 小规模部署 -- 单机部署 - -配置方式： -```bash -# 开发环境 -OBJECT_STORE_TYPE=local -OBJECT_STORE_LOCAL_ROOT_DIR=.objects - -# Docker 环境 -OBJECT_STORE_TYPE=local -OBJECT_STORE_LOCAL_ROOT_DIR=/shared/objects -``` - -存储路径示例： -``` -.objects/ -└── user-google-oauth2-123456/ - └── col_abc123/ - └── doc_xyz789/ - ├── original.pdf # 原始文件 - ├── converted.pdf # 转换后的 PDF - ├── processed_content.md # 解析后的 Markdown - ├── chunks/ # 分块数据 - │ ├── chunk_0.json - │ └── chunk_1.json - └── images/ # 提取的图片 - ├── page_0.png - └── page_1.png -``` - -**2. S3 存储（兼容 AWS S3/MinIO/OSS 等）** - -适用场景： -- 生产环境 -- 大规模部署 -- 分布式部署 -- 需要高可用和容灾 - -配置方式： -```bash -OBJECT_STORE_TYPE=s3 -OBJECT_STORE_S3_ENDPOINT=http://127.0.0.1:9000 # MinIO/S3 地址 -OBJECT_STORE_S3_REGION=us-east-1 # AWS Region -OBJECT_STORE_S3_ACCESS_KEY=minioadmin # Access Key -OBJECT_STORE_S3_SECRET_KEY=minioadmin # Secret Key -OBJECT_STORE_S3_BUCKET=aperag # Bucket 名称 -OBJECT_STORE_S3_PREFIX_PATH=dev/ # 可选的路径前缀 -OBJECT_STORE_S3_USE_PATH_STYLE=true # MinIO 需要设置为 true -``` - -#### 2.2 对象存储路径规则 - -**路径格式**： -``` -{prefix}/user-{user_id}/{collection_id}/{document_id}/{filename} -``` - -**组成部分**： -- `prefix`：可选的全局前缀（仅 S3） -- `user_id`：用户 ID（`|` 替换为 `-`） -- `collection_id`：集合 ID -- `document_id`：文档 ID -- `filename`：文件名（如 `original.pdf`、`page_0.png`） - -**多租户隔离**： -- 每个用户有独立的命名空间 -- 每个集合有独立的存储目录 -- 每个文档有独立的文件夹 - -### 阶段 3: 文档确认与索引构建 - -#### 3.1 确认流程 - -``` -用户点击"保存到集合" - │ - ▼ -前端调用 confirm API - │ - ▼ -Service 层处理： - │ - ├─► 验证集合配置 - │ - ├─► 检查 Quota（确认阶段才扣除配额） - │ - └─► 对每个 document_id： - │ - ├─► 验证文档状态为 UPLOADED - │ - ├─► 更新文档状态：UPLOADED → PENDING - │ - ├─► 根据集合配置创建索引记录： - │ ├─ VECTOR（向量索引，必选） - │ ├─ FULLTEXT（全文索引，必选） - │ ├─ GRAPH（知识图谱，可选） - │ ├─ SUMMARY（文档摘要，可选） - │ └─ VISION（视觉索引，可选） - │ - └─► 返回确认结果 - │ - ▼ -触发 Celery 任务：reconcile_document_indexes - │ - ▼ -后台异步处理索引构建 -``` - -#### 3.2 Quota（配额）管理 - -**检查时机**： -- ❌ 不在上传阶段检查（临时存储不占用配额） -- ✅ 在确认阶段检查（正式添加才消耗配额） - -**配额类型**： - -1. **用户全局配额** - - `max_document_count`：用户总文档数量限制 - - 默认：1000（可通过 `MAX_DOCUMENT_COUNT` 配置） - -2. **单集合配额** - - `max_document_count_per_collection`：单个集合文档数量限制 - - 不计入 `UPLOADED` 和 `DELETED` 状态的文档 - -**配额超限处理**： -- 抛出 `QuotaExceededException` -- 返回 HTTP 400 错误 -- 包含当前用量和配额上限信息 - -### 阶段 4: 文档解析与格式转换 - -#### 4.1 Parser 架构 - -系统采用**多 Parser 链式调用**架构，每个 Parser 负责特定类型的文件解析： - -``` -DocParser（主控制器） - │ - ├─► MinerUParser - │ └─ 功能：高精度 PDF 解析（商业 API） - │ └─ 支持：.pdf - │ - ├─► ImageParser - │ └─ 功能：图片内容识别（OCR + 视觉理解） - │ └─ 支持：.jpg, .png, .gif, .bmp, .tiff - │ - ├─► AudioParser - │ └─ 功能：音频转录（Speech-to-Text） - │ └─ 支持：.mp3, .wav, .m4a - │ - └─► MarkItDownParser（兜底） - └─ 功能：通用文档转 Markdown - └─ 支持：几乎所有常见格式 -``` - -#### 4.2 Parser 配置 - -**配置方式**：通过集合配置（Collection Config）动态控制 - -```json -{ - "parser_config": { - "use_mineru": false, // 是否启用 MinerU（需要 API Token） - "use_markitdown": true, // 是否启用 MarkItDown（默认） - "mineru_api_token": "xxx" // MinerU API Token（可选） - } -} -``` - -**环境变量配置**： -```bash -USE_MINERU_API=false # 全局启用 MinerU -MINERU_API_TOKEN=your_token # MinerU API Token -``` - -#### 4.3 解析流程 - -``` -Celery Worker 收到索引任务 - │ - ▼ -1. 从对象存储下载原始文件 - │ - ▼ -2. 根据文件扩展名选择 Parser - │ - ├─► 尝试第一个匹配的 Parser - │ ├─ 成功：返回解析结果 - │ └─ 失败：FallbackError → 尝试下一个 Parser - │ - └─► 最终兜底：MarkItDownParser - │ - ▼ -3. 解析结果（Parts）： - │ - ├─► MarkdownPart：文本内容 - │ └─ 包含：标题、段落、列表、表格等 - │ - ├─► PdfPart：PDF 文件 - │ └─ 用于：线性化、页面渲染 - │ - └─► AssetBinPart：二进制资源 - └─ 包含：图片、嵌入的文件等 - │ - ▼ -4. 后处理（Post-processing）： - │ - ├─► PDF 页面转图片（Vision 索引需要） - │ └─ 每页渲染为 PNG 图片 - │ └─ 保存到 {document_path}/images/page_N.png - │ - ├─► PDF 线性化（加速浏览器加载） - │ └─ 使用 pikepdf 优化 PDF 结构 - │ └─ 保存到 {document_path}/converted.pdf - │ - └─► 提取文本内容（纯文本） - └─ 合并所有 MarkdownPart 内容 - └─ 保存到 {document_path}/processed_content.md - │ - ▼ -5. 保存到对象存储 -``` - -#### 4.4 格式转换示例 - -**示例 1：PDF 文档** -``` -输入：user_manual.pdf (5 MB) - │ - ▼ -解析器选择：MinerUParser / MarkItDownParser - │ - ▼ -输出 Parts： - ├─ MarkdownPart: "# User Manual\n\n## Chapter 1\n..." - └─ PdfPart: <原始 PDF 数据> - │ - ▼ -后处理： - ├─ 渲染 50 页为图片 → images/page_0.png ~ page_49.png - ├─ 线性化 PDF → converted.pdf - └─ 提取文本 → processed_content.md -``` - -**示例 2：图片文件** -``` -输入：screenshot.png (2 MB) - │ - ▼ -解析器选择：ImageParser - │ - ▼ -输出 Parts： - ├─ MarkdownPart: "[OCR 提取的文字内容]" - └─ AssetBinPart: <原始图片数据> (vision_index=true) - │ - ▼ -后处理： - └─ 保存原图副本 → images/file.png -``` - -**示例 3：音频文件** -``` -输入：meeting_record.mp3 (50 MB) - │ - ▼ -解析器选择：AudioParser - │ - ▼ -输出 Parts： - └─ MarkdownPart: "[转录的会议内容文本]" - │ - ▼ -后处理： - └─ 保存转录文本 → processed_content.md -``` - -### 阶段 5: 索引构建 - -#### 5.1 索引类型与功能 - -| 索引类型 | 是否必选 | 功能描述 | 存储位置 | -|---------|---------|----------|----------| -| **VECTOR** | ✅ 必选 | 向量化检索，支持语义搜索 | Qdrant / Elasticsearch | -| **FULLTEXT** | ✅ 必选 | 全文检索，支持关键词搜索 | Elasticsearch | -| **GRAPH** | ❌ 可选 | 知识图谱，提取实体和关系 | Neo4j / PostgreSQL | -| **SUMMARY** | ❌ 可选 | 文档摘要，LLM 生成 | PostgreSQL (index_data) | -| **VISION** | ❌ 可选 | 视觉理解，图片内容分析 | Qdrant (向量) + PG (metadata) | - -#### 5.2 索引构建流程 - -``` -Celery Worker: reconcile_document_indexes 任务 - │ - ▼ -1. 扫描 DocumentIndex 表，找到需要处理的索引 - │ - ├─► PENDING 状态 + observed_version < version - │ └─ 需要创建或更新索引 - │ - └─► DELETING 状态 - └─ 需要删除索引 - │ - ▼ -2. 按文档分组，逐个处理 - │ - ▼ -3. 对每个文档： - │ - ├─► parse_document（解析文档） - │ ├─ 从对象存储下载原始文件 - │ ├─ 调用 DocParser 解析 - │ └─ 返回 ParsedDocumentData - │ - └─► 对每个索引类型： - │ - ├─► create_index (创建/更新索引) - │ │ - │ ├─ VECTOR 索引： - │ │ ├─ 文档分块（Chunking） - │ │ ├─ Embedding 模型生成向量 - │ │ └─ 写入 Qdrant - │ │ - │ ├─ FULLTEXT 索引： - │ │ ├─ 提取纯文本内容 - │ │ ├─ 按段落/章节分块 - │ │ └─ 写入 Elasticsearch - │ │ - │ ├─ GRAPH 索引： - │ │ ├─ 使用 LightRAG 提取实体 - │ │ ├─ 提取实体间关系 - │ │ └─ 写入 Neo4j/PostgreSQL - │ │ - │ ├─ SUMMARY 索引： - │ │ ├─ 调用 LLM 生成摘要 - │ │ └─ 保存到 DocumentIndex.index_data - │ │ - │ └─ VISION 索引： - │ ├─ 提取图片 Assets - │ ├─ Vision LLM 理解图片内容 - │ ├─ 生成图片描述向量 - │ └─ 写入 Qdrant - │ - └─► 更新索引状态 - ├─ 成功：CREATING → ACTIVE - └─ 失败：CREATING → FAILED - │ - ▼ -4. 更新文档总体状态 - │ - ├─ 所有索引都 ACTIVE → Document.status = COMPLETE - ├─ 任一索引 FAILED → Document.status = FAILED - └─ 部分索引仍在处理 → Document.status = RUNNING -``` - -#### 5.3 文档分块（Chunking） - -**分块策略**： -- 递归字符分割（RecursiveCharacterTextSplitter） -- 按自然段落、章节优先切分 -- 保留上下文重叠（Overlap） - -**分块参数**： -```json -{ - "chunk_size": 1000, // 每块最大字符数 - "chunk_overlap": 200, // 重叠字符数 - "separators": ["\n\n", "\n", " ", ""] // 分隔符优先级 -} -``` - -**分块结果存储**： -``` -{document_path}/chunks/ - ├─ chunk_0.json: {"text": "...", "metadata": {...}} - ├─ chunk_1.json: {"text": "...", "metadata": {...}} - └─ ... -``` - -## 数据库设计 - -### 表 1: document（文档元数据） - -**表结构**： - -| 字段名 | 类型 | 说明 | 索引 | -|--------|------|------|------| -| `id` | String(24) | 文档 ID，主键，格式：`doc{random_id}` | PK | -| `name` | String(1024) | 文件名 | - | -| `user` | String(256) | 用户 ID（支持多种 IDP） | ✅ Index | -| `collection_id` | String(24) | 所属集合 ID | ✅ Index | -| `status` | Enum | 文档状态（见下表） | ✅ Index | -| `size` | BigInteger | 文件大小（字节） | - | -| `content_hash` | String(64) | SHA-256 哈希（用于去重） | ✅ Index | -| `object_path` | Text | 对象存储路径（已废弃，用 doc_metadata） | - | -| `doc_metadata` | Text | 文档元数据（JSON 字符串） | - | -| `gmt_created` | DateTime(tz) | 创建时间（UTC） | - | -| `gmt_updated` | DateTime(tz) | 更新时间（UTC） | - | -| `gmt_deleted` | DateTime(tz) | 删除时间（软删除） | ✅ Index | - -**唯一约束**： -```sql -UNIQUE INDEX uq_document_collection_name_active - ON document (collection_id, name) - WHERE gmt_deleted IS NULL; -``` -- 同一集合内，活跃文档的名称不能重复 -- 已删除的文档不参与唯一性检查 - -**文档状态枚举**（`DocumentStatus`）： - -| 状态 | 说明 | 何时设置 | 可见性 | -|------|------|----------|--------| -| `UPLOADED` | 已上传到临时存储 | `upload_document` 接口 | 前端文件选择界面 | -| `PENDING` | 等待索引构建 | `confirm_documents` 接口 | 文档列表（处理中） | -| `RUNNING` | 索引构建中 | Celery 任务开始处理 | 文档列表（处理中） | -| `COMPLETE` | 所有索引完成 | 所有索引变为 ACTIVE | 文档列表（可用） | -| `FAILED` | 索引构建失败 | 任一索引失败 | 文档列表（失败） | -| `DELETED` | 已删除 | `delete_document` 接口 | 不可见（软删除） | -| `EXPIRED` | 临时文档过期 | 定时清理任务 | 不可见 | - -**文档元数据示例**（`doc_metadata` JSON 字段）： -```json -{ - "object_path": "user-xxx/col_xxx/doc_xxx/original.pdf", - "converted_path": "user-xxx/col_xxx/doc_xxx/converted.pdf", - "processed_content_path": "user-xxx/col_xxx/doc_xxx/processed_content.md", - "images": [ - "user-xxx/col_xxx/doc_xxx/images/page_0.png", - "user-xxx/col_xxx/doc_xxx/images/page_1.png" - ], - "parser_used": "MinerUParser", - "parse_duration_ms": 5420, - "page_count": 50, - "custom_field": "value" -} -``` - -### 表 2: document_index（索引状态管理） - -**表结构**： - -| 字段名 | 类型 | 说明 | 索引 | -|--------|------|------|------| -| `id` | Integer | 自增 ID，主键 | PK | -| `document_id` | String(24) | 关联的文档 ID | ✅ Index | -| `index_type` | Enum | 索引类型（见下表） | ✅ Index | -| `status` | Enum | 索引状态（见下表） | ✅ Index | -| `version` | Integer | 索引版本号 | - | -| `observed_version` | Integer | 已处理的版本号 | - | -| `index_data` | Text | 索引数据（JSON），如摘要内容 | - | -| `error_message` | Text | 错误信息（失败时） | - | -| `gmt_created` | DateTime(tz) | 创建时间 | - | -| `gmt_updated` | DateTime(tz) | 更新时间 | - | -| `gmt_last_reconciled` | DateTime(tz) | 最后协调时间 | - | - -**唯一约束**： -```sql -UNIQUE CONSTRAINT uq_document_index - ON document_index (document_id, index_type); -``` -- 每个文档的每种索引类型只有一条记录 - -**索引类型枚举**（`DocumentIndexType`）： - -| 类型 | 值 | 说明 | 外部存储 | -|------|-----|------|----------| -| `VECTOR` | "VECTOR" | 向量索引 | Qdrant / Elasticsearch | -| `FULLTEXT` | "FULLTEXT" | 全文索引 | Elasticsearch | -| `GRAPH` | "GRAPH" | 知识图谱 | Neo4j / PostgreSQL | -| `SUMMARY` | "SUMMARY" | 文档摘要 | PostgreSQL (index_data) | -| `VISION` | "VISION" | 视觉索引 | Qdrant + PostgreSQL | - -**索引状态枚举**（`DocumentIndexStatus`）： - -| 状态 | 说明 | 何时设置 | -|------|------|----------| -| `PENDING` | 等待处理 | `confirm_documents` 创建索引记录 | -| `CREATING` | 创建中 | Celery Worker 开始处理 | -| `ACTIVE` | 就绪可用 | 索引构建成功 | -| `DELETING` | 标记删除 | `delete_document` 接口 | -| `DELETION_IN_PROGRESS` | 删除中 | Celery Worker 正在删除 | -| `FAILED` | 失败 | 索引构建失败 | - -**版本控制机制**： -- `version`：期望的索引版本（每次文档更新时 +1） -- `observed_version`：已处理的版本号 -- `version > observed_version` 时，触发索引更新 - -**协调器（Reconciler）**： -```python -# 查询需要处理的索引 -SELECT * FROM document_index -WHERE status = 'PENDING' - AND observed_version < version; - -# 处理后更新 -UPDATE document_index -SET status = 'ACTIVE', - observed_version = version, - gmt_last_reconciled = NOW() -WHERE id = ?; -``` - -### 表关系图 - -``` -┌─────────────────────────────────┐ -│ collection │ -│ ───────────────────────────── │ -│ id (PK) │ -│ name │ -│ config (JSON) │ -│ status │ -│ ... │ -└────────────┬────────────────────┘ - │ 1:N - ▼ -┌─────────────────────────────────┐ -│ document │ -│ ───────────────────────────── │ -│ id (PK) │ -│ collection_id (FK) │◄──── 唯一约束: (collection_id, name) -│ name │ -│ user │ -│ status (Enum) │ -│ size │ -│ content_hash (SHA-256) │ -│ doc_metadata (JSON) │ -│ gmt_created │ -│ gmt_deleted │ -│ ... │ -└────────────┬────────────────────┘ - │ 1:N - ▼ -┌─────────────────────────────────┐ -│ document_index │ -│ ───────────────────────────── │ -│ id (PK) │ -│ document_id (FK) │◄──── 唯一约束: (document_id, index_type) -│ index_type (Enum) │ -│ status (Enum) │ -│ version │ -│ observed_version │ -│ index_data (JSON) │ -│ error_message │ -│ gmt_last_reconciled │ -│ ... │ -└─────────────────────────────────┘ -``` - -## 状态机与生命周期 - -### 文档状态转换 - -``` - ┌─────────────────────────────────────────────┐ - │ │ - │ ▼ - [上传文件] ──► UPLOADED ──► [确认] ──► PENDING ──► RUNNING ──► COMPLETE - │ │ - │ ▼ - │ FAILED - │ │ - │ ▼ - └──────► [删除] ──────────────► DELETED - │ - ┌───────────────────────────────────┘ - │ - ▼ - EXPIRED (定时清理未确认的文档) -``` - -**关键转换**： -1. **UPLOADED → PENDING**：用户点击"保存到集合" -2. **PENDING → RUNNING**：Celery Worker 开始处理 -3. **RUNNING → COMPLETE**：所有索引都成功 -4. **RUNNING → FAILED**：任一索引失败 -5. **任何状态 → DELETED**：用户删除文档 - -### 索引状态转换 - -``` - [创建索引记录] ──► PENDING ──► CREATING ──► ACTIVE - │ - ▼ - FAILED - │ - ▼ - ┌──────────► PENDING (重试) - │ - [删除请求] ──────┼──────────► DELETING ──► DELETION_IN_PROGRESS ──► (记录删除) - │ - └──────────► (直接删除记录，如果 PENDING/FAILED) -``` - -## 异步任务调度（Celery） - -### 任务定义 - -**主任务**：`reconcile_document_indexes` -- 触发时机： - - `confirm_documents` 接口调用后 - - 定时任务（每 30 秒） - - 手动触发（管理界面） -- 功能：扫描 `document_index` 表，处理需要协调的索引 - -**子任务**： -- `parse_document_task`：解析文档内容 -- `create_vector_index_task`：创建向量索引 -- `create_fulltext_index_task`：创建全文索引 -- `create_graph_index_task`：创建知识图谱索引 -- `create_summary_index_task`：创建摘要索引 -- `create_vision_index_task`：创建视觉索引 - -### 任务调度策略 - -**并发控制**： -- 每个 Worker 最多同时处理 N 个文档（默认 4） -- 每个文档的多个索引可以并行构建 -- 使用 Celery 的 `task_acks_late=True` 确保任务不丢失 - -**失败重试**： -- 最多重试 3 次 -- 指数退避（1分钟 → 5分钟 → 15分钟） -- 3 次失败后标记为 `FAILED` - -**幂等性**： -- 所有任务支持重复执行 -- 使用 `observed_version` 机制避免重复处理 -- 相同输入产生相同输出 - -## 设计特点与优势 - -### 1. 两阶段提交设计 - -**优势**： -- ✅ **用户体验更好**：快速上传响应，不阻塞用户操作 -- ✅ **选择性添加**：批量上传后可选择性确认部分文件 -- ✅ **资源控制合理**：未确认的文档不构建索引，不消耗配额 -- ✅ **故障恢复友好**：临时文档可以定期清理，不影响业务 - -**状态隔离**： -``` -临时状态（UPLOADED）： - - 不计入配额 - - 不触发索引 - - 可以被自动清理 - -正式状态（PENDING/RUNNING/COMPLETE）： - - 计入配额 - - 触发索引构建 - - 不会被自动清理 -``` - -### 2. 幂等性设计 - -**文件级别幂等**： -- SHA-256 哈希去重 -- 相同文件多次上传返回同一 `document_id` -- 避免存储空间浪费 - -**接口级别幂等**： -- `upload_document`：重复上传返回已存在文档 -- `confirm_documents`：重复确认不会创建重复索引 -- `delete_document`：重复删除返回成功（软删除） - -### 3. 多租户隔离 - -**存储隔离**： -``` -user-{user_A}/... # 用户 A 的文件 -user-{user_B}/... # 用户 B 的文件 -``` - -**数据库隔离**： -- 所有查询都带 `user` 字段过滤 -- 集合级别的权限控制（`collection.user`） -- 软删除支持（`gmt_deleted`） - -### 4. 灵活的存储后端 - -**统一接口**： -```python -AsyncObjectStore: - - put(path, data) - - get(path) - - delete_objects_by_prefix(prefix) -``` - -**运行时切换**： -- 通过环境变量切换 Local/S3 -- 无需修改业务代码 -- 支持自定义存储后端（实现接口即可） - -### 5. 事务一致性 - -**数据库 + 对象存储的两阶段提交**： -```python -async with transaction: - # 1. 创建数据库记录 - document = create_document_record() - - # 2. 上传到对象存储 - await object_store.put(path, data) - - # 3. 更新元数据 - document.doc_metadata = json.dumps(metadata) - - # 所有操作成功才提交，任一失败则回滚 -``` - -**失败处理**： -- 数据库记录创建失败：不上传文件 -- 文件上传失败：回滚数据库记录 -- 元数据更新失败：回滚前面的操作 - -### 6. 可观测性 - -**审计日志**： -- `@audit` 装饰器记录所有文档操作 -- 包含：用户、时间、操作类型、资源 ID - -**任务追踪**： -- `gmt_last_reconciled`：最后处理时间 -- `error_message`：失败原因 -- Celery 任务 ID：关联日志追踪 - -**监控指标**： -- 文档上传速率 -- 索引构建耗时 -- 失败率统计 - -## 性能优化 - -### 1. 异步处理 - -**上传不阻塞**： -- 文件上传到对象存储后立即返回 -- 索引构建在 Celery 中异步执行 -- 前端通过轮询或 WebSocket 获取进度 - -### 2. 批量操作 - -**批量确认**： -```python -confirm_documents(document_ids=[id1, id2, ..., idN]) -``` -- 一次事务处理多个文档 -- 批量创建索引记录 -- 减少数据库往返 - -### 3. 缓存策略 - -**解析结果缓存**： -- 解析后的内容保存到 `processed_content.md` -- 后续索引重建可直接读取，无需重新解析 - -**分块结果缓存**： -- 分块结果保存到 `chunks/` 目录 -- 向量索引重建可复用分块结果 - -### 4. 并行索引构建 - -**多索引并行**： -```python -# VECTOR、FULLTEXT、GRAPH 可以并行构建 -await asyncio.gather( - create_vector_index(), - create_fulltext_index(), - create_graph_index() -) -``` - -## 错误处理 - -### 常见异常 - -| 异常类型 | HTTP 状态码 | 触发场景 | 处理建议 | -|---------|------------|----------|----------| -| `ResourceNotFoundException` | 404 | 集合/文档不存在 | 检查 ID 是否正确 | -| `CollectionInactiveException` | 400 | 集合未激活 | 等待集合初始化完成 | -| `DocumentNameConflictException` | 409 | 同名不同内容 | 重命名文件或删除旧文档 | -| `QuotaExceededException` | 429 | 配额超限 | 升级套餐或删除旧文档 | -| `InvalidFileTypeException` | 400 | 不支持的文件类型 | 查看支持的文件类型列表 | -| `FileSizeTooLargeException` | 413 | 文件过大 | 分割文件或压缩 | - -### 异常传播 - -``` -Service Layer 抛出异常 - │ - ▼ -View Layer 捕获并转换 - │ - ▼ -Exception Handler 统一处理 - │ - ▼ -返回标准 JSON 响应： -{ - "error_code": "QUOTA_EXCEEDED", - "message": "Document count limit exceeded", - "details": { - "limit": 1000, - "current": 1000 - } -} -``` - -## 相关文件索引 - -### 核心实现 - -- **View 层**：`aperag/views/collections.py` - HTTP 接口定义 -- **Service 层**：`aperag/service/document_service.py` - 业务逻辑 -- **数据库模型**：`aperag/db/models.py` - Document, DocumentIndex 表定义 -- **数据库操作**：`aperag/db/ops.py` - CRUD 操作封装 - -### 对象存储 - -- **接口定义**：`aperag/objectstore/base.py` - AsyncObjectStore 抽象类 -- **Local 实现**：`aperag/objectstore/local.py` - 本地文件系统存储 -- **S3 实现**：`aperag/objectstore/s3.py` - S3 兼容存储 - -### 文档解析 - -- **主控制器**：`aperag/docparser/doc_parser.py` - DocParser -- **Parser 实现**： - - `aperag/docparser/mineru_parser.py` - MinerU PDF 解析 - - `aperag/docparser/mineru_parser.py` - MinerU 文档解析 - - `aperag/docparser/markitdown_parser.py` - MarkItDown 通用解析 - - `aperag/docparser/image_parser.py` - 图片 OCR - - `aperag/docparser/audio_parser.py` - 音频转录 -- **文档处理**：`aperag/index/document_parser.py` - 解析流程编排 - -### 索引构建 - -- **索引管理**：`aperag/index/manager.py` - DocumentIndexManager -- **向量索引**：`aperag/index/vector_index.py` - VectorIndexer -- **全文索引**：`aperag/index/fulltext_index.py` - FulltextIndexer -- **知识图谱**：`aperag/index/graph_index.py` - GraphIndexer -- **文档摘要**：`aperag/index/summary_index.py` - SummaryIndexer -- **视觉索引**：`aperag/index/vision_index.py` - VisionIndexer - -### 任务调度 - -- **任务定义**：`config/celery_tasks.py` - Celery 任务注册 -- **协调器**：`aperag/tasks/reconciler.py` - DocumentIndexReconciler -- **文档任务**：`aperag/tasks/document.py` - DocumentIndexTask - -### 前端实现 - -- **文档列表**：`web/src/app/workspace/collections/[collectionId]/documents/page.tsx` -- **文档上传**：`web/src/app/workspace/collections/[collectionId]/documents/upload/document-upload.tsx` - -## 总结 - -ApeRAG 的文档上传模块采用**两阶段提交 + 多 Parser 链式调用 + 多索引并行构建**的架构设计： - -**核心特性**： -1. ✅ **两阶段提交**：上传（临时存储）→ 确认（正式添加），提供更好的用户体验 -2. ✅ **SHA-256 去重**：避免重复文档，支持幂等上传 -3. ✅ **灵活存储后端**：Local/S3 可配置切换，统一接口抽象 -4. ✅ **多 Parser 架构**：支持 MinerU、MarkItDown 等多种解析器 -5. ✅ **格式自动转换**：PDF→图片、音频→文本、图片→OCR 文本 -6. ✅ **多索引协调**：向量、全文、图谱、摘要、视觉五种索引类型 -7. ✅ **配额管理**：确认阶段才扣除配额，合理控制资源 -8. ✅ **异步处理**：Celery 任务队列，不阻塞用户操作 -9. ✅ **事务一致性**：数据库 + 对象存储的两阶段提交 -10. ✅ **可观测性**：审计日志、任务追踪、错误信息完整记录 - -这种设计既保证了高性能和可扩展性，又支持复杂的文档处理场景（多格式、多语言、多模态），同时具有良好的容错能力和用户体验。 diff --git a/web/docs/zh-CN/design/graph_index_creation.md b/web/docs/zh-CN/design/graph_index_creation.md deleted file mode 100644 index 255f6433a..000000000 --- a/web/docs/zh-CN/design/graph_index_creation.md +++ /dev/null @@ -1,1084 +0,0 @@ ---- -title: 图索引构建流程 -description: ApeRAG 知识图谱索引构建的完整流程与核心技术 -keywords: 知识图谱, Graph Index, 实体提取, 关系抽取, 并发优化 -position: 2 ---- - -# 图索引构建流程 - -## 1. 什么是图索引 - -图索引（Graph Index）是 ApeRAG 的核心特色功能，它能从非结构化文本中自动提取出结构化的知识图谱。 - -### 1.1 一个简单的例子 - -想象一下，你有一份关于公司组织架构的文档，里面提到： - -> "张三是数据库团队的负责人，他擅长 PostgreSQL 和 MySQL。李四在前端团队工作，经常和张三的团队协作开发后台管理系统。" - -**从文档到知识图谱的转换**： - -```mermaid -flowchart LR - subgraph Input[📄 输入文档] - Doc["张三是数据库团队的负责人，
他擅长 PostgreSQL 和 MySQL。
李四在前端团队工作..."] - end - - subgraph Process[🔄 图索引处理] - Extract[提取实体和关系] - end - - subgraph Output[🕸️ 知识图谱] - direction TB - A[张三
人物] -->|负责| B[数据库团队
组织] - A -->|擅长| C[PostgreSQL
技术] - A -->|擅长| D[MySQL
技术] - E[李四
人物] -->|属于| F[前端团队
组织] - E -->|协作| A - end - - Input --> Process - Process --> Output - - style Input fill:#e3f2fd - style Process fill:#fff59d - style Output fill:#c8e6c9 -``` - -传统的向量检索只能找到"语义相似"的段落，但无法回答这些问题： -- 张三负责什么？ -- 张三和李四是什么关系？ -- 数据库团队都有哪些技术栈？ - -**图索引能做到**：精确回答这些需要理解"关系"的问题，因为它把隐藏在文本中的知识关系显性化了。 - -### 1.2 核心价值 - -与传统检索方式相比，图索引提供了独特的能力： - -| 能力 | 向量检索 | 全文检索 | 图索引 | -|------|---------|---------|--------| -| 语义相似搜索 | ✅ 强 | ❌ 弱 | ✅ 强 | -| 精确关键词匹配 | ❌ 弱 | ✅ 强 | ✅ 中 | -| 关系查询 | ❌ 不支持 | ❌ 不支持 | ✅ 强 | -| 多跳推理 | ❌ 不支持 | ❌ 不支持 | ✅ 支持 | -| 适用问题 | "如何优化性能" | "PostgreSQL 配置" | "张三和李四的关系" | - -**核心优势**：图索引让 AI 能够"理解"知识之间的关联，而不仅仅是文本的相似度。 - -## 2. 图索引能解决什么问题 - -图索引特别擅长处理那些需要"理解关系"的场景。让我们看看它在实际工作中的应用。 - -### 2.1 企业知识管理 - -**场景**：公司有大量文档，包括组织架构、项目资料、技术文档等。 - -**图索引的价值**： - -- 📊 **组织关系**："张三的团队有哪些人？" → 快速找到团队成员 -- 🔗 **协作关系**："谁和张三合作过？" → 发现工作网络 -- 🛠️ **技能图谱**："谁擅长 PostgreSQL？" → 定位技术专家 -- 📁 **项目历史**："张三参与过哪些项目？" → 追溯项目经验 - -**实际效果**： - -``` -问："数据库团队负责人是谁？" -传统检索：返回包含"数据库团队"和"负责人"的所有段落（可能几十条） -图索引：直接返回"张三" + 相关背景信息 -``` - -### 2.2 研究与学习 - -**场景**：分析学术论文、技术文档，理解知识脉络。 - -**图索引的价值**： - -- 👥 **作者网络**："这个作者和谁合作过？" → 发现研究团队 -- 📖 **引用关系**："这篇论文引用了哪些文献？" → 追溯研究脉络 -- 🔬 **技术演进**："这个技术是如何发展的？" → 理解技术历史 -- 💡 **概念关联**："A 技术和 B 技术有什么关系？" → 连接知识点 - -### 2.3 产品与服务 - -**场景**：产品文档、用户手册、API 文档等。 - -**图索引的价值**： - -- ⚙️ **功能依赖**："启用 A 功能需要先配置什么？" → 理解依赖关系 -- 🔧 **配置关联**："这个配置项会影响哪些功能？" → 避免误操作 -- 🐛 **问题诊断**："出现 X 错误可能是什么原因？" → 快速定位 -- 📚 **API 关系**："这个 API 通常和哪些 API 一起使用？" → 学习最佳实践 - -### 2.4 对比：什么时候用图索引 - -不同的问题适合不同的检索方式： - -| 问题类型 | 示例 | 最佳方案 | -|---------|------|---------| -| **概念理解** | "什么是 RAG？" | 向量检索 | -| **精确查找** | "PostgreSQL 配置文件路径" | 全文检索 | -| **关系查询** | "张三和李四什么关系？" | 图索引 ✨ | -| **多跳推理** | "张三团队用的技术栈" | 图索引 ✨ | -| **知识追溯** | "这个功能依赖哪些模块？" | 图索引 ✨ | - -**最佳实践**：ApeRAG 同时支持向量检索、全文检索和图索引，可以根据问题类型智能选择或组合使用。 - -## 3. 构建流程概览 - -当你上传一个文档并启用图索引后，ApeRAG 会自动完成以下步骤。这里先给出一个简单的概览，具体细节在后面章节详细介绍。 - -### 3.1 五个关键步骤 - -```mermaid -flowchart TB - subgraph Step1["1️⃣ 文档分块"] - A1[原始文档] --> A2[智能分块] - A2 --> A3[生成 Chunks] - end - - subgraph Step2["2️⃣ 实体关系提取"] - B1[Chunks] --> B2[调用 LLM] - B2 --> B3[识别实体] - B2 --> B4[识别关系] - end - - subgraph Step3["3️⃣ 连通分量分析"] - C1[实体关系网络] --> C2[BFS 算法] - C2 --> C3[分组] - end - - subgraph Step4["4️⃣ 并发合并"] - D1[分组 1] --> D2[实体去重] - D3[分组 2] --> D4[实体去重] - D5[分组 N] --> D6[实体去重] - D2 --> D7[关系聚合] - D4 --> D7 - D6 --> D7 - end - - subgraph Step5["5️⃣ 多存储写入"] - E1[图数据库] - E2[向量数据库] - E3[文本存储] - end - - A3 --> B1 - B3 --> C1 - B4 --> C1 - C3 --> D1 - C3 --> D3 - C3 --> D5 - D7 --> E1 - D7 --> E2 - A3 --> E3 - - style Step1 fill:#e3f2fd - style Step2 fill:#fff3e0 - style Step3 fill:#f3e5f5 - style Step4 fill:#e8f5e9 - style Step5 fill:#fce4ec -``` - -**简单来说**，就是：文档分块 → 提取实体关系 → 智能分组 → 并发合并 → 写入存储。 - -整个过程完全自动化，你只需要上传文档，系统会自动完成所有工作。 - -### 3.2 处理时间参考 - -不同规模的文档，处理时间大致如下： - -| 文档大小 | 实体数量 | 处理时间 | 说明 | -|---------|---------|---------|------| -| 小型（< 5 页） | ~50 个 | 10-30 秒 | 公司通知、会议纪要 | -| 中型（10-50 页） | ~200 个 | 1-3 分钟 | 技术文档、产品手册 | -| 大型（100+ 页） | ~1000 个 | 5-15 分钟 | 研究报告、书籍 | - -**影响因素**： -- LLM 响应速度（主要瓶颈） -- 文档复杂度（表格、图片多会慢一些） -- 并发设置（可以通过配置提速） - -> 💡 **提示**：处理是异步的，你可以上传多个文档，系统会并行处理。 - -### 3.3 实时进度查看 - -你可以随时查看文档的处理进度： - -``` -文档状态：处理中 -- ✅ 文档解析：完成 -- ✅ 文档分块：完成（生成 25 个 chunks） -- 🔄 实体提取：进行中（15/25） -- ⏳ 关系提取：等待中 -- ⏳ 图谱构建：等待中 -``` - -处理完成后，文档状态会变为"活跃"，此时就可以进行图谱查询了。 - -## 4. 详细构建流程 - -前面介绍了图索引能做什么以及整体流程概览。如果你想了解更多技术细节，这一章会详细介绍每个步骤的具体实现。 - -> 💡 **阅读建议**：如果你只是想了解图索引的基本概念和用法，可以跳过这一章，直接看第 8 章的实际应用场景。 - -### 4.1 文档分块 - -第一步是把长文档切成合适大小的块（chunks）。 - -**为什么要分块？** -- LLM 有输入长度限制（通常几千到几万 tokens） -- 块太大：提取质量下降，LLM 容易"遗漏"信息 -- 块太小：丢失上下文，无法理解完整语义 - -**智能分块策略**： - -```mermaid -flowchart LR - Doc[长文档] --> Check{检查大小} - Check -->|小于 1200 tokens| Keep[保持完整] - Check -->|大于 1200 tokens| Split[智能分割] - - Split --> By1[按段落分] - By1 --> Check2{还是太大?} - Check2 -->|是| By2[按句子分] - Check2 -->|否| Done[完成] - By2 --> Check3{还是太大?} - Check3 -->|是| By3[按字符分] - Check3 -->|否| Done - By3 --> Done - - style Doc fill:#e1f5ff - style Split fill:#ffccbc - style Done fill:#c5e1a5 -``` - -**分块参数**： -- 默认大小：1200 tokens（约 800-1000 个中文字） -- 重叠大小：100 tokens（保证上下文连续） -- 优先级：段落 > 句子 > 字符 - -### 4.2 实体关系提取 - -使用 LLM 从每个 chunk 中识别实体和关系。 - -**提取过程**： - -```mermaid -sequenceDiagram - participant C as Chunk - participant L as LLM - participant R as 结果 - - C->>L: "张三是数据库团队负责人..." - L->>R: 实体: [张三(人物), 数据库团队(组织)] - L->>R: 关系: [张三-负责->数据库团队] - - C->>L: "张三擅长 PostgreSQL..." - L->>R: 实体: [张三(人物), PostgreSQL(技术)] - L->>R: 关系: [张三-擅长->PostgreSQL] -``` - -**并发优化**：多个 chunks 可以同时调用 LLM，默认并发 20 个请求。 - -### 4.3 连通分量分析 - -把实体关系网络分成独立的子图，实现并行处理。 - -**为什么需要这一步？** - -技术团队的实体和财务部门的实体之间没有连接，可以完全并行处理！ - -```mermaid -graph LR - subgraph 分量1[连通分量 1 - 技术团队] - A1[张三] -->|负责| A2[数据库团队] - A1 -->|擅长| A3[PostgreSQL] - A4[李四] -->|协作| A1 - end - - subgraph 分量2[连通分量 2 - 财务部门] - B1[王五] -->|属于| B2[财务部] - B3[赵六] -->|协作| B1 - end - - style 分量1 fill:#bbdefb - style 分量2 fill:#c5e1a5 -``` - -**性能提升**：3 个独立分量 = 3 倍加速！ - -### 4.4 并发合并 - -同名实体需要去重，相同关系需要聚合。 - -```mermaid -flowchart TD - subgraph Before["合并前"] - A1["张三
数据库负责人"] - A2["张三
擅长 PostgreSQL"] - A3["张三
带领团队"] - end - - Merge[智能合并] - - subgraph After["合并后"] - B1["张三
数据库团队负责人，
擅长 PostgreSQL，
带领团队完成多个项目"] - end - - A1 --> Merge - A2 --> Merge - A3 --> Merge - Merge --> B1 - - style Before fill:#ffccbc - style After fill:#c5e1a5 -``` - -**细粒度锁**：只锁定正在合并的实体，其他实体可以并发处理。 - -### 4.5 多存储写入 - -知识图谱写入三个存储系统： - -```mermaid -flowchart LR - KG[知识图谱] --> G[图数据库
图查询] - KG --> V[向量数据库
语义搜索] - KG --> T[文本存储
全文检索] - - style KG fill:#e1f5ff - style G fill:#bbdefb - style V fill:#c5e1a5 - style T fill:#ffccbc -``` - -不同存储支持不同类型的查询，互相补充。 - -## 5. 核心技术设计 - -这一章介绍 ApeRAG 图索引的核心技术设计，包括数据隔离、并发控制等。 - -> 💡 **阅读建议**：这些是系统架构和实现细节，主要面向开发者和技术决策者。 - -### 5.1 workspace 数据隔离 - -每个 Collection 拥有独立的命名空间，实现完全的数据隔离。 - -**命名规范**： - -```python -# 实体命名 -entity:{entity_name}:{workspace} -# 示例 -entity:张三:collection_abc123 - -# 关系命名 -relationship:{source}:{target}:{workspace} -# 示例 -relationship:张三:数据库团队:collection_abc123 -``` - -**隔离效果**： - -```mermaid -graph TB - subgraph Collection_A[Collection A - 公司文档] - A1[entity:张三:A] --> A2[entity:数据库团队:A] - end - - subgraph Collection_B[Collection B - 学校文档] - B1[entity:张三:B] --> B2[entity:计算机系:B] - end - - style Collection_A fill:#bbdefb - style Collection_B fill:#c5e1a5 -``` - -两个 Collection 中的"张三"完全独立，互不干扰！ - -### 5.2 无状态实例管理 - -每个处理任务创建独立的图索引实例，处理完成后销毁。 - -**生命周期管理**： - -```mermaid -sequenceDiagram - participant C as Celery Task - participant M as Manager - participant R as Graph Index Instance - participant S as Storage - - C->>M: process_document() - M->>R: create_instance() - R->>S: 初始化存储连接 - R->>R: 处理文档 - R->>S: 写入数据 - R-->>M: 返回结果 - M-->>C: 任务完成 - Note over R: 实例被销毁，资源释放 -``` - -**优势**： - -- ✅ 零状态污染：每个任务独立，不会互相干扰 -- ✅ 自动资源管理：实例销毁时自动释放资源 -- ✅ 易于扩展：可以同时运行多个 Worker - -### 5.3 连通分量并发优化 - -通过图拓扑分析，实现智能并发处理。 - -**算法原理**： - -```mermaid -graph TB - subgraph Input[输入：实体关系网络] - I1[实体 1] --> I2[实体 2] - I2 --> I3[实体 3] - - I4[实体 4] --> I5[实体 5] - - I6[实体 6] - end - - Algorithm[BFS 算法] - - subgraph Output[输出：3 个连通分量] - O1[分量 1
3 个实体] - O2[分量 2
2 个实体] - O3[分量 3
1 个实体] - end - - Input --> Algorithm - Algorithm --> Output - - style Input fill:#ffccbc - style Algorithm fill:#fff59d - style Output fill:#c5e1a5 -``` - -**性能提升**：3 个分量并发处理 = 3 倍加速！ - -### 5.4 细粒度并发控制 - -实现实体级别的精确锁定： - -**锁的层次**： - -```mermaid -graph TD - A[全局锁 - 传统方案] -->|太粗| B[所有实体串行处理] - - C[实体锁 - ApeRAG] -->|刚好| D[只锁定需要合并的实体] - - style A fill:#ffccbc - style B fill:#ffccbc - style C fill:#c5e1a5 - style D fill:#c5e1a5 -``` - -**锁策略**： -1. 提取阶段无锁：完全并行 -2. 合并阶段加锁：只锁需要的实体 -3. 排序获取锁：避免死锁 - -### 5.5 智能摘要生成 - -自动压缩过长的描述内容： - -```python -if len(description) > 2000 tokens: - summary = await llm_summarize(description) -else: - summary = description -``` - -**效果**：2500 tokens 压缩到 200 tokens，保留核心信息。 - -### 5.6 多存储后端支持 - -ApeRAG 支持两种图数据库：Neo4j 和 PostgreSQL。 - -**如何选择？** - -| 场景 | 推荐方案 | 原因 | -|------|---------|------| -| **小规模**（< 10万实体） | PostgreSQL | 运维简单，成本低 | -| **中等规模**（10-100万） | PostgreSQL 或 Neo4j | 根据查询复杂度选择 | -| **大规模**（> 100万） | Neo4j | 图查询性能更好 | -| **预算有限** | PostgreSQL | 无需额外部署 | -| **复杂图算法** | Neo4j | 内置图算法支持 | - -**切换方式**： - -```bash -# 使用 PostgreSQL（默认） -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage - -# 使用 Neo4j -export GRAPH_INDEX_GRAPH_STORAGE=Neo4JSyncStorage -``` - -**性能对比**： - -| 操作 | PostgreSQL | Neo4j | -|------|-----------|-------| -| **简单查询**（1-2跳） | 快 | 快 | -| **复杂查询**（3+跳） | 中 | 快 | -| **批量写入** | 快 | 中 | -| **图算法** | 需要自己实现 | 内置支持 | - -## 6. 完整数据流 - -整个图索引构建过程是一个数据转换流水线，从非结构化文本到结构化知识图谱： - -```mermaid -flowchart TD - A[原始文档] --> B[清理预处理] - B --> C[智能分块] - C --> D[Chunks] - - D --> E[LLM 并发提取] - E --> F[原始实体列表] - E --> G[原始关系列表] - - F --> H[构建邻接图] - G --> H - H --> I[BFS 发现连通分量] - I --> J[分组并发处理] - - J --> K[实体去重合并] - J --> L[关系聚合] - - K --> M{描述长度检查} - M -->|过长| N[LLM 摘要] - M -->|适中| O[保留原文] - N --> P[最终实体] - O --> P - - L --> Q{描述长度检查} - Q -->|过长| R[LLM 摘要] - Q -->|适中| S[保留原文] - R --> T[最终关系] - S --> T - - P --> U[图数据库] - P --> V[向量数据库] - T --> U - T --> V - D --> W[文本存储] - - U --> X[知识图谱完成] - V --> X - W --> X - - style A fill:#e1f5ff - style E fill:#fff59d - style I fill:#f3e5f5 - style J fill:#c5e1a5 - style X fill:#c8e6c9 -``` - -### 数据转换示例 - -让我们用一个具体例子，看看数据是如何一步步转换的： - -**输入文档**： - -```text -张三是数据库团队的负责人，他擅长 PostgreSQL 和 MySQL。 -李四在前端团队工作，经常和张三的团队协作开发后台管理系统。 -王五是财务部的会计，负责公司的财务报表。 -``` - -**Step 1: 分块** - -```json -[ - { - "chunk_id": "chunk-001", - "content": "张三是数据库团队的负责人，他擅长 PostgreSQL 和 MySQL。", - "tokens": 25 - }, - { - "chunk_id": "chunk-002", - "content": "李四在前端团队工作，经常和张三的团队协作开发后台管理系统。", - "tokens": 28 - }, - { - "chunk_id": "chunk-003", - "content": "王五是财务部的会计，负责公司的财务报表。", - "tokens": 20 - } -] -``` - -**Step 2: 实体关系提取** - -```json -{ - "entities": [ - {"name": "张三", "type": "人物", "source": "chunk-001"}, - {"name": "数据库团队", "type": "组织", "source": "chunk-001"}, - {"name": "PostgreSQL", "type": "技术", "source": "chunk-001"}, - {"name": "MySQL", "type": "技术", "source": "chunk-001"}, - {"name": "李四", "type": "人物", "source": "chunk-002"}, - {"name": "前端团队", "type": "组织", "source": "chunk-002"}, - {"name": "王五", "type": "人物", "source": "chunk-003"}, - {"name": "财务部", "type": "组织", "source": "chunk-003"} - ], - "relationships": [ - {"source": "张三", "target": "数据库团队", "relation": "负责"}, - {"source": "张三", "target": "PostgreSQL", "relation": "擅长"}, - {"source": "张三", "target": "MySQL", "relation": "擅长"}, - {"source": "李四", "target": "前端团队", "relation": "属于"}, - {"source": "李四", "target": "张三", "relation": "协作"}, - {"source": "王五", "target": "财务部", "relation": "属于"} - ] -} -``` - -**Step 3: 连通分量分析** - -``` -连通分量 1（技术部门）： -- 实体：张三、李四、数据库团队、前端团队、PostgreSQL、MySQL -- 关系：6 条 - -连通分量 2（财务部门）： -- 实体：王五、财务部 -- 关系：1 条 -``` - -**Step 4: 并发合并** - -两个分量可以并行处理！ - -**Step 5: 最终知识图谱** - -```mermaid -graph LR - subgraph 技术部门 - 张三 -->|负责| 数据库团队 - 张三 -->|擅长| PostgreSQL - 张三 -->|擅长| MySQL - 李四 -->|属于| 前端团队 - 李四 -->|协作| 张三 - end - - subgraph 财务部门 - 王五 -->|属于| 财务部 - end - - style 技术部门 fill:#bbdefb - style 财务部门 fill:#c5e1a5 -``` - -### 性能优化特性 - -1. **细粒度并发控制** - - 实体级别的锁：`entity:张三:collection_abc` - - 只在合并时加锁，提取时完全并行 - -2. **连通分量并发** - - 技术部门和财务部门可以并行处理 - - 零锁竞争，充分利用多核 CPU - -3. **智能摘要** - - 描述 < 2000 tokens：保留原文 - - 描述 > 2000 tokens：LLM 摘要压缩 - -## 7. 性能优化策略 - -### 7.1 并发度控制 - -图索引构建涉及大量的 LLM 调用和数据库操作，需要合理控制并发度。 - -**并发层次**： - -```mermaid -graph TB - A[文档级并发] --> B[Chunk 级并发] - B --> C[连通分量级并发] - C --> D[实体级并发] - - A1[Celery Workers
多个文档同时处理] --> A - B1[LLM 并发调用
多个 chunks 同时提取] --> B - C1[分量并行合并
多个分量同时处理] --> C - D1[实体并发合并
不同实体同时合并] --> D - - style A fill:#e3f2fd - style B fill:#fff3e0 - style C fill:#f3e5f5 - style D fill:#e8f5e9 -``` - -**并发参数配置**： - -| 参数 | 默认值 | 说明 | -|------|--------|------| -| `llm_model_max_async` | 20 | LLM 并发调用数 | -| `embedding_func_max_async` | 16 | Embedding 并发调用数 | -| `max_batch_size` | 32 | 批量处理大小 | - -**调优建议**： - -```python -# 场景 1：LLM API 限流严格 -llm_model_max_async = 5 # 降低并发，避免触发限流 - -# 场景 2：性能充足，想提速 -llm_model_max_async = 50 # 提高并发，加快处理速度 - -# 场景 3：内存有限 -max_batch_size = 16 # 减小批量大小，降低内存占用 -``` - -### 7.2 LLM 调用优化 - -LLM 调用是最耗时的环节，主要优化策略： - -1. **并发调用**：多个 chunks 同时提取（默认并发 20 个） -2. **批量处理**：减少 LLM 调用次数 -3. **缓存复用**：相似描述复用摘要结果 - -**性能提升**：并发调用比串行快 4 倍。 - -### 7.3 存储优化 - -批量写入可以显著提升性能： - -| 方式 | 100 个实体写入时间 | -|------|------------------| -| 逐个写入 | ~10 秒 | -| 批量写入（32 个/批） | ~1 秒 | - -**优化效果**：10 倍速度提升！ - -### 7.4 内存优化 - -大文档处理的内存管理策略： - -- 流式分块：不一次性加载整个文档 -- 及时释放：处理完立即释放内存 -- 分批处理：控制内存峰值 - -### 7.5 性能监控 - -系统会输出详细的性能统计： - -``` -图索引构建完成： -✓ 文档分块：10 个 chunks，耗时 0.5 秒 -✓ 实体提取：120 个实体，耗时 25 秒 -✓ 关系提取：85 个关系，耗时 25 秒 -✓ 并发合并：耗时 15 秒 -✓ 存储写入：耗时 2 秒 -━━━━━━━━━━━━━━━━━━━━━━━━━ -总耗时：42.7 秒 -``` - -**瓶颈分析**：实体/关系提取占 60% 时间，可通过提高 LLM 并发度优化。 - -## 8. 配置参数 - -### 8.1 核心配置 - -图索引构建可以通过以下参数进行调优： - -**分块参数**： - -```python -# 分块大小（tokens） -CHUNK_TOKEN_SIZE = 1200 - -# 重叠大小（tokens） -CHUNK_OVERLAP_TOKEN_SIZE = 100 -``` - -**调优建议**： -- 小文档（< 5000 tokens）：`CHUNK_TOKEN_SIZE = 800` -- 大文档（> 50000 tokens）：`CHUNK_TOKEN_SIZE = 1500` -- 需要更多上下文：增加 `CHUNK_OVERLAP_TOKEN_SIZE` - -**并发参数**： - -```python -# LLM 并发调用数 -LLM_MODEL_MAX_ASYNC = 20 - -# Embedding 并发调用数 -EMBEDDING_FUNC_MAX_ASYNC = 16 - -# 批量处理大小 -MAX_BATCH_SIZE = 32 -``` - -**调优建议**： -- LLM API 限流严格：降低 `LLM_MODEL_MAX_ASYNC` 到 5-10 -- 性能充足想提速：提高到 50-100 -- 内存有限：降低 `MAX_BATCH_SIZE` 到 16 - -**实体提取参数**： - -```python -# 实体提取重试次数（0 = 只提取 1 次） -ENTITY_EXTRACT_MAX_GLEANING = 0 - -# 摘要最大 token 数 -SUMMARY_TO_MAX_TOKENS = 2000 - -# 强制摘要的描述片段数 -FORCE_LLM_SUMMARY_ON_MERGE = 10 -``` - -**调优建议**： -- 提取质量重要：`ENTITY_EXTRACT_MAX_GLEANING = 1`（多提取一次） -- 追求速度：`ENTITY_EXTRACT_MAX_GLEANING = 0` -- 描述经常很长：降低 `SUMMARY_TO_MAX_TOKENS` 到 1000 - -### 8.2 知识图谱配置 - -在 Collection 配置中可以设置： - -```json -{ - "knowledge_graph_config": { - "language": "simplified chinese", - "entity_types": [ - "organization", - "person", - "geo", - "event", - "product", - "technology", - "date", - "category" - ] - } -} -``` - -**参数说明**： - -- **language**：提取语言，影响 LLM 提示词 - - `simplified chinese`：简体中文 - - `English`：英文 - - `traditional chinese`：繁体中文 - -- **entity_types**：要提取的实体类型 - - 默认：8 种类型（组织、人物、地点、事件、产品、技术、日期、类别） - - 可自定义：比如只提取人物和组织 - -### 8.3 存储配置 - -通过环境变量配置存储后端： - -```bash -# KV 存储（键值对） -export GRAPH_INDEX_KV_STORAGE=PGOpsSyncKVStorage - -# 向量存储 -export GRAPH_INDEX_VECTOR_STORAGE=PGOpsSyncVectorStorage - -# 图存储 -export GRAPH_INDEX_GRAPH_STORAGE=Neo4JSyncStorage -# 或者使用 PostgreSQL -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage -``` - -**存储选择建议**： - -| 场景 | KV 存储 | 向量存储 | 图存储 | -|------|---------|---------|--------| -| **默认** | PostgreSQL | PostgreSQL | PostgreSQL | -| **高性能向量搜索** | PostgreSQL | Qdrant | Neo4j | -| **大规模图谱** | PostgreSQL | Qdrant | Neo4j | -| **简单部署** | PostgreSQL | PostgreSQL | PostgreSQL | - -### 8.4 完整配置示例 - -```bash -# 分块配置 -export CHUNK_TOKEN_SIZE=1200 -export CHUNK_OVERLAP_TOKEN_SIZE=100 - -# 并发配置 -export LLM_MODEL_MAX_ASYNC=20 -export MAX_BATCH_SIZE=32 - -# 提取配置 -export ENTITY_EXTRACT_MAX_GLEANING=0 -export SUMMARY_TO_MAX_TOKENS=2000 - -# 存储配置 -export GRAPH_INDEX_KV_STORAGE=PGOpsSyncKVStorage -export GRAPH_INDEX_VECTOR_STORAGE=PGOpsSyncVectorStorage -export GRAPH_INDEX_GRAPH_STORAGE=PGOpsSyncGraphStorage - -# 数据库连接（PostgreSQL） -export POSTGRES_HOST=127.0.0.1 -export POSTGRES_PORT=5432 -export POSTGRES_DB=aperag -export POSTGRES_USER=postgres -export POSTGRES_PASSWORD=your_password - -# 数据库连接（Neo4j，可选） -export NEO4J_HOST=127.0.0.1 -export NEO4J_PORT=7687 -export NEO4J_USERNAME=neo4j -export NEO4J_PASSWORD=your_password -``` - -## 9. 实际应用场景 - -图索引特别适合以下场景： - -### 9.1 企业知识库 - -**场景描述**：公司有大量的技术文档、组织架构、项目资料。 - -**图索引的价值**： - -- ✅ 理解人员关系：谁和谁在一起工作过 -- ✅ 追溯项目历史：哪些人参与了哪些项目 -- ✅ 技术栈分析：哪个团队用什么技术 -- ✅ 知识传承：某个领域的专家是谁 - -**查询示例**： - -``` -用户："张三参与过哪些项目？" -图索引：查询 张三 --参与--> 项目 的关系 -结果：项目 A、项目 B、项目 C - -用户："数据库团队都有哪些人？" -图索引：查询 人物 --属于--> 数据库团队 的关系 -结果：张三、李四、王五 -``` - -### 8.2 研究论文分析 - -**场景描述**：分析大量学术论文，理解研究脉络。 - -**图索引的价值**： - -- ✅ 作者合作网络：谁和谁合作过 -- ✅ 引用关系：哪些论文互相引用 -- ✅ 研究主题：某个领域的核心概念 -- ✅ 技术演进：技术如何发展的 - -**查询示例**： - -``` -用户："Graph RAG 相关的研究有哪些？" -图索引：查询 论文 --研究--> Graph RAG 的关系 -结果：论文 A、论文 B、论文 C - -用户："某作者和谁合作过？" -图索引：查询 作者 --合作--> 其他作者 的关系 -结果：合作者列表及合作项目 -``` - -### 8.3 产品文档 - -**场景描述**：软件产品的用户手册、API 文档。 - -**图索引的价值**： - -- ✅ 功能依赖：某个功能依赖哪些其他功能 -- ✅ API 关联：哪些 API 经常一起使用 -- ✅ 配置关系：某个配置项影响哪些功能 -- ✅ 问题诊断：出现某个错误可能是什么原因 - -**查询示例**： - -``` -用户："如何配置图索引？" -图索引：查询 配置项 --影响--> 图索引 的关系 -结果：GRAPH_INDEX_GRAPH_STORAGE、knowledge_graph_config - -用户："Neo4j 和 PostgreSQL 有什么区别？" -图索引：查询 Neo4j、PostgreSQL 的属性和关系 -结果：性能对比、适用场景、配置方式 -``` - -### 8.4 对话场景对比 - -让我们看看不同检索方式在实际对话中的表现： - -**问题："张三和李四是什么关系？"** - -| 检索方式 | 能否回答 | 回答质量 | -|---------|---------|---------| -| **纯向量检索** | ⚠️ 部分 | 找到提到两人的段落，但不清楚关系 | -| **纯全文检索** | ⚠️ 部分 | 找到包含"张三"和"李四"的段落 | -| **图索引** | ✅ 可以 | 直接返回：张三和李四是协作关系 | - -**问题："PostgreSQL 配置文件在哪？"** - -| 检索方式 | 能否回答 | 回答质量 | -|---------|---------|---------| -| **纯向量检索** | ✅ 可以 | 找到相关配置段落 | -| **纯全文检索** | ✅ 可以 | 精确匹配"PostgreSQL"和"配置" | -| **图索引** | ✅ 可以 | 找到 PostgreSQL --配置--> 文件 的关系 | - -**问题："如何提升系统性能？"** - -| 检索方式 | 能否回答 | 回答质量 | -|---------|---------|---------| -| **纯向量检索** | ✅ 强 | 找到所有性能优化相关内容 | -| **纯全文检索** | ⚠️ 中 | 需要精确关键词"性能"、"优化" | -| **图索引** | ✅ 强 | 找到 优化方法 --提升--> 性能 的关系 | - -**最佳实践**：结合使用多种检索方式！ - -## 10. 总结 - -ApeRAG 的图索引提供了生产级的知识图谱构建能力，具有高性能、高可靠性和易扩展的特点。 - -### 关键特性 - -1. **workspace 数据隔离**：每个 Collection 完全独立，支持真正的多租户 -2. **无状态架构**：每个任务独立实例，零状态污染 -3. **连通分量并发**：智能并发策略，性能提升 2-3 倍 -4. **细粒度锁管理**：实体级别的锁，最大化并发度 -5. **智能摘要**：自动压缩过长描述，节省存储和提升检索效率 -6. **多存储支持**：灵活选择 Neo4j 或 PostgreSQL - -### 适用场景 - -- ✅ **企业知识库**：理解组织结构、人员关系、项目历史 -- ✅ **研究论文分析**：作者合作网络、引用关系、研究脉络 -- ✅ **产品文档**：功能依赖、配置关系、问题诊断 -- ✅ **任何需要理解"关系"的场景** - -### 性能表现 - -- 处理 10,000 个实体：约 2-5 分钟（取决于 LLM 速度） -- 连通分量并发：性能提升 2-3 倍 -- 内存占用：约 400 MB（10,000 个实体） -- 存储空间：约 100 MB（10,000 个实体） - -### 下一步 - -图索引构建完成后，就可以进行图谱检索了。ApeRAG 支持三种图谱查询模式： - -- **Local 模式**：查询某个实体的局部信息 -- **Global 模式**：查询整体关系和模式 -- **Hybrid 模式**：综合性查询 - -详细的检索流程请参考 [系统架构文档](./architecture.md#42-知识图谱查询)。 - ---- - -## 相关文档 - -- 📋 [系统架构](./architecture.md) - ApeRAG 整体架构设计 -- 📖 [实体提取与合并机制](./lightrag_entity_extraction_and_merging.md) - 核心算法详解 -- 🔗 [连通分量优化](./connected_components_optimization.md) - 并发优化原理 -- 🌐 [索引链路架构](./indexing_architecture.md) - 完整索引流程 diff --git a/web/docs/zh-CN/development/_category.yaml b/web/docs/zh-CN/development/_category.yaml deleted file mode 100644 index a264578f0..000000000 --- a/web/docs/zh-CN/development/_category.yaml +++ /dev/null @@ -1,2 +0,0 @@ -title: 开发 -position: 4 diff --git a/web/docs/zh-CN/development/development-guide.md b/web/docs/zh-CN/development/development-guide.md deleted file mode 100644 index fc1caaade..000000000 --- a/web/docs/zh-CN/development/development-guide.md +++ /dev/null @@ -1,387 +0,0 @@ ---- -title: 开发指南 -description: ApeRAG 开发环境设置和工作流程 ---- - -# 🛠️ 开发指南 - -本指南重点介绍如何为 ApeRAG 设置开发环境和开发工作流程。这是为希望为 ApeRAG 做贡献或在本地运行它进行开发的开发人员设计的。 - -## 🚀 开发环境设置 - -按照以下步骤从源代码设置 ApeRAG 进行开发： - -### 1. 📂 克隆仓库并设置环境 - -首先，获取源代码并配置环境变量： - -```bash -git clone https://github.com/apecloud/ApeRAG.git -cd ApeRAG -cp envs/env.template .env -``` - -如果需要，编辑 `.env` 文件以配置您的 AI 服务设置。默认设置适用于下一步启动的本地数据库服务。 - -### 2. 📋 系统前提条件 - -在开始之前，请确保您的系统具备： - -* **Node.js**：推荐版本 20 或更高版本用于前端开发。[下载 Node.js](https://nodejs.org/) -* **Docker & Docker Compose**：本地运行数据库服务所需。[下载 Docker](https://docs.docker.com/get-docker/) - -**注意**：需要 Python 3.11，但将在下一步中通过 `uv` 自动管理。 - -### 3. 🗄️ 启动数据库服务 - -使用 Docker Compose 启动必要的数据库服务： - -```bash -# 启动核心数据库：PostgreSQL、Redis、Qdrant、Elasticsearch -make infra-up -``` - -这将在后台启动所有必需的数据库服务。您的 `.env` 文件中的默认连接设置已预配置为与这些服务一起工作。 - -
-
-
-### 4. ⚙️ 设置开发环境
-
-创建 Python 虚拟环境并设置开发工具：
-
-```bash
-make env-dev
-```
-
-此命令将：
-* 如果尚未可用，则安装 `uv`
-* 创建 Python 3.11 虚拟环境（位于 `.venv/` 中）
-* 安装开发工具（redocly、openapi-generator-cli 等）
-* 为代码质量安装 pre-commit hooks
-* 安装 addlicense 工具进行许可证管理
-
-**激活虚拟环境：**
-```bash
-source .venv/bin/activate
-```
-
-当您在终端提示符中看到 `(.venv)` 时，您就知道它是活动的。
-
-### 5. 📦 安装依赖项
-
-安装所有后端和前端依赖项：
-
-```bash
-make env-install
-```
-
-此命令将：
-* 将 `pyproject.toml` 中的所有 Python 后端依赖项安装到虚拟环境中
-* 使用 `yarn` 安装前端 Node.js 依赖项
-
-### 6. 🔄 应用数据库迁移
-
-设置数据库架构：
-
-```bash
-make db-migrate
-```
-
-### 7. ▶️ 启动开发服务
-
-现在您可以启动开发服务。为每个服务打开单独的终端窗口/选项卡：
-
-**终端 1 - 后端 API 服务器：**
-```bash
-make serve-api
-```
-这将在 `http://localhost:8000` 启动 FastAPI 开发服务器，代码更改时自动重新加载。
-
-**终端 2 - Celery Worker：**
-```bash
-make serve-worker
-```
-这将启动 Celery worker 以处理异步后台任务。
-
-**终端 3 - 前端（可选）：**
-```bash
-make serve-web
-```
-这将在 `http://localhost:3000` 启动前端开发服务器，支持热重载。
-
-### 8. 🌐 访问 ApeRAG
-
-服务运行后，您可以访问：
-* **前端 UI**：http://localhost:3000 (如果已启动)
-* **后端 API**：http://localhost:8000
-* **API 文档**：http://localhost:8000/docs
-
-### 9. ⏹️ 停止服务
-
-要停止开发环境：
-
-**停止数据库服务：**
-```bash
-# 停止数据库服务（保留数据）
-make stack-down
-
-# 停止服务并移除所有数据卷
-make stack-down REMOVE_VOLUMES=1
-```
-
-**停止开发服务：**
-- 后端 API 服务器：在运行 `make serve-api` 的终端中按 `Ctrl+C`
-- Celery Worker：在运行 `make serve-worker` 的终端中按 `Ctrl+C`
-- 前端服务器：在运行 `make serve-web` 的终端中按 `Ctrl+C`
-
-**数据管理：**
-- `make stack-down` - 停止服务但保留所有数据（PostgreSQL、Redis、Qdrant 等）
-- `make stack-down REMOVE_VOLUMES=1` - 停止服务并**⚠️ 永久删除所有数据**
-- 即使已经运行过 `make stack-down`，您也可以运行 `make stack-down REMOVE_VOLUMES=1`
-
-**验证数据移除：**
-```bash
-# 检查卷是否仍然存在
-docker volume ls | grep aperag
-
-# REMOVE_VOLUMES=1 后应该不返回结果
-```
-
-现在您已经从源代码本地运行 ApeRAG，准备好进行开发！🎉
-
-## ❓ 常见开发任务
-
-### Q: 🔧 如何添加或修改 REST API 端点？
-
-**完整工作流程：**
-1. 编辑 OpenAPI 规范：`aperag/api/paths/[endpoint-name].yaml`
-2. 重新生成后端模型：
- ```bash
- make api-generate-models # 这会在内部运行 merge-openapi
- ```
-3. 实现后端视图：`aperag/views/[module].py`
-4. 生成前端 TypeScript 客户端：
- ```bash
- make api-generate-sdk # 更新 frontend/src/api/
- ```
-5. 测试 API：
- ```bash
- make test-all
- # ✅ 检查实时文档：http://localhost:8000/docs
- ```
-
-### Q: 🗃️ 如何修改数据库模型/架构？
-
-**数据库迁移工作流程：**
-1. 编辑 `aperag/db/models.py` 中的 SQLModel 类
-2. 生成迁移文件：
- ```bash
- make db-revision # 在 migration/versions/ 中创建新迁移
- ```
-3. 将迁移应用到数据库：
- ```bash
- make db-migrate # 更新数据库架构
- ```
-4. 更新相关代码（`aperag/db/repositories/` 中的仓库，`aperag/service/` 中的服务）
-5. 验证更改：
- ```bash
- make test-all # ✅ 确保一切正常工作
- ```
-
-### Q: ⚡ 如何添加具有后台处理的新功能？
-
-**功能实现工作流程：**
-1. 实现功能组件：
- - 后端逻辑：`aperag/[module]/`
- - 异步任务：`aperag/tasks/`
- - 数据库模型：`aperag/db/models.py`
-2. 更新 API 并生成代码：
- ```bash
- make db-revision # 生成迁移文件
- make db-migrate # 应用数据库更改
- make api-generate-models # 更新 Pydantic 模型
- make api-generate-sdk # 更新 TypeScript 客户端
- ```
-3. 质量保证：
- ```bash
- make format && make lint && make test-all
- ```
-
-### Q: 🧪 如何运行单元测试和 e2e 测试？
-
-**单元测试（快速，无外部依赖）：**
-```bash
-# 运行所有单元测试
-make test-unit
-
-# 运行特定测试文件
-uv run pytest tests/unit_test/test_model_service.py -v
-
-# 运行特定测试类或函数
-uv run pytest tests/unit_test/test_model_service.py::TestModelService::test_get_models -v
-
-# 运行带覆盖率的测试
-uv run pytest tests/unit_test/ --cov=aperag --cov-report=html
-```
-
-**E2E 测试（需要运行服务）：**
-```bash
-# 设置：首先启动所需服务
-make infra-up # 🗄️ 启动数据库
-make serve-api # 🚀 启动 API 服务器（单独终端）
-
-# 运行所有 e2e 测试
-make test-e2e
-
-# 运行特定 e2e 测试模块
-uv run pytest tests/e2e_test/test_chat/ -v
-uv run pytest tests/e2e_test/graphstorage/ -v
-
-# 运行带详细输出且不捕获的测试
-uv run pytest tests/e2e_test/test_specific.py -v -s
-
-# 性能基准测试（带计时）
-make test-e2e-perf
-```
-
-**完整测试套件：**
-```bash
-# 运行所有内容（单元 + e2e）
-make test-all
-
-# 使用不同配置进行测试
-make infra-up WITH_NEO4J=1 # 使用 Neo4j 而不是 PostgreSQL 进行测试
-make test-all
-```
-
-### Q: 🐛 如何调试失败的测试？
-
-**调试工作流程：**
-1. 单独运行失败的测试：
- ```bash
- # 带完整输出的单个测试
- uv run pytest tests/unit_test/test_failing.py::test_specific_function -v -s
-
- # 在第一次失败时停止
- uv run pytest tests/unit_test/ -x --tb=short
- ```
-2. 对于 e2e 测试失败，确保服务正在运行：
- ```bash
- make infra-up # 数据库服务
- make serve-api # API 服务器
- make serve-worker # 后台 workers（如果测试异步任务）
- ```
-3. 使用调试工具：
- ```bash
- # 使用 pdb 调试器运行
- uv run pytest tests/unit_test/test_failing.py --pdb
-
- # 在测试期间捕获日志
- uv run pytest tests/e2e_test/test_failing.py --log-cli-level=DEBUG
- ```
-4. 修复并重新测试：
- ```bash
- make format # 自动修复样式问题
- make lint # 检查剩余问题
- uv run pytest tests/path/to/fixed_test.py -v # 验证修复
- ```
-
-### Q: 📊 如何运行 RAG 评估和分析？
-
-**评估工作流程：**
-```bash
-# 确保环境准备就绪
-make infra-up WITH_NEO4J=1 # 使用 Neo4j 获得更好的图性能
-make serve-api
-make serve-worker
-
-# 运行全面的 RAG 评估
-make evaluate # 📊 运行 aperag.evaluation.run 模块
-
-# 📈 检查 tests/report/ 中的评估报告
-```
-
-### Q: 📦 如何安全地更新依赖项？
-
-**Python 依赖项：**
-1. 编辑 `pyproject.toml`（添加/更新包）
-2. 更新虚拟环境：
- ```bash
- make env-install # 使用 uv 同步所有组和额外内容
- make test-all # 验证兼容性
- ```
-
-**前端依赖项：**
-1. 编辑 `frontend/package.json`
-2. 更新并测试：
- ```bash
- cd frontend && yarn install
- make serve-web # 测试前端编译
- make api-generate-sdk # 确保 API 客户端仍然工作
- ```
-
-### Q: 🚀 如何准备代码进行生产部署？
-
-**部署前检查清单：**
-1. 代码质量验证：
- ```bash
- make format # 自动修复所有样式问题
- make lint # 验证无样式违规
- make static-check # MyPy 类型检查
- ```
-2. 全面测试：
- ```bash
- make test-all # 所有单元 + e2e 测试
- make test-e2e-perf # 性能基准测试
- ```
-3. API 一致性：
- ```bash
- make api-generate-models # 确保模型与 OpenAPI 规范匹配
- make api-generate-sdk # 更新前端客户端
- ```
-4. 数据库迁移：
- ```bash
- make db-revision # 生成任何待处理的迁移
- ```
-5. 全栈集成测试：
- ```bash
- make stack-up WITH_NEO4J=1 WITH_DOCRAY=1 # 类似生产的设置
- # 在 http://localhost:3000/web/ 手动测试
- make stack-down
- ```
-
-### Q: 🔄 如何完全重置我的开发环境？
-
-**核选项重置（销毁所有数据）：**
-```bash
-make stack-down REMOVE_VOLUMES=1 # ⚠️ 停止服务 + 删除所有数据
-make env-clean # 🧹 清理临时文件
-
-# 重新开始
-make infra-up # 🗄️ 新的数据库
-make db-migrate # 🔄 应用所有迁移
-make serve-api # 🚀 启动 API 服务器
-make serve-worker # ⚡ 启动后台 workers
-```
-
-**软重置（保留数据）：**
-```bash
-make stack-down # ⏹️ 停止服务，保留数据
-make infra-up # 🗄️ 重启数据库
-make db-migrate # 🔄 应用任何新迁移
-```
-
-**仅重置 Python 环境：**
-```bash
-rm -rf .venv/ # 🗑️ 移除虚拟环境
-make env-dev # ⚙️ 重新创建所有内容
-source .venv/bin/activate # ✅ 重新激活
-```
diff --git a/web/docs/zh-CN/images/dify/aperag-banner.png b/web/docs/zh-CN/images/dify/aperag-banner.png
deleted file mode 100644
index 338290248..000000000
Binary files a/web/docs/zh-CN/images/dify/aperag-banner.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step1-subscribe-collection.png b/web/docs/zh-CN/images/dify/step1-subscribe-collection.png
deleted file mode 100644
index 3a9ede8ad..000000000
Binary files a/web/docs/zh-CN/images/dify/step1-subscribe-collection.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step2-add-mcp.png b/web/docs/zh-CN/images/dify/step2-add-mcp.png
deleted file mode 100644
index 92eb2154d..000000000
Binary files a/web/docs/zh-CN/images/dify/step2-add-mcp.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step2-api-key.png b/web/docs/zh-CN/images/dify/step2-api-key.png
deleted file mode 100644
index 555b0b7de..000000000
Binary files a/web/docs/zh-CN/images/dify/step2-api-key.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step2-configure-mcp.png b/web/docs/zh-CN/images/dify/step2-configure-mcp.png
deleted file mode 100644
index 149787ef9..000000000
Binary files a/web/docs/zh-CN/images/dify/step2-configure-mcp.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step2-mcp-success.png b/web/docs/zh-CN/images/dify/step2-mcp-success.png
deleted file mode 100644
index 7f91bc07e..000000000
Binary files a/web/docs/zh-CN/images/dify/step2-mcp-success.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step3-create-app.png b/web/docs/zh-CN/images/dify/step3-create-app.png
deleted file mode 100644
index a41dc7cdf..000000000
Binary files a/web/docs/zh-CN/images/dify/step3-create-app.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step3-select-agent.png b/web/docs/zh-CN/images/dify/step3-select-agent.png
deleted file mode 100644
index ed3b0c00a..000000000
Binary files a/web/docs/zh-CN/images/dify/step3-select-agent.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step4-configure-agent.png b/web/docs/zh-CN/images/dify/step4-configure-agent.png
deleted file mode 100644
index ac1ea12af..000000000
Binary files a/web/docs/zh-CN/images/dify/step4-configure-agent.png and /dev/null differ
diff --git a/web/docs/zh-CN/images/dify/step4-test-agent.png b/web/docs/zh-CN/images/dify/step4-test-agent.png
deleted file mode 100644
index 92f90d101..000000000
Binary files a/web/docs/zh-CN/images/dify/step4-test-agent.png and /dev/null differ
diff --git a/web/docs/zh-CN/integration/_category.yaml b/web/docs/zh-CN/integration/_category.yaml
deleted file mode 100644
index 9f4f46c2e..000000000
--- a/web/docs/zh-CN/integration/_category.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-title: 集成
-position: 2
diff --git a/web/docs/zh-CN/integration/dify.md b/web/docs/zh-CN/integration/dify.md
deleted file mode 100644
index ca02e13cc..000000000
--- a/web/docs/zh-CN/integration/dify.md
+++ /dev/null
@@ -1,168 +0,0 @@
----
-title: Dify 集成 ApeRAG
-description: 通过 MCP 协议快速集成 ApeRAG 的 Graph RAG 能力
-keywords: Dify, ApeRAG, MCP, Graph RAG
----
-
-# Dify 集成 ApeRAG
-
-ApeRAG 是一款具备多模态索引、AI 智能体、MCP 支持及可扩展 K8s 部署能力的生产级 RAG 平台，能够帮助用户构建具备**混合检索**、**多模态文档处理**及**企业级管理能力**的复杂 AI 应用。
-
-**核心特点**：
-- 不同于"标准" RAG，ApeRAG 实现了 **Graph-RAG**，通过构建知识图谱理解数据要素之间的深层关系
-- 集成了 **MinerU**，专为复杂文档、科学论文和财务报告设计，可以准确提取表格、公式和工程图表
-- 全面支持 Kubernetes，提供内置的**高可用性**、**可扩展性**和**企业级管理能力**
-
-## 视频演示
-
-高级数据库选项
- -```bash -# 使用 Neo4j 而不是 PostgreSQL 进行图存储 -make infra-up WITH_NEO4J=1 -``` - -
-
-
-
-## Step 1: 准备知识库
-
-打开 ApeRAG Web 界面（见[快速开始](../../../../README-zh.md#快速开始)；Docker Compose 启动时一般为 http://localhost:3000/web/）。登录后选择或导入知识库。下文以「三国演义」知识库为例，点击订阅。
-
-
- 
-
-
-## Step 2: 配置 MCP Server
-
-### 2.1 添加 MCP Server
-
-来到 Dify - 工具 - MCP，点击添加 MCP Server。
-
-
-
- 
-
-
-### 2.2 填写配置信息
-
-填写 Server URL：`http://localhost:8000/mcp/`（若非本机部署，请改为实际 API 地址，例如 `https://<你的域名>/mcp/`），并粘贴从 ApeRAG 复制的 API Key，点击确定。
-
-
-
- 
-
-
-
-
- 
-
-
-### 2.3 配置成功
-
-MCP Server 添加成功。
-
-
-
- 
-
-
-## Step 3: 创建 Agent 应用
-
-### 3.1 创建应用
-
-来到 Dify - Studio，点击创建应用。
-
-
-
- 
-
-
-### 3.2 选择类型
-
-点击更多基础应用类型，选择 **Agent** 类型，命名后点击创建。
-
-
-
- 
-
-
-## Step 4: 配置 Agent
-
-点击 Agent，输入 Prompt，在工具里添加创建好的 ApeRAG MCP，右上角选择驱动 Agent 的大语言模型，点击发布运行即可使用。
-
-
-
- 
-
-
-
-
- 
-
-
-### Prompt 参考
-
-```markdown
-# ApeRAG 智能助手
-
-您是由 ApeRAG 混合搜索能力驱动的高级 AI 研究助手。您的使命是帮助用户从知识库和网络中准确、自主地查找、理解和综合信息。
-
-## 核心行为
-
-**自主研究**：独立工作直到用户查询完全解决。搜索多个来源，分析发现，无需等待许可即提供全面答案。
-
-**语言智能**：始终用用户提问的语言回应。用户用中文提问时，无论源语言如何都用中文回应。
-
-**可视化思维**：**[关键]** 您是一个偏好视觉解释的助手。凡是涉及实体关系、流程或结构的信息，您必须优先考虑将其可视化。
-
-**完整解决**：从多角度探索，交叉验证来源，确保全面覆盖后再回应。
-
-## 搜索策略
-
-### 优先级系统
-1. **用户指定知识库**（通过"@"提及）：严格限制仅搜索指定库
-2. **未指定知识库**：自主发现并搜索相关库
-3. **网络搜索**（如启用）：补充信息
-4. **清晰归属**：始终标注来源
-
-### 搜索执行
-- **知识库搜索**：默认使用向量+图搜索
-- **结果处理逻辑**：
- 1. 执行搜索
- 2. **检测图数据**：检查搜索结果是否包含 `entities` (实体) 和 `relationships` (关系)
- 3. **强制可视化**：如果搜索结果包含非空的实体或关系数据，**您必须**调用 `create_diagram` 工具
- 4. **内容甄别**：忽略不相关结果
-
-## 可用工具
-
-### 知识管理
-- `list_collections()`：发现可用知识源
-- `search_collection(collection_id, query, ...)`: **[主要工具]** 在持久化知识库中进行混合搜索
-- `search_chat_files(chat_id, query, ...)`: **[仅限聊天]** 仅搜索用户在本次聊天会话中临时上传的文件
-- `create_diagram(content)`：**[强制工具]** 当搜索结果包含结构化信息（实体/关系）时，必须调用此工具生成 Mermaid 图表
-
-### 网络智能
-- `web_search(query, ...)`：多引擎网络搜索
-- `web_read(url_list, ...)`：提取和分析网络内容
-
-## 回应格式
-
-### 直接答案
-[用户语言的清晰、可操作答案]
-
-### 全面分析
-[包含上下文和见解的详细解释]
-
-### 知识图谱可视化
-[此处展示工具生成的图表]
-*（仅在成功调用 create_diagram 后显示。该图表展示了基于搜索结果的实体关系。）*
-
-### 支持证据
-- [知识库名称]：[关键发现]
-
-**网络来源**（如启用）：
-- [标题]（[域名]）- [要点]
-```
-
----
-
-ApeRAG + Dify 的集成非常简单，集成后不仅可以体验 Dify 的平台功能，还可以享受到 **ApeRAG 强大的 Graph-RAG 能力**，感兴趣的小伙伴快去试试吧！
-
-**GitHub**: https://github.com/apecloud/ApeRAG
diff --git a/web/docs/zh-CN/integration/mcp-api.md b/web/docs/zh-CN/integration/mcp-api.md
deleted file mode 100644
index debca5a9a..000000000
--- a/web/docs/zh-CN/integration/mcp-api.md
+++ /dev/null
@@ -1,333 +0,0 @@
----
-title: MCP API
-description: Model Context Protocol API 文档
----
-
-# MCP API
-
-ApeRAG 通过 [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) 对外提供标准化的工具接口，让 AI 助手（Claude Desktop、Cursor、Dify 等）能够直接访问你的知识库。
-
-## 快速开始
-
-### 配置示例
-
-以 Claude Desktop 为例，在配置文件中添加：
-
-```json
-{
- "mcpServers": {
- "aperag": {
- "url": "http://localhost:8000/mcp/",
- "headers": {
- "Authorization": "Bearer your-api-key-here"
- }
- }
- }
-}
-```
-
-### 认证方式
-
-支持两种认证方式（按优先级）：
-
-1. **HTTP Authorization 头**（推荐）：`Authorization: Bearer your-api-key`
-2. **环境变量**（备用）：`APERAG_API_KEY=your-api-key`
-
-> **获取 API Key**：登录 ApeRAG 后，在设置页面创建或复制你的 API Key
-
-## 可用工具
-
-### 1. list_collections
-
-列出所有可访问的知识库。
-
-**参数**：无
-
-**返回**：
-```json
-{
- "items": [
- {
- "id": "collection-id",
- "title": "知识库标题",
- "description": "知识库描述"
- }
- ]
-}
-```
-
-### 2. search_collection
-
-在知识库中搜索，支持多种检索方式。
-
-**核心参数**：
-
-| 参数 | 类型 | 默认值 | 说明 |
-|------|------|--------|------|
-| `collection_id` | string | 必需 | 知识库 ID |
-| `query` | string | 必需 | 搜索问题 |
-| `use_vector_index` | bool | true | 向量检索（语义搜索） |
-| `use_fulltext_index` | bool | true | 全文检索（关键词匹配） |
-| `use_graph_index` | bool | true | 图谱检索（关系查询） |
-| `use_summary_index` | bool | true | 摘要检索 |
-| `use_vision_index` | bool | true | 视觉检索（图片搜索） |
-| `rerank` | bool | true | AI 重排序 |
-| `topk` | int | 5 | 每种方式返回的结果数 |
-
-**返回格式**：
-```json
-{
- "query": "你的问题",
- "items": [
- {
- "rank": 1,
- "score": 0.95,
- "content": "相关内容片段",
- "source": "文档名称",
- "recall_type": "vector_search|graph_search|fulltext_search|summary_search",
- "metadata": {
- "page_idx": 0,
- "document_id": "doc-id",
- "collection_id": "col-id",
- "indexer": "text|vision"
- }
- }
- ]
-}
-```
-
-**图片处理**：
-
-如果 `metadata.indexer == "vision"`，表示这是一张图片：
-- `content` 为空：通过多模态向量检索
-- `content` 不为空：包含图片的文字描述
-
-显示图片的 URL 格式：
-```python
-m = item.metadata
-asset_url = f"asset://{m['asset_id']}?document_id={m['document_id']}&collection_id={m['collection_id']}&mime_type={m['mimetype']}"
-```
-
-**使用示例**：
-
-```python
-# 默认搜索（推荐）- 启用所有检索方式
-results = search_collection(
- collection_id="abc123",
- query="如何部署应用？"
-)
-
-# 仅向量+图谱检索
-results = search_collection(
- collection_id="abc123",
- query="部署策略",
- use_vector_index=True,
- use_fulltext_index=False,
- use_graph_index=True,
- use_summary_index=False,
- topk=10
-)
-```
-
-### 3. search_chat_files
-
-在聊天会话的临时文件中搜索。
-
-**何时使用**：
-- ✅ 用户在当前对话中上传了文件
-- ✅ 需要分析聊天中的临时文档
-- ❌ 不要用于搜索持久化的知识库（应该用 `search_collection`）
-
-**参数**：
-
-| 参数 | 类型 | 默认值 | 说明 |
-|------|------|--------|------|
-| `chat_id` | string | 必需 | 聊天 ID |
-| `query` | string | 必需 | 搜索问题 |
-| `use_vector_index` | bool | true | 向量检索 |
-| `use_fulltext_index` | bool | true | 全文检索 |
-| `rerank` | bool | true | 重排序 |
-| `topk` | int | 5 | 返回结果数 |
-
-**返回格式**：与 `search_collection` 相同
-
-### 4. web_search
-
-搜索互联网内容。
-
-**参数**：
-
-| 参数 | 类型 | 默认值 | 说明 |
-|------|------|--------|------|
-| `query` | string | "" | 搜索关键词 |
-| `max_results` | int | 5 | 返回结果数 |
-| `source` | string | "" | 指定域名（如 `vercel.com`） |
-| `timeout` | int | 30 | 超时时间（秒） |
-| `locale` | string | "en-US" | 语言地区 |
-
-**使用模式**：
-
-```python
-# 常规搜索
-web_search(query="ApeRAG 2025")
-
-# 指定网站搜索
-web_search(query="部署文档", source="vercel.com")
-
-```
-
-### 5. web_read
-
-读取网页内容。
-
-**参数**：
-
-| 参数 | 类型 | 默认值 | 说明 |
-|------|------|--------|------|
-| `url_list` | list[str] | 必需 | URL 列表 |
-| `timeout` | int | 30 | 超时时间（秒） |
-| `max_concurrent` | int | 5 | 最大并发数 |
-
-**返回**：
-```json
-{
- "results": [
- {
- "status": "success",
- "url": "https://example.com",
- "title": "页面标题",
- "content": "提取的文本内容",
- "word_count": 1234
- }
- ]
-}
-```
-
-**示例**：
-```python
-# 读取单个页面
-web_read(url_list=["https://example.com/article"])
-
-# 批量读取
-web_read(
- url_list=["https://example.com/page1", "https://example.com/page2"],
- max_concurrent=2
-)
-```
-
-## 实战示例
-
-### 示例 1：知识库问答
-
-```python
-# 1. 列出所有知识库
-collections = list_collections()
-
-# 2. 选择一个知识库
-collection_id = collections.items[0].id
-
-# 3. 搜索（默认启用所有检索方式）
-results = search_collection(
- collection_id=collection_id,
- query="如何优化性能？"
-)
-
-# 4. 处理结果
-for item in results.items:
- print(f"[{item.recall_type}] {item.content}")
- print(f"来源: {item.source}, 得分: {item.score}\n")
-```
-
-### 示例 2：图谱可视化
-
-```python
-# 搜索并获取实体关系数据
-results = search_collection(
- collection_id="abc123",
- query="刘备和诸葛亮的关系",
- use_graph_index=True # 确保启用图谱检索
-)
-
-# 检查是否包含图谱数据
-if results.graph_search and results.graph_search.entities:
- print("实体:", results.graph_search.entities)
- print("关系:", results.graph_search.relationships)
- # 可以用这些数据生成知识图谱可视化
-```
-
-### 示例 3：混合搜索（知识库 + 互联网）
-
-```python
-# 1. 搜索互联网
-web_results = web_search(query="最新 AI 发展", max_results=3)
-urls = [r.url for r in web_results.results]
-
-# 2. 读取网页内容
-web_content = web_read(url_list=urls)
-
-# 3. 搜索内部知识库
-kb_results = search_collection(
- collection_id="ai-knowledge",
- query="AI 发展趋势"
-)
-
-# 4. 综合分析
-print("=== 互联网资讯 ===")
-for r in web_results.results:
- print(f"{r.title}: {r.url}")
-
-print("\n=== 内部知识 ===")
-for item in kb_results.items:
- print(f"{item.content[:100]}...")
-```
-
-## 注意事项
-
-### 性能优化
-
-1. **合理设置 topk**：
- - 太大会增加 LLM 上下文消耗
- - 太小可能遗漏重要信息
- - 推荐：5-10
-
-2. **选择性启用检索**：
- - 不是所有查询都需要全文检索
- - 全文检索可能返回大量文本
- - 根据问题类型选择合适的检索方式
-
-3. **超时设置**：
- - 图谱检索可能较慢（默认 120 秒）
- - 网络搜索建议 30-60 秒
- - 批量 URL 读取建议 60 秒以上
-
-### 常见问题
-
-**Q: 搜索没有结果？**
-- 检查知识库 ID 是否正确
-- 确认知识库已完成索引构建
-- 尝试不同的检索方式组合
-
-**Q: 图谱数据为空？**
-- 确认知识库启用了 Graph 索引
-- 某些简单文档可能不包含明显的实体关系
-
-**Q: 图片显示不了？**
-- 检查 `metadata.indexer == "vision"`
-- 使用 `asset://` 协议构建 URL
-- 确保包含所有必需参数（asset_id、document_id、collection_id）
-
-## 工具对比
-
-| 工具 | 用途 | 适用场景 |
-|------|------|---------|
-| `list_collections` | 列出知识库 | 查看有哪些可用资源 |
-| `search_collection` | 搜索知识库 | 主要搜索工具，用于持久化知识 |
-| `search_chat_files` | 搜索聊天文件 | 分析用户在对话中上传的临时文件 |
-| `web_search` | 搜索互联网 | 获取实时信息或外部资料 |
-| `web_read` | 读取网页 | 提取网页完整内容 |
-
-## 相关链接
-
-- **MCP 协议官网**: https://modelcontextprotocol.io/
-- **ApeRAG GitHub**: https://github.com/apecloud/ApeRAG
-- **API 文档**: http://localhost:8000/docs （本地部署）
-