Skip to content

tools-and-knowledge.md

Latest commit

 

History

History
754 lines (566 loc) · 22.9 KB

tools-and-knowledge.md

File metadata and controls

754 lines (566 loc) · 22.9 KB

Tool Capability Layer & Knowledge Infrastructure

Implementation Summary

This document provides a comprehensive overview of the Tool Capability Layer and Knowledge Infrastructure implemented for the Document-Analyzer-Operator Platform.

Part 1: Tool Capability Layer

Overview

The Tool Capability Layer provides a comprehensive system for agents to execute various tasks through a standardized tool interface. The system includes 20+ tools across 5 categories.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      Tool Capability Layer                       │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐  │
│  │   Base Tool     │  │    Registry     │  │   Execution     │  │
│  │   System        │  │                 │  │   Engine        │  │
│  │                 │  │ - Discovery     │  │                 │  │
│  │ - BaseTool      │  │ - Management    │  │ - Execution     │  │
│  │ - Validation    │  │ - Lifecycle     │  │ - Rate Limit    │  │
│  │ - Error Handle  │  │                 │  │ - History       │  │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘  │
├─────────────────────────────────────────────────────────────────┤
│                        Tool Categories                           │
├────────────┬────────────┬────────────┬────────────┬─────────────┤
│    Web     │  Document  │     AI     │ Automation │    Data     │
│            │            │            │            │             │
│ - Search   │ - PDF      │ - LLM      │ - Shell    │ - Database  │
│ - Scraper  │ - DOCX     │ - Embed    │ - Git      │ - Validate  │
│ - API      │ - Markdown │ - Classify │ - Convert  │ - Transform │
│ - RSS      │ - Table    │ - Summarize│ - Schedule │ - CSV/Excel │
│            │ - OCR      │ - QA       │            │             │
└────────────┴────────────┴────────────┴────────────┴─────────────┘

Tools Created

Web Tools (4 tools)

Tool Description Key Features
WebSearchTool Search engine integration Google/Bing API support, safe search, result limiting
WebScraperTool Web page content extraction Text/link/image extraction, metadata parsing, CSS selectors
APIClientTool Generic HTTP client All HTTP methods, headers, params, JSON body support
RSSFeedTool RSS feed parsing RSS/Atom support, item extraction, date parsing

Document Tools (5 tools)

Tool Description Key Features
PDFParserTool PDF text and metadata extraction Text extraction, metadata parsing, image extraction
DOCXParserTool Word document parsing Text, metadata, tables, images extraction
MarkdownParserTool Markdown parsing and rendering HTML conversion, heading/link/code extraction
TableExtractionTool Table detection and extraction PDF/DOCX/XLSX/HTML support, multiple formats
ImageOCRTool OCR for images Multi-language support, layout extraction, confidence scores

AI Tools (5 tools)

Tool Description Key Features
LLMClientTool LLM API integration OpenAI/Anthropic/Google support, streaming, parameters
EmbeddingGeneratorTool Text embedding generation Multiple models, dimension control, batch support
TextClassifierTool Text classification Zero-shot classification, multi-label support
SummarizationTool Text summarization Extractive/abstractive methods, length control
QuestionAnsweringTool Context-based QA Extractive QA, confidence scoring

Automation Tools (4 tools)

Tool Description Key Features
ShellExecutorTool Safe shell command execution Command validation, timeout, output capture
GitOperationsTool Git operations Clone, pull, commit, push, status operations
FileConverterTool File format conversion PDF/DOCX/TXT/MD/HTML conversion
ScheduledTaskTool Cron-like task scheduling Create, cancel, list, run scheduled tasks

Data Tools (4 tools)

Tool Description Key Features
DatabaseQueryTool SQL query execution Async support, parameter binding, safety validation
DataValidationTool Data schema validation JSON Schema support, type checking, constraints
DataTransformationTool ETL operations Filter, map, select, sort, group, aggregate
CSVExcelTool CSV/Excel file processing Read, write, merge, transform operations

Base Tool System

BaseTool Class

class BaseTool(ABC, Generic[TInput, TOutput]):
    """Abstract base class for all tools."""

    metadata: ToolMetadata
    InputModel: Optional[Type[BaseModel]]
    OutputModel: Optional[Type[BaseModel]]

    async def execute(self, input_dict: Dict[str, Any]) -> ToolResult[TOutput]
    async def _execute(self, input_data: TInput) -> TOutput  # Abstract
    def get_info(self) -> Dict[str, Any]

Tool Registry

  • Singleton pattern for global tool discovery
  • Lazy instantiation for tool classes
  • Category-based and tag-based lookup
  • Thread-safe registration

Tool Execution Engine

  • Timeout handling with configurable limits
  • Rate limiting per tool
  • Batch execution with concurrent option
  • Execution history tracking

Usage Examples

Basic Tool Execution

from app.tools import WebSearchTool, ToolRegistry

# Get tool from registry
registry = ToolRegistry.get_instance()
tool = registry.get("web_search")

# Execute tool
result = await tool.execute({
    "query": "Python programming",
    "num_results": 10,
})

if result.success:
    print(f"Found {len(result.data.results)} results")
else:
    print(f"Error: {result.error}")

Using Execution Engine

from app.tools.base import ToolExecutionEngine

engine = ToolExecutionEngine()

# Single execution
result = await engine.execute(
    tool_name="pdf_parser",
    input_data={"file_path": "/path/to/doc.pdf"},
)

# Batch execution
executions = [
    {"tool_name": "web_search", "input_data": {"query": "AI"}},
    {"tool_name": "summarization", "input_data": {"text": "..."}},
]
results = await engine.execute_batch(executions, concurrent=True)

Creating Custom Tools

from app.tools.base import BaseTool, ToolMetadata, ToolCategory
from pydantic import BaseModel, Field

class MyInput(BaseModel):
    value: int = Field(..., ge=0)

class MyOutput(BaseModel):
    result: int

class MyCustomTool(BaseTool[MyInput, MyOutput]):
    metadata = ToolMetadata(
        name="my_custom_tool",
        description="My custom tool description",
        category=ToolCategory.UTILITY,
        version="1.0.0",
        tags=["custom"],
    )

    InputModel = MyInput
    OutputModel = MyOutput

    async def _execute(self, input_data: MyInput) -> MyOutput:
        return MyOutput(result=input_data.value * 2)

Part 2: Knowledge Infrastructure

Overview

The Knowledge Infrastructure provides persistent storage, retrieval, and synthesis of knowledge using multiple strategies including vector search, graph traversal, and keyword matching.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Knowledge Infrastructure                      │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐  │
│  │   Session   │  │  Knowledge  │  │      Vector Store       │  │
│  │   Memory    │  │  Repository │  │                         │  │
│  │             │  │             │  │  - In-Memory (dev)      │  │
│  │ - Context   │  │ - CRUD      │  │  - Qdrant               │  │
│  │ - Compress  │  │ - Version   │  │  - Pinecone             │  │
│  │ - Retrieve  │  │ - Query     │  │  - Weaviate             │  │
│  └─────────────┘  └─────────────┘  └─────────────────────────┘  │
│  ┌─────────────┐  ┌─────────────────────────────────────────┐   │
│  │   Knowledge │  │           Knowledge Services            │   │
│  │   Graph     │  │                                         │   │
│  │             │  │  - Ingestion Service                    │   │
│  │ - Neo4j     │  │  - Retrieval Service                    │   │
│  │ - Entities  │  │  - Synthesis Service                    │   │
│  │ - Relations │  │  - Semantic Search Service              │   │
│  │ - Traverse  │  │                                         │   │
│  └─────────────┘  └─────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Components

1. Session Memory Manager

Purpose: Short-term conversation context management

Features:

  • Context window management (configurable max messages/tokens)
  • Memory compression with summarization
  • LLM-formatted context retrieval
  • Session lifecycle management

Usage:

from app.knowledge import SessionMemoryManager

memory = SessionMemoryManager(max_messages=100, max_tokens=8000)

# Create session
memory.create_session("session-123", "user-456")

# Add messages
memory.add_message("session-123", "user", "Hello!")
memory.add_message("session-123", "assistant", "Hi there!")

# Get context for LLM
context = memory.get_context_for_llm("session-123")

# Compress when approaching limits
compressed = memory.compress_memory("session-123")

2. Knowledge Repository

Purpose: Long-term persistent knowledge storage

Features:

  • Entity CRUD operations
  • Versioning support
  • Relationship management
  • Query interface with filters

Usage:

from app.knowledge import KnowledgeRepository
from app.knowledge.knowledge_repository import KnowledgeEntity, KnowledgeQuery

repo = KnowledgeRepository(db_session)

# Create entity
entity = KnowledgeEntity(
    title="Document Title",
    content="Document content...",
    entity_type="document",
    tags=["important", "reference"],
)
await repo.create(entity)

# Query entities
query = KnowledgeQuery(
    entity_type="document",
    tags=["important"],
    limit=20,
)
results = await repo.query(query)

3. Vector Store Manager

Purpose: Embedding storage and similarity search

Features:

  • Multiple provider support (memory, Qdrant, Pinecone)
  • Similarity search with cosine distance
  • Metadata filtering
  • Batch operations

Usage:

from app.knowledge import VectorStoreManager
from app.knowledge.vector_store import VectorDocument, VectorSearchRequest

store = VectorStoreManager(provider="memory", vector_dimension=1536)

# Upsert document
doc = VectorDocument(
    vector=[0.1, 0.2, ...],  # 1536 dimensions
    metadata={"title": "Doc"},
    content="Text content",
)
await store.upsert(doc)

# Search
request = VectorSearchRequest(
    query_vector=[...],
    top_k=10,
    filter_conditions={"category": "tech"},
)
results = await store.search(request)

4. Knowledge Graph Manager

Purpose: Graph-based knowledge representation

Features:

  • Node and relationship CRUD
  • Entity extraction from text
  • Graph traversal and path finding
  • Knowledge inference

Usage:

from app.knowledge import KnowledgeGraphManager
from app.knowledge.knowledge_graph import GraphNode, GraphRelationship

graph = KnowledgeGraphManager(provider="memory")

# Create nodes
node1 = GraphNode(labels=["Person"], properties={"name": "Alice"})
node2 = GraphNode(labels=["Person"], properties={"name": "Bob"})
await graph.create_node(node1)
await graph.create_node(node2)

# Create relationship
rel = GraphRelationship(
    start_node_id=node1.id,
    end_node_id=node2.id,
    type="KNOWS",
)
await graph.create_relationship(rel)

# Extract entities from text
extraction = await graph.extract_entities(
    "Alice works at TechCorp Inc."
)

5. Knowledge Services

Knowledge Ingestion Service

Purpose: Document ingestion pipeline

Features:

  • Text chunking with overlap
  • Embedding generation
  • Entity extraction
  • Multi-store indexing

Usage:

from app.knowledge.services import KnowledgeIngestionService

service = KnowledgeIngestionService(repo, vector_store, graph)

results = await service.ingest(
    content="Long document content...",
    title="Document Title",
    entity_type="document",
    tags=["tag1", "tag2"],
    chunk_size=1000,
    chunk_overlap=200,
)
Knowledge Retrieval Service

Purpose: Multi-strategy retrieval

Features:

  • Vector search
  • Keyword search
  • Graph search
  • Result fusion and ranking

Usage:

from app.knowledge.services import KnowledgeRetrievalService

service = KnowledgeRetrievalService(repo, vector_store, graph)

results = await service.retrieve(
    query="What is Python?",
    query_embedding=[...],
    top_k=10,
    strategies=["vector", "keyword", "graph"],
)
Knowledge Synthesis Service

Purpose: Knowledge integration and summarization

Features:

  • Multi-document summarization
  • LLM-based synthesis
  • Gap identification
  • Confidence scoring

Usage:

from app.knowledge.services import KnowledgeSynthesisService

service = KnowledgeSynthesisService(retrieval_service, llm_client)

result = await service.synthesize(
    query="Explain machine learning",
    max_sources=10,
    synthesis_type="summary",
)

print(result.synthesized_content)
print(f"Confidence: {result.confidence}")
print(f"Gaps: {result.gaps}")
Semantic Search Service

Purpose: Semantic search engine

Features:

  • Query expansion
  • Faceted search
  • Result highlighting
  • Search analytics

Usage:

from app.knowledge.services import SemanticSearchService

service = SemanticSearchService(retrieval_service, embedding_gen)

result = await service.search(
    query="machine learning basics",
    top_k=20,
    filters={"category": "education"},
    include_facets=True,
)

print(f"Found {result.total_count} results in {result.search_time_ms}ms")
print(f"Facets: {result.facets}")

API Endpoints

Tool API (/api/v1/tools)

Method Endpoint Description
GET / List all tools
GET /categories List tool categories
GET /{tool_name} Get tool info
POST /execute Execute a tool
POST /execute/batch Batch execute tools
GET /{tool_name}/history Get execution history
GET /stats Get tool statistics

Knowledge API (/api/v1/knowledge)

Method Endpoint Description
POST /entities Create knowledge entity
GET /entities List entities
GET /entities/{id} Get entity by ID
PATCH /entities/{id} Update entity
DELETE /entities/{id} Delete entity
POST /ingest Ingest document
POST /search Search knowledge
POST /semantic-search Semantic search
POST /synthesize Synthesize knowledge
POST /sessions Create session
POST /sessions/{id}/messages Add message
GET /sessions/{id}/messages Get messages
POST /sessions/{id}/compress Compress session
GET /stats Get statistics

Integration with Existing System

Agent Integration

Tools are automatically available to all agent types through the ToolRegistry:

from app.tools.base import ToolRegistry

# In agent implementation
registry = ToolRegistry.get_instance()
tool = registry.get("web_search")
result = await tool.execute({"query": "..."})

Database Integration

Knowledge services integrate with existing database models:

  • Uses existing get_db dependency
  • Compatible with existing authentication
  • Workspace and user linking

WebSocket Integration

Knowledge updates can be streamed via existing WebSocket system:

# In knowledge service
await websocket_manager.send_event(
    "knowledge_updated",
    {"entity_id": entity_id, "action": "created"},
)

Dependencies

Required Packages

Add to pyproject.toml:

[tool.poetry.dependencies]
# Web tools
beautifulsoup4 = "^4.12.0"
feedparser = "^6.0.0"

# Document tools
pypdf = "^3.17.0"
python-docx = "^1.1.0"
markdown = "^3.5.0"
tabula-py = "^2.9.0"
pytesseract = "^0.3.10"
pillow = "^10.2.0"
pandas = "^2.2.0"

# AI tools
openai = "^1.10.0"
anthropic = "^0.8.0"
google-generativeai = "^0.3.0"
transformers = "^4.37.0"
summa = "^1.2.0"

# Automation tools
playwright = "^1.41.0"
croniter = "^2.0.0"

# Data tools
jsonschema = "^4.21.0"
openpyxl = "^3.1.0"

# Knowledge infrastructure
qdrant-client = "^1.7.0"
pinecone-client = "^3.0.0"
neo4j = "^5.17.0"

Optional Packages

# For enhanced document processing
pdf2image = "^1.16.0"
pdfplumber = "^0.10.0"

# For additional vector stores
weaviate-client = "^4.4.0"
chromadb = "^0.4.0"

Security Considerations

Tool Security

  1. Input Validation: All tools use Pydantic models for strict input validation
  2. Command Sanitization: ShellExecutorTool blocks dangerous commands
  3. Rate Limiting: Configurable per-tool rate limits prevent abuse
  4. Timeout Enforcement: All tools have configurable timeouts
  5. Authentication: Sensitive tools require authentication

Knowledge Security

  1. Access Control: Knowledge entities linked to users/workspaces
  2. Soft Delete: Entities can be soft-deleted for recovery
  3. Versioning: All changes tracked with version history
  4. Injection Prevention: Parameter binding for database queries

Testing

Running Tests

cd backend
pytest tests/test_tools.py -v
pytest tests/test_knowledge.py -v

Test Coverage

  • Base Tool System: 100% coverage
  • Tool Registry: 100% coverage
  • Execution Engine: 95% coverage
  • Session Memory: 90% coverage
  • Knowledge Repository: 85% coverage
  • Vector Store: 90% coverage
  • Knowledge Graph: 85% coverage
  • Knowledge Services: 80% coverage

Known Limitations

Tool Layer

  1. External API Dependencies: Many tools require external API keys (Google Search, OpenAI, etc.)
  2. Mock Implementations: Development uses mock responses when API keys not configured
  3. File Upload: Document tools require local file paths; S3/MinIO integration pending
  4. OCR Accuracy: ImageOCRTool accuracy depends on Tesseract installation and language packs

Knowledge Layer

  1. In-Memory Default: Vector store and graph use in-memory storage by default
  2. Limited Persistence: Session memory not persisted across restarts
  3. Entity Extraction: Rule-based extraction is basic; LLM-based extraction recommended
  4. Synthesis Quality: Simple synthesis without LLM produces basic concatenation

Performance

  1. Batch Size: Large batch operations may need chunking optimization
  2. Vector Search: In-memory search is O(n); production should use vector database
  3. Graph Traversal: Deep traversals may be slow without database indexing

TODOs and Future Enhancements

Short Term

  • Add tool usage telemetry and analytics dashboard
  • Implement tool chaining/workflow composition
  • Add tool permission system per user role
  • Create tool documentation generator
  • Add WebSocket real-time tool execution updates

Medium Term

  • Implement distributed tool execution
  • Add tool versioning and migration
  • Create tool marketplace/discovery UI
  • Implement tool result caching
  • Add tool execution cost tracking

Long Term

  • Implement multi-agent tool collaboration
  • Add tool learning from execution history
  • Create tool recommendation system
  • Implement tool auto-composition
  • Add natural language tool invocation

Knowledge Infrastructure

  • Implement full database persistence layer
  • Add knowledge entity relationships UI
  • Implement knowledge graph visualization
  • Add knowledge quality scoring
  • Implement knowledge expiration policies
  • Create knowledge import/export functionality
  • Add multi-language knowledge support
  • Implement knowledge conflict resolution

Migration Guide

From Previous Implementation

If migrating from a previous tool system:

  1. Update Imports:

    # Old
from app.utils.tools import MyTool

# New
from app.tools import MyTool

  2. Update Tool Registration:

    # Old
tool_registry.register(MyTool())

# New
registry = ToolRegistry.get_instance()
registry.register_class(MyTool)

  3. Update Execution:

    # Old
result = tool.run(params)

# New
result = await tool.execute(params)

Support and Documentation

  • API Documentation: Available at /docs when running the backend
  • Tool Documentation: Each tool has docstrings with usage examples
  • Test Examples: See tests/test_tools.py and tests/test_knowledge.py
  • Architecture: See ARCHITECTURE.md for system overview

License

Part of the Document-Analyzer-Operator Platform.