feat: harden extraction pipeline and overhaul documentation

Author: dev-dami

README.MD

@@ -9,144 +9,80 @@
 
 </div>
 
-Sophisticated NLP library for comprehensive text processing, tokenization, and analysis with LLM integration.
+Qirrel is a TypeScript NLP library for tokenization, rule-based entity extraction, optional LLM enrichment, and pipeline orchestration.
 
 [GitHub](https://github.com/dev-dami/qirrel) | [NPM](https://www.npmjs.com/package/qirrel) | Author: [Damilare Osibanjo](https://github.com/dev-dami)
 
-## Quick Start
+## Install
 
 ```bash
-npm install qirrel
+bun add qirrel
 ```
 
-### Extract Multiple Entity Types
+## Quick Start
 
 ```ts
 import { processText } from 'qirrel';
-const result = await processText('Contact John at john@example.com or call +1-555-123-4567');
+
+const result = await processText('Contact John at john@example.com or call +1 415 555 2671');
 console.log(result.data?.entities);
-// [
-//   { type: 'email', value: 'john@example.com', ... },
-//   { type: 'phone', value: '+1-555-123-4567', ... }
-// ]
 ```
 
-### Extract URLs and Numbers
+## Batch Processing
 
 ```ts
-import { processText } from 'qirrel';
-const result = await processText('Visit https://example.com for more info. Price: $29.99');
-console.log(result.data?.entities);
-// [
-//   { type: 'url', value: 'https://example.com', ... },
-//   { type: 'number', value: '29.99', ... }
-// ]
-```
+import { processTexts, Pipeline } from 'qirrel';
 
-### Advanced Pipeline Usage
+const results = await processTexts(['first input', 'second input'], undefined, { concurrency: 2 });
 
-```ts
-import { Pipeline } from 'qirrel';
 const pipeline = new Pipeline();
-const result = await pipeline.process('Check out https://github.com and email support@example.com');
-console.log(result.data?.entities);
+const ordered = await pipeline.processBatch(['a', 'b', 'c'], { concurrency: 3 });
 ```
 
 ## LLM Integration
 
-Connect to large language models for enhanced text analysis:
-
 ```ts
 import { Pipeline } from 'qirrel';
 
-// Enable LLM in your config
 const pipeline = new Pipeline('./config-with-llm.yaml');
-const llmAdapter = pipeline.getLLMAdapter();
+await pipeline.init();
 
+const llmAdapter = pipeline.getLLMAdapter();
 if (llmAdapter) {
-  // Use LLM for advanced analysis
-  const response = await llmAdapter.generateContent('Analyze this text...');
-  console.log(response);
+  const response = await llmAdapter.generate('Summarize this text: Hello world');
+  console.log(response.content);
 }
 ```
 
 ## Caching
 
-Qirrel includes built-in caching functionality to improve performance by storing frequently accessed results:
-
 ```ts
-import { Pipeline, LruCacheManager } from 'qirrel';
+import { Pipeline } from 'qirrel';
 
-// The pipeline automatically uses caching based on configuration
 const pipeline = new Pipeline();
 
-// Access the cache manager directly if needed
-const cacheManager = pipeline.getCacheManager();
-
-// Custom cache usage
-const cache = new LruCacheManager({
-  maxEntries: 1000,  // Maximum number of cached items
-  ttl: 300000        // Time-to-live in milliseconds (5 minutes)
-});
-
-// Check if a result is already cached
 if (pipeline.isCached('some text')) {
-  const cachedResult = pipeline.getCached('some text');
-  console.log('Using cached result');
+  console.log(pipeline.getCached('some text'));
 } else {
   const result = await pipeline.process('some text');
-  console.log('Processing new text');
+  console.log(result.data?.entities);
 }
 ```
 
-## Pipeline Lifecycle Events
-
-Monitor and react to pipeline execution with comprehensive lifecycle events:
-
-```ts
-import { Pipeline, PipelineEvent } from 'qirrel';
-
-const pipeline = new Pipeline();
-
-// Subscribe to pipeline events
-pipeline.on(PipelineEvent.RunStart, (payload) => {
-  console.log('Pipeline started:', payload.context.meta?.requestId);
-});
-
-pipeline.on(PipelineEvent.ProcessorEnd, (payload) => {
-  console.log(`Processor ${payload.processorName} completed in ${payload.duration}ms`);
-});
-
-pipeline.on(PipelineEvent.RunEnd, (payload) => {
-  console.log('Pipeline completed in', payload.duration, 'ms');
-});
-
-const result = await pipeline.process('Your text here');
-```
-
-## Use Cases
-
-Qirrel is perfect for developers building solutions for:
-
-- **Business Applications**: Extract customer contact details, lead generation from documents, document processing for legal/medical/financial use cases
-- **Social Media Analysis**: Process social content for mentions, hashtags, links, and user-generated content patterns
-- **Research & Content**: Extract structured data from academic papers, news articles, or documentation
-- **Data Processing**: Unstructured data extraction, log file analysis, preprocessing for ML pipelines
-- **Communication Tools**: Email processing, chat applications, content moderation
-
 ## Documentation
+
 - [API Reference](./docs/api.md)
 - [Configuration Guide](./docs/configuration.md)
 - [Usage Examples](./docs/examples.md)
-- [Code Walkthrough](./docs/walkthrough.md)
 - [Basic Usage](./docs/usage/basic.md)
-- [Advanced Usage](./docs/usage/advanced.md)
 - [LLM Integration](./docs/integrations/llm.md)
 - [Pipeline Events](./docs/events.md)
 - [Caching](./docs/usage/caching.md)
 
 ## Contributing
-We welcome contributions from the community! Please read our [Contributing Guide](./docs/contributing.md) for more information.
+
+See [CONTRIBUTING.md](./CONTRIBUTING.md).
 
 ## License
-This project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details.
+
+MIT. See [LICENSE](./LICENSE).

default.yaml

@@ -25,4 +25,5 @@ llm:
   model: gemini-2.5-flash
   temperature: 0.7
   maxTokens: 1024
-  timeout: 30000
+  timeout: 30000
+  cacheTtl: 300000