ZeusDB
diff --git a/‎CHANGELOG.md‎
Lines changed: 124 additions & 0 deletions b/‎CHANGELOG.md‎
Lines changed: 124 additions & 0 deletions
diff --git a/‎README.md‎
Lines changed: 167 additions & 8 deletions b/‎README.md‎
Lines changed: 167 additions & 8 deletions
@@ -7,6 +7,130 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.2.0] - 2025-07-25
+
+### Added
+- Product Quantization (PQ) Support
+  - Quantized vector storage with configurable compression ratios (4x-256x)
+  - Automatic training pipeline with intelligent threshold detection
+  - 3-path storage architecture for optimal memory usage:
+    - Path A: Raw storage (no quantization)
+    - Path B: Raw storage + ID collection (pre-training)
+    - Path C: Quantized storage (post-training)
+
+- Quantized Search API
+  - Unified search interface supports both raw and quantized vectors transparently.
+- Automatic fallback to raw search if quantization is not yet trained.
+- Quantization-aware batch addition for efficient ingestion at scale.
+- Detailed quantization diagnostics via get_quantization_info() (e.g., codebook stats, compression ratio, memory footprint).
+- Debug logging macro (ZEUSDB_DEBUG) for controlled diagnostic output in Rust backend.
+- Thread safety diagnostics in get_stats() (e.g., "thread_safety": "RwLock+Mutex").
+- Improved test coverage for quantized and raw modes, including edge cases and error handling.
+
+- Asymmetric Distance Computation (ADC) for fast quantized search
+- Memory-efficient k-means clustering for codebook generation
+- Configurable quantization parameters:
+  - `subvectors`: Number of vector subspaces (divisor of dimension)
+  - `bits`: Bits per quantized code (1-8)
+  - `training_size`: Vectors needed for training (minimum 1000)
+  - `max_training_vectors`: Maximum vectors used for training
+
+- Enhanced Vector Database API
+- Quantization configuration support in create() method
+- Training progress monitoring with get_training_progress()
+- Storage mode detection with get_storage_mode()
+- Quantization status methods:
+  - `has_quantization()`: Check if quantization is configured
+  - `can_use_quantization()`: Check if PQ model is trained
+  - `is_quantized()`: Check if index is using quantized storage
+- Quantization info retrieval with `get_quantization_info()`
+- Training readiness check with `is_training_ready()`
+- Training vectors needed with `training_vectors_needed()`
+
+- Performance Monitoring
+  - Compression ratio calculation and reporting
+  - Memory usage estimation for raw vs compressed storage
+  - Training time measurement and optimization
+  - Search performance metrics for quantized vs raw modes
+  - Detailed statistics in 'get_stats()' method
+
+- Input Handling
+ - Enhanced dictionary input parsing with comprehensive error handling
+ - Flexible metadata support for various Python object types
+ - Automatic type detection and conversion for metadata
+ - Graceful handling of None values and edge cases
+ - Comprehensive input validation with descriptive error messages
+
+- Performance Optimizations
+ - Batch processing for large-scale vector additions
+ - Optimized memory allocation during training and storage
+ - Efficient vector reconstruction from quantized codes
+ - Fast ADC search implementation with SIMD optimizations
+ - Automatic performance scaling post-training (up to 8x faster additions)
+
+### Changed
+- Vector Addition Behavior
+ - Automatic training trigger when threshold is reached during vector addition
+ - Dynamic storage mode switching from raw to quantized seamlessly
+ - Enhanced error reporting with detailed failure information in AddResult
+ - Improved batch processing with better memory management
+
+- Search Performance
+ - Adaptive search strategy based on storage mode (raw vs quantized)
+ - Optimized distance calculations for quantized vectors
+ - Enhanced result quality with proper score normalization
+
+- Index Architecture
+ - 3-path storage system replaces simple raw storage
+ - Intelligent memory management with automatic cleanup
+ - Robust state transitions between storage modes
+ - Enhanced concurrency handling with proper lock management
+
+- Statistics and Monitoring
+ - Extended statistics including quantization metrics
+ - Real-time progress tracking during training operations
+ - Enhanced memory usage reporting with compression analysis
+ - Detailed timing information for performance optimization
+
+- Default search parameters tuned for quantized and L1/L2 spaces (e.g., higher default ef_search for L1/L2).
+- Improved error messages for quantization-related failures and configuration issues.
+- Consistent handling of vector normalization (cosine) vs. raw (L1/L2) in all input/output paths.
+
+### Fixed
+- Memory Management
+ - Fixed temporary value lifetime issues in PyO3 integration
+ - Resolved borrow checker conflicts in quantization pipeline
+ - Corrected memory leaks during large-scale operations
+ - Fixed reference counting for Python object handling
+
+- Vector Processing
+ - Fixed input format parsing for edge cases and invalid data
+ - Resolved metadata conversion issues for complex Python objects
+ - Corrected vector dimension validation with proper error messages
+ - Fixed batch processing memory allocation issues
+
+- Performance Issues
+ - Optimized training memory usage to prevent out-of-memory errors
+ - Fixed search performance degradation in large indexes
+ - Resolved training stability issues with improved k-means initialization
+ - Corrected distance calculation accuracy in quantized mode
+
+- Error Handling
+ - Enhanced validation for quantization configuration parameters
+ - Improved error propagation from Rust to Python
+ - Fixed panic conditions in edge cases
+ - Better handling of invalid input combinations
+
+- Fixed rare edge case where quantization training could stall with duplicate vectors.
+- Resolved non-deterministic search results in small datasets with L1/L2 metrics by tuning search parameters.
+- Fixed debug output leaking to production logs (now controlled by environment variable).
+
+### Removed
+- Removed legacy single-path storage logic (now fully 3-path).
+- Deprecated or removed any old quantization/test hooks that are no longer needed.
+
+---
+
 ## [0.1.2] - 2025-07-17
 
 ### Added
 
@@ -37,22 +37,30 @@ ZeusDB leverages the HNSW (Hierarchical Navigable Small World) algorithm for spe
 
 ## ⭐ Features
 
-🔍 Approximate Nearest Neighbor (ANN) search with HNSW
+🐍 User-friendly Python API for adding vectors and running similarity searches
 
-📋 Supports multiple distance metrics: `cosine`, `L1`, `L2` 
+🔥 High-performance Rust backend optimized for speed and concurrency
 
-🔥 High-performance Rust backend 
+🔍 Approximate Nearest Neighbor (ANN) search using HNSW for fast, accurate results
+
+📦 Product Quantization (PQ) for compact storage, faster distance computations, and scalability for Big Data
+
+📥 Flexible input formats, including native Python types and zero-copy NumPy arrays
+
+🗂️ Metadata-aware filtering for precise and contextual querying
 
-📥 Supports multiple input formats using a single, easy-to-use Python method
 
-🗂️ Metadata-aware filtering at query time
 
-🐍 Simple and intuitive Python API
+
+<!-- 
+📋 Supports multiple distance metrics: `cosine`, `L1`, `L2` 
+
+📥 Supports multiple input formats using a single, easy-to-use Python method
 
 ⚡ Smart multi-threaded inserts that automatically speed up large batch uploads
 
 🚀 Fast, concurrent searches so you can run multiple queries at the same time
-
+-->
 
 <br/>
 
@@ -196,10 +204,11 @@ index = vdb.create(
 |------------------|--------|-----------|-----------------------------------------------------------------------------|
 | `index_type`     | `str`  | `"hnsw"`  | The type of vector index to create. Currently supports `"hnsw"`. Future options include `"ivf"`, `"flat"`, etc. Case-insensitive. |
 | `dim`            | `int`  | `1536`    | Dimensionality of the vectors to be indexed. Each vector must have this length. The default dim=1536 is chosen to match the output dimensionality of OpenAI’s text-embedding-ada-002 model. |
-| `space`          | `str`  | `"cosine"`| Distance metric used for similarity search. Options include `"cosine"`. Additional metrics such as `"l2"`, and `"dot"` will be added in future versions. |
+| `space`          | `str`  | `"cosine"`| Distance metric used for similarity search. Options include `"cosine"`, `"L1"` and `"L2"`.|
 | `m`              | `int`  | `16`      | Number of bi-directional connections created for each new node. Higher `m` improves recall but increases index size and build time. |
 | `ef_construction`| `int`  | `200`     | Size of the dynamic list used during index construction. Larger values increase indexing time and memory, but improve quality. |
 | `expected_size`  | `int`  | `10000`   | Estimated number of elements to be inserted. Used for preallocating internal data structures. Not a hard limit. |
+| `quantization_config` | `dict` | `None` | Product Quantization configuration for memory-efficient vector compression. |
 
 <br/>
 
@@ -536,6 +545,156 @@ print(partial)
 
 ⚠️ `get_records()` only returns results for IDs that exist in the index. Missing IDs are silently skipped.
 
+<br />
+
+
+## 🗜️ Product Quantization
+
+Product Quantization (PQ) is a vector compression technique that significantly reduces memory usage while preserving high search accuracy. Commonly used in HNSW-based vector databases, PQ works by dividing each vector into subvectors and quantizing them independently. This enables compression ratios of 4× to 256×, making it ideal for large-scale, high-dimensional datasets.
+
+ZeusDB Vector Database’s PQ implementation features:
+
+✅ Intelligent Training – PQ model trains automatically at defined thresholds
+
+✅ Efficient Memory Use – Store 4× to 256× more vectors in the same RAM footprint
+
+✅ Fast Approximate Search – Uses Asymmetric Distance Computation (ADC) for high-speed search computation
+
+✅ Seamless Operation – Index automatically switches from raw to quantized storage modes
+
+<br />
+
+### 📘 Quantization Configuration Parameters
+
+To enable PQ, pass a `quantization_config` dictionary to the `.create()` index method:
+
+| Parameter | Type | Description | Valid Range | Default |
+|-----------|------|-------------|-------------|---------|
+| `type` | `str` | Quantization algorithm type | `"pq"` | *required* |
+| `subvectors` | `int` | Number of vector subspaces (must divide dimension evenly) | 1 to dimension | `8` |
+| `bits` | `int` | Bits per quantized code (controls centroids per subvector) | 1-8 | `8` |
+| `training_size` | `int` | Minimum vectors needed for stable k-means clustering | ≥ 1000 | 1000 |
+| `max_training_vectors` | `int` | Maximum vectors used during training (optional limit) | ≥ training_size | `None` |
+
+
+<br/>
+
+
+### 🔧 Usage Example
+
+```python
+from zeusdb_vector_database import VectorDatabase
+import numpy as np
+
+# Create index with product quantization
+vdb = VectorDatabase()
+
+# Configure quantization for memory efficiency
+quantization_config = {
+    'type': 'pq',                  # `pq` for Product Quantization
+    'subvectors': 8,               # Divide 1536-dim vectors into 8 subvectors of 192 dims each
+    'bits': 8,                     # 256 centroids per subvector (2^8)
+    'training_size': 10000,        # Train when 10k vectors are collected
+    'max_training_vectors': 50000  # Use max 50k vectors for training
+}
+
+# Create index with quantization
+# This will automatically handle training when enough vectors are added
+index = vdb.create(
+    index_type="hnsw",
+    dim=1536,                                  # OpenAI `text-embedding-3-small` dimension
+    quantization_config=quantization_config    # Add the compression configuration
+)
+
+# Add vectors - training triggers automatically at threshold
+documents = [
+    {
+        "id": f"doc_{i}", 
+        "values": np.random.rand(1536).astype(float).tolist(),
+        "metadata": {"category": "tech", "year": 2026}
+    }
+    for i in range(15000) 
+]
+
+# Training will trigger automatically when 10k vectors are added
+result = index.add(documents)
+print(f"Added {result.total_inserted} vectors")
+
+# Check quantization status
+print(f"Training progress: {index.get_training_progress():.1f}%")
+print(f"Storage mode: {index.get_storage_mode()}")
+print(f"Is quantized: {index.is_quantized()}")
+
+# Get compression statistics
+quant_info = index.get_quantization_info()
+if quant_info:
+    print(f"Compression ratio: {quant_info['compression_ratio']:.1f}x")
+    print(f"Memory usage: {quant_info['memory_mb']:.1f} MB")
+
+# Search works seamlessly with quantized storage
+query_vector = np.random.rand(1536).astype(float).tolist()
+results = index.search(vector=query_vector, top_k=3)
+
+# Simply print raw results
+print(results)
+```
+
+Results
+```python
+[
+{'id': 'doc_9719', 'score': 0.5133496522903442, 'metadata': {'category': 'tech', 'year': 2026}},
+{'id': 'doc_8148', 'score': 0.5139288306236267, 'metadata': {'category': 'tech', 'year': 2026}}, 
+{'id': 'doc_7822', 'score': 0.5151920914649963, 'metadata': {'category': 'tech', 'year': 2026}}, 
+]
+```
+
+<br />
+
+### ⚙️ Configuration Guidelines
+
+For Balanced Memory & Accuracy (Recommended to start with)
+```python
+quantization_config = {
+    'type': 'pq',
+    'subvectors': 8,      # Balanced: moderate compression, good accuracy
+    'bits': 8,            # 256 centroids per subvector (high precision)
+    'training_size': 10000  # Or higher for large datasets
+}
+# Achieves ~16x–32x compression with strong recall for most applications
+```
+
+
+For Memory Optimization:
+```python
+quantization_config = {
+    'type': 'pq',
+    'subvectors': 16,      # More subvectors = better compression
+    'bits': 6,             # Fewer bits = less memory per centroid
+    'training_size': 20000
+}
+# Achieves ~32x compression ratio
+```
+
+For Accuracy Optimization:
+```python
+quantization_config = {
+    'type': 'pq',
+    'subvectors': 4,       # Fewer subvectors = better accuracy
+    'bits': 8,             # More bits = more precise quantization
+    'training_size': 50000 # More training data = better centroids
+}
+# Achieves ~4x compression ratio with minimal accuracy loss
+```
+
+### 📊 Performance Characteristics
+
+- Training: Occurs once when threshold is reached (typically 1-5 minutes for 50k vectors)
+- Memory Reduction: 4x-256x depending on configuration
+- Search Speed: Comparable or faster than raw vectors due to ADC optimization
+- Accuracy Impact: Typically 1-5% recall reduction with proper tuning
+
+Quantization is ideal for production deployments with large vector datasets (100k+ vectors) where memory efficiency is critical.
+
 
 <br/>