Release v0.2.1

* Bump version to 0.2.1 for development

* feat: add configurable storage modes for product quantization

* Add: bincode v2.0.1 dependency for binary serialization

* feat: implement Rust backend support for storage mode configuration

* 📄 docs(changelog): update for v0.2.1 release

* Add: latest uv.lock file for reproducible Python dependency management

* test: add comprehensive storage mode tests for quantization configurations

* 📄 docs(readme): updated the content information

* 📄 docs(readme): updated the content information

Author: doubleinfinity

CHANGELOG.md

@@ -7,6 +7,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.2.1] - 2025-
+
+### Added
+- Storage mode configuration for product quantization: New storage_mode parameter in quantization config allows users to choose between:
+  - '"quantized_only"' (default): Maximum memory efficiency by discarding raw vectors after quantization
+  - '"quantized_with_raw"': Keep both quantized codes and raw vectors for exact reconstruction
+- Case-insensitive storage mode validation: Accepts variations like "Quantized_Only", "QUANTIZED_WITH_RAW"
+- Automatic memory usage warnings: Users are warned when `quantized_with_raw` mode will use significantly more memory
+- Enhanced subvector divisor suggestions: `_suggest_subvector_divisors()` now returns `list[int]` for programmatic use
+- StorageMode enum: Rust backend support for `quantized_only` and `quantized_with_raw` storage modes with JSON serialization
+- Storage mode parsing: Complete quantization config parsing in HNSWIndex constructor with proper error handling
+- Intelligent vector retrieval: `get_records()` method now prioritizes raw vectors over PQ reconstruction when available
+- Enhanced statistics: `get_stats()` now reports storage mode, memory usage breakdown, and storage strategy information
+- Memory usage tracking: Real-time memory usage calculations for both raw vectors and quantized codes
+
+### Changed
+- Quantization config validation: Now includes comprehensive validation and normalization of all parameters
+- Error messages: Improved clarity for storage mode validation with sorted mode suggestions
+- Defensive programming: Added final safety checks to ensure complete configuration before passing to Rust backend
+- QuantizationConfig struct: Now includes `storage_mode` field with backward-compatible defaults
+- add_quantized_vector logic: Respects storage mode configuration to conditionally store raw vectors
+- get_stats output: Enhanced with storage strategy descriptions ("memory_optimized" vs "quality_optimized")
+- Vector storage behavior: `quantized_only` mode stops storing raw vectors after PQ training for maximum memory efficiency
+
+### Fixed
+- Configuration completeness: All quantization parameters now have guaranteed defaults to prevent missing key errors
+- None value handling: Python config cleaning now properly removes `None` values before passing to Rust backend
+- Constructor parameter validation: Improved error handling for missing or invalid quantization parameters
+- Memory statistics accuracy: Corrected memory usage calculations based on actual storage mode behavior
+
+### Removed
+<!-- Add removals/deprecations here -->
+
+---
+
 ## [0.2.0] - 2025-07-28
 
 ### Added

README.md

@@ -575,12 +575,13 @@ To enable PQ, pass a `quantization_config` dictionary to the `.create()` index m
 | `bits` | `int` | Bits per quantized code (controls centroids per subvector) | 1-8 | `8` |
 | `training_size` | `int` | Minimum vectors needed for stable k-means clustering | ≥ 1000 | 1000 |
 | `max_training_vectors` | `int` | Maximum vectors used during training (optional limit) | ≥ training_size | `None` |
+| `storage_mode` | `str` | Storage strategy: "quantized_only" (memory optimized) or "quantized_with_raw" (keep raw vectors for exact reconstruction) | "quantized_only", "quantized_with_raw" | `"quantized_only"` |
 
 
 <br/>
 
 
-### 🔧 Usage Example
+### 🔧 Usage Example 1
 
 ```python
 from zeusdb_vector_database import VectorDatabase
@@ -646,6 +647,36 @@ Results
 {'id': 'doc_8148', 'score': 0.5139288306236267, 'metadata': {'category': 'tech', 'year': 2026}}, 
 {'id': 'doc_7822', 'score': 0.5151920914649963, 'metadata': {'category': 'tech', 'year': 2026}}, 
 ]
+```
+<br />
+
+### 🔧 Usage Example 2 - with explicit storage mode
+
+```python
+from zeusdb_vector_database import VectorDatabase
+import numpy as np
+
+# Create index with product quantization
+vdb = VectorDatabase()
+
+# Configure quantization for memory efficiency
+quantization_config = {
+    'type': 'pq',                  # `pq` for Product Quantization
+    'subvectors': 8,               # Divide 1536-dim vectors into 8 subvectors of 192 dims each
+    'bits': 8,                     # 256 centroids per subvector (2^8)
+    'training_size': 10000,        # Train when 10k vectors are collected
+    'max_training_vectors': 50000,  # Use max 50k vectors for training
+    'storage_mode': 'quantized_only'  # Explicitly set storage mode to only keep quantized values
+}
+
+# Create index with quantization
+# This will automatically handle training when enough vectors are added
+index = vdb.create(
+    index_type="hnsw",
+    dim=3072,                                  # OpenAI `text-embedding-3-large` dimension
+    quantization_config=quantization_config    # Add the compression configuration
+)
+
 ```
 
 <br />
@@ -658,7 +689,8 @@ quantization_config = {
     'type': 'pq',
     'subvectors': 8,      # Balanced: moderate compression, good accuracy
     'bits': 8,            # 256 centroids per subvector (high precision)
-    'training_size': 10000  # Or higher for large datasets
+    'training_size': 10000,  # Or higher for large datasets
+    'storage_mode': 'quantized_only'  # Default, memory efficient
 }
 # Achieves ~16x–32x compression with strong recall for most applications
 ```
@@ -670,7 +702,8 @@ quantization_config = {
     'type': 'pq',
     'subvectors': 16,      # More subvectors = better compression
     'bits': 6,             # Fewer bits = less memory per centroid
-    'training_size': 20000
+    'training_size': 20000,
+    'storage_mode': 'quantized_only'
 }
 # Achieves ~32x compression ratio
 ```
@@ -682,6 +715,7 @@ quantization_config = {
     'subvectors': 4,       # Fewer subvectors = better accuracy
     'bits': 8,             # More bits = more precise quantization
     'training_size': 50000 # More training data = better centroids
+    'storage_mode': 'quantized_with_raw'  # Keep raw vectors for exact recall
 }
 # Achieves ~4x compression ratio with minimal accuracy loss
 ```
@@ -695,6 +729,10 @@ quantization_config = {
 
 Quantization is ideal for production deployments with large vector datasets (100k+ vectors) where memory efficiency is critical.
 
+`"quantized_only"` is recommended for most use cases and maximizes memory savings.
+
+`"quantized_with_raw"` keeps both quantized and raw vectors for exact reconstruction, but uses more memory.
+
 
 <br/>
 

benchmarks/29.new_storage_mode_configs_test.py

@@ -0,0 +1,120 @@
+# Usage Examples for Storage Mode Configuration
+
+from zeusdb_vector_database import VectorDatabase
+import numpy as np
+
+# Example 1: Default (Memory Efficient) - quantized_only
+vdb = VectorDatabase()
+index_memory_efficient = vdb.create(
+    "hnsw", 
+    dim=768,
+    quantization_config={
+        "type": "pq",
+        "subvectors": 8,
+        "bits": 8,
+        "training_size": 10000
+        # storage_mode defaults to "quantized_only"
+    }
+)
+
+# Example 2: Explicit quantized_only mode
+index_explicit = vdb.create(
+    "hnsw", 
+    dim=768,
+    quantization_config={
+        "type": "pq",
+        "subvectors": 8,
+        "bits": 8,
+        "training_size": 10000,
+        "storage_mode": "quantized_only"
+    }
+)
+
+# Example 3: Keep raw vectors for exact reconstruction
+index_with_raw = vdb.create(
+    "hnsw", 
+    dim=768,
+    quantization_config={
+        "type": "pq",
+        "subvectors": 8,
+        "bits": 8,
+        "training_size": 10000,
+        "storage_mode": "quantized_with_raw"  # Keep both quantized + raw
+    }
+)
+# This will show a warning about increased memory usage
+
+# Testing the different modes
+def test_storage_modes():
+    # Generate test data
+    vectors = np.random.random((15000, 768)).astype(np.float32)
+    
+    # Test quantized_only mode
+    print("=== Testing quantized_only mode ===")
+    index1 = vdb.create("hnsw", dim=768, quantization_config={
+        "type": "pq", "subvectors": 8, "bits": 8, 
+        "training_size": 10000, "storage_mode": "quantized_only"
+    })
+    
+    # Add vectors (will trigger training)
+    result1 = index1.add(vectors.tolist())
+    print(f"Added: {result1.total_inserted}, Errors: {result1.total_errors}")
+    
+    # Check stats
+    stats1 = index1.get_stats()
+    print(f"Storage mode: {stats1['storage_mode']}")
+    print(f"Raw vectors stored: {stats1['raw_vectors_stored']}")
+    print(f"Quantized codes stored: {stats1['quantized_codes_stored']}")
+    
+    # Get records (will use PQ reconstruction)
+    records1 = index1.get_records(["vec_1"], return_vector=True)
+    print(f"Vector available: {'vector' in records1[0] if records1 else False}")
+
+    if records1 and 'vector' in records1[0]:
+        print(f"Vector shape: {len(records1[0]['vector'])}")
+    
+    print("\n=== Testing quantized_with_raw mode ===")
+    index2 = vdb.create("hnsw", dim=768, quantization_config={
+        "type": "pq", "subvectors": 8, "bits": 8, 
+        "training_size": 10000, "storage_mode": "quantized_with_raw"
+    })
+    
+    # Add vectors (will trigger training)
+    result2 = index2.add(vectors.tolist())
+    print(f"Added: {result2.total_inserted}, Errors: {result2.total_errors}")
+    
+    # Check stats
+    stats2 = index2.get_stats()
+    print(f"Storage mode: {stats2['storage_mode']}")
+    print(f"Raw vectors stored: {stats2['raw_vectors_stored']}")
+    print(f"Quantized codes stored: {stats2['quantized_codes_stored']}")
+    
+    # Get records (will use exact raw vectors)
+    records2 = index2.get_records(["vec_1"], return_vector=True)
+    print(f"Vector available: {'vector' in records2[0] if records2 else False}")
+
+    if records2 and 'vector' in records2[0]:
+        print(f"Vector shape: {len(records2[0]['vector'])}")
+    
+    # Compare memory usage
+    print("\nMemory comparison:")
+    print(f"quantized_only - Raw vectors: {stats1['raw_vectors_stored']}")
+    print(f"quantized_with_raw - Raw vectors: {stats2['raw_vectors_stored']}")
+
+# Error handling test
+def test_invalid_storage_mode():
+    try:
+        vdb = VectorDatabase()
+        vdb.create("hnsw", dim=768, quantization_config={
+            "type": "pq", 
+            "subvectors": 8, 
+            "bits": 8, 
+            "training_size": 10000,
+            "storage_mode": "invalid_mode"  # This should fail
+        })
+    except ValueError as e:
+        print(f"Expected error: {e}")
+
+if __name__ == "__main__":
+    test_storage_modes()
+    test_invalid_storage_mode()

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "zeusdb-vector-database"
-version = "0.2.0"
+version = "0.2.1"
 description = "Blazing-fast vector DB with real-time similarity search and metadata filtering."
 readme = "README.md"
 authors = [

src/zeusdb_vector_database/__init__.py

@@ -1,7 +1,7 @@
 """
 ZeusDB Vector Database Module
 """
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 from .vector_database import VectorDatabase # imports the VectorDatabase class from the vector_database.py file
 

src/zeusdb_vector_database/vector_database.py

@@ -56,7 +56,8 @@ def create(self, index_type: str = "hnsw", quantization_config: Optional[Dict[st
                     'subvectors': 8,           # Number of subvectors (must divide dim evenly, default: 8)
                     'bits': 8,                 # Bits per subvector (1-8, controls centroids, default: 8)
                     'training_size': None,     # Auto-calculated based on subvectors & bits (or specify manually)
-                    'max_training_vectors': None  # Optional limit on training vectors used
+                    'max_training_vectors': None,  # Optional limit on training vectors used
+                    'storage_mode': 'quantized_only' # Storage mode for quantized vectors (or 'quantized_with_raw')  
                 }
 
             Note: Quantization reduces memory usage (typically 4-32x compression) but may 
@@ -88,7 +89,8 @@ def create(self, index_type: str = "hnsw", quantization_config: Optional[Dict[st
                 'type': 'pq',
                 'subvectors': 16,         # More subvectors = better compression
                 'bits': 6,                # Fewer bits = less memory per centroid
-                'training_size': 75000    # Override auto-calculation
+                'training_size': 75000,    # Override auto-calculation
+                'storage_mode': 'quantized_only'  # Only store quantized vectors
             }
             index = vdb.create(
                 index_type="hnsw",
@@ -126,11 +128,12 @@ def create(self, index_type: str = "hnsw", quantization_config: Optional[Dict[st
 
         try:
             # Always pass quantization_config parameter
-            clean_config = None
             if quantization_config is not None:
-                # Clean quantization_config before passing to Rust (remove internal keys)
-                clean_config = {k: v for k, v in quantization_config.items() if not k.startswith('_')}
-            
+                # Remove keys with None values and internal keys
+                clean_config = {k: v for k, v in quantization_config.items() if not k.startswith('_') and v is not None}
+            else:
+                clean_config = None
+
             return constructor(quantization_config=clean_config, **kwargs)
         except Exception as e:
             raise RuntimeError(f"Failed to create {index_type.upper()} index: {e}") from e
@@ -172,7 +175,7 @@ def _validate_quantization_config(self, config: Dict[str, Any], dim: int) -> Dic
         if dim % subvectors != 0:
             raise ValueError(
                 f"subvectors ({subvectors}) must divide dimension ({dim}) evenly. "
-                f"Consider using subvectors: {self._suggest_subvector_divisors(dim)}"
+                f"Consider using subvectors: {', '.join(map(str, self._suggest_subvector_divisors(dim)))}"
             )
 
         if subvectors > dim:
@@ -206,9 +209,38 @@ def _validate_quantization_config(self, config: Dict[str, Any], dim: int) -> Dic
                 )
             validated_config['max_training_vectors'] = max_training_vectors
 
+        # Validate storage mode
+        storage_mode = str(validated_config.get('storage_mode', 'quantized_only')).lower()
+        valid_modes = {'quantized_only', 'quantized_with_raw'}
+        if storage_mode not in valid_modes:
+            raise ValueError(
+                f"Invalid storage_mode: '{storage_mode}'. Supported modes: {', '.join(sorted(valid_modes))}"
+            )
+        
+        validated_config['storage_mode'] = storage_mode
+
         # Calculate and warn about memory usage
         self._check_memory_usage(validated_config, dim)
+
+        # Add helpful warnings about storage mode
+        if storage_mode == 'quantized_with_raw':
+            import warnings
+            compression_ratio = validated_config.get('__memory_info__', {}).get('compression_ratio', 1.0)
+            warnings.warn(
+                f"storage_mode='quantized_with_raw' will use ~{compression_ratio:.1f}x more memory "
+                f"than 'quantized_only' but enables exact vector reconstruction.",
+                UserWarning,
+                stacklevel=2
+            )
 
+        # Final safety check: ensure all expected keys are present
+        # This is a final defensive programming - all the keys should already be set above, but added just in case
+        validated_config.setdefault('type', 'pq')
+        validated_config.setdefault('subvectors', 8)
+        validated_config.setdefault('bits', 8)
+        validated_config.setdefault('max_training_vectors', None)
+        validated_config.setdefault('storage_mode', 'quantized_only')
+
         return validated_config
 
     def _calculate_smart_training_size(self, subvectors: int, bits: int) -> int:
@@ -236,13 +268,14 @@ def _calculate_smart_training_size(self, subvectors: int, bits: int) -> int:
 
         return min(max(statistical_minimum, reasonable_minimum), reasonable_maximum)
 
-    def _suggest_subvector_divisors(self, dim: int) -> str:
-        """Suggest valid subvector counts that divide the dimension evenly."""
-        divisors = []
-        for i in range(1, min(33, dim + 1)):  # Common subvector counts up to 32
-            if dim % i == 0:
-                divisors.append(str(i))
-        return ', '.join(divisors[:8])  # Show first 8 suggestions
+    
+    def _suggest_subvector_divisors(self, dim: int) -> list[int]:
+        """Return valid subvector counts that divide the dimension evenly (up to 32)."""
+        return [i for i in range(1, min(33, dim + 1)) if dim % i == 0]
+    
+
+
+
 
     def _check_memory_usage(self, config: Dict[str, Any], dim: int) -> None:
         """