Add changelog, optimize performance, and enhance CLI features for version 0.1.2

- Introduced CHANGELOG.md to document notable changes and updates.
- Added performance optimizations including auto-detection of system resources and new flags for CLI commands.
- Enhanced progress tracking with a unified progress bar and improved terminal output.
- Updated README.md with new features and installation instructions.
- Integrated memory management and batch processing improvements for better efficiency.
- Added tests for optimized configuration and memory management features.

Author: starkbaknet

CHANGELOG.md

@@ -0,0 +1,211 @@
+# Changelog
+
+All notable changes to Project Vectorizer will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.2] - 2025-10-13
+
+### Added
+
+#### Performance & Optimization
+- **Optimized Config Generation** - New `Config.create_optimized()` method that auto-detects optimal settings based on system resources
+  - Auto-detects CPU cores and sets optimal `max_workers`
+  - Calculates safe batch sizes based on available RAM
+  - Dynamically adjusts memory thresholds and GC intervals
+  - Use with `pv init --optimize` for permanent optimization
+- **Max Resources Flag** - New `--max-resources` flag for `index` and `index-git` commands
+  - Temporarily overrides config with optimized settings
+  - Perfect for one-time performance boosts
+  - Example: `pv index . --max-resources`
+- **psutil Integration** - Added `psutil` library for system resource detection
+  - Automatic CPU core detection
+  - Memory availability monitoring
+  - Smart resource allocation
+
+#### Progress & UI Improvements
+- **Unified Progress Tracking** - Clean, single-line progress bar for all indexing operations
+  - Replaces cluttered per-file logging
+  - Real-time progress updates with file names
+  - Shows file status tags: [New], [Modified], [Deleted]
+  - Callback-based architecture for flexible progress reporting
+- **Library Progress Bar Suppression** - Suppressed sentence-transformers batch progress bars
+  - Added `show_progress_bar=False` parameter to embedding generation
+  - Eliminates "Batches: 0%|..." clutter during indexing
+- **Timing Information** - All indexing operations now display elapsed time
+  - Shows duration in seconds (< 1 min) or minutes+seconds (≥ 1 min)
+  - Displayed in completion panels
+  - Example: "Time taken: 2m 17s"
+- **Clean Terminal Output** - Professional, easy-to-read output
+  - Automatic log suppression when progress callback is active
+  - Warnings and errors still shown
+  - Verbose mode available with `--verbose` flag
+
+#### Documentation
+- **Comprehensive Documentation Index** - New `docs/README.md` with organized navigation
+  - Links to all feature guides
+  - Quick reference section
+  - CLI command reference
+  - Configuration reference
+  - Troubleshooting guide
+- **Performance Guides** - Three detailed guides added:
+  - `MAX_RESOURCES_GUIDE.md` - Using maximum system resources
+  - `OPTIMIZED_CONFIG.md` - Auto-optimization features
+  - `CLEAN_PROGRESS_OUTPUT.md` - Progress tracking system
+- **Changelog** - This file! Track all version changes
+
+### Changed
+
+#### Configuration
+- **Chunk Size Enforcement** - Engine now enforces max 128-token chunks (line 35 in engine.py)
+  - Improved precision for searches
+  - Better granularity for code matching
+  - Configurable chunk size still respected for larger values
+- **Default Chunk Overlap** - Optimized overlap settings
+  - Better balance between precision and performance
+  - Reduced from 32 to 16 tokens for 128-chunk configs
+
+#### Performance
+- **Batch Processing** - Memory-based batch size validation
+  - Safe batch sizes calculated based on available RAM
+  - Prevents out-of-memory errors on large projects
+- **Parallel Processing** - Improved worker management
+  - Auto-detection of optimal worker count
+  - Better CPU utilization
+  - Configurable via `max_workers` setting
+
+#### CLI
+- **Progress Output** - All index commands now use unified progress bar
+  - `pv index` - Shows clean progress with timing
+  - `pv index-git` - Shows git reference and timing
+  - `pv sync --watch` - Real-time progress updates
+
+### Fixed
+
+- **Progress Bar Clutter** - Fixed multiple progress bars appearing during indexing
+- **Sentence-Transformers Output** - Suppressed library-internal batch progress bars
+- **Memory Usage** - Better memory management with GC intervals and monitoring
+- **Log Suppression** - Informational logs now properly suppressed when progress callback is active
+
+### Performance
+
+#### Benchmarks (with --max-resources)
+- **Indexing Speed**: ~2m 16s for 48 files (9,222 chunks) on 8-core system
+- **Chunk Size Impact**: Virtually identical performance between 128 and 512 token chunks
+- **Optimal Settings**: Auto-detected based on system (16 workers, 400 batch size on 8-core/16GB RAM)
+
+#### Improvements
+- **Smart Incremental**: 60-70% faster than full re-indexing
+- **Git-Aware**: 80-90% faster for recent changes only
+- **Parallel Processing**: Linear scaling with CPU cores
+
+### Documentation
+
+All documentation has been reorganized and consolidated:
+- New central `docs/README.md` with comprehensive navigation
+- Updated main `README.md` with latest features
+- Added performance optimization guides
+- Added troubleshooting sections
+- Removed overly technical implementation documents
+
+---
+
+## [0.1.1] - 2025-10-12
+
+### Added
+- **Enhanced Single-Word Search** - Optimized search for single keywords with high precision
+- **Multi-Level Chunking** - Added micro-chunks and word-level chunks for better granularity
+- **Adaptive Search Thresholds** - Automatically adjusts thresholds for optimal results
+- **Improved Word Matching** - Enhanced exact word boundary matching with relevance boosting
+- **Programming Keyword Detection** - Special handling for common programming terms
+- **Search Result Prioritization** - Better ranking for single-word queries
+- **Smaller Default Chunk Size** - Reduced to 256 tokens for better precision
+
+### Changed
+- Search algorithm prioritizes exact matches over semantic similarity
+- Chunk size defaults changed for better single-word search performance
+
+---
+
+## [0.1.0] - 2025-10-10
+
+### Added
+- Initial release
+- Code vectorization with sentence-transformers and OpenAI embeddings
+- Smart incremental indexing with file categorization
+- Git-aware indexing for commit-based updates
+- Watch mode with configurable debouncing
+- MCP (Model Context Protocol) server
+- HTTP fallback API
+- ChromaDB backend for vector storage
+- Support for 30+ programming languages
+- Real-time file monitoring
+- Semantic search with similarity thresholds
+- Project status tracking
+- Comprehensive CLI interface
+
+### Features
+- Parse and vectorize code files
+- Track changes with git integration
+- Multiple indexing modes (full, incremental, smart, git-aware)
+- Search with natural language queries
+- MCP server for AI agent integration
+- File watching with debounce control
+- Configurable embeddings (sentence-transformers or OpenAI)
+- Multi-language support
+- Vector similarity search
+
+---
+
+## Release Notes
+
+### Upgrading to 0.1.2
+
+**New installations:**
+```bash
+pip install project-vectorizer
+pv init /path/to/project --optimize
+```
+
+**Existing projects:**
+```bash
+pip install --upgrade project-vectorizer
+
+# Optional: Regenerate config with optimization
+pv init /path/to/project --optimize
+
+# Re-index with max resources
+pv index /path/to/project --max-resources
+```
+
+### Breaking Changes
+
+None. Version 0.1.2 is fully backward compatible with 0.1.1 and 0.1.0.
+
+### Deprecations
+
+None.
+
+### Migration Guide
+
+No migration needed. Existing projects will continue to work with their current configurations. To take advantage of new optimization features:
+
+1. **Enable optimization** (optional):
+   ```bash
+   pv init /path/to/project --optimize
+   ```
+
+2. **Use max resources** (optional):
+   ```bash
+   pv index /path/to/project --max-resources
+   ```
+
+---
+
+## Links
+
+- [GitHub Repository](https://github.com/starkbaknet/project-vectorizer)
+- [PyPI Package](https://pypi.org/project/project-vectorizer/)
+- [Documentation](docs/README.md)
+- [Issue Tracker](https://github.com/starkbaknet/project-vectorizer/issues)