Add comprehensive README for Rust BSON extension (PYTHON-5683)

- Document implementation history and architecture
- Compare with Copilot POC from PR #2689
- Explain performance analysis (~0.21x vs C extension)
- Clarify why POC appeared faster (incomplete functionality)
- Detail build system differences (cargo vs maturin)
- Provide steps to achieve performance parity
- Include building, testing, and usage instructions

Author: aclark4life

bson/_rbson/README.md

@@ -0,0 +1,277 @@
+# Rust BSON Extension Module
+
+This directory contains a Rust-based implementation of BSON encoding/decoding for PyMongo, developed as part of [PYTHON-5683](https://jira.mongodb.org/browse/PYTHON-5683).
+
+## Overview
+
+The Rust extension (`_rbson`) provides the same interface as the C extension (`_cbson`) but is implemented in Rust using:
+- **PyO3**: Python bindings for Rust
+- **bson crate**: MongoDB's official Rust BSON library
+- **Maturin**: Build tool for Rust Python extensions
+
+## Implementation History
+
+This implementation was developed through [PR #2695](https://github.com/mongodb/mongo-python-driver/pull/2695) to investigate using Rust as an alternative to C for Python extension modules.
+
+### Key Milestones
+
+1. **Initial Implementation** - Complete BSON type support with 100% test compatibility (88/88 tests passing)
+2. **Performance Optimizations** - Type caching, fast paths for common types, direct byte operations
+3. **Architectural Analysis** - Identified fundamental performance differences between Rust and C approaches
+
+## Features
+
+### Supported BSON Types
+
+The Rust extension supports all BSON types:
+- **Primitives**: Double, String, Int32, Int64, Boolean, Null
+- **Complex Types**: Document, Array, Binary, ObjectId, DateTime
+- **Special Types**: Regex, Code, Timestamp, Decimal128, MinKey, MaxKey
+- **Deprecated Types**: DBPointer (decodes to DBRef)
+
+### CodecOptions Support
+
+Full support for PyMongo's `CodecOptions`:
+- `document_class` - Custom document classes
+- `tz_aware` - Timezone-aware datetime handling
+- `tzinfo` - Timezone conversion
+- `uuid_representation` - UUID encoding/decoding modes
+- `datetime_conversion` - DateTime handling modes (AUTO, CLAMP, MS)
+- `unicode_decode_error_handler` - UTF-8 error handling
+
+### Runtime Selection
+
+The Rust extension can be enabled via environment variable:
+```bash
+export PYMONGO_USE_RUST=1
+python your_script.py
+```
+
+Without this variable, PyMongo uses the C extension by default.
+
+## Performance Analysis
+
+### Current Performance: ~0.21x (5x slower than C)
+
+**Benchmark Results** (from PR #2695):
+```
+Simple documents:  C: 100%  | Rust: 21%
+Mixed types:       C: 100%  | Rust: 20%
+Nested documents:  C: 100%  | Rust: 18%
+Lists:             C: 100%  | Rust: 22%
+```
+
+### Root Cause: Architectural Difference
+
+The performance gap is due to a fundamental architectural difference:
+
+**C Extension Architecture:**
+```
+Python objects → BSON bytes (direct)
+```
+- Writes BSON bytes directly from Python objects
+- No intermediate data structures
+- Minimal memory allocations
+
+**Rust Extension Architecture:**
+```
+Python objects → Rust Bson enum → BSON bytes
+```
+- Converts Python objects to Rust `Bson` enum
+- Then serializes `Bson` to bytes
+- Extra conversion layer adds overhead
+
+### Optimization Attempts
+
+Multiple optimization strategies were attempted in PR #2695:
+
+1. **Type Caching** - Cache frequently used Python types (UUID, datetime, etc.)
+2. **Fast Paths** - Special handling for common types (int, str, bool, None)
+3. **Direct Byte Writing** - Write BSON bytes directly without intermediate `Document`
+4. **PyDict Fast Path** - Use `PyDict_Next` for efficient dict iteration
+
+**Result**: These optimizations improved performance from ~0.15x to ~0.21x, but the fundamental architectural difference remains.
+
+## Comparison with Copilot POC (PR #2689)
+
+The current implementation evolved significantly from the initial Copilot-generated proof-of-concept in PR #2689:
+
+### Copilot POC (PR #2689) - Initial Spike
+**Status**: 53/88 tests passing (60%)
+
+**Build System**: `cargo build --release` (manual copy of .so file)
+- Used raw `cargo` commands
+- Manual file copying to project root
+- No wheel generation
+- Located in `rust/` directory
+
+**What it had:**
+- ✅ Basic BSON type support (int, float, string, bool, bytes, dict, list, null)
+- ✅ ObjectId, DateTime, Regex encoding/decoding
+- ✅ Binary, Code, Timestamp, Decimal128, MinKey, MaxKey support
+- ✅ DBRef and DBPointer decoding
+- ✅ Int64 type marker support
+- ✅ Basic CodecOptions (tz_aware, uuid_representation)
+- ✅ Buffer protocol support (memoryview, array)
+- ✅ _id field ordering at top level
+- ✅ Benchmark scripts and performance analysis
+- ✅ Comprehensive documentation (RUST_SPIKE_RESULTS.md)
+- ✅ **Same Rust architecture**: PyO3 0.27 + bson 2.13 crate (Python → Bson enum → bytes)
+
+**What it lacked:**
+- ❌ Only 60% test pass rate (53/88 tests)
+- ❌ Incomplete datetime handling (no DATETIME_CLAMP, DATETIME_AUTO, DATETIME_MS modes)
+- ❌ Missing unicode_decode_error_handler support
+- ❌ No document_class support from CodecOptions
+- ❌ No tzinfo conversion support
+- ❌ Missing BSON validation (size checks, null terminator)
+- ❌ No performance optimizations (type caching, fast paths)
+- ❌ Located in `rust/` directory instead of `bson/_rbson/`
+
+**Performance Claims**: 2.89x average speedup over C (from benchmarks in POC)
+
+**Why the POC appeared faster:**
+The Copilot POC's claimed 2.89x speedup was likely due to:
+1. **Limited test scope** - Benchmarks only tested simple documents that passed (53/88 tests)
+2. **Missing validation** - No BSON size checks, null terminator validation, or extra bytes detection
+3. **Incomplete CodecOptions** - Skipped expensive operations like:
+   - Timezone conversions (`tzinfo` with `astimezone()`)
+   - DateTime mode handling (CLAMP, AUTO, MS)
+   - Unicode error handler fallbacks to Python
+   - Custom document_class instantiation
+4. **Optimistic measurements** - May have measured only the fast path without edge cases
+5. **Different test methodology** - POC used custom benchmarks vs production testing with full PyMongo test suite
+
+When these missing features were added to achieve 100% compatibility, the true performance cost of the Rust `Bson` enum architecture became apparent.
+
+### Current Implementation (PR #2695) - Production-Ready
+**Status**: 88/88 tests passing (100%)
+
+**Build System**: `maturin build --release` (proper wheel generation)
+- Uses Maturin for proper Python packaging
+- Generates wheels with correct metadata
+- Extracts .so file to `bson/` directory
+- Located in `bson/_rbson/` directory (proper module structure)
+
+**Improvements over Copilot POC:**
+- ✅ **100% test compatibility** (88/88 vs 53/88)
+- ✅ **Complete CodecOptions support**:
+  - `document_class` - Custom document classes
+  - `tzinfo` - Timezone conversion with astimezone()
+  - `datetime_conversion` - All modes (AUTO, CLAMP, MS)
+  - `unicode_decode_error_handler` - Fallback to Python for non-strict handlers
+- ✅ **BSON validation** (size checks, null terminator, extra bytes detection)
+- ✅ **Performance optimizations**:
+  - Type caching (UUID, datetime, Pattern, etc.)
+  - Fast paths for common types (int, str, bool, None)
+  - Direct byte operations where possible
+  - PyDict fast path with pre-allocation
+- ✅ **Production-ready error handling** (matches C extension error messages exactly)
+- ✅ **Proper module structure** (`bson/_rbson/` with build.sh and maturin)
+- ✅ **Runtime selection** via PYMONGO_USE_RUST environment variable
+- ✅ **Comprehensive testing** (cross-compatibility tests, performance benchmarks)
+- ✅ **Same Rust architecture**: PyO3 0.23 + bson 2.13 crate (Python → Bson enum → bytes)
+
+**Performance Reality**: ~0.21x (5x slower than C) - see Performance Analysis section
+
+**Key Insights**:
+1. **Same Architecture, Different Results**: Both implementations use the same Rust architecture (PyO3 + bson crate with intermediate `Bson` enum), so the build system (cargo vs maturin) is not the cause of the performance difference.
+2. **Incomplete vs Complete**: The POC's speed claims were based on incomplete functionality (60% test pass rate). Achieving 100% compatibility revealed the true performance cost of:
+   - Complete CodecOptions handling (timezone conversions, datetime modes, etc.)
+   - BSON validation (size checks, null terminators, extra bytes)
+   - Production-ready error handling
+   - Edge case handling for all 88 tests
+3. **The Fundamental Issue**: Both implementations suffer from the same architectural limitation (Python → Bson enum → bytes), but it only becomes a significant bottleneck when you implement all the features required for production use.
+
+## Steps to Achieve Performance Parity with C Extensions
+
+Based on the analysis in PR #2695, here are the steps needed to match C extension performance:
+
+### 1. Eliminate Intermediate Bson Enum (High Impact)
+**Current**: Python → Bson → bytes
+**Target**: Python → bytes (direct)
+
+Implement direct BSON byte writing from Python objects without converting to `Bson` enum first. This would require:
+- Custom serialization logic for each Python type
+- Manual BSON format handling (type bytes, length prefixes, etc.)
+- Bypassing the `bson` crate's serialization layer
+
+**Estimated Impact**: 3-4x performance improvement
+
+### 2. Optimize Python API Calls (Medium Impact)
+- Reduce `getattr()` calls by caching attribute lookups
+- Use `PyDict_GetItem` instead of `dict.get_item()`
+- Minimize Python exception handling overhead
+- Use `PyTuple_GET_ITEM` for tuple access
+
+**Estimated Impact**: 1.2-1.5x performance improvement
+
+### 3. Memory Allocation Optimization (Low-Medium Impact)
+- Pre-allocate buffers based on estimated document size
+- Reuse buffers across multiple encode operations
+- Use arena allocation for temporary objects
+
+**Estimated Impact**: 1.1-1.3x performance improvement
+
+### 4. SIMD Optimizations (Low Impact)
+- Use SIMD for byte copying operations
+- Vectorize validation checks
+- Optimize string encoding/decoding
+
+**Estimated Impact**: 1.05-1.1x performance improvement
+
+### Combined Potential
+If all optimizations are implemented successfully:
+- Current: 0.21x (5x slower)
+- Target: 0.21x × 3.5 × 1.3 × 1.2 × 1.05 = **~1.13x** (13% faster than C)
+
+However, achieving this would require:
+- Significant engineering effort (weeks to months)
+- Bypassing the `bson` crate (losing its benefits)
+- Complex low-level code (harder to maintain)
+
+## Building
+
+```bash
+cd bson/_rbson
+./build.sh
+```
+
+Or using maturin directly:
+```bash
+maturin develop --release
+```
+
+## Testing
+
+Run the test suite with the Rust extension:
+```bash
+PYMONGO_USE_RUST=1 python -m pytest test/
+```
+
+Run performance benchmarks:
+```bash
+python test/performance/perf_test.py
+```
+
+## Conclusion
+
+The Rust extension demonstrates that:
+1. ✅ **Rust can provide a complete, production-ready BSON implementation**
+2. ✅ **100% compatibility with existing tests and APIs is achievable**
+3. ❌ **Performance parity with C requires bypassing the `bson` crate**
+4. ❌ **The engineering effort may not justify the benefits**
+
+### Recommendation
+
+The Rust extension is **production-ready** from a correctness standpoint but **not recommended** for performance-critical applications. The C extension remains the better choice for performance.
+
+**Use Cases for Rust Extension:**
+- Platforms where C compilation is difficult (e.g., WebAssembly)
+- Development environments without C toolchain
+- Testing and validation purposes
+- Future exploration if `bson` crate performance improves
+
+For more details, see:
+- [PYTHON-5683 JIRA ticket](https://jira.mongodb.org/browse/PYTHON-5683)
+- [PR #2695](https://github.com/mongodb/mongo-python-driver/pull/2695)