docs: add comprehensive Metrics package section to API_REFERENCE.md

Added extensive documentation for pkg/metrics package (721 lines):

Core Types:
- Metrics - Internal metrics collector (not exported)
- Stats - Performance statistics snapshot with 16 fields

Stats Fields (16 total):
- Basic counts: TokenizeOperations, TokenizeErrors, ErrorRate
- Performance: AverageDuration, OperationsPerSecond
- Pool metrics: PoolGets, PoolPuts, PoolBalance, PoolMissRate
- Query size: MinQuerySize, MaxQuerySize, AverageQuerySize, TotalBytesProcessed
- Timing: Uptime, LastOperationTime
- Errors: ErrorsByType map

Configuration Functions:
- Enable() - Activate metrics collection
- Disable() - Deactivate metrics collection
- IsEnabled() - Check if collection is active

Recording Functions (automatic):
- RecordTokenization() - Record tokenization operation
- RecordPoolGet() - Record pool retrieval
- RecordPoolPut() - Record pool return

Query Functions:
- GetStats() - Get current performance statistics
- LogStats() - Alias for GetStats (logging convenience)
- Reset() - Clear all metrics (testing)

Usage Examples:
- Basic metrics collection
- Production monitoring with periodic reporting
- Error tracking and analysis
- Pool efficiency monitoring
- Query size analysis
- JSON export for APIs
- HTTP metrics endpoint
- Prometheus integration
- Performance alerting with SLOs

Integration Patterns:
- Pattern 1: Application startup (enable early, disable late)
- Pattern 2: Periodic reporting (ticker-based)
- Pattern 3: Testing with metrics (reset before test)

Performance Characteristics:
- Thread Safety: Lock-free atomic operations, RWMutex for error map
- Memory Overhead: ~200 bytes + error map (fixed footprint)
- Performance Impact: ~50ns enabled, ~1ns disabled, O(n) GetStats

Best Practices:
- Enable at application startup (not per-operation)
- Use periodic reporting (1min intervals)
- Monitor pool efficiency (>95% hit rate target)
- Set performance SLOs (error rate, throughput, latency, pool efficiency)

Production Monitoring:
- HTTP /metrics endpoint
- Prometheus integration
- Alert on: high error rate (>1%), slow duration (>1ms), low pool hit rate (<90%), low throughput (<1k ops/sec)
- Metrics dashboard example with formatted output

JSON Export Support:
- All Stats fields have json tags
- Direct marshaling to JSON
- Ready for monitoring systems

Completes API_REFERENCE.md expansion with:
- High-Level API (338 lines)
- Keywords Package (631 lines)
- Errors Package (670 lines)
- Metrics Package (721 lines)

Total new documentation: 2,360 lines across 4 major sections

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: 2 people