update readme

Author: LoneWandererProductions

README.md

@@ -1,177 +1,82 @@
 # MemoryPrototype
 
-**MemoryPrototype** is a prototype memory allocation and handle system built to experiment with custom memory management in C#. Inspired by techniques found in game engines and real-time systems, it explores handle indirection, dual-tier memory lanes, and optional compaction.
+**MemoryPrototype** is a high-performance, dual-tier unmanaged memory arena and handle system built to bypass the .NET Garbage Collector for performance-critical subsystems. Inspired by data-oriented memory management architecture in low-latency runtime engines, it implements stable handle indirection, generational version validation, automated lifecycle eviction policies, and low-level compacting memory lanes.
 
-> ⚠️ **Note:** This is a **prototype for learning and experimentation**. Use at your own risk.
-> Validated via high-stress memory pressure tests and performance benchmarks against the .NET Garbage Collector.
+> ⚠️ **Engineering Status:** This is an **experimental sandbox prototype for learning and systems engineering**. While optimized to achieve exceptional throughput and zero GC allocation on hot paths, it is currently undergoing architectural hardening. See the [Production Finish-Line Roadmap](#-the-production-finish-line-roadmap) below for critical hurdles to address before running this engine in production environments.
 
 ---
 
-### 🚀 The Killer Features
+### 🚀 Key Features
 
 1. **Stable Handle Indirection (Relocatable Memory)**
-   Instead of handing you a raw pointer that permanently pins memory, MemoryLane gives you an O(1) `MemoryHandle`. This means the engine can physically move your data in the background to defragment the heap, and your references will never break.
-2. **The "Janitor" (Automated Lifecycle Management)**
-   You don't have to decide where data lives. Hot, short-lived data stays in the zero-allocation `FastLane`. If data survives too long or goes "Cold," the automated Janitor seamlessly migrates it to the persistent `SlowLane`, leaving a lightweight redirection stub behind.
-3. **Live Compaction (Zero External Fragmentation)**
-   Standard freelist allocators suffer from "Swiss Cheese" memory over time, leaving gaps you can't use for large objects. MemoryLane features Live Compaction, physically sliding memory blocks together to eliminate gaps and reclaim 100% of your wasted space without pausing the application.
-4. **Pluggable Allocation Strategies**
-   Not all workloads are the same. MemoryLane allows you to hot-swap the internal allocation engine of the `FastLane` via a simple config toggle. Choose between a safe, hole-reusing **Free-List Allocator** or a lightning-fast, O(1) **Linear Bump Allocator**.
+   Instead of handing out raw pointers that permanently pin your layout blocks in memory, `MemoryArena` yields opaque, lightweight $O(1)$ `MemoryHandle` tokens. This layer of indirection permits the background engine to safely move your data to defragment the unmanaged heap without ever breaking user references.
 
----
-
-## ⚠️ Known Limitations
-
-- Does not manage C# objects directly — only unmanaged memory blocks.
-- No garbage collection integration.
-- No automatic bounds checking — `MemoryHandle` is mostly safe *if* used correctly via the API.
-- Designed for low-level experimentation, not high-level safety.
-
----
-
-## ✨ Features
-
-- 🧠 **Dual Memory Lanes** - `FastLane`: low-latency, short-lived allocations (e.g., frame-local).  
-  - `SlowLane`: large, persistent allocations (e.g., background assets).
-
-- 🔁 **Stable Handle System** - `MemoryHandle` provides opaque, safe references to internal memory entries.  
-  - Redirection via lightweight stubs for safe relocation.
-
-- 🔌 **Modular Strategy Pattern** - Toggle the core allocation logic (`AllocatorStrategy.FreeList` vs `AllocatorStrategy.LinearBump`) to perfectly match your application's allocation tempo.
-
-- 🧹 **Optional Live Compaction** - Reduces fragmentation and reclaims space dynamically.
+2. **Generational Version Validation (Zombie Protection)**
+   To defeat the dangling pointer problem common in custom allocators, each `MemoryHandle` carries a tracking `Id` and a generation `Version`. When handles are recycled, their index version increments. Any legacy handle trying to access an old address will instantly trigger a safe access violation instead of silently corrupting recycled memory.
 
-- 🔄 **Stub-Based Indirection** - Moves memory without invalidating existing handles.
+3. **The "Janitor" (Automated Tiered Eviction)**
+   You do not need to micro-manage object lifespans. Hot, short-lived data is allocated inside a lean, high-speed `FastLane`. If an allocation outstays its welcome (based on frame age), grows too large, or is explicitly tagged with `AllocationHints.Cold`, the automated Janitor moves it to a persistent `SlowLane` during quiet frames, replacing the original site with a seamless redirection stub.
 
-- 🧪 **Safety Checks** - Includes `TryGet<T>`, `IsValid`, and `GetHandleState`.
+4. **Pluggable Fast-Lane Strategies**
+   Not all workloads share the same structural tempo. The arena config file lets you hot-swap the allocation engine backing the `FastLane`:
+   * **`AllocatorStrategy.FreeList`**: Tracks fragmented allocations with a block recycling manager; ideal for chaotic, highly variable lifespans.
+   * **`AllocatorStrategy.Linear`**: A blazing-fast, true $O(1)$ sequential bump allocator; ideal for transient per-frame scratch pads.
 
-- 🛤️ **One-Way Lane Transfer** - `OneWayLane` shifts data from `FastLane` to `SlowLane`.  
-  - Uses `Span<T>` and `Buffer.MemoryCopy` to avoid breaking references.  
-  - Can be plugged into compaction or run manually.
-
-- 📦 **High-Level Managed Types** - Includes ArenaList<T>, a resizable collection that lives entirely in unmanaged memory.
-  - Automatically handles Growth and Migration through the handle system.
-  - Integrates with the Janitor for automatic defragmentation when lists are resized or freed.
- 
-- 🔤 **Unmanaged String Support** - Store strings as UTF-8 byte arrays to save 50% memory over C# UTF-16 strings and bypass the GC.
-
-- ✅ **Robust Unit Tests** - Includes structural validation, performance benchmarks, and multi-lane stress testing.
+5. **Atomic Safe Copying**
+   Features a native, thread-safe `BulkSet<T>` utility that safely accepts `ReadOnlySpan<T>` payloads, guaranteeing that block data transfers, size validation, and pointer resolution occur in a single atomic phase safely isolated from compaction interrupts.
 
 ---
 
-## 🎯 Learning Objectives
+## 📐 Architecture Overview
 
-This project was created to:
+MemoryPrototype establishes a strict separation of concerns across multiple specialized unmanaged memory tiers.
 
-- Understand manual memory management strategies.
-- Explore handle indirection and pointer safety.
-- Learn arena-style memory management, paging, and pooling.
-- Gain insights into low-level system design using C#.
+* **⚡ FastLane**: Optimized for high-frequency, short-lived "hot" transient structures. Uses sequential **positive integers** (`1, 2, 3...`) for allocation IDs. Version tracking is backed by an ultra-fast, growable native pointer array (`uint*`) that maps handle IDs directly to generation counts via an $O(1)$ array offset calculation.
+* **🐌 SlowLane**: Optimized for large, persistent "cold" allocations or long-lived structural blocks. Uses sequential **negative integers** (`-1, -2, -3...`) for allocation IDs. Maps negative handles cleanly to positive array coordinates using an absolute value index converter (`-id`), retaining raw pointer lookup performance.
+* **🪐 BlobManager (Slow-Lane Sub-Allocator)**: Optimized for isolating tiny, chaotic allocations (e.g., $\le$ 256 bytes) away from the primary free block table to prevent memory fragmentation.
+* **走 OneWayLane (The Bridge)**: Orchestrates seamless data migration. It safely reads the original entry metadata to pull telemetry (preserving original frame age, priorities, and usage hints) and executes a vector-speed `System.Buffer.MemoryCopy` across the lane boundaries before converting the old site into a routing redirection stub.
 
 ---
 
-## ⚙️ Requirements
+## 🛑 The Production Finish-Line Roadmap
 
-- **.NET 9.0** (or compatible runtime)
+To move this system out of the prototyping phase and into a hardened, production-grade systems engine, the following structural limitations must be addressed:
 
----
+### 1. Multi-Threaded Cache-Line Contention (The Global Lock Bottleneck)
+* **Current State:** The system achieves thread safety by acquiring a heavy global `lock (_lock)` on the wrapper for nearly every execution entry point (`Allocate`, `Resolve`, `Free`, `BulkSet`). Under high thread contention, parallel worker threads will stall waiting for this single lock, destroying core scaling.
+* **Production Fix:** Transition to a **Thread-Local Arena** architecture. Each thread should allocate out of its own dedicated lock-free local scratch ring, only accessing the synchronized global arena when local pages are completely exhausted.
 
-## 🧭 Future Work / Ideas
+### 2. "Stop-the-World" Compaction Latency
+* **Current State:** When compaction triggers, the compactor engine allocates an entirely new unmanaged heap block via `Marshal.AllocHGlobal` and slides all survivors over via memory block copying. For massive heaps (e.g., 512MB+), this causes noticeable block stutters (latency spikes) and temporarily forces a **double memory footprint**.
+* **Production Fix:** Implement an **Incremental/Phased Compactor** that only relocates a small chunk of fragmented blocks per frame tick, or enforce strict zero-compaction rules where arenas are completely cleared and cycled out at deterministic boundaries (e.g., scene changes).
 
-These are conceptual features or areas for future exploration:
+### 3. Hardware Memory Alignment Blindness
+* **Current State:** The bump allocators currently increment layouts directly by byte size (`_nextFreeOffset += size`). If a user allocates an odd structural layout (e.g., a 7-byte struct), subsequent items are placed on unaligned memory addresses, causing the CPU to fetch multiple cache-lines for single read instructions, tanking cache line efficiency.
+* **Production Fix:** Force all lane offsets to snap cleanly to hardware alignment boundaries (e.g., 16-byte, 32-byte, or 64-byte boundaries for SIMD alignment) using bitwise masking masks:
+  $$\text{alignedOffset} = (\text{offset} + (\text{alignment} - 1)) \ \& \ \sim(\text{alignment} - 1)$$
 
-- [ ] **Thread Safety** — Replace global lock with `ReaderWriterLockSlim` for parallel Resolve operations.
-- [ ] **Handle ID Pooling** — Implement a stack-based pool for `MemoryHandle` IDs to ensure O(1) allocation without metadata overhead.
-- [ ] **SIMD Alignment** — Add `AlignTo(int boundary)` to the allocation logic for cache-line and SIMD-friendly memory offsets.
-- [ ] **Visual Profiler** — Export internal memory maps to a heatmap (JSON/HTML) for real-time fragmentation monitoring.
-- [ ] **Bidirectional Transfer** — (Advanced) Allow the Janitor to "pull" frequently accessed data back into the `FastLane`.
-- [ ] **The "Lounge" (Pool Allocator)** — A dedicated memory tier for uniform, homogeneous data blocks (e.g., arrays of identical structs). Eliminates fragmentation entirely using an O(1) index stack. Includes a strict Janitor policy to evict "liars" to the `SlowLane` if their temporary frame data overstays its welcome.
+### 4. Zero Defensive Buffer-Overrun Safety Guardrails
+* **Current State:** Resolving a handle exposes a raw native `nint` address space. If code reads or writes outside the structural bounds of that specific entry allocation, it will silently overwrite adjacent allocations or metadata arrays without an error, resulting in impossible-to-trace heap corruption.
+* **Production Fix:** In `#if DEBUG` configurations, implement **Canary Guard Bands**. Inject a known sequence of invariant diagnostic bytes (e.g., `0xDEADBEEF`) directly before and after every physical allocation chunk. During `Free()` or compaction loops, validate these bands to instantly catch memory overruns.
 
 ---
 
-## 📐 Planned Architecture Overview
-
-MemoryLane utilizes a dual-tier strategy to bypass the .NET Garbage Collector for high-frequency or large-scale unmanaged data. By using Handle Indirection, the system can physically move memory (defragmentation) without breaking user-held references.
-
-### 🔧 Tier Responsibilities
-
-#### ✅ FastLane
-**Optimized for:** High-frequency, short-lived "hot" data.  
-**Backend:** Pluggable Strategy (`LinearBump` for maximum O(1) speed, or `FreeList` for highly-variable, chaotic lifespans).  
-**Convention:** Uses Positive Allocation IDs.  
-**The Janitor:** Automatically promotes stale, oversized, or "Cold-tagged" entries to the SlowLane during maintenance cycles to keep the FastLane lean.
-
-#### ✅ SlowLane
-**Optimized for:** Large, persistent "cold" data or background assets.  
-**Backend:** Hybrid Segregated Allocator (Free-Block Manager for large assets, Bump Allocator for tiny blobs).  
-**Convention:** Uses Negative Allocation IDs.  
-**Maintenance:** Performs deep compaction when fragmentation exceeds configured thresholds.
-
-#### ✅ OneWayLane (The Bridge)
-**Responsibility:** Seamlessly migrates memory from FastLane to SlowLane.  
-**Mechanism:** Direct pointer-to-pointer `Buffer.MemoryCopy`.  
-**Stub System:** Replaces the FastLane entry with a Stub that redirects all future Resolve calls to the new SlowLane address.
-
----
+## 🧩 Config Presets & Advanced Usage
 
-## 🧩 Example Usage
+MemoryPrototype includes optimized configuration presets out of the box to instantly tune execution properties for specific workloads:
 
 ```csharp
 using MemoryManager;
-using MemoryManager.Types;
-
-// --- 1. Setup Configuration ---
-// Define your sandbox boundaries. 10MB SlowLane, 1MB FastLane.
-var config = new MemoryManagerConfig(slowLaneSize: 10 * 1024 * 1024) 
-{
-    EnableAutoCompaction = true,
-    FastLaneUsageThreshold = 0.90, // Trigger maintenance at 90% full
-    MaxFastLaneAgeFrames = 600,    // Janitor evicts "stale" data after 10 seconds (60fps)
-    FastLaneStrategy = AllocatorStrategy.LinearBump // O(1) lightning speed
-};
-
-var arena = new MemoryArena(config);
-
-// --- 2. High-Level Collections (The Lounge) ---
-// Use ArenaList for resizable, unmanaged collections.
-// It feels like a standard List<T>, but lives in your FastLane.
-var list = new ArenaList<int>(arena, initialCapacity: 16);
-for(int i = 0; i < 20; i++) list.Add(i); 
-
-// --- 3. Stable Handles & Syntactic Sugar ---
-// Store a single value. The handle remains valid even if the Janitor 
-// moves this integer to the SlowLane later!
-var healthHandle = arena.AllocateAndStore(100);
-
-// --- 4. Bulk Data & Memory "Slamming" ---
-int[] sourceData = { 10, 20, 30, 40, 50 };
-var arrayHandle = arena.AllocateArray<int>(sourceData.Length);
-
-// Vectorized copy from managed to unmanaged memory
-arena.BulkSet(arrayHandle, sourceData);
-
-// --- 5. Unmanaged Strings (Save 50% RAM) ---
-// Store text as UTF-8 in the Arena, bypassing C# UTF-16 overhead and the GC.
-var stringHandle = arena.AllocateString("Hello from the Arena!");
-
-// --- 6. Pointer-Speed Access via Spans ---
-// Get a Span for zero-allocation, high-speed iteration.
-// This is the "hot path" for game loops.
-var span = arena.GetSpan<int>(arrayHandle, sourceData.Length);
-foreach(ref var val in span) 
-{
-    val *= 2; // Direct memory manipulation at CPU cache speeds
-}
-
-// --- 7. Policy-Driven Maintenance ---
-// Call this once per frame. It tracks the age of allocations.
-arena.TickFrame(); 
-
-// Periodically runs the Janitor (eviction) and Compaction (defragmentation)
-// to ensure your FastLane never turns into "Swiss Cheese."
-arena.RunMaintenanceCycle(); 
-
-// --- 8. Manual Cleanup ---
-arena.Free(healthHandle);
-arena.Free(arrayHandle);
-// Note: ArenaList and String handles are cleaned up similarly.
+using MemoryManager.Core;
+
+// --- PROFILE 1: Ultra-Fast Real-Time Game Loop ---
+// Initializes a high-frequency layout budget utilizing the 
+// ultra-fast O(1) Linear Bump strategy with aggressive frame aging.
+var gameConfig = MemoryManagerConfig.CreateForGameLoop(totalBudget: 32 * 1024 * 1024);
+var gameArena = new MemoryArena(gameConfig);
+
+// --- PROFILE 2: Heavy I/O Stream / Asset Chunking ---
+// Instantiates a robust Free-List strategy engine capable of 
+// managing out-of-order data transfers with larger block thresholds.
+var streamConfig = MemoryManagerConfig.CreateForBulkProcessing(totalBudget: 128 * 1024 * 1024);
+var streamArena = new MemoryArena(streamConfig);