[docs] cb memory management (#45587)

* docs

* feedback

* update

Author: stevhliu

docs/source/en/continuous_batching_architecture.md

@@ -81,6 +81,59 @@ Set the scheduler in [`ContinuousBatchingConfig`].
 cb_config = ContinuousBatchingConfig(scheduler_type="prefill_first")
 ```
 
+## Memory management
+
+The KV cache is paged so requests of different lengths can share a fixed memory pool without fragmentation. The cache divides memory into fixed-size blocks, and each request holds a list of block IDs that the attention kernel reads from and writes to.
+
+```text
+Paged KV cache (num_blocks × block_size tokens per block)
+
+    0   1   2   3   4   5   6   7   8   9   …
+  ┌───┬───┬───┬───┬───┬───┬───┬───┬───┬───┐
+  │ B │ · │ A │ B │ · │ A │ · │ A │ · │ · │
+  └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘
+  A  request A: 2, 5, 7   B  request B: 0, 3   ·  free: 1, 4, 6, 8, 9, …
+```
+
+A page holds the key/value state for one token in one layer. A block is a span of `block_size` pages (default 256) and is the unit of allocation. Blocks are allocated per layer group. Layers within a group share a block ID, which keeps bookkeeping uniform for mixed-attention models.
+
+### Cache sizing
+
+The manager infers the number of blocks at startup from free GPU memory. The manager solves an equation that accounts for KV tensors, attention masks, activations, and bookkeeping indices, then sizes the pool to fit inside `max_memory_percent` (default 0.9) of the available memory.
+
+You can pin the values explicitly in [`ContinuousBatchingConfig`].
+
+```py
+cb_config = ContinuousBatchingConfig(
+    block_size=256,
+    num_blocks=4096,
+    max_batch_tokens=512,
+    max_memory_percent=0.8,
+)
+```
+
+### Admission
+
+Before a request joins the batch, the scheduler checks that enough free blocks exist for every layer group. If any group would fall short, the request is rejected and nothing is allocated.
+
+To avoid filling the cache until [offloading](#offloading) is the only option, the default FIFO scheduler enforces a safety margin. Once free blocks fall below 20% of the total, new prefills are held back and only active decodes continue.
+
+### Prefix caching
+
+When two requests share a prompt prefix, they can share the blocks that hold the KV for that prefix. Each completed block is content-hashed. A later request with a matching prefix reuses the block and skips the prefill for those tokens. Shared blocks are reference-counted and only return to the free pool once every request using them has finished.
+
+Prefix caching is enabled by default and is active only when a model has exclusively full-attention layers. Set `allow_block_sharing=False` in [`ContinuousBatchingConfig`] for workloads with short prompts and long generations, where the bookkeeping outweighs the savings.
+
+### Eviction
+
+Freed blocks aren't wiped and they stay "initialized" with their content and hash intact so a later prefix match can reuse them. When the uninitialized pool runs low, the manager lazily demotes the most recent initialized block back to uninitialized, dropping their hash. The prefix cache is a best-effort layer that the allocator discards to keep serving new requests.
+
+### Soft reset
+
+If the KV cache fills completely during a long session, because requests are generating very long outputs, the manager triggers offloading rather than crashing. It selects the oldest active request (or the newest, if a previous soft reset already blocked new requests from joining), appends its generated tokens to its original prompt, frees its cache blocks, and adds it back to the queue.
+
+When the cache has space again, the request resumes from where it left off with its generation history encoded in the prompt.
+
 ## CUDA graphs
 
 Every generation step involves CPU overhead, from assembling the batch to dispatching GPU kernels and reading the results. CUDA graphs eliminate the overhead by recording the full GPU execution sequence once and replaying it for batches with matching shapes.