Skip to content

Commit c83747b

Browse files
kmbandyclaude
andcommitted
kv-cache-tiered: implement real per-layer K/V tensor data movement
Replace placeholder warm tier (flat single-buffer) with per-layer TieredKVLayer structs. Add public get_layer_k_raw/v_raw/get_num_kv_layers/is_v_transposed accessors to llama_kv_cache. Wire set_kv_layers_from_cache() from server init_slot so tensor pointers are populated at startup. Hot→warm migration now: - Copies K contiguously via hipMemcpy (VRAM→RAM or VRAM→eGPU D2D) - Gathers transposed V column at pos via hipMemcpy2D (VRAM→RAM) or element loop - Warm→hot restores with scatter for v_trans=true (hipMemcpy2D) Metadata-only fallback retained when set_kv_layers_from_cache() not called. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent a9280d1 commit c83747b

6 files changed

Lines changed: 395 additions & 189 deletions

File tree

0 commit comments

Comments
 (0)