Commit c83747b
kv-cache-tiered: implement real per-layer K/V tensor data movement
Replace placeholder warm tier (flat single-buffer) with per-layer TieredKVLayer
structs. Add public get_layer_k_raw/v_raw/get_num_kv_layers/is_v_transposed
accessors to llama_kv_cache. Wire set_kv_layers_from_cache() from server init_slot
so tensor pointers are populated at startup.
Hot→warm migration now:
- Copies K contiguously via hipMemcpy (VRAM→RAM or VRAM→eGPU D2D)
- Gathers transposed V column at pos via hipMemcpy2D (VRAM→RAM) or element loop
- Warm→hot restores with scatter for v_trans=true (hipMemcpy2D)
Metadata-only fallback retained when set_kv_layers_from_cache() not called.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>1 parent a9280d1 commit c83747b
6 files changed
Lines changed: 395 additions & 189 deletions
0 commit comments