Skip to content

[WIP] feat: Add storage checkpoint index for KV offloads#668

Draft
albertoperdomo2 wants to merge 7 commits into
llm-d:mainfrom
albertoperdomo2:feat/storage-kv-cache-index
Draft

[WIP] feat: Add storage checkpoint index for KV offloads#668
albertoperdomo2 wants to merge 7 commits into
llm-d:mainfrom
albertoperdomo2:feat/storage-kv-cache-index

Conversation

@albertoperdomo2

Copy link
Copy Markdown
Contributor

Summary

Adds a write-path-only storage checkpoint index so the Go indexer can track which KV blocks exist on shared storage (FS backend). This is the foundation for storage-aware scoring, pre-fetch signals... and any work related to it.

Python side:

  • StorageEventPublisher now requires a block_size parameter, populated from hash_block_size * gpu_blocks_per_file in the FS backend manager. This value is emitted in field [4] of BlockStored events so the Go side can derive stride at runtime without duplicated configuration.
  • PVC evictor passes block_size=0 explicitly (it only emits removal events).

Go side:

  • New StorageIndex interface (AddCheckpoint, HasCheckpoint, RemoveCheckpoint, Clear, SetStride, Stride) backed by a Cuckoo filter in pkg/kvcache/kvblock/storage_index.go.
  • Indexer creates and exposes the StorageIndex when StorageIndexConfig.Enabled is true.
  • Pool.processEventBatch() detects storage-tier events via isStorageTier() and routes them to the storage index, bypassing the normal engine-key-to-request-key resolution path. Storage events carry canonical request-key checkpoints directly.
  • Stride is learned from the first BlockStored event: event.BlockSize / tokenProcessor.BlockSize(). Thread-safe initialization via storageStrideMu mutex with consistency validation on subsequent events.
  • Storage indexing is opt-in: when storageIndex is nil, storage events are silently dropped and never fall through into GPU/CPU processing.
  • SHARED_STORAGE pod identifier is excluded from scoring to prevent it from appearing as a routable pod.

Testing

Unit tests covered, now benchmarking for potential regressions (WIP).

Related Issues

Closes #520

@github-actions github-actions Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 15, 2026
@albertoperdomo2
albertoperdomo2 force-pushed the feat/storage-kv-cache-index branch from 6b934e1 to 404fbea Compare June 15, 2026 13:23
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
@albertoperdomo2
albertoperdomo2 force-pushed the feat/storage-kv-cache-index branch from 25edee1 to c223ae8 Compare June 28, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add storage KV-cache indexing support

1 participant