Skip to content

Commit e6ca062

Browse files
authored
[Fix] Update KVCacheLayout to set row counts. (#920)
## Purpose Fix the `IndexError: list index out of range` issue when running `GLM-4.7-W8A8-floatmtp`. ## Modification Use the number of unique layer IDs, `len(set(self.layer_name_to_id.values()))`, as the row count in KV cache layout construction instead of `self.local_num_hidden_layers`.
1 parent 42afd8b commit e6ca062

1 file changed

Lines changed: 4 additions & 2 deletions

File tree

ucm/integration/vllm/ucm_connector.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,8 +88,10 @@ def __init__(
8888
self._build_layout(kvcaches)
8989

9090
def _build_layout(self, kvcaches):
91-
raw_ptr_rows = [[] for _ in range(self.local_num_hidden_layers)]
92-
stride_rows = [[] for _ in range(self.local_num_hidden_layers)]
91+
92+
num_rows = len(set(self.layer_name_to_id.values()))
93+
raw_ptr_rows = [[] for _ in range(num_rows)]
94+
stride_rows = [[] for _ in range(num_rows)]
9395

9496
for layer_name, kv_layer in kvcaches.items():
9597
ptrs = []

0 commit comments

Comments
 (0)