Skip to content

Commit bf3f81a

Browse files
committed
add sparse method patches for vllm 0.11.0
1 parent 6f90147 commit bf3f81a

2 files changed

Lines changed: 866 additions & 1 deletion

File tree

examples/offline_inference_kvcomphbm.py renamed to examples/offline_inference_gsa_on_device.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ def build_llm_with_uc(module_path: str, name: str, model: str):
7777
},
7878
}
7979
],
80-
"ucm_sparse_config": {"KvCompOnDevice": {}},
80+
"ucm_sparse_config": {"GSAOnDevice": {}},
8181
},
8282
)
8383

0 commit comments

Comments
 (0)