Skip to content

Commit 1965d5f

Browse files
committed
refactor(mtmd): move multimodal handlers to separate module llama_multimodal
- Move MTMDChatHandler, GenericMTMDChatHandler, and model-specific multimodal chat handlers out of llama_chat_format.py into llama_multimodal.py. - llama_chat_format.py has grown too large and difficult to maintain, especially as MTMD support expands beyond image-only use cases. Splitting multimodal handling into its own module makes the chat formatting layer smaller and keeps media loading, MTMD tokenization, multimodal KV-cache bookkeeping, and handler implementations in a dedicated place. - This also prepares the codebase for broader multimodal support and future video frame / image batch evaluation, where the media-processing path will need to evolve independently from text-only chat formatting. - Keep backward-compatible re-exports from llama_chat_format.py so existing imports continue to work. - Also keep `clip_model_path` as a deprecated initialization alias for `mmproj_path` in the base MTMD handler. Signed-off-by: JamePeng <jame_peng@sina.com>
1 parent 29b3f3a commit 1965d5f

3 files changed

Lines changed: 3690 additions & 3602 deletions

File tree

llama_cpp/llama.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
from .llama_tokenizer import BaseLlamaTokenizer, LlamaTokenizer
4646
import llama_cpp.llama_cpp as llama_cpp_lib
4747
import llama_cpp.llama_chat_format as llama_chat_format
48+
import llama_cpp.llama_multimodal as llama_multimodal
4849

4950
from llama_cpp.llama_speculative import LlamaDraftModel
5051

@@ -711,20 +712,19 @@ def __init__(
711712
self.metadata = {}
712713
if self.verbose:
713714
print(f"Failed to load metadata: {e}", file=sys.stderr)
714-
715-
if self.verbose:
716-
print(f"Model metadata: {self.metadata}", file=sys.stderr)
717715

718716
if mmproj_path is not None:
719717
if self.chat_handler is not None and self.verbose:
720718
print("Warning: Both `chat_handler` and `mmproj_path` are not null. Chat handler will be overwritten.", flush = True)
721719

722-
self.chat_handler = llama_chat_format.GenericMTMDChatHandler(
720+
self.chat_handler = llama_multimodal.GenericMTMDChatHandler(
723721
chat_format = self.metadata.get("tokenizer.chat_template", None),
724722
mmproj_path = mmproj_path,
725723
verbose = self.verbose,
726724
**chat_handler_kwargs
727725
)
726+
727+
if self.verbose:
728728
print(f"Model desc: {self.model_desc}, "
729729
f"Model size: {self.model_size / (1024 * 1024):.2f} MB, "
730730
f"Model metadata: {self.metadata}",

0 commit comments

Comments
 (0)