Overview

Adds native support for https://huggingface.co/collections/google/paligemma2-release-67500e1e1dbfdd4dee27ba48
(SigLIP vision encoder + Gemma2 backbone) to llama.cpp / libmtmd.

PaliGemma2 is Google's vision-language model family (3B/10B/28B, 224px/448px) designed for OCR, VQA, image
captioning, object detection, and document understanding.

Architecture

PaliGemma2 is simpler than Gemma3 vision:

• Vision encoder: SigLIP (already in llama.cpp via siglip.cpp)
• Projector: single nn.Linear(in=1152, out=hidden_size) with bias — no pooling, no RMS norm
• LM backbone: Gemma2 (already in llama.cpp)
• Attention: image prefix uses bidirectional attention (same path as Gemma3, via
  llama_set_causal_attn(false))

Supported variants:

┌────────────────────────────────────────────────┬────────────┬──────────────┐
│ Variant │ Resolution │ Image tokens │
├────────────────────────────────────────────────┼────────────┼──────────────┤
│ paligemma2-3b-pt-224, paligemma2-3b-mix-224, … │ 224px │ 256 (16×16) │
├────────────────────────────────────────────────┼────────────┼──────────────┤
│ paligemma2-3b-pt-448, paligemma2-10b-pt-448, … │ 448px │ 1024 (32×32) │
└────────────────────────────────────────────────┴────────────┴──────────────┘

Changes

GGUF conversion (convert_hf_to_gguf.py)

• Add PaliGemma2VisionModel (MmprojModel) for PaliGemmaForConditionalGeneration
• Extend Gemma2Model.modify_tensors to strip language_model.* prefix from PaliGemma2 checkpoints
• Extend Gemma2Model.set_vocab to fall back to _set_vocab_gpt2 when tokenizer.model is absent

GGUF types (gguf-py/)

• Add PALIGEMMA2 = "paligemma2" to VisionProjectorType
• Add multi_modal_projector.linear to V_MM_INP_PROJ tensor mapping

libmtmd (tools/mtmd/)

• clip-impl.h: PROJECTOR_TYPE_PALIGEMMA2 enum + TN_MM_INP_PROJ_B constant
• clip-model.h: mm_input_proj_b field (projector has bias)
• clip.cpp: load_hparams (n_merge=1, bilinear), n_patches (no pooling), n_mmproj_embd, set_input, weight
  loading
• siglip.cpp: PROJECTOR_TYPE_PALIGEMMA2 projector branch (direct linear + bias, no pool/norm)
• mtmd.cpp: init_vision (fixed-size preprocessor), mtmd_decode_use_non_causal (bidirectional)
• mtmd-cli.cpp: warn instead of exit(1) when no chat template; use raw prompt mode for PT/mix models

Usage

Step 1: Convert (requires tokenizer.model from gemma-2-2b if not present)

python convert_hf_to_gguf.py google/paligemma2-3b-mix-224 \
--mmproj --outfile pali2-mmproj.gguf
python convert_hf_to_gguf.py google/paligemma2-3b-mix-224 \
--outtype bf16 --outfile pali2-text.gguf

Step 2: Run inference (prefix completion format, no chat template)

./llama-mtmd-cli -m pali2-text.gguf --mmproj pali2-mmproj.gguf
--image photo.jpg -p $'\nanswer en What is in this image? '

Caption

./llama-mtmd-cli -m pali2-text.gguf --mmproj pali2-mmproj.gguf
--image photo.jpg -p $'\ncaption en '

Notes

• PaliGemma2 uses prefix completion format (not chat/instruction format). The --chat-template flag is not
  required; the CLI now warns instead of exiting when no template is found.
• tokenizer.model is not included in PaliGemma2's HF repo; use one from google/gemma-2-2b (same vocabulary).
• This PR closes Add PaliGemma Support #7553 which attempted PaliGemma support using the pre-libmtmd architecture.

Testing

Verified with paligemma2-3b-pt-224 and paligemma2-3b-mix-224 on CPU (GCP e2, no GPU):

• GGUF conversion: mmproj 833MB, text 5.2GB ✓
• SigLIP shape: [1, 256, 1152], matches HuggingFace transformers output ✓
• Projector shape: [256, 2304], numerically verified against HF weights ✓
• 448px: GGUF position_embd.weight [1152, 1024] (1024 tokens) ✓
• paligemma2-3b-mix-224 correctly identifies colors: red→"red", blue→"blue, 0" ✓
• Full pipeline: EXIT 0, image encoded in ~8s (CPU), 256 tokens decoded ✓

cc @ngxson (SigLIP/Gemma3 author)