Skip to content

feat: Support sarashina2.2-vision-3b model#22103

Merged
ngxson merged 1 commit intoggml-org:masterfrom
samuraieng:samuraieng/sarashina22vl-00
May 7, 2026
Merged

feat: Support sarashina2.2-vision-3b model#22103
ngxson merged 1 commit intoggml-org:masterfrom
samuraieng:samuraieng/sarashina22vl-00

Conversation

@samuraieng
Copy link
Copy Markdown
Contributor

Overview

This PR adds new PROJECTOR_TYPE to support for Sarashina2.2 Vision 3B model to llama.cpp, both llama-cli, llama-server and connvert_hf_to_gguf.py.
This implementation is updated to support the Sarashina model with minimum modifications.

Additional information

Changes

This PR updates six (6) files

  • convert_hf_to_gguf.py
    • Add Sarashina2VisionForCausalLM
    • Add some logic to proceed Sarashina2.2-vision-3B model
  • gguf-py/gguf/constants.py
    • Add SARASHINA2VL
  • tools/mtmd/clip-impl.h
  • tools/mtmd/clip.cpp
  • tools/mtmd/mtmd.cpp
    • Add PROJECTOR_TYPE_SARASHINA2VL_MERGER
    • Modify switch-case to support the type
  • tools/mtmd/tests.sh
    • Add sarashina2.2-vision-3b:Q4_K_M as add_test_vision

Testing

  • ctest -L main passed
  • tools/mtmd/tests.sh passed (with this modification)
  • Manual test with using llama-cli and llama-server with following models
    • Sarashina2.2-vision-3b-gguf (Target to support)
    • Qwen3-VL-2B-Instruct-gguf (For regression testing)
    • Qwen2.5-VL-3B-Instruct (Locally converted to gguf, to check convert_hf_to_gguf.py)

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - AI assisted with codeing. All code was human-authored, manually modified, reviewed and tested.

@samuraieng samuraieng requested review from a team and CISC as code owners April 19, 2026 02:24
@github-actions github-actions Bot added examples python python script changes labels Apr 19, 2026
Comment thread convert_hf_to_gguf.py Outdated
Comment thread gguf-py/gguf/constants.py Outdated
Comment thread tools/mtmd/clip-impl.h Outdated
Comment thread convert_hf_to_gguf.py Outdated
Comment thread tools/mtmd/tests.sh Outdated
@CISC
Copy link
Copy Markdown
Member

CISC commented Apr 21, 2026

@samuraieng Bad merge.

@CISC CISC removed request for a team and ggerganov April 21, 2026 11:08
@github-actions github-actions Bot added devops improvements to build systems and github actions server ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language Apple Metal https://en.wikipedia.org/wiki/Metal_(API) Hexagon OpenVINO WebGPU labels Apr 22, 2026
@samuraieng
Copy link
Copy Markdown
Contributor Author

Would you please review again? I think I captured all your comments in the latest update.

@CISC
Copy link
Copy Markdown
Member

CISC commented Apr 23, 2026

@samuraieng Bad merge.

@CISC CISC removed request for a team April 23, 2026 07:09
@CISC CISC removed documentation Improvements or additions to documentation model Model specific script Script related testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs Vulkan Issues specific to the Vulkan backend labels Apr 23, 2026
@samuraieng
Copy link
Copy Markdown
Contributor Author

Sorry for made this bad merge. I pushed again with new code. Would you please reviwe again?
As I wrote, only convert_hf_to_gguf.py is the file to review. Most of modification are made inside new classes.

Comment thread convert_hf_to_gguf.py Outdated
Comment thread convert_hf_to_gguf.py Outdated
Comment thread convert_hf_to_gguf.py Outdated
Comment thread convert_hf_to_gguf.py Outdated
Comment thread convert_hf_to_gguf.py Outdated
@samuraieng
Copy link
Copy Markdown
Contributor Author

Mostly, MMPROJ side was updated. The new class follows Qwen2VLVisionModel

Comment thread convert_hf_to_gguf.py Outdated
Comment thread convert_hf_to_gguf.py Outdated
@samuraieng
Copy link
Copy Markdown
Contributor Author

@ngxson , would you please review the latest code?

@masabamba
Copy link
Copy Markdown

@ngxson , would you please review the latest code? The code I modified are same as before. Sync with the master branch.

Comment thread convert_hf_to_gguf.py Outdated
@masabamba
Copy link
Copy Markdown

I updated to follow this review comment. Would you please review again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python python script changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants