Skip to content

[OpenVINO] Support allenai/MolmoWeb-4B (Molmo2)#1812

Open
openvino-agent wants to merge 1 commit into
huggingface:mainfrom
openvino-agent:molmoweb_support
Open

[OpenVINO] Support allenai/MolmoWeb-4B (Molmo2)#1812
openvino-agent wants to merge 1 commit into
huggingface:mainfrom
openvino-agent:molmoweb_support

Conversation

@openvino-agent

Copy link
Copy Markdown
Contributor

What does this PR do?

Installation instructions:

pip install optimum-intel@
pip install transformers==4.57.3

Cmd-line for export:

optimum-cli export openvino -m allenai/MolmoWeb-4B MolmoWeb-4B --trust-remote-code

The script for inference:

from transformers import AutoProcessor
from optimum.intel.openvino import OVModelForVisualCausalLM

model_id = "./MolmoWeb-4B"
#model_id = "allenai/MolmoWeb-4B"
model = OVModelForVisualCausalLM.from_pretrained(model_id, trust_remote_code=True)

processor = AutoProcessor.from_pretrained(model_id, padding_side="left", trust_remote_code=True)

url = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4="
messages = [
    {
        "role": "user", "content": [
            {"type": "image", "url": url},
            {"type": "text", "text": "What is shown in this image?"},
        ]
    },
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
)

output = model.generate(**inputs, max_new_tokens=50)
print(processor.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Add OpenVINO export and inference support for the Molmo2 vision-language
model architecture (allenai/MolmoWeb-4B). This is a trust_remote_code model
with a unique vision pipeline: ViT encoder + 2D pooling attention + MLP
projector, with features merged into text embeddings at image_patch_id positions.

- Add Molmo2OpenVINOConfig export configuration with custom behaviors
- Add DummyMolmo2VisionInputGenerator for 3D pixel_values + pooling indices
- Add Molmo2ImageEmbeddingModelPatcher for vision backbone export
- Add _OVMolmo2ForCausalLM runtime class with vision/text merge logic
- Register molmo2 in MULTI_MODAL_TEXT_GENERATION_MODELS
- Fix AutoModelForImageTextToText loading for remote code models
- Add tests and update supported models documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openvino-agent openvino-agent changed the title [OpenVINO] Add support for allenai/MolmoWeb-4B (Molmo2) [OpenVINO] Support allenai/MolmoWeb-4B (Molmo2) Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants