[OpenVINO] Support MiniCPM-V-4.6

[openvino-agent]

What does this PR do?

Installation instructions

pip install optimum-intel@
pip install transformers==5.7.0

Cmd-line for export:

optimum-cli export openvino -m openbmb/MiniCPM-V-4.6 MiniCPM-V-4.6

The script for inference:

from transformers import AutoProcessor
from optimum.intel.openvino import OVModelForVisualCausalLM

model_id = "./MiniCPM-V-4.6"
#model_id = "openbmb/MiniCPM-V-4.6"
model = OVModelForVisualCausalLM.from_pretrained(model_id)

processor = AutoProcessor.from_pretrained(model_id, padding_side="left")

url = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4="
messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are a helpful assistant."}
        ]
    },
    {
        "role": "user", "content": [
            {"type": "image", "url": url},
            {"type": "text", "text": "What is shown in this image?"},
        ]
    },
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
)

output = model.generate(**inputs, max_new_tokens=50)
print(processor.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[as-suvorov]

@rkazants @popovaan WWB gives 0.92 accuracy hf vs optimum.
Could you please take a look at reproducer:
optimum-cli export openvino --weight-format fp32 -m openbmb/MiniCPM-V-4.6 MiniCPM-V-4.6

from transformers import AutoProcessor, AutoModelForImageTextToText
from optimum.intel.openvino import OVModelForVisualCausalLM
from pathlib import Path

prompt = "List all the states marked in the purple color in the map."
image_url = "https://textvision-data-quality.s3.us-west-1.amazonaws.com/infographics_images/33051.jpeg"
model_id = "openbmb/MiniCPM-V-4.6"

processor = AutoProcessor.from_pretrained(model_id, padding_side="left")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": image_url},
            {"type": "text", "text": prompt},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
)


def run_optimum(inputs):
    model_path = Path("./MiniCPM-V-4.6")
    ov_config = {"INFERENCE_PRECISION_HINT": "f32", "KV_CACHE_PRECISION": "f32", "DYNAMIC_QUANTIZATION_GROUP_SIZE": 0}
    model = OVModelForVisualCausalLM.from_pretrained(model_path, ov_config=ov_config)

    output = model.generate(**inputs, max_new_tokens=100, do_sample=False)
    return processor.decode(output[0, inputs.input_ids.shape[1] :], skip_special_tokens=True)


def run_transformers(inputs):
    model = AutoModelForImageTextToText.from_pretrained(model_id)
    output = model.generate(**inputs, max_new_tokens=100, do_sample=False)
    return processor.decode(output[0, inputs.input_ids.shape[1] :], skip_special_tokens=True)


result_transformers = run_transformers(inputs)
result_optimum = run_optimum(inputs)

print(f"Results transformers:\n{result_transformers}\n")
print(f"Results optimum:\n{result_optimum}\n")
assert result_optimum == result_transformers

# Results transformers:
# Based on the map titled "WE CAN DO BETTER," the states marked in purple are:

# - TX (Texas)
# - LA (Louisiana) — note: LA is shown in blue, but the box says "STATE SIZE DISTORTED" and LA is in purple? Wait, let’s check the map carefully.

# Actually, looking at the map, the states colored **purple** are:

# - Texas (TX)
# - Louisiana (LA) — but wait, the

# Results optimum:
# Looking at the map under the "WE CAN DO BETTER" section and the state color-coded in purple, the states marked in purple are:

# - TX (Texas)
# - LA (Louisiana) — wait, no, looking again, LA is blue (top states), but let’s check the purple ones.

# Actually, reviewing the map, the states marked in **purple** (which according to the legend likely represent bottom states or lower science adoption) include:

# - TX (

# Traceback (most recent call last):
#   File "/home/asuvorov/projects/openvino.genai/reproducer.py", line 49, in <module>
#     assert result_optimum == result_transformers
#            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# AssertionError

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.