fixing more typos (#45689)

* squash commit

* tmp

* maybe no

* use height first format!

* more clean-up

* and even more

* fix(minicpmv4_6): respect _is_hf_initialized in ViTWindowAttentionMerger and cast target_sizes to int

* fix(processing): handle batched video paths from apply_chat_template

* no post init

* fix: remove extra merge_kernel_size arg passed to DownsampleMLP

* fix(minicpmv4_6): fix image processor output format and processor placeholder expansion

* fix(minicpmv4_6): sync image processor with Option B and add skip_tensor_conversion

* fix: remove empty fallbacks

* fix: adjust tokenizer config

* fix: forget import init

* fix: tested video support and fix multimodal pipeline
- Add video_token_id to config and model forward for video token masking
- Add _process_visual_features helper to handle NaViT repacking and
  beam search expansion for both image and video inputs
- Cast pixel_values to vision tower dtype to fix float32/bfloat16 mismatch
- Fix video placeholder in processor: compute per-frame tokens correctly
  (patches_per_frame × num_frames) and restore slice_start/slice_end structure
- Video processor now outputs grids_videos and num_frames metadata
- Pass pixel_values_videos/target_sizes_videos through generate pipeline

* fix(minicpmv4_6): default image_token_id to None and skip metadata tensor conversion

* fix: replace video_start/end_token with image_start/end_token (align training)

* fix(processor): derive token divisor from downsample_mode at call time

* fix: align video processing with legacy semantics
- Fix concat_frames_as_image canvas color from white (255) to black (0)
  to match the legacy _concat_images line_color=(0,0,0) separator lines
- Add missing `self` parameter to concat_frames_as_image
- Fix refine_size return order (height, width) in resize_and_split_patches
- Remove unused imports (group_videos_by_shape, reorder_videos)
- Rework _preprocess_video to process each visual unit independently
  with per-frame grids and target sizes instead of a single shared grid
- Update processor placeholder generation to iterate per-frame with
  individual grid/token counts via num_patches_per_frame

* fix(video_processor): align sub-frame stacking with sample_frames semantics

Reconstruct sub_timestamps using the same double-loop as sample_frames
and apply identical np.linspace downsampling, then group by second via
cursor-based loop (matching legacy _group_stacked_by_second) instead of
fixed-stride slicing — ensures composite-per-second interleaving is
faithful to the original pipeline for all video durations.

* refactor(minicpmv4_6): add get_video_features public API, remove _process_visual_features

* refactor(minicpmv4_6): move beam expansion handling to _expand_inputs_for_generation

* fix: remove redundant conversion

* fix: Infer num_beams from pixel tensor shapes instead of passing via kwargs

* add comment in code

* Fix MiniCPMV4_6 test, image/processor batching and video sampling tests

* fix: consolidate MiniCPMV4_6 video metadata validation

* fix: address MiniCPM-V 4.6 review cleanups

* vision attention as in qwen

* skip the test, it will block merge otherwise

* move around

* .

* return new!

* frogot to delete from sig

* last fixes

* fix

* fix repo

* hmm

* fix

* come on

* aya

---------

Co-authored-by: raushan <raushan@huggingface.co>
Co-authored-by: airlsyn <airlsyn.ai@gmail.com>

Author: 3 people

docs/source/en/_toctree.yml

@@ -1287,6 +1287,8 @@
         title: MetaCLIP 2
       - local: model_doc/mgp-str
         title: MGP-STR
+      - local: model_doc/minicpmv4_6
+        title: MiniCPM-V
       - local: model_doc/mistral3
         title: Mistral3
       - local: model_doc/mistral4

docs/source/en/model_doc/minicpmv4_6.md

@@ -0,0 +1,214 @@
+<!--Copyright 2026 OpenBMB and the HuggingFace Inc. team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+*This model was released on 2025-09-16 and added to Hugging Face Transformers on 2026-04-28.*
+
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+        <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+        <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
+        <img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
+    </div>
+</div>
+
+# MiniCPM-V
+
+[MiniCPM-V](https://huggingface.co/papers/2509.18154) is a series of efficient multimodal large language models developed by [OpenBMB](https://github.com/OpenBMB). The MiniCPM-V 4.6 architecture uses a [SigLIP](siglip) vision encoder with a window-attention merger and a [Qwen3.5](qwen3_5) language model backbone, supporting both 4x and 16x visual downsampling modes.
+
+This model was contributed by [OpenBMB](https://huggingface.co/openbmb).
+The original code can be found [here](https://github.com/OpenBMB/MiniCPM-V).
+
+## Usage example
+
+### Inference with Pipeline
+
+```python
+from transformers import pipeline
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
+            },
+            {"type": "text", "text": "Describe this image."},
+        ],
+    },
+]
+
+pipe = pipeline("image-text-to-text", model="openbmb/MiniCPM-V-4_6")
+outputs = pipe(text=messages, max_new_tokens=50, return_full_text=False)
+outputs[0]["generated_text"]
+```
+
+### Inference on a single image
+
+> [!NOTE]
+> The model has been trained with a specific prompt format for chatting. Use `processor.apply_chat_template(my_conversation_dict)` to correctly format your prompts.
+
+```python
+from transformers import AutoProcessor, AutoModelForImageTextToText
+
+model_checkpoint = "openbmb/MiniCPM-V-4_6"
+processor = AutoProcessor.from_pretrained(model_checkpoint)
+model = AutoModelForImageTextToText.from_pretrained(model_checkpoint, device_map="auto")
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
+            {"type": "text", "text": "Describe this image."},
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt",
+).to(model.device, dtype=model.dtype)
+
+output = model.generate(**inputs, max_new_tokens=100)
+decoded_output = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(decoded_output)
+```
+
+### Downsampling mode
+
+MiniCPM-V 4.6 supports two visual downsampling modes:
+
+- **16x** (default): More aggressive downsampling, fewer visual tokens, faster inference.
+- **4x**: Less downsampling, more visual tokens, better for detail-rich tasks.
+
+You can change the downsampling mode at runtime by passing `downsample_mode` via `processor_kwargs` and to `model.generate`:
+
+```python
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt",
+    processor_kwargs={"downsample_mode": "4x"},
+).to(model.device, dtype=model.dtype)
+
+output = model.generate(**inputs, max_new_tokens=100, downsample_mode="4x")
+```
+
+### Thinking mode
+
+The model supports a thinking mode controlled by `enable_thinking` in the chat template. When enabled, the model generates internal reasoning before providing the final answer:
+
+```python
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt",
+    enable_thinking=True,
+).to(model.device, dtype=model.dtype)
+
+output = model.generate(**inputs, max_new_tokens=1024)
+```
+
+To disable thinking (default for evaluation):
+
+```python
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt",
+    enable_thinking=False,
+).to(model.device, dtype=model.dtype)
+```
+
+### Image processing backend
+
+MiniCPM-V 4.6 provides two image processing backends:
+
+- **torchvision** (default): Uses `torchvision.transforms` for image resizing.
+- **pil**: Uses `PIL.Image.resize`, matching the original implementation.
+
+To use the PIL backend:
+
+```python
+from transformers import AutoProcessor, AutoImageProcessor
+
+processor = AutoProcessor.from_pretrained(model_checkpoint)
+processor.image_processor = AutoImageProcessor.from_pretrained(model_checkpoint, backend="pil")
+```
+
+### Video inference
+
+MiniCPM-V 4.6 supports video understanding.
+
+```python
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "video", "video": "path/to/video.mp4"},
+            {"type": "text", "text": "Describe what happens in this video."},
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt",
+).to(model.device, dtype=model.dtype)
+
+output = model.generate(**inputs, max_new_tokens=200)
+decoded_output = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(decoded_output)
+```
+
+If you already have the rendered prompt string, you can call `processor(text=..., videos=[...])` directly instead.
+
+## MiniCPMV4_6Config
+
+[[autodoc]] MiniCPMV4_6Config
+
+## MiniCPMV4_6VisionConfig
+
+[[autodoc]] MiniCPMV4_6VisionConfig
+
+## MiniCPMV4_6Model
+
+[[autodoc]] MiniCPMV4_6Model
+    - forward
+    - get_image_features
+
+## MiniCPMV4_6ForConditionalGeneration
+
+[[autodoc]] MiniCPMV4_6ForConditionalGeneration
+    - forward
+    - get_image_features
+
+## MiniCPMV4_6Processor
+
+[[autodoc]] MiniCPMV4_6Processor
+    - __call__
+
+## MiniCPMV4_6ImageProcessor
+
+[[autodoc]] MiniCPMV4_6ImageProcessor
+    - preprocess
+
+## MiniCPMV4_6ImageProcessorPil
+
+[[autodoc]] MiniCPMV4_6ImageProcessorPil
+    - preprocess
+
+## MiniCPMV4_6VideoProcessor
+
+[[autodoc]] MiniCPMV4_6VideoProcessor
+    - preprocess

src/transformers/activations.py

@@ -354,6 +354,7 @@ def get_activation(activation_string):
 gelu_new = get_activation("gelu_new")
 gelu = get_activation("gelu")
 gelu_fast = get_activation("gelu_fast")
+gelu_pytorch_tanh = get_activation("gelu_pytorch_tanh")
 quick_gelu = get_activation("quick_gelu")
 silu = get_activation("silu")
 mish = get_activation("mish")

src/transformers/image_transforms.py

@@ -837,24 +837,26 @@ def split_to_tiles(images: "torch.Tensor", num_tiles_height: int, num_tiles_widt
 
 
 def divide_to_patches(
-    image: Union[np.ndarray, "torch.Tensor"], patch_size: int
+    image: Union[np.ndarray, "torch.Tensor"], patch_size: int | tuple[int, int]
 ) -> list[Union[np.ndarray, "torch.Tensor"]]:
     """
     Divides an image into patches of a specified size.
 
     Args:
         image (`np.array | "torch.Tensor"`):
             The input image.
-        patch_size (`int`):
-            The size of each patch.
+        patch_size (`int` or `tuple[int, int]`):
+            The size of each patch. If an int, patches are square. If a tuple,
+            it is interpreted as `(patch_height, patch_width)`.
     Returns:
         list: A list of `np.array | "torch.Tensor"` representing the patches.
     """
+    patch_h, patch_w = (patch_size, patch_size) if isinstance(patch_size, int) else patch_size
     patches = []
     height, width = get_image_size(image, channel_dim=ChannelDimension.FIRST)
-    for i in range(0, height, patch_size):
-        for j in range(0, width, patch_size):
-            patch = image[:, i : i + patch_size, j : j + patch_size]
+    for i in range(0, height, patch_h):
+        for j in range(0, width, patch_w):
+            patch = image[..., i : i + patch_h, j : j + patch_w]
             patches.append(patch)
 
     return patches

src/transformers/models/__init__.py

@@ -250,6 +250,7 @@
     from .metaclip_2 import *
     from .mgp_str import *
     from .mimi import *
+    from .minicpmv4_6 import *
     from .minimax import *
     from .minimax_m2 import *
     from .ministral import *

src/transformers/models/auto/auto_mappings.py

@@ -331,6 +331,8 @@
         ("metaclip_2_vision_model", "MetaClip2VisionConfig"),
         ("mgp-str", "MgpstrConfig"),
         ("mimi", "MimiConfig"),
+        ("minicpmv4_6", "MiniCPMV4_6Config"),
+        ("minicpmv4_6_vision", "MiniCPMV4_6VisionConfig"),
         ("minimax", "MiniMaxConfig"),
         ("minimax_m2", "MiniMaxM2Config"),
         ("ministral", "MinistralConfig"),
@@ -740,6 +742,7 @@
         ("metaclip_2_text_model", "metaclip_2"),
         ("metaclip_2_vision_model", "metaclip_2"),
         ("mgp-str", "mgp_str"),
+        ("minicpmv4_6_vision", "minicpmv4_6"),
         ("mlcd_vision_model", "mlcd"),
         ("mllama_text_model", "mllama"),
         ("mllama_vision_model", "mllama"),
@@ -901,6 +904,7 @@
         ("llava_onevision", {"pil": "LlavaOnevisionImageProcessorPil", "torchvision": "LlavaOnevisionImageProcessor"}),
         ("mask2former", {"pil": "Mask2FormerImageProcessorPil", "torchvision": "Mask2FormerImageProcessor"}),
         ("maskformer", {"pil": "MaskFormerImageProcessorPil", "torchvision": "MaskFormerImageProcessor"}),
+        ("minicpmv4_6", {"pil": "MiniCPMV4_6ImageProcessorPil", "torchvision": "MiniCPMV4_6ImageProcessor"}),
         ("mllama", {"pil": "MllamaImageProcessorPil", "torchvision": "MllamaImageProcessor"}),
         ("mobilenet_v1", {"pil": "MobileNetV1ImageProcessorPil", "torchvision": "MobileNetV1ImageProcessor"}),
         ("mobilenet_v2", {"pil": "MobileNetV2ImageProcessorPil", "torchvision": "MobileNetV2ImageProcessor"}),
@@ -967,6 +971,7 @@
         ("internvl", "InternVLVideoProcessor"),
         ("llava_next_video", "LlavaNextVideoVideoProcessor"),
         ("llava_onevision", "LlavaOnevisionVideoProcessor"),
+        ("minicpmv4_6", "MiniCPMV4_6VideoProcessor"),
         ("pe_video", "PeVideoVideoProcessor"),
         ("perception_lm", "PerceptionLMVideoProcessor"),
         ("qwen2_vl", "Qwen2VLVideoProcessor"),

src/transformers/models/auto/modeling_auto.py

@@ -285,6 +285,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("metaclip_2", "MetaClip2Model"),
         ("mgp-str", "MgpstrForSceneTextRecognition"),
         ("mimi", "MimiModel"),
+        ("minicpmv4_6", "MiniCPMV4_6Model"),
         ("minimax", "MiniMaxModel"),
         ("minimax_m2", "MiniMaxM2Model"),
         ("ministral", "MinistralModel"),
@@ -1008,6 +1009,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("llava_next", "LlavaNextForConditionalGeneration"),
         ("llava_next_video", "LlavaNextVideoForConditionalGeneration"),
         ("llava_onevision", "LlavaOnevisionForConditionalGeneration"),
+        ("minicpmv4_6", "MiniCPMV4_6ForConditionalGeneration"),
         ("mistral3", "Mistral3ForConditionalGeneration"),
         ("mistral4", "Mistral4ForCausalLM"),
         ("mllama", "MllamaForConditionalGeneration"),

src/transformers/models/auto/processing_auto.py

@@ -119,6 +119,7 @@
             ("markuplm", "MarkupLMProcessor"),
             ("metaclip_2", "CLIPProcessor"),
             ("mgp-str", "MgpstrProcessor"),
+            ("minicpmv4_6", "MiniCPMV4_6Processor"),
             ("mistral3", "PixtralProcessor"),
             ("mllama", "MllamaProcessor"),
             ("mm-grounding-dino", "GroundingDinoProcessor"),

src/transformers/models/auto/tokenization_auto.py

@@ -186,6 +186,7 @@
         ("megatron-bert", "BertTokenizer" if is_tokenizers_available() else None),
         ("metaclip_2", "XLMRobertaTokenizer" if is_tokenizers_available() else None),
         ("mgp-str", "MgpstrTokenizer"),
+        ("minicpmv4_6", "TokenizersBackend" if is_tokenizers_available() else None),
         (
             "ministral",
             "MistralCommonBackend"

src/transformers/models/minicpmv4_6/__init__.py

@@ -0,0 +1,33 @@
+# Copyright 2026 OpenBMB and the HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_minicpmv4_6 import *
+    from .image_processing_minicpmv4_6 import *
+    from .image_processing_pil_minicpmv4_6 import *
+    from .modeling_minicpmv4_6 import *
+    from .processing_minicpmv4_6 import *
+    from .video_processing_minicpmv4_6 import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)