fix(mtmd/qwen3vl): grid selection rewrite + --image-max-tiles override

Rewrite Qwen3VL multi-tile grid selection: enumerate CxR grids that downscale
the image in both dimensions (col*tile_px<=nx, row*tile_px<=ny), exclude 1x1
(equivalent to dyn_size), then pick within a ratio-error tolerance band by max
tile count and min log-ratio error. Fall back to dyn_size when no grid fits. Add
a global overview thumbnail at entries[0].

Add the --image-max-tiles CLI/env override (1..256), applied after load_hparams
so GGUF defaults don't clobber it, and clamp preproc_min_tiles so a low override
can't produce an empty candidate set on InternVL-family models. Guard the
log-ratio division against zero-dimension images.

Hardening:
- Validate clip.vision.preproc_max_tiles read from the GGUF: clamp to [1,256]
  (the --image-max-tiles range) and fall back to 4 with a warning, matching the
  InternVL branch. A crafted/corrupt value could otherwise drive an oversized
  fitting.reserve() and OOM on load or first inference.
- Size fitting.reserve() to max_tiles*ln(max_tiles) (the actual candidate count)
  instead of max_tiles*2, which undershot ~3x at the top of the range.
- Fail a zero-dimension image gracefully (LOG_ERR + return false) instead of
  GGML_ASSERT, so one malformed upload fails its own request rather than
  aborting the whole server. Callers already treat false as a per-request error.
- Doc/log consistency: reflow --image-max-tiles help to the sibling --image-*
  style, add the --image-tile-mode/--image-max-tiles rows to the cli/server
  READMEs, and log preproc_max_tiles as "(custom value)" when overridden.

Author: Rita Law

common/arg.cpp

@@ -2270,6 +2270,17 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             else { throw std::invalid_argument("unknown --image-tile-mode: " + value + " (use batched, sequential, or disabled)"); }
         }
     ).set_examples(mmproj_examples).set_env("LLAMA_ARG_IMAGE_TILE_MODE"));
+    add_opt(common_arg(
+        {"--image-max-tiles"}, "N",
+        "maximum number of tiles for multi-tile vision models (e.g. Qwen3VL), overrides the\n"
+        "value from the GGUF; use when the GGUF lacks the key or the model default is wrong\n"
+        "for your model size (Qwen3VL defaults to 4, too low for 8B+ variants)",
+        [](common_params & params, int value) {
+            if (value < 1)   { throw std::invalid_argument("--image-max-tiles must be >= 1"); }
+            if (value > 256) { throw std::invalid_argument("--image-max-tiles must be <= 256"); }
+            params.image_max_tiles = value;
+        }
+    ).set_examples(mmproj_examples).set_env("LLAMA_ARG_IMAGE_MAX_TILES"));
     if (llama_supports_rpc()) {
         add_opt(common_arg(
             {"--rpc"}, "SERVERS",

common/common.h

@@ -595,6 +595,7 @@ struct common_params {
     int image_min_tokens = -1;
     int image_max_tokens = -1;
     common_image_tile_mode image_tile_mode = COMMON_IMAGE_TILE_MODE_SEQUENTIAL;
+    int image_max_tiles = -1;  // override preproc_max_tiles from GGUF; -1 = use model default
 
     // finetune
     struct lr_opt lr;

tools/cli/README.md

@@ -165,6 +165,8 @@
 | `--image, --audio FILE` | path to an image or audio file. use with multimodal models, use comma-separated values for multiple files |
 | `--image-min-tokens N` | minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model)<br/>(env: LLAMA_ARG_IMAGE_MIN_TOKENS) |
 | `--image-max-tokens N` | maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model)<br/>(env: LLAMA_ARG_IMAGE_MAX_TOKENS) |
+| `--image-tile-mode MODE` | tile encoding mode for multi-tile vision models (e.g. Qwen3VL):<br/>  batched    - all tiles in one forward pass<br/>  sequential - tiles encoded one-by-one (default)<br/>  disabled   - tiling disabled, single tile only<br/>(env: LLAMA_ARG_IMAGE_TILE_MODE) |
+| `--image-max-tiles N` | maximum number of tiles for multi-tile vision models (e.g. Qwen3VL), overrides the<br/>value from the GGUF; use when the GGUF lacks the key or the model default is wrong<br/>for your model size (Qwen3VL defaults to 4, too low for 8B+ variants)<br/>(env: LLAMA_ARG_IMAGE_MAX_TILES) |
 | `--chat-template-kwargs STRING` | sets additional params for the json template parser, must be a valid json object string, e.g. '{"key1":"value1","key2":"value2"}'<br/>(env: LLAMA_CHAT_TEMPLATE_KWARGS) |
 | `--jinja, --no-jinja` | whether to use jinja template engine for chat (default: enabled)<br/>(env: LLAMA_ARG_JINJA) |
 | `--reasoning-format FORMAT` | controls whether thought tags are allowed and/or extracted from the response, and in which format they're returned; one of:<br/>- none: leaves thoughts unparsed in `message.content`<br/>- deepseek: puts thoughts in `message.reasoning_content`<br/>- deepseek-legacy: keeps `<think>` tags in `message.content` while also populating `message.reasoning_content`<br/>(default: auto)<br/>(env: LLAMA_ARG_THINK) |

tools/mtmd/clip.cpp

@@ -1416,8 +1416,26 @@ struct clip_model_loader {
                         hparams.image_resize_algo = RESIZE_ALGO_BILINEAR;
                         get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false);
                         get_u32(KEY_WIN_ATTN_PATTERN, hparams.n_wa_pattern, model.proj_type == PROJECTOR_TYPE_QWEN25VL); // only 2.5 requires it
-                        // optional multi-tile cap; absent in GGUF → stays 0 and the qwen3vl preprocessor falls back to 4
+                        // SigLIP2-Large ViT (n_head=16, n_layer=24) is used for 2B/4B — max_tiles=4 confirmed.
+                        // Larger ViT variants (8B+) use SigLIP2-SO-400M with unknown max_tiles.
+                        // Read from GGUF if present; otherwise default to 4 and warn for unknown ViTs.
+                        hparams.preproc_max_tiles = 4;
+                        const bool has_max_tiles_key = gguf_find_key(ctx_gguf.get(), KEY_PREPROC_MAX_TILES) >= 0;
                         get_u32(KEY_PREPROC_MAX_TILES, hparams.preproc_max_tiles, false);
+                        if (has_max_tiles_key && (hparams.preproc_max_tiles < 1 || hparams.preproc_max_tiles > 256)) {
+                            LOG_WRN("%s: clip.vision.preproc_max_tiles=%d out of range [1,256] in GGUF — defaulting to 4\n",
+                                    __func__, hparams.preproc_max_tiles);
+                            hparams.preproc_max_tiles = 4;
+                        }
+                        if (!has_max_tiles_key && (hparams.n_head > 16 || hparams.n_layer > 24)) {
+                            const char * model_name =
+                                model.proj_type == PROJECTOR_TYPE_QWEN2VL  ? "Qwen2VL"  :
+                                model.proj_type == PROJECTOR_TYPE_QWEN25VL ? "Qwen2.5VL" : "Qwen3VL";
+                            LOG_WRN("%s: %s large ViT detected (n_head=%d, n_layer=%d) but "
+                                    "clip.vision.preproc_max_tiles not in GGUF — defaulting to 4. "
+                                    "If using an 8B+ model, set the correct value with --image-max-tiles.\n",
+                                    __func__, model_name, hparams.n_head, hparams.n_layer);
+                        }
                         // ref: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/blob/main/preprocessor_config.json
                         hparams.set_limit_image_tokens(8, 4096);
                         hparams.warmup_image_size = hparams.image_size; // warmup at actual tile size to match inference graph shape
@@ -2744,6 +2762,18 @@ struct clip_init_result clip_init(const char * fname, struct clip_context_params
         if (loader.has_vision) {
             ctx_vision = new clip_ctx(ctx_params);
             loader.load_hparams(ctx_vision->model, CLIP_MODALITY_VISION);
+            // apply CLI overrides after load_hparams so GGUF defaults don't clobber them
+            if (ctx_params.image_max_tiles != -1) {
+                ctx_vision->model.hparams.preproc_max_tiles = ctx_params.image_max_tiles;
+                LOG_INF("%s: preproc_max_tiles: %d (custom value)\n", __func__, ctx_vision->model.hparams.preproc_max_tiles);
+                // models with a min-tiles floor (e.g. InternVL) would produce an empty candidate
+                // set if the override drops max below min; clamp min so the invariant holds.
+                if (ctx_vision->model.hparams.preproc_min_tiles > ctx_vision->model.hparams.preproc_max_tiles) {
+                    LOG_WRN("%s: --image-max-tiles=%d is below the model's preproc_min_tiles=%d; clamping min\n",
+                            __func__, ctx_params.image_max_tiles, ctx_vision->model.hparams.preproc_min_tiles);
+                    ctx_vision->model.hparams.preproc_min_tiles = ctx_vision->model.hparams.preproc_max_tiles;
+                }
+            }
             loader.load_tensors(*ctx_vision);
             if (ctx_params.warmup) {
                 loader.warmup(*ctx_vision);

tools/mtmd/clip.h

@@ -51,7 +51,8 @@ struct clip_context_params {
     ggml_backend_sched_eval_callback cb_eval;
     void * cb_eval_user_data;
     const char * backend_device; // optional, if null will use env var or default GPU backend
-    int image_tile_mode; // 0=batched (default), 1=sequential, 2=disabled
+    int image_tile_mode;   // 0=batched (default), 1=sequential, 2=disabled
+    int image_max_tiles;   // override preproc_max_tiles; -1 = use GGUF/model default
 };
 
 struct clip_init_result {

tools/mtmd/mtmd-cli.cpp

@@ -143,7 +143,8 @@ struct mtmd_cli_context {
         mparams.warmup           = params.warmup;
         mparams.image_min_tokens       = params.image_min_tokens;
         mparams.image_max_tokens       = params.image_max_tokens;
-        mparams.image_tile_mode = (int)params.image_tile_mode;
+        mparams.image_tile_mode  = (int)params.image_tile_mode;
+        mparams.image_max_tiles  = params.image_max_tiles;
         if (std::getenv("MTMD_DEBUG_GRAPH") != nullptr) {
             mparams.cb_eval_user_data = &cb_data;
             mparams.cb_eval = common_debug_cb_eval;

tools/mtmd/mtmd-image.cpp

@@ -935,39 +935,69 @@ bool mtmd_image_preprocessor_qwen3vl::preprocess(const clip_image_u8 & img, clip
     GGML_ASSERT(tile_px > 0 && "image_size not set in model hparams");
 
     GGML_ASSERT(hparams.patch_size > 0);
+    // Guard the log-ratio division below; the public API asserts this in mtmd.cpp, but a direct
+    // caller (fuzz harness, unit test, malformed upload reaching this path) could reach here with
+    // a zero-dimension image. Fail this one image gracefully instead of aborting the whole process.
+    if (img.nx <= 0 || img.ny <= 0) {
+        LOG_ERR("%s: image has zero dimension (%dx%d)\n", __func__, img.nx, img.ny);
+        return false;
+    }
 
-    // Pick the grid (n_col × n_row with n_col*n_row <= max_tiles, excluding 1×1) whose aspect
-    // ratio is closest to the image's (log-ratio distance), minimising the stretch needed to
-    // fill the tile canvas. Using log-ratio so e.g. 2:1 and 1:2 are symmetric around 1:1.
-    std::vector<clip_image_size> candidate_grids;
+    // Select the grid (n_col × n_row, n_col*n_row <= max_tiles, excluding 1×1) that:
+    //   1. downscales the image in both dimensions (upscaling gains no detail)
+    //   2. maximises tile count (more tiles = more spatial detail preserved)
+    //   3. among equal tile counts, minimises log-ratio error (less stretch distortion)
+    // If no grid downscales, fall back to dyn_size (single-tile at native resolution).
+    const float img_log_ratio = std::log((float)img.nx / (float)img.ny);
+
+    struct grid_cand { int col, row; float ratio_err; };
+    std::vector<grid_cand> fitting;
+    // candidate count is sum_{col} floor(max_tiles/col) ≈ max_tiles*ln(max_tiles), not max_tiles²;
+    // a small reserve avoids the early reallocations without over-allocating at large max_tiles.
+    fitting.reserve((size_t)(max_tiles * std::log((float)std::max(max_tiles, 2))));
     for (int col = 1; col <= max_tiles; col++) {
         for (int row = 1; col * row <= max_tiles; row++) {
-            if (col == 1 && row == 1) { continue; } // 1×1 == the dyn_size fallback below
-            candidate_grids.push_back({col, row});
+            if (col == 1 && row == 1) { continue; } // 1×1 == dyn_size below
+            if (col * tile_px <= img.nx && row * tile_px <= img.ny) {
+                const float err = std::abs(img_log_ratio - std::log((float)col / (float)row));
+                fitting.push_back({col, row, err});
+            }
         }
     }
-    const float img_log_ratio = std::log((float)img.nx / (float)img.ny);
-    const clip_image_size best_grid = pick_grid_by_log_ratio(candidate_grids, img_log_ratio);
-    const int best_col = best_grid.width;
-    const int best_row = best_grid.height;
-
-    const int target_w = best_col * tile_px;
-    const int target_h = best_row * tile_px;
-
-    // Only tile when the canvas DOWNSCALES the image. If the chosen tile canvas is larger than
-    // the image in either dimension, tiling would upscale it — no real detail is gained, it just
-    // spends several full-tile embeddings (plus an overview) on an image a single dyn_size pass
-    // represents fully and more cheaply. This subsumes the old "fits in one tile" check (any such
-    // image maps to a canvas that upscales it) and avoids the medium-image token blow-up, while
-    // large images the canvas downscales still tile to preserve local detail.
-    if (target_w > img.nx || target_h > img.ny) {
-        LOG_INF("%s: %dx%d would upscale into a %dx%d tile canvas — using dyn_size instead\n",
-                __func__, img.nx, img.ny, target_w, target_h);
+    if (fitting.empty()) {
+        LOG_INF("%s: %dx%d fits no tile grid (tile_px=%d, max_tiles=%d) — using dyn_size\n",
+                __func__, img.nx, img.ny, tile_px, max_tiles);
         preprocess_dyn_size_aligned(img, output, hparams, *this, dyn_size_align_px(hparams));
         // grid_x/grid_y left at 0 → single-tile path; tokenizer treats this as 1×1
         return true;
     }
 
+    // Tolerance band: find the best (minimum) ratio error, then allow any candidate within
+    // TOLERANCE of it to compete on tile count. This prevents a high-tile-count grid from
+    // winning when a near-perfect-ratio lower-tile-count grid exists (e.g. 2×2 over 2×1
+    // for a 2:1 image, which would squash the image 2× in one dimension).
+    static constexpr float RATIO_ERR_TOLERANCE = 0.2f;
+    const float best_ratio_err = std::min_element(fitting.begin(), fitting.end(),
+        [](const grid_cand & a, const grid_cand & b) { return a.ratio_err < b.ratio_err; })->ratio_err;
+    const float eligible_threshold = best_ratio_err + RATIO_ERR_TOLERANCE;
+
+    auto best = std::min_element(fitting.begin(), fitting.end(), [eligible_threshold](const grid_cand & a, const grid_cand & b) {
+        const bool ea = a.ratio_err <= eligible_threshold;
+        const bool eb = b.ratio_err <= eligible_threshold;
+        if (ea != eb) { return ea; }        // eligible always beats ineligible
+        const int ta = a.col * a.row;
+        const int tb = b.col * b.row;
+        if (ta != tb) { return ta > tb; }   // more tiles first
+        return a.ratio_err < b.ratio_err;   // then closer ratio
+    });
+
+    const int use_col = best->col;
+    const int use_row = best->row;
+    LOG_INF("%s: %dx%d — selected %dx%d grid (%d tiles, ratio_err=%.3f)\n",
+            __func__, img.nx, img.ny, use_col, use_row, use_col * use_row, best->ratio_err);
+    const int target_w = use_col * tile_px;
+    const int target_h = use_row * tile_px;
+
     // Resize to the tile canvas by stretching. With aspect-ratio-aware grid selection the chosen
     // grid's ratio is close to the image's ratio, so the residual stretch is small.
     // pad=false avoids black bars that confuse the model in tile inputs.
@@ -977,11 +1007,28 @@ bool mtmd_image_preprocessor_qwen3vl::preprocess(const clip_image_u8 & img, clip
                      /* pad */ PAD_NONE,
                      hparams.image_pad_color);
 
-    // Split into best_col × best_row tiles, packed row-major into the batch.
+    // LLaVA-style global overview (thumbnail): a downscaled full image (tile_px×tile_px) placed as
+    // entries[0]. It carries the whole image uncut, so text split across a tile seam stays readable;
+    // on DocVQA this recovers seam-truncated text for a sizeable ANLS gain. Encoded as a separate
+    // 1×1 chunk in mtmd.cpp; the tiles are untouched (no resolution cost to them). Built before the
+    // tiles so it lands at entries[0] via push_back, avoiding an O(N) insert-at-begin pointer shift.
+    {
+        clip_image_u8 thumb_u8;
+        img_tool::resize(img, thumb_u8, {tile_px, tile_px},
+                         hparams.image_resize_algo,
+                         /* pad */ PAD_NONE,
+                         hparams.image_pad_color);
+        clip_image_f32_ptr thumb_f32(clip_image_f32_init());
+        img_u8_to_f32(thumb_u8, *thumb_f32, hparams.image_mean, hparams.image_std);
+        output.entries.push_back(std::move(thumb_f32));
+        output.has_overview = true;
+    }
+
+    // Split into use_col × use_row tiles, packed row-major into the batch after the thumbnail.
     clip_image_u8 tile_u8;
 
-    for (int tr = 0; tr < best_row; tr++) {
-        for (int tc = 0; tc < best_col; tc++) {
+    for (int tr = 0; tr < use_row; tr++) {
+        for (int tc = 0; tc < use_col; tc++) {
             const int src_x0 = tc * tile_px;
             const int src_y0 = tr * tile_px;
             img_tool::crop(resized, tile_u8, src_x0, src_y0, tile_px, tile_px);
@@ -992,24 +1039,8 @@ bool mtmd_image_preprocessor_qwen3vl::preprocess(const clip_image_u8 & img, clip
         }
     }
 
-    // LLaVA-style global overview (thumbnail): a downscaled full image (tile_px×tile_px) prepended
-    // as entries[0]. It carries the whole image uncut, so text split across a tile seam stays
-    // readable; on DocVQA this recovers seam-truncated text for a sizeable ANLS gain. Encoded as a
-    // separate 1×1 chunk in mtmd.cpp; the tiles are untouched (no resolution cost to them).
-    {
-        clip_image_u8 thumb_u8;
-        img_tool::resize(img, thumb_u8, {tile_px, tile_px},
-                         hparams.image_resize_algo,
-                         /* pad */ PAD_NONE,
-                         hparams.image_pad_color);
-        clip_image_f32_ptr thumb_f32(clip_image_f32_init());
-        img_u8_to_f32(thumb_u8, *thumb_f32, hparams.image_mean, hparams.image_std);
-        output.entries.insert(output.entries.begin(), std::move(thumb_f32));
-        output.has_overview = true;
-    }
-
-    output.grid_x = best_col;
-    output.grid_y = best_row;
+    output.grid_x = use_col;
+    output.grid_y = use_row;
     return true;
 }
 

tools/mtmd/mtmd.cpp

@@ -124,6 +124,7 @@ mtmd_context_params mtmd_context_params_default() {
         /* cb_eval_user_data */ nullptr,
         /* backend_device    */ nullptr,
         /* image_tile_mode   */ 1, // 0=batched, 1=sequential (default), 2=disabled
+        /* image_max_tiles   */ -1,
     };
     return params;
 }
@@ -195,6 +196,7 @@ struct mtmd_context {
             /* cb_eval_user_data */ ctx_params.cb_eval_user_data,
             /* backend_device    */ ctx_params.backend_device,
             /* image_tile_mode   */ ctx_params.image_tile_mode,
+            /* image_max_tiles   */ ctx_params.image_max_tiles,
         };
 
         auto res = clip_init(mmproj_fname, ctx_clip_params);

tools/mtmd/mtmd.h

@@ -102,6 +102,10 @@ struct mtmd_context_params {
 
     // tile encoding mode for multi-tile vision models (Qwen3VL): 0=batched (default), 1=sequential, 2=disabled
     int image_tile_mode;
+
+    // override preproc_max_tiles from GGUF; -1 = use model default (4 for Qwen3VL 2B/4B)
+    // needed for 8B+ models whose GGUFs may lack the clip.vision.preproc_max_tiles key
+    int image_max_tiles;
 };
 
 MTMD_API const char * mtmd_default_marker(void);

tools/server/README.md

@@ -181,6 +181,8 @@ For the full list of features, please refer to [server's changelog](https://gith
 | `--mmproj-offload, --no-mmproj-offload` | whether to enable GPU offloading for multimodal projector (default: enabled)<br/>(env: LLAMA_ARG_MMPROJ_OFFLOAD) |
 | `--image-min-tokens N` | minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model)<br/>(env: LLAMA_ARG_IMAGE_MIN_TOKENS) |
 | `--image-max-tokens N` | maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model)<br/>(env: LLAMA_ARG_IMAGE_MAX_TOKENS) |
+| `--image-tile-mode MODE` | tile encoding mode for multi-tile vision models (e.g. Qwen3VL):<br/>  batched    - all tiles in one forward pass<br/>  sequential - tiles encoded one-by-one (default)<br/>  disabled   - tiling disabled, single tile only<br/>(env: LLAMA_ARG_IMAGE_TILE_MODE) |
+| `--image-max-tiles N` | maximum number of tiles for multi-tile vision models (e.g. Qwen3VL), overrides the<br/>value from the GGUF; use when the GGUF lacks the key or the model default is wrong<br/>for your model size (Qwen3VL defaults to 4, too low for 8B+ variants)<br/>(env: LLAMA_ARG_IMAGE_MAX_TILES) |
 | `-a, --alias STRING` | set model name aliases, comma-separated (to be used by API)<br/>(env: LLAMA_ARG_ALIAS) |
 | `--tags STRING` | set model tags, comma-separated (informational, not used for routing)<br/>(env: LLAMA_ARG_TAGS) |
 | `--embd-normalize N` | normalisation for embeddings (default: 2) (-1=none, 0=max absolute int16, 1=taxicab, 2=euclidean, >2=p-norm) |