Add typed multimodal chat surface (§2.1, Java-only)

The original §2.1 scope assumed new JNI plumbing was needed for image
input. On inspection the upstream OAI chat path already handles
{"type":"image_url","image_url":{"url":"data:..."}} blocks and
routes them through the compiled-in mtmd pipeline, and our
handleChatCompletions JNI method forwards the request JSON intact. The
only gap was a typed Java surface to emit the multipart-array form of
the message content.

New API
- ContentPart value type (TEXT / IMAGE_URL) with factories text(),
  imageUrl(), imageBytes(bytes, mime), imageFile(Path). imageFile
  detects png/jpeg/webp/gif from the file extension and produces a
  base64 data URI.
- ChatMessage(String role, List<ContentPart> parts) constructor plus
  ChatMessage.userMultimodal(ContentPart...) factory. getParts() returns
  an unmodifiable list (null for legacy text-only messages); hasParts()
  is the boolean accessor. getContent() concatenates text parts so
  legacy readers stay functional.
- InferenceParameters.setMessages(List<ChatMessage>) overload routing
  through ParameterJsonSerializer.buildMessages(List<ChatMessage>) which
  emits array-form content only when a message has parts.

Backwards compat
- Existing ChatMessage(role, content) constructor unchanged. Existing
  setMessages(String, List<Pair>) unchanged. Existing serializer
  buildMessages(String, List<Pair>) unchanged. All 123 prior tests in
  ChatMessageTest / ParameterJsonSerializerTest / InferenceParametersTest
  still pass.

Tests
- ContentPartTest (14): factory contracts, MIME detection, data URI shape.
- MultimodalMessagesTest (11): hasParts/getParts split, concatText
  behaviour, OAI array-form serialisation, mixed legacy+parts message
  lists, and end-to-end round-trip into the InferenceParameters JSON.
All 25 new tests run without a model (pure JSON shape verification).

Zero new JNI symbols. Mmproj wiring (ModelParameters.setMmproj) already
shipped previously and is untouched.

Author: claude

docs/feature-investigation-llama-stack-client-kotlin.md

@@ -62,8 +62,44 @@ branch unless noted.
 
 ### 2.1 Multimodal image input (mtmd) — **L**
 
-**Status: OPEN.** `ModelParameters.setMmproj` already wires the projector; no
-typed Java image API yet. Same gap as issues #103 / #34.
+**Status: SHIPPED (typed Java surface).** The original L-effort scope assumed
+new JNI plumbing was required, but on inspection the upstream OAI chat path
+(`oaicompat_chat_params_parse` in `server-common.cpp`) already detects
+`{"type":"image_url","image_url":{"url":"data:..."}}` blocks and routes them
+through the compiled-in `mtmd` pipeline, and the project's
+`handleChatCompletions` JNI method forwards the request JSON intact. Only the
+Java-side convenience to emit the multipart-array `content` was missing.
+
+This pass adds:
+- **`ContentPart`** value type (`TEXT` / `IMAGE_URL`) with static factories
+  `text(...)`, `imageUrl(...)`, `imageBytes(byte[], mime)`, and
+  `imageFile(Path)` (auto-detects png/jpeg/webp/gif from the extension and
+  base64-encodes into a `data:` URI).
+- **`ChatMessage(String role, List<ContentPart> parts)`** constructor plus
+  `userMultimodal(ContentPart...)` factory, `getParts()`, and `hasParts()`.
+  The legacy `ChatMessage(role, content)` ctor and existing serializer path
+  are unchanged.
+- **`InferenceParameters.setMessages(List<ChatMessage>)`** overload that
+  routes through a new `ParameterJsonSerializer.buildMessages(List<ChatMessage>)`
+  emitting array-form `content` only when a message has parts.
+- 25 unit tests in `ContentPartTest` and `MultimodalMessagesTest` cover the
+  factory contracts, the parts/legacy split, and the OAI multipart JSON shape;
+  the 123 existing `ChatMessage` / `InferenceParameters` /
+  `ParameterJsonSerializer` tests still pass.
+
+A multimodal call from Java now looks like:
+```java
+LlamaModel model = new LlamaModel(new ModelParameters()
+    .setModel("vision-model.gguf")
+    .setMmproj("vision-projector.gguf"));
+String reply = model.chatCompleteText(new InferenceParameters("")
+    .setMessages(java.util.Collections.singletonList(
+        ChatMessage.userMultimodal(
+            ContentPart.text("What is in this image?"),
+            ContentPart.imageFile(java.nio.file.Paths.get("photo.jpg"))))));
+```
+
+Zero new JNI symbols; zero risk to existing text-only chat callers.
 
 **Gap.** Upstream llama.cpp ships `mtmd` (vision + audio for some models) and
 the compiled-in server already pulls it in via `mtmd.h` / `mtmd-helper.h`. No

src/main/java/net/ladenthin/llama/ChatMessage.java

@@ -4,6 +4,7 @@
 
 package net.ladenthin.llama;
 
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
 
@@ -17,13 +18,21 @@
  * {@link #getToolCalls()} list. Tool-result turns have role {@code "tool"}, the tool's
  * output as content, and {@link #getToolCallId()} pointing back at the originating call.
  * </p>
+ * <p>
+ * Multimodal turns carry a non-null {@link #getParts()} list of {@link ContentPart}s
+ * (text and image references). When parts are present they take precedence over
+ * {@link #getContent()} during serialization; the upstream OAI chat path
+ * (see {@link InferenceParameters#setMessages(java.util.List)}) emits an array-form
+ * {@code content} field that the compiled-in {@code mtmd} pipeline understands.
+ * </p>
  */
 public final class ChatMessage {
 
     private final String role;
     private final String content;
     private final String toolCallId;
     private final List<ToolCall> toolCalls;
+    private final List<ContentPart> parts;
 
     /**
      * Plain user/assistant/system message.
@@ -32,7 +41,7 @@ public final class ChatMessage {
      * @param content the message text
      */
     public ChatMessage(String role, String content) {
-        this(role, content, null, Collections.<ToolCall>emptyList());
+        this(role, content, null, Collections.<ToolCall>emptyList(), null);
     }
 
     /**
@@ -44,10 +53,49 @@ public ChatMessage(String role, String content) {
      * @param toolCalls  for assistant tool-call turns, the list of calls; empty otherwise
      */
     public ChatMessage(String role, String content, String toolCallId, List<ToolCall> toolCalls) {
+        this(role, content, toolCallId, toolCalls, null);
+    }
+
+    /**
+     * Multimodal constructor: build a message whose content is a list of
+     * {@link ContentPart}s (text and/or image references). The {@link #getContent()}
+     * accessor returns the concatenation of the text parts for legacy callers that
+     * cannot consume the array form.
+     *
+     * @param role  the message role
+     * @param parts ordered list of content parts (must not be {@code null} or empty)
+     */
+    public ChatMessage(String role, List<ContentPart> parts) {
+        this(role, concatText(parts), null, Collections.<ToolCall>emptyList(),
+             Collections.unmodifiableList(new java.util.ArrayList<ContentPart>(requireNonEmpty(parts))));
+    }
+
+    private ChatMessage(String role, String content, String toolCallId,
+                        List<ToolCall> toolCalls, List<ContentPart> parts) {
         this.role = role;
         this.content = content;
         this.toolCallId = toolCallId;
         this.toolCalls = toolCalls == null ? Collections.<ToolCall>emptyList() : toolCalls;
+        this.parts = parts;
+    }
+
+    private static List<ContentPart> requireNonEmpty(List<ContentPart> parts) {
+        if (parts == null || parts.isEmpty()) {
+            throw new IllegalArgumentException("parts must not be null or empty");
+        }
+        return parts;
+    }
+
+    private static String concatText(List<ContentPart> parts) {
+        if (parts == null) return "";
+        StringBuilder sb = new StringBuilder();
+        for (ContentPart p : parts) {
+            if (p.getType() == ContentPart.Type.TEXT) {
+                if (sb.length() > 0) sb.append('\n');
+                sb.append(p.getText());
+            }
+        }
+        return sb.toString();
     }
 
     /**
@@ -72,6 +120,17 @@ public static ChatMessage assistantToolCalls(String content, List<ToolCall> tool
         return new ChatMessage("assistant", content == null ? "" : content, null, toolCalls);
     }
 
+    /**
+     * Convenience factory for a {@code "user"} turn mixing text and one or more
+     * images. Equivalent to {@code new ChatMessage("user", parts)}.
+     *
+     * @param parts ordered text and image parts; at least one is required
+     * @return a multimodal user message
+     */
+    public static ChatMessage userMultimodal(ContentPart... parts) {
+        return new ChatMessage("user", Arrays.asList(parts));
+    }
+
     /**
      * Message role accessor.
      * @return the message role string
@@ -104,6 +163,24 @@ public List<ToolCall> getToolCalls() {
         return toolCalls;
     }
 
+    /**
+     * Multimodal content parts accessor.
+     * @return an unmodifiable list of text and image parts, or {@code null} for
+     *         legacy text-only messages built via {@link #ChatMessage(String, String)}
+     */
+    public List<ContentPart> getParts() {
+        return parts;
+    }
+
+    /**
+     * Whether this message carries multimodal parts (i.e. was constructed via
+     * {@link #ChatMessage(String, List)} or {@link #userMultimodal(ContentPart...)}).
+     * @return {@code true} when {@link #getParts()} is non-null
+     */
+    public boolean hasParts() {
+        return parts != null;
+    }
+
     @Override
     public String toString() {
         if (!toolCalls.isEmpty()) return role + " (tool_calls=" + toolCalls.size() + "): " + content;

src/main/java/net/ladenthin/llama/ContentPart.java

@@ -0,0 +1,150 @@
+// SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com>
+//
+// SPDX-License-Identifier: MIT
+
+package net.ladenthin.llama;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Base64;
+import java.util.Locale;
+import java.util.Objects;
+
+/**
+ * One piece of a {@link ChatMessage}'s multimodal content array: either a text
+ * fragment or an image URL (typically a {@code data:image/...;base64,...} URI).
+ * Mirrors the OpenAI-compatible {@code content} part shape that the upstream
+ * {@code llama.cpp} server already understands, so no new JNI plumbing is
+ * required &#x2014; an image-bearing message is serialized to
+ * <pre>
+ * {"role":"user","content":[
+ *   {"type":"text","text":"What is in this image?"},
+ *   {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}
+ * ]}
+ * </pre>
+ * and the upstream {@code oaicompat_chat_params_parse} routes it through the
+ * compiled-in {@code mtmd} pipeline (requires
+ * {@link ModelParameters#setMmproj(String)} to be wired).
+ * <p>
+ * Instances are immutable and safe to share across threads. Use the static
+ * factories &#x2014; the constructor is private.
+ * </p>
+ */
+public final class ContentPart {
+
+    /** Discriminator for the two part kinds the OAI multipart schema supports. */
+    public enum Type {
+        /** A plain-text fragment. */
+        TEXT,
+        /** An image reference (data URI or remote URL). */
+        IMAGE_URL
+    }
+
+    private final Type type;
+    private final String text;
+    private final String imageUrl;
+
+    private ContentPart(Type type, String text, String imageUrl) {
+        this.type = type;
+        this.text = text;
+        this.imageUrl = imageUrl;
+    }
+
+    /**
+     * Build a text part.
+     *
+     * @param text the text fragment (must not be {@code null})
+     * @return a TEXT part wrapping {@code text}
+     */
+    public static ContentPart text(String text) {
+        Objects.requireNonNull(text, "text");
+        return new ContentPart(Type.TEXT, text, null);
+    }
+
+    /**
+     * Build an image part from a pre-formed URL or data URI. Pass either an
+     * HTTP(S) URL (if the server is configured to fetch it) or a complete
+     * {@code data:image/...;base64,...} string.
+     *
+     * @param url image URL or data URI (must not be {@code null})
+     * @return an IMAGE_URL part wrapping {@code url}
+     */
+    public static ContentPart imageUrl(String url) {
+        Objects.requireNonNull(url, "url");
+        return new ContentPart(Type.IMAGE_URL, null, url);
+    }
+
+    /**
+     * Build an image part from raw bytes plus an explicit MIME type. The bytes
+     * are base64-encoded and wrapped in a {@code data:} URI.
+     *
+     * @param bytes raw image bytes (must not be {@code null})
+     * @param mimeType MIME type, e.g. {@code "image/png"} (must not be {@code null} or empty)
+     * @return an IMAGE_URL part carrying the data URI
+     */
+    public static ContentPart imageBytes(byte[] bytes, String mimeType) {
+        Objects.requireNonNull(bytes, "bytes");
+        Objects.requireNonNull(mimeType, "mimeType");
+        if (mimeType.isEmpty()) {
+            throw new IllegalArgumentException("mimeType must not be empty");
+        }
+        String encoded = Base64.getEncoder().encodeToString(bytes);
+        return new ContentPart(Type.IMAGE_URL, null, "data:" + mimeType + ";base64," + encoded);
+    }
+
+    /**
+     * Build an image part by reading a file from disk and detecting its MIME
+     * type from the file extension. Recognised extensions: {@code .png},
+     * {@code .jpg}, {@code .jpeg}, {@code .webp}, {@code .gif}. Anything else
+     * throws {@link IllegalArgumentException}; use {@link #imageBytes(byte[], String)}
+     * to force a MIME type explicitly.
+     *
+     * @param imagePath path to the image file (must not be {@code null})
+     * @return an IMAGE_URL part carrying the data URI
+     * @throws IOException if the file cannot be read
+     */
+    public static ContentPart imageFile(Path imagePath) throws IOException {
+        Objects.requireNonNull(imagePath, "imagePath");
+        String name = imagePath.getFileName().toString().toLowerCase(Locale.ROOT);
+        String mimeType;
+        if (name.endsWith(".png")) {
+            mimeType = "image/png";
+        } else if (name.endsWith(".jpg") || name.endsWith(".jpeg")) {
+            mimeType = "image/jpeg";
+        } else if (name.endsWith(".webp")) {
+            mimeType = "image/webp";
+        } else if (name.endsWith(".gif")) {
+            mimeType = "image/gif";
+        } else {
+            throw new IllegalArgumentException(
+                    "Cannot infer MIME type from extension: " + imagePath
+                    + " &#x2014; use ContentPart.imageBytes(bytes, mimeType) instead");
+        }
+        return imageBytes(Files.readAllBytes(imagePath), mimeType);
+    }
+
+    /**
+     * Part-kind accessor.
+     * @return the discriminator selecting {@link #getText()} or {@link #getImageUrl()}
+     */
+    public Type getType() {
+        return type;
+    }
+
+    /**
+     * Text accessor (only set for {@link Type#TEXT}).
+     * @return the text fragment, or {@code null} for {@link Type#IMAGE_URL} parts
+     */
+    public String getText() {
+        return text;
+    }
+
+    /**
+     * Image URL accessor (only set for {@link Type#IMAGE_URL}).
+     * @return the URL or data URI, or {@code null} for {@link Type#TEXT} parts
+     */
+    public String getImageUrl() {
+        return imageUrl;
+    }
+}

src/main/java/net/ladenthin/llama/InferenceParameters.java

@@ -554,6 +554,28 @@ public InferenceParameters setMessages(String systemMessage, List<Pair<String, S
         return this;
     }
 
+    /**
+     * Multimodal-capable variant. Accepts {@link ChatMessage} objects so messages
+     * with non-null {@link ChatMessage#getParts()} are serialized as OAI array-form
+     * {@code content} (text + image_url parts). Plain text messages emit the legacy
+     * string-form {@code content}, so this overload is also a drop-in replacement
+     * for the {@code List&lt;Pair&gt;} variant when callers prefer the typed
+     * {@link ChatMessage} surface.
+     * <p>
+     * Image parts require the model to have a multimodal projector loaded via
+     * {@link ModelParameters#setMmproj(String)}. The upstream OAI chat parser
+     * routes {@code image_url} blocks through the compiled-in {@code mtmd}
+     * pipeline; no additional JNI configuration is needed on the Java side.
+     * </p>
+     *
+     * @param messages ordered messages, including any {@code "system"} prelude
+     * @return this builder
+     */
+    public InferenceParameters setMessages(List<ChatMessage> messages) {
+        parameters.put(PARAM_MESSAGES, serializer.buildMessages(messages).toString());
+        return this;
+    }
+
     /**
      * Set the {@code messages} array directly from a pre-built JSON string. Use this
      * for the typed chat API (see {@link ChatRequest#buildMessagesJson()}) when the