Wire CI for end-to-end multimodal regression (closes #103 / #34)

Adds vision-capable model + matching mmproj + a CC0/PD test image to all
four Java test jobs (Linux x86_64, macOS arm64 with/without Metal,
Windows x86_64) and a model-gated MultimodalIntegrationTest that proves
the typed ChatMessage(role, List<ContentPart>) surface from PR #189
round-trips through the upstream mtmd pipeline end-to-end.

CI changes (.github/workflows/publish.yml)
- New env vars: VISION_MODEL_URL / VISION_MODEL_NAME pointing at
  ggml-org/SmolVLM-500M-Instruct-Q8_0.gguf (smallest reliable vision
  GGUF on community ggml-org), VISION_MMPROJ_URL / _NAME for the
  matching mmproj, VISION_IMAGE_URL / _NAME for a small PD red-apple
  image from Wikimedia Commons.
- Each of the four Java test jobs gains three download steps and three
  -D system properties on the mvn test invocation:
  -Dnet.ladenthin.llama.vision.model / .mmproj / .image.

Validation scripts
- validate-models.sh refactored into validate_gguf() + validate_image()
  helpers with a 'required' vs 'optional' mode. Required models still
  fail-fast; the new vision GGUFs and PD image are validated only when
  present so jobs that skip them keep passing.
- validate-models.bat extended with a parallel OPTIONAL_MODELS loop.

Test (src/test/java/.../MultimodalIntegrationTest.java)
- Self-skips via Assume when any of the three -D paths is unset or its
  file is missing, so local mvn test stays green without the artifacts.
- multimodalRequestProducesNonEmptyReply: builds a ChatMessage.userMultimodal
  with ContentPart.text(...) + ContentPart.imageFile(Paths.get(image)),
  calls chatCompleteText, asserts non-empty reply. Does NOT assert reply
  semantics &#x2014; a 500M model can caption inaccurately and CI must
  not flap on model quality.
- multimodalThenTextOnSameModel: sanity check that a multimodal call
  followed by a text-only call on the same model both succeed (catches
  any parts/legacy split poisoning the inference context).

TestConstants gains PROP_VISION_MODEL_PATH / PROP_VISION_MMPROJ_PATH /
PROP_VISION_IMAGE_PATH so the test reads the system properties via the
same naming pattern as PROP_NOMIC_MODEL_PATH.

Docs
- docs/history/49be664_open_issues.md: #103 and #34 PARTIALLY FIXED ->
  FIXED in the per-issue blocks, the verdict guide, the status overview
  table, the deep-dive table, the cannot-be-closed-by-unit-tests-alone
  table, and the recommended-sequencing list. Bottom-line summary
  updated to reflect that 0 of the original LIKELY/PARTIALLY FIXED items
  remain partially fixed.
- (docs/feature-investigation-llama-stack-client-kotlin.md §2.1 was
  already updated in the PR-189 typed-multimodal-surface commit.)

Verified locally
- mvn test-compile: clean.
- mvn test -Dtest=MultimodalIntegrationTest: SKIPPED (no -D properties
  set; expected self-skip path).
- mvn javadoc:jar: BUILD SUCCESS.

Author: claude

.github/validate-models.bat

@@ -11,7 +11,12 @@ setlocal enabledelayedexpansion
 
 set "MODELS=models\codellama-7b.Q2_K.gguf" "models\jina-reranker-v1-tiny-en-Q4_0.gguf" "models\AMD-Llama-135m-code.Q2_K.gguf" "models\Qwen3-0.6B-Q4_K_M.gguf"
 
-echo Validating model files...
+REM Vision GGUFs are validated only when present (the Windows job downloads
+REM them too, but the validation step must not fail when a future job opts
+REM out of the vision matrix).
+set "OPTIONAL_MODELS=models\SmolVLM-500M-Instruct-Q8_0.gguf" "models\mmproj-SmolVLM-500M-Instruct-Q8_0.gguf"
+
+echo Validating required model files...
 for %%M in (%MODELS%) do (
   if not exist "%%M" (
     echo ERROR: Model not found: %%M
@@ -37,5 +42,24 @@ for %%M in (%MODELS%) do (
   echo OK: %%M ^(!size! bytes^)
 )
 
+echo Validating optional vision model files...
+for %%M in (%OPTIONAL_MODELS%) do (
+  if not exist "%%M" (
+    echo SKIP: %%M not present
+  ) else (
+    for /f %%S in ('powershell -Command "(Get-Item '%%M').Length"') do set "size=%%S"
+    if !size! lss 4 (
+      echo ERROR: Model file too small (likely corrupted^): %%M (size: !size! bytes^)
+      exit /b 1
+    )
+    for /f %%H in ('powershell -Command "[System.BitConverter]::ToString((Get-Content '%%M' -Encoding Byte -ReadCount 4)[0]) -replace '-',''"') do set "magic=%%H"
+    if not "!magic!"=="47475546" (
+      echo ERROR: Invalid GGUF magic bytes in %%M (got: !magic!, expected: 47475546^)
+      exit /b 1
+    )
+    echo OK: %%M ^(!size! bytes^)
+  )
+)
+
 echo All models validated successfully!
 exit /b 0

.github/validate-models.sh

@@ -17,28 +17,80 @@ MODELS=(
   "models/Qwen3-0.6B-Q4_K_M.gguf"
 )
 
-echo "Validating model files..."
-for model in "${MODELS[@]}"; do
+# Optional GGUFs and image, validated only when present so jobs that do not
+# download them (e.g. cross-compile smoke runs) still pass.
+OPTIONAL_MODELS=(
+  "models/nomic-embed-text-v1.5.f16.gguf"
+  "models/SmolVLM-500M-Instruct-Q8_0.gguf"
+  "models/mmproj-SmolVLM-500M-Instruct-Q8_0.gguf"
+)
+
+OPTIONAL_IMAGES=(
+  "models/Red_Apple.jpg"
+)
+
+validate_gguf() {
+  local model="$1"
+  local required="$2"
   if [[ ! -f "$model" ]]; then
-    echo "ERROR: Model not found: $model"
-    exit 1
+    if [[ "$required" == "required" ]]; then
+      echo "ERROR: Model not found: $model"
+      exit 1
+    else
+      echo "- $model (optional, skipped: not present)"
+      return
+    fi
   fi
-
-  # Check file size (must be > 4 bytes for magic header)
+  local size
   size=$(stat -f%z "$model" 2>/dev/null || stat -c%s "$model" 2>/dev/null)
   if [[ $size -lt 4 ]]; then
     echo "ERROR: Model file too small (likely corrupted): $model (size: $size bytes)"
     exit 1
   fi
-
-  # Check GGUF magic bytes: 0x47 0x47 0x55 0x46
+  local magic
   magic=$(xxd -p -l 4 "$model")
   if [[ "$magic" != "47475546" ]]; then
     echo "ERROR: Invalid GGUF magic bytes in $model (got: $magic, expected: 47475546)"
     exit 1
   fi
-
   echo "✓ $model ($(numfmt --to=iec-i --suffix=B $size 2>/dev/null || echo $size bytes))"
+}
+
+validate_image() {
+  local img="$1"
+  if [[ ! -f "$img" ]]; then
+    echo "- $img (optional, skipped: not present)"
+    return
+  fi
+  local size
+  size=$(stat -f%z "$img" 2>/dev/null || stat -c%s "$img" 2>/dev/null)
+  if [[ $size -lt 100 ]]; then
+    echo "ERROR: Image file too small (likely an HTML error page): $img (size: $size bytes)"
+    exit 1
+  fi
+  # Accept JPEG (FF D8 FF), PNG (89 50 4E 47), WebP RIFF (52 49 46 46), GIF (47 49 46 38)
+  local magic
+  magic=$(xxd -p -l 4 "$img")
+  case "$magic" in
+    ffd8ff*|89504e47|52494646|47494638)
+      echo "✓ $img ($(numfmt --to=iec-i --suffix=B $size 2>/dev/null || echo $size bytes))"
+      ;;
+    *)
+      echo "ERROR: Unrecognised image magic in $img (got: $magic)"
+      exit 1
+      ;;
+  esac
+}
+
+echo "Validating model files..."
+for model in "${MODELS[@]}"; do
+  validate_gguf "$model" required
+done
+for model in "${OPTIONAL_MODELS[@]}"; do
+  validate_gguf "$model" optional
+done
+for img in "${OPTIONAL_IMAGES[@]}"; do
+  validate_image "$img"
 done
 
 echo "All models validated successfully!"

.github/workflows/publish.yml

@@ -22,6 +22,20 @@ env:
   REASONING_MODEL_NAME: "Qwen3-0.6B-Q4_K_M.gguf"
   NOMIC_EMBED_MODEL_URL: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.f16.gguf"
   NOMIC_EMBED_MODEL_NAME: "nomic-embed-text-v1.5.f16.gguf"
+  # Vision model + mmproj for MultimodalIntegrationTest (issues #103 / #34).
+  # SmolVLM-500M is the smallest community vision GGUF that loads reliably
+  # under the upstream mtmd pipeline. Total download ~600 MB across model
+  # plus mmproj; matches the existing per-test-job download budget.
+  VISION_MODEL_URL: "https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/SmolVLM-500M-Instruct-Q8_0.gguf"
+  VISION_MODEL_NAME: "SmolVLM-500M-Instruct-Q8_0.gguf"
+  VISION_MMPROJ_URL: "https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/mmproj-SmolVLM-500M-Instruct-Q8_0.gguf"
+  VISION_MMPROJ_NAME: "mmproj-SmolVLM-500M-Instruct-Q8_0.gguf"
+  # Small CC0 / public-domain test image from Wikimedia Commons. A simple
+  # subject (red apple, ~12 KB) so a 500M vision model has a fair chance of
+  # producing recognisable output; the test only asserts a non-empty reply
+  # so model accuracy is not the gating signal.
+  VISION_IMAGE_URL: "https://upload.wikimedia.org/wikipedia/commons/1/15/Red_Apple.jpg"
+  VISION_IMAGE_NAME: "Red_Apple.jpg"
 permissions:
   contents: read
 jobs:
@@ -393,6 +407,12 @@ jobs:
         run: curl -L --fail ${REASONING_MODEL_URL} --create-dirs -o models/${REASONING_MODEL_NAME}
       - name: Download nomic embedding model (issue #98 regression)
         run: curl -L --fail ${NOMIC_EMBED_MODEL_URL} --create-dirs -o models/${NOMIC_EMBED_MODEL_NAME}
+      - name: Download vision model (issues #103 / #34)
+        run: curl -L --fail ${VISION_MODEL_URL} --create-dirs -o models/${VISION_MODEL_NAME}
+      - name: Download vision mmproj
+        run: curl -L --fail ${VISION_MMPROJ_URL} --create-dirs -o models/${VISION_MMPROJ_NAME}
+      - name: Download CC0 / public-domain test image
+        run: curl -L --fail -A "java-llama.cpp-ci/1.0" "${VISION_IMAGE_URL}" --create-dirs -o models/${VISION_IMAGE_NAME}
       - name: List files in models directory
         run: ls -l models/
       - name: Validate model files
@@ -408,7 +428,12 @@ jobs:
           ulimit -c unlimited
           echo "${{ github.workspace }}/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern
       - name: Run tests
-        run: mvn --no-transfer-progress test -Dnet.ladenthin.llama.nomic.path=models/${NOMIC_EMBED_MODEL_NAME}
+        run: |
+          mvn --no-transfer-progress test \
+            -Dnet.ladenthin.llama.nomic.path=models/${NOMIC_EMBED_MODEL_NAME} \
+            -Dnet.ladenthin.llama.vision.model=models/${VISION_MODEL_NAME} \
+            -Dnet.ladenthin.llama.vision.mmproj=models/${VISION_MMPROJ_NAME} \
+            -Dnet.ladenthin.llama.vision.image=models/${VISION_IMAGE_NAME}
       - uses: actions/upload-artifact@v7
         if: success()
         with:
@@ -455,6 +480,12 @@ jobs:
         run: curl -L --fail ${DRAFT_MODEL_URL} --create-dirs -o models/${DRAFT_MODEL_NAME}
       - name: Download reasoning model
         run: curl -L --fail ${REASONING_MODEL_URL} --create-dirs -o models/${REASONING_MODEL_NAME}
+      - name: Download vision model (issues #103 / #34)
+        run: curl -L --fail ${VISION_MODEL_URL} --create-dirs -o models/${VISION_MODEL_NAME}
+      - name: Download vision mmproj
+        run: curl -L --fail ${VISION_MMPROJ_URL} --create-dirs -o models/${VISION_MMPROJ_NAME}
+      - name: Download CC0 / public-domain test image
+        run: curl -L --fail -A "java-llama.cpp-ci/1.0" "${VISION_IMAGE_URL}" --create-dirs -o models/${VISION_IMAGE_NAME}
       - name: List files in models directory
         run: ls -l models/
       - name: Validate model files
@@ -468,7 +499,11 @@ jobs:
       - name: Enable core dumps
         run: ulimit -c unlimited
       - name: Run tests
-        run: mvn --no-transfer-progress -Dnet.ladenthin.llama.test.ngl=0 test
+        run: |
+          mvn --no-transfer-progress -Dnet.ladenthin.llama.test.ngl=0 test \
+            -Dnet.ladenthin.llama.vision.model=models/${VISION_MODEL_NAME} \
+            -Dnet.ladenthin.llama.vision.mmproj=models/${VISION_MMPROJ_NAME} \
+            -Dnet.ladenthin.llama.vision.image=models/${VISION_IMAGE_NAME}
       - name: Memory after tests
         if: always()
         run: vm_stat && sysctl hw.memsize hw.physmem
@@ -508,6 +543,12 @@ jobs:
         run: curl -L --fail ${DRAFT_MODEL_URL} --create-dirs -o models/${DRAFT_MODEL_NAME}
       - name: Download reasoning model
         run: curl -L --fail ${REASONING_MODEL_URL} --create-dirs -o models/${REASONING_MODEL_NAME}
+      - name: Download vision model (issues #103 / #34)
+        run: curl -L --fail ${VISION_MODEL_URL} --create-dirs -o models/${VISION_MODEL_NAME}
+      - name: Download vision mmproj
+        run: curl -L --fail ${VISION_MMPROJ_URL} --create-dirs -o models/${VISION_MMPROJ_NAME}
+      - name: Download CC0 / public-domain test image
+        run: curl -L --fail -A "java-llama.cpp-ci/1.0" "${VISION_IMAGE_URL}" --create-dirs -o models/${VISION_IMAGE_NAME}
       - name: List files in models directory
         run: ls -l models/
       - name: Validate model files
@@ -521,7 +562,11 @@ jobs:
       - name: Enable core dumps
         run: ulimit -c unlimited
       - name: Run tests
-        run: mvn --no-transfer-progress test
+        run: |
+          mvn --no-transfer-progress test \
+            -Dnet.ladenthin.llama.vision.model=models/${VISION_MODEL_NAME} \
+            -Dnet.ladenthin.llama.vision.mmproj=models/${VISION_MMPROJ_NAME} \
+            -Dnet.ladenthin.llama.vision.image=models/${VISION_IMAGE_NAME}
       - name: Memory after tests
         if: always()
         run: vm_stat && sysctl hw.memsize hw.physmem
@@ -564,6 +609,12 @@ jobs:
         run: curl -L --fail $env:DRAFT_MODEL_URL --create-dirs -o models/$env:DRAFT_MODEL_NAME
       - name: Download reasoning model
         run: curl -L --fail $env:REASONING_MODEL_URL --create-dirs -o models/$env:REASONING_MODEL_NAME
+      - name: Download vision model (issues #103 / #34)
+        run: curl -L --fail $env:VISION_MODEL_URL --create-dirs -o models/$env:VISION_MODEL_NAME
+      - name: Download vision mmproj
+        run: curl -L --fail $env:VISION_MMPROJ_URL --create-dirs -o models/$env:VISION_MMPROJ_NAME
+      - name: Download CC0 / public-domain test image
+        run: curl -L --fail -A "java-llama.cpp-ci/1.0" $env:VISION_IMAGE_URL --create-dirs -o models/$env:VISION_IMAGE_NAME
       - name: List files in models directory
         run: ls -l models/
       - name: Validate model files
@@ -576,7 +627,11 @@ jobs:
         run: Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory,TotalVisibleMemorySize | Format-List
         shell: pwsh
       - name: Run tests
-        run: mvn --no-transfer-progress test
+        run: |
+          mvn --no-transfer-progress test `
+            "-Dnet.ladenthin.llama.vision.model=models/$env:VISION_MODEL_NAME" `
+            "-Dnet.ladenthin.llama.vision.mmproj=models/$env:VISION_MMPROJ_NAME" `
+            "-Dnet.ladenthin.llama.vision.image=models/$env:VISION_IMAGE_NAME"
       - name: Memory after tests
         if: always()
         run: Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory,TotalVisibleMemorySize | Format-List