Commit 33ec4a8
feat(vision): add vision tool using MiniCPM-V 4.6 (1.3B) via llama-mtmd-cli (#22)
* feat(vision): add vision tool using MiniCPM-V 4.6 (1.3B) via llama-mtmd-cli
Adds a `vision` built-in tool that analyses images and videos locally using
MiniCPM-V 4.6 — a 1.3B multimodal model running via llama.cpp's
llama-mtmd-cli, with no cloud API required.
## What's new
**Tool (`vision`)**
- Accepts images (JPEG, PNG, GIF, WebP, BMP) and videos (MP4, MOV, AVI,
MKV, WebM)
- Videos: ffprobe reads duration, ffmpeg extracts N evenly-spaced frames,
all frames sent as a multi-image call to the model (configurable via
`video_frames`, default 8)
- Security: O_NOFOLLOW open (symlink protection), danger.CheckOperation
classification, all output wrapped in wrapUntrusted() with provenance tag
- Setup instructions in every error path (missing binary, missing model,
missing mmproj, missing ffmpeg)
**Docker (`docker/Dockerfile`)**
- New `minicpm` multi-stage build: downloads pre-built llama-mtmd-cli
(llama.cpp b9549) for amd64/arm64 from the official GitHub release, then
fetches MiniCPM-V-4_6-Q4_K_M.gguf (529 MB) and mmproj-model-f16.gguf
(1.1 GB) from HuggingFace into /usr/local/share/minicpm-v/models/
- Overridable via --build-arg MINICPM_QUANT=Q8_0 and LLAMA_VERSION
- Runtime stage copies binary + models; no new runtime deps (libstdc++6
already present for whisper)
**Config (`internal/config/loader.go`)**
- New VisionConfig struct: ModelsDir, BinaryPath, VideoFrames
- Wired into FileConfig, ResolvedConfig, resolveVision(), mergeFile()
**Tests**
- 13 tests in cmd/odek/vision_tool_test.go: empty path, invalid JSON, file
not found, symlink rejected, missing binary, missing model, missing mmproj,
mock happy-path image (4 extensions), custom prompt, mock happy-path video
(with mock ffprobe+ffmpeg via PATH override), missing ffmpeg fallback,
schema shape
- 3 tests in internal/config/vision_test.go: resolveVision defaults,
zero-frames backfill, custom values round-trip
**Docs**
- docs/CHEATSHEET.md: new Image & Video Understanding section with config
snippet and field reference
- docs/SECURITY.md: vision added to untrusted-content table, always-untrusted
list, and skills provenance gate paragraph
- docs/CONFIG.md + docs/TELEGRAM.md: smart-previews bullet updated
- docker/README.md: new Image & video understanding (out of the box) section
- README.md: vision added to external-content ingestion list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vision): pin MiniCPM-V HuggingFace downloads to a specific revision hash
Replace `resolve/main/` with `resolve/<sha>/` (78e02f0) so Docker builds
are reproducible — a future model update on the main branch won't silently
change the binary image.
vprotocol auto-repair: finding D001 (Axis 2.6 Dependency Integrity).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>1 parent 1128396 commit 33ec4a8
21 files changed
Lines changed: 839 additions & 19 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | | - | |
| 247 | + | |
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
779 | 779 | | |
780 | 780 | | |
781 | 781 | | |
782 | | - | |
| 782 | + | |
783 | 783 | | |
784 | 784 | | |
785 | 785 | | |
| |||
1054 | 1054 | | |
1055 | 1055 | | |
1056 | 1056 | | |
1057 | | - | |
| 1057 | + | |
1058 | 1058 | | |
1059 | 1059 | | |
1060 | 1060 | | |
| |||
1089 | 1089 | | |
1090 | 1090 | | |
1091 | 1091 | | |
| 1092 | + | |
1092 | 1093 | | |
1093 | 1094 | | |
1094 | 1095 | | |
| |||
1598 | 1599 | | |
1599 | 1600 | | |
1600 | 1601 | | |
1601 | | - | |
| 1602 | + | |
1602 | 1603 | | |
1603 | 1604 | | |
1604 | 1605 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
206 | | - | |
| 206 | + | |
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
| 80 | + | |
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
570 | 570 | | |
571 | 571 | | |
572 | 572 | | |
573 | | - | |
| 573 | + | |
574 | 574 | | |
575 | 575 | | |
576 | 576 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
270 | | - | |
| 270 | + | |
271 | 271 | | |
272 | 272 | | |
273 | 273 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
291 | 291 | | |
292 | 292 | | |
293 | 293 | | |
294 | | - | |
| 294 | + | |
295 | 295 | | |
296 | 296 | | |
297 | 297 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
320 | 320 | | |
321 | 321 | | |
322 | 322 | | |
323 | | - | |
| 323 | + | |
324 | 324 | | |
325 | 325 | | |
326 | 326 | | |
| |||
338 | 338 | | |
339 | 339 | | |
340 | 340 | | |
341 | | - | |
| 341 | + | |
342 | 342 | | |
343 | 343 | | |
344 | 344 | | |
| |||
432 | 432 | | |
433 | 433 | | |
434 | 434 | | |
435 | | - | |
| 435 | + | |
436 | 436 | | |
437 | 437 | | |
438 | 438 | | |
| |||
0 commit comments