Skip to content

Commit 33ec4a8

Browse files
jkyberneeesclaude
andauthored
feat(vision): add vision tool using MiniCPM-V 4.6 (1.3B) via llama-mtmd-cli (#22)
* feat(vision): add vision tool using MiniCPM-V 4.6 (1.3B) via llama-mtmd-cli Adds a `vision` built-in tool that analyses images and videos locally using MiniCPM-V 4.6 — a 1.3B multimodal model running via llama.cpp's llama-mtmd-cli, with no cloud API required. ## What's new **Tool (`vision`)** - Accepts images (JPEG, PNG, GIF, WebP, BMP) and videos (MP4, MOV, AVI, MKV, WebM) - Videos: ffprobe reads duration, ffmpeg extracts N evenly-spaced frames, all frames sent as a multi-image call to the model (configurable via `video_frames`, default 8) - Security: O_NOFOLLOW open (symlink protection), danger.CheckOperation classification, all output wrapped in wrapUntrusted() with provenance tag - Setup instructions in every error path (missing binary, missing model, missing mmproj, missing ffmpeg) **Docker (`docker/Dockerfile`)** - New `minicpm` multi-stage build: downloads pre-built llama-mtmd-cli (llama.cpp b9549) for amd64/arm64 from the official GitHub release, then fetches MiniCPM-V-4_6-Q4_K_M.gguf (529 MB) and mmproj-model-f16.gguf (1.1 GB) from HuggingFace into /usr/local/share/minicpm-v/models/ - Overridable via --build-arg MINICPM_QUANT=Q8_0 and LLAMA_VERSION - Runtime stage copies binary + models; no new runtime deps (libstdc++6 already present for whisper) **Config (`internal/config/loader.go`)** - New VisionConfig struct: ModelsDir, BinaryPath, VideoFrames - Wired into FileConfig, ResolvedConfig, resolveVision(), mergeFile() **Tests** - 13 tests in cmd/odek/vision_tool_test.go: empty path, invalid JSON, file not found, symlink rejected, missing binary, missing model, missing mmproj, mock happy-path image (4 extensions), custom prompt, mock happy-path video (with mock ffprobe+ffmpeg via PATH override), missing ffmpeg fallback, schema shape - 3 tests in internal/config/vision_test.go: resolveVision defaults, zero-frames backfill, custom values round-trip **Docs** - docs/CHEATSHEET.md: new Image & Video Understanding section with config snippet and field reference - docs/SECURITY.md: vision added to untrusted-content table, always-untrusted list, and skills provenance gate paragraph - docs/CONFIG.md + docs/TELEGRAM.md: smart-previews bullet updated - docker/README.md: new Image & video understanding (out of the box) section - README.md: vision added to external-content ingestion list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(vision): pin MiniCPM-V HuggingFace downloads to a specific revision hash Replace `resolve/main/` with `resolve/<sha>/` (78e02f0) so Docker builds are reproducible — a future model update on the main branch won't silently change the binary image. vprotocol auto-repair: finding D001 (Axis 2.6 Dependency Integrity). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 1128396 commit 33ec4a8

21 files changed

Lines changed: 839 additions & 19 deletions

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ odek is not a framework. It's a **runtime** — the smallest possible surface ar
3636
Every session can run in an isolated Docker container: no network, no host mounts beyond the working directory, zero capabilities, destroyed on exit. `odek serve` enables the sandbox **by default**; `odek run` keeps it opt-in but warns when running unsandboxed. `--ctx` files are auto-injected into the container at `/workspace/`. Full security model in [docs/SANDBOXING.md](docs/SANDBOXING.md).
3737

3838
### 🛡️ Prompt-Injection-Aware
39-
External content the agent ingests (`browser`, `read_file`, `shell`, `search_files`, `multi_grep`, `transcribe`, `session_search`, MCP tools) is wrapped in per-call nonce'd `<untrusted_content>` boundaries so the model can distinguish data from instructions. Redirect hops are re-classified (`browser`/`http_batch`), MCP tool descriptions are scanned for injection at registration, and the MCP error channel is wrapped too. The danger classifier resists 8 known shell-evasion tricks (`$()`, backticks, `$IFS`, `command`/`exec`, `\rm`, basenamed absolute paths). Approvers engage friction mode after 3 same-class approvals in 60 s. Memory episodes from tainted sessions are stored but never auto-replayed. Skill auto-save tracks provenance and pins untrusted suggestions for explicit `odek skill promote`. `odek audit <session-id>` surfaces every ingest + per-turn divergence heuristic. Full threat model in [docs/SECURITY.md](docs/SECURITY.md).
39+
External content the agent ingests (`browser`, `read_file`, `shell`, `search_files`, `multi_grep`, `transcribe`, `vision`, `session_search`, MCP tools) is wrapped in per-call nonce'd `<untrusted_content>` boundaries so the model can distinguish data from instructions. Redirect hops are re-classified (`browser`/`http_batch`), MCP tool descriptions are scanned for injection at registration, and the MCP error channel is wrapped too. The danger classifier resists 8 known shell-evasion tricks (`$()`, backticks, `$IFS`, `command`/`exec`, `\rm`, basenamed absolute paths). Approvers engage friction mode after 3 same-class approvals in 60 s. Memory episodes from tainted sessions are stored but never auto-replayed. Skill auto-save tracks provenance and pins untrusted suggestions for explicit `odek skill promote`. `odek audit <session-id>` surfaces every ingest + per-turn divergence heuristic. Full threat model in [docs/SECURITY.md](docs/SECURITY.md).
4040

4141
### 🧩 Sub-Agent Delegation
4242
Parallel OS-process sub-agents via `delegate_tasks`. True isolation — each sub-agent is a fresh `odek subagent` process with its own config, tools, and termination timeout. Up to 8 concurrent workers. [docs/SUBAGENTS.md](docs/SUBAGENTS.md)

cmd/odek/injection_hardening_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,7 +244,7 @@ func TestBuiltinTools_SessionSearchWrappedAsUntrusted(t *testing.T) {
244244
store, cleanup := seedSessionStore(t)
245245
defer cleanup()
246246

247-
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 4, "", config.TranscriptionConfig{}, store)
247+
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 4, "", config.TranscriptionConfig{}, config.VisionConfig{}, store)
248248

249249
var ss odek.Tool
250250
for _, tool := range tools {

cmd/odek/main.go

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -779,7 +779,7 @@ func run(args []string) error {
779779

780780
// Sandbox setup
781781
var sandboxCleanup func() error
782-
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, resolved.Transcription, nil)
782+
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, resolved.Transcription, resolved.Vision, nil)
783783

784784
// MCP server tools
785785
var mcpCleanup func()
@@ -1054,7 +1054,7 @@ func setupSandbox(tools []odek.Tool, cfg sandboxConfig) (containerName string, c
10541054
return containerName, cleanup, nil
10551055
}
10561056

1057-
func builtinTools(dc danger.DangerousConfig, sm *skills.SkillManager, approver danger.Approver, maxConcurrency int, apiKey string, tc config.TranscriptionConfig, store *session.Store) []odek.Tool {
1057+
func builtinTools(dc danger.DangerousConfig, sm *skills.SkillManager, approver danger.Approver, maxConcurrency int, apiKey string, tc config.TranscriptionConfig, vc config.VisionConfig, store *session.Store) []odek.Tool {
10581058
tools := []odek.Tool{
10591059
&shellTool{
10601060
dangerousConfig: dc,
@@ -1089,6 +1089,7 @@ func builtinTools(dc danger.DangerousConfig, sm *skills.SkillManager, approver d
10891089
&trTool{dangerousConfig: dc},
10901090
&wordCountTool{dangerousConfig: dc},
10911091
newTranscribeTool(dc, tc),
1092+
newVisionTool(dc, vc),
10921093
// session_search returns content from arbitrary past sessions —
10931094
// including sessions that ingested untrusted content. That path
10941095
// otherwise bypasses the memory taint gate and the audit log, so
@@ -1598,7 +1599,7 @@ func continueCmd(args []string) error {
15981599
"./.odek/skills",
15991600
)
16001601
}
1601-
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, resolved.Transcription, store)
1602+
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, resolved.Transcription, resolved.Vision, store)
16021603
var sandboxCleanup func() error
16031604

16041605
// MCP server tools

cmd/odek/main_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,7 +203,7 @@ func TestRun_NoAPIKey(t *testing.T) {
203203
}
204204

205205
func TestBuiltinTools(t *testing.T) {
206-
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, nil)
206+
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, config.VisionConfig{}, nil)
207207
if len(tools) == 0 {
208208
t.Fatal("builtinTools() returned empty slice")
209209
}

cmd/odek/mcp.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ Flags:
7373
}
7474

7575
// Build tools
76-
toolSet := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, nil)
76+
toolSet := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, config.VisionConfig{}, nil)
7777

7878
// MCP server tools — connect and discover before sandbox
7979
var mcpCleanup func()

cmd/odek/repl.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ func replCmd(args []string) error {
7777
"./.odek/skills",
7878
)
7979
}
80-
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, nil)
80+
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, config.VisionConfig{}, nil)
8181
var sandboxCleanup func() error
8282

8383
// MCP server tools

cmd/odek/schedule.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -570,7 +570,7 @@ func runTaskHeadless(ctx context.Context, resolved config.ResolvedConfig, system
570570
resolved.Dangerous.NonInteractive = &deny
571571
}
572572

573-
tools := builtinTools(resolved.Dangerous, nil, nil, resolved.MaxConcurrency, resolved.APIKey, resolved.Transcription, nil)
573+
tools := builtinTools(resolved.Dangerous, nil, nil, resolved.MaxConcurrency, resolved.APIKey, resolved.Transcription, resolved.Vision, nil)
574574
tools = append(tools, mcpTools...)
575575

576576
// Capture cumulative token usage from the final iteration so the Runner

cmd/odek/serve.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -267,7 +267,7 @@ func newServeAgent(resolved config.ResolvedConfig, system string, sendFn func(v
267267
approver := newWSApprover(sendFn)
268268
resolved.Dangerous.Approver = approver
269269

270-
tools := builtinTools(resolved.Dangerous, sm, approver, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, nil)
270+
tools := builtinTools(resolved.Dangerous, sm, approver, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, config.VisionConfig{}, nil)
271271

272272
// Find the delegateTasksTool to wire up sub-agent log streaming
273273
var subagentTool *delegateTasksTool

cmd/odek/subagent.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -291,7 +291,7 @@ func subagentCmd(args []string) error {
291291
"./.odek/skills",
292292
)
293293
}
294-
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, nil)
294+
tools := builtinTools(resolved.Dangerous, sm, nil, resolved.MaxConcurrency, resolved.APIKey, config.TranscriptionConfig{}, config.VisionConfig{}, nil)
295295
var sandboxCleanup func() error
296296

297297
// MCP server tools

cmd/odek/subagent_contract_test.go

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -320,7 +320,7 @@ func TestSubagent_ExitCodeThree(t *testing.T) {
320320
// ── 4. delegate_tasks Tool Schema ───────────────────────────────────
321321

322322
func TestDelegateTasksTool_Exists(t *testing.T) {
323-
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, nil)
323+
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, config.VisionConfig{}, nil)
324324
if len(tools) == 0 {
325325
t.Fatal("builtinTools() returned empty slice")
326326
}
@@ -338,7 +338,7 @@ func TestDelegateTasksTool_Exists(t *testing.T) {
338338
}
339339

340340
func TestDelegateTasksTool_HasSchema(t *testing.T) {
341-
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, nil)
341+
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, config.VisionConfig{}, nil)
342342

343343
var tool odek.Tool
344344
for _, t2 := range tools {
@@ -432,7 +432,7 @@ func TestDelegateTasksTool_HasSchema(t *testing.T) {
432432
}
433433

434434
func TestDelegateTasksTool_Description(t *testing.T) {
435-
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, nil)
435+
tools := builtinTools(danger.DangerousConfig{}, nil, nil, 3, "", config.TranscriptionConfig{}, config.VisionConfig{}, nil)
436436

437437
var tool odek.Tool
438438
for _, t2 := range tools {

0 commit comments

Comments
 (0)