You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* feat(telegram): unique photo filenames + caption-aware auto-vision
Two fixes for the Telegram photo flow:
1) Filename collision ("image already processed"). DownloadPhoto/DownloadVoice
named files photo_<fileID[:16]>.<ext>, but Telegram file_ids share a long
constant prefix (e.g. "AgACAgIAAxkBAAI…") — the distinguishing bytes come
*after* char 16. Truncating kept only the shared prefix, so every photo
mapped to the same filename and overwrote the last one. Now we hash the full
file_id (SHA-256, first 16 hex chars) for a genuinely unique suffix. Adds a
prefix-collision regression test.
2) Caption-aware vision. Photos can carry a caption (the user's request), which
was silently dropped, and the agent had to discover/call vision itself. Now:
- Message gains a Caption field; OnPhotoMessage receives it.
- New vision.auto_describe config (default true, mirrors auto_transcribe).
- On a photo, the bot runs the vision model FIRST (focused by the caption if
present) to extract a description, then injects "[description] + caption"
to the agent so it answers the request. Falls back to the path-based
message when auto-describe is off or vision fails.
Docker configs ship vision.auto_describe=true. Docs (CHEATSHEET, TELEGRAM)
updated. All packages build, vet clean, tests pass under -race.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(telegram): extract photo message composition into tested pure funcs
vprotocol auto-repair (§6.2 property tests). The photo-handler message
composition lived inline in an untested closure in package main, leaving the
new branching logic (caption present/absent, vision success/fallback)
unexercised — the binding weakness in the verification η.
Extract three pure functions — photoVisionPrompt, photoVisionMessage,
photoFallbackMessage — and cover them with unit tests, including a regression
that the <untrusted_content> wrapping is preserved verbatim when the
description is injected into the agent (axis 2.8). No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
returnfmt.Sprintf("🖼 Photo saved to %q with this message from the user: %q. Use the vision tool to analyze the image, then respond.", localPath, caption)
2040
+
}
2041
+
returnfmt.Sprintf("🖼 Photo received and saved to %q. Use the vision tool or shell commands to analyze and respond.", localPath)
2042
+
}
2043
+
1968
2044
// mediaTypeFromExt returns the Telegram media type for a file extension.
- Videos are sampled into evenly-spaced frames with ffmpeg; all frames analysed in one call
92
92
- Model files: `model.gguf` (~529 MB, Q4\_K\_M) + `mmproj.gguf` (~1.1 GB) — bundled in the Docker image at `/usr/local/share/minicpm-v/models/`
93
+
-**Telegram photos auto-describe** (`auto_describe`, default on): a received photo is run through the vision model first to extract a description, then the agent answers using it. Any caption you send with the photo becomes your request and focuses the extraction.
93
94
- Configure via `vision` section in config:
94
95
95
96
```json
96
97
{
97
98
"vision": {
99
+
"auto_describe": true,
98
100
"models_dir": "~/.odek/minicpm-v/models",
99
101
"binary_path": "/usr/local/bin/llama-mtmd-cli",
100
102
"video_frames": 8
101
103
}
102
104
}
103
105
```
104
106
105
-
Settings: `models_dir` (dir with `model.gguf` + `mmproj.gguf`), `binary_path` (llama-mtmd-cli path), `video_frames` (frames to sample from video, default 8).
107
+
Settings: `auto_describe` (Telegram photo → description before the agent answers, default true), `models_dir` (dir with `model.gguf` + `mmproj.gguf`), `binary_path` (llama-mtmd-cli path), `video_frames` (frames to sample from video, default 8).
All callbacks return a response string (may be empty) and an error. The `Handle` method:
@@ -294,8 +294,25 @@ Media files are saved to `~/.odek/media/` (created automatically on first downlo
294
294
295
295
- Takes a slice of `PhotoSize` IDs (Telegram sends multiple sizes)
296
296
- Uses the last (largest) photo size
297
-
- Saves as `photo_<truncated_fileID>.<ext>` (default extension: `.jpg`)
298
-
- Same fileID truncation as voice downloads
297
+
- Saves as `photo_<hash>.<ext>` (default extension: `.jpg`), where `<hash>` is the first 16 hex chars of the SHA-256 of the full Telegram `file_id`
298
+
- Hashing the **full** id avoids a collision: Telegram photo `file_id`s share a long constant prefix (e.g. `AgACAgIAAxkBAAI…`), so raw-truncating to 16 chars produced identical filenames for different photos — each overwrote the last, making the bot report a photo as "already processed". Voice downloads use the same scheme.
299
+
300
+
### Auto-Describe (Photo → Vision)
301
+
302
+
When `vision.auto_describe: true` is set in config (default) and the MiniCPM-V model is available, photos are automatically run through the local vision model before reaching the agent:
303
+
304
+
```
305
+
Photo received → DownloadPhoto (largest size to disk)
306
+
→ vision tool (llama-mtmd-cli, focused by the caption if any)
307
+
→ extracted description + the caption injected as the user message
308
+
→ agent answers the request using the description
309
+
```
310
+
311
+
If the photo has a **caption**, that text becomes the user's request and also focuses the vision extraction. The description is wrapped in `<untrusted_content>` boundaries (image text is untrusted input).
312
+
313
+
**Fallback:** If auto-describe is disabled or the vision model fails, the agent receives the file path (and caption, if any) with a suggestion to use the `vision` tool manually.
314
+
315
+
**Docker:** the official image bundles `llama-mtmd-cli` and MiniCPM-V 4.6, with `auto_describe` enabled in the shipped configs — so photo understanding works out of the box. See [../docker/README.md](../docker/README.md#image--video-understanding-out-of-the-box).
299
316
300
317
### Auto-Transcribe (Voice → Text)
301
318
@@ -535,7 +552,7 @@ The Telegram package is exhaustively tested under `-race`. Tests use:
535
552
-`httptest.NewServer` to mock Telegram API responses
536
553
- HTTP handler functions for each API endpoint (getFile, sendMessage, sendDocument, etc.)
537
554
-`t.TempDir()` + `t.Setenv("HOME", ...)` for filesystem isolation
538
-
-Long fileID truncation tests for voice/photo downloads
555
+
-Hashed fileID suffix tests for voice/photo downloads (incl. prefix-collision regression)
539
556
- Plan CRUD tests with prefix matching, ambiguous matches, and error paths
540
557
- Session manager tests with TTL expiry and cache behavior
0 commit comments