Skip to content

Commit 2a5da61

Browse files
committed
docs: refresh media auto-detect refs
1 parent 459ede5 commit 2a5da61

3 files changed

Lines changed: 31 additions & 14 deletions

File tree

docs/nodes/audio.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,15 @@ title: "Audio and Voice Notes"
2323
If you **don’t configure models** and `tools.media.audio.enabled` is **not** set to `false`,
2424
OpenClaw auto-detects in this order and stops at the first working option:
2525

26-
1. **Local CLIs** (if installed)
26+
1. **Active reply model** when its provider supports audio understanding.
27+
2. **Local CLIs** (if installed)
2728
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
2829
- `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
2930
- `whisper` (Python CLI; downloads models automatically)
30-
2. **Gemini CLI** (`gemini`) using `read_many_files`
31-
3. **Provider keys** (OpenAI → Groq → Deepgram → Google)
31+
3. **Gemini CLI** (`gemini`) using `read_many_files`
32+
4. **Provider auth**
33+
- Configured `models.providers.*` entries that support audio are tried first
34+
- Bundled fallback order: OpenAI → Groq → Deepgram → Google → Mistral
3235

3336
To disable auto-detection, set `tools.media.audio.enabled: false`.
3437
To customize, set `tools.media.audio.models`.

docs/nodes/images.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ The WhatsApp channel runs via **Baileys Web**. This document captures the curren
4848
- Media understanding (if configured via `tools.media.*` or shared `tools.media.models`) runs before templating and can insert `[Image]`, `[Audio]`, and `[Video]` blocks into `Body`.
4949
- Audio sets `{{Transcript}}` and uses the transcript for command parsing so slash commands still work.
5050
- Video and image descriptions preserve any caption text for command parsing.
51+
- If the active primary image model already supports vision natively, OpenClaw skips the `[Image]` summary block and passes the original image to the model instead.
5152
- By default only the first matching image/audio/video attachment is processed; set `tools.media.<cap>.attachments` to process multiple attachments.
5253

5354
## Limits & Errors

docs/nodes/media-understanding.md

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,9 @@ Rules:
133133
- Audio files smaller than **1024 bytes** are treated as empty/corrupt and skipped before provider/CLI transcription.
134134
- If the model returns more than `maxChars`, output is trimmed.
135135
- `prompt` defaults to simple “Describe the {media}.” plus the `maxChars` guidance (image/video only).
136+
- If the active primary image model already supports vision natively, OpenClaw
137+
skips the `[Image]` summary block and passes the original image into the
138+
model instead.
136139
- If `<capability>.enabled: true` but no models are configured, OpenClaw tries the
137140
**active reply model** when its provider supports the capability.
138141

@@ -142,15 +145,22 @@ If `tools.media.<capability>.enabled` is **not** set to `false` and you haven’
142145
configured models, OpenClaw auto-detects in this order and **stops at the first
143146
working option**:
144147

145-
1. **Local CLIs** (audio only; if installed)
148+
1. **Active reply model** when its provider supports the capability.
149+
2. **`agents.defaults.imageModel`** primary/fallback refs (image only).
150+
3. **Local CLIs** (audio only; if installed)
146151
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
147152
- `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
148153
- `whisper` (Python CLI; downloads models automatically)
149-
2. **Gemini CLI** (`gemini`) using `read_many_files`
150-
3. **Provider keys**
151-
- Audio: OpenAI → Groq → Deepgram → Google → Mistral
152-
- Image: OpenAI → Anthropic → Google → MiniMax → MiniMax Portal → Z.AI
153-
- Video: Google
154+
4. **Gemini CLI** (`gemini`) using `read_many_files`
155+
5. **Provider auth**
156+
- Configured `models.providers.*` entries that support the capability are
157+
tried before the bundled fallback order.
158+
- Image-only config providers with an image-capable model auto-register for
159+
media understanding even when they are not a bundled vendor plugin.
160+
- Bundled fallback order:
161+
- Audio: OpenAI → Groq → Deepgram → Google → Mistral
162+
- Image: OpenAI → Anthropic → Google → MiniMax → MiniMax Portal → Z.AI
163+
- Video: Google → Moonshot
154164

155165
To disable auto-detection, set:
156166

@@ -190,22 +200,25 @@ lists, OpenClaw can infer defaults:
190200
- `openai`, `anthropic`, `minimax`: **image**
191201
- `minimax-portal`: **image**
192202
- `moonshot`: **image + video**
203+
- `openrouter`: **image**
193204
- `google` (Gemini API): **image + audio + video**
194205
- `mistral`: **audio**
195206
- `zai`: **image**
196207
- `groq`: **audio**
197208
- `deepgram`: **audio**
209+
- Any `models.providers.<id>.models[]` catalog with an image-capable model:
210+
**image**
198211

199212
For CLI entries, **set `capabilities` explicitly** to avoid surprising matches.
200213
If you omit `capabilities`, the entry is eligible for the list it appears in.
201214

202215
## Provider support matrix (OpenClaw integrations)
203216

204-
| Capability | Provider integration | Notes |
205-
| ---------- | -------------------------------------------------- | ----------------------------------------------------------------------- |
206-
| Image | OpenAI, Anthropic, Google, MiniMax, Moonshot, Z.AI | Vendor plugins register image support against core media understanding. |
207-
| Audio | OpenAI, Groq, Deepgram, Google, Mistral | Provider transcription (Whisper/Deepgram/Gemini/Voxtral). |
208-
| Video | Google, Moonshot | Provider video understanding via vendor plugins. |
217+
| Capability | Provider integration | Notes |
218+
| ---------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
219+
| Image | OpenAI, OpenRouter, Anthropic, Google, MiniMax, Moonshot, Z.AI, config providers | Vendor plugins register image support; image-capable config providers auto-register. |
220+
| Audio | OpenAI, Groq, Deepgram, Google, Mistral | Provider transcription (Whisper/Deepgram/Gemini/Voxtral). |
221+
| Video | Google, Moonshot | Provider video understanding via vendor plugins. |
209222

210223
## Model selection guidance
211224

0 commit comments

Comments
 (0)