Skip to content

Commit f6dbcf4

Browse files
steipetesteipete
committed
docs: document music generation async flow
1 parent 3027f0d commit f6dbcf4

13 files changed

Lines changed: 253 additions & 46 deletions

File tree

‎docs/concepts/models.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ Related:
3030
falls back to `agents.defaults.imageModel`, then the resolved session/default
3131
model.
3232
- `agents.defaults.imageGenerationModel` is used by the shared image-generation capability. If omitted, `image_generate` can still infer an auth-backed provider default. It tries the current default provider first, then the remaining registered image-generation providers in provider-id order. If you set a specific provider/model, also configure that provider's auth/API key.
33+
- `agents.defaults.musicGenerationModel` is used by the shared music-generation capability. If omitted, `music_generate` can still infer an auth-backed provider default. It tries the current default provider first, then the remaining registered music-generation providers in provider-id order. If you set a specific provider/model, also configure that provider's auth/API key.
3334
- `agents.defaults.videoGenerationModel` is used by the shared video-generation capability. If omitted, `video_generate` can still infer an auth-backed provider default. It tries the current default provider first, then the remaining registered video-generation providers in provider-id order. If you set a specific provider/model, also configure that provider's auth/API key.
3435
- Per-agent defaults can override `agents.defaults.model` via `agents.list[].model` plus bindings (see [/concepts/multi-agent](/concepts/multi-agent)).
3536

@@ -253,5 +254,6 @@ This applies whenever OpenClaw regenerates `models.json`, including command-driv
253254
- [Model Providers](/concepts/model-providers) — provider routing and auth
254255
- [Model Failover](/concepts/model-failover) — fallback chains
255256
- [Image Generation](/tools/image-generation) — image model configuration
257+
- [Music Generation](/tools/music-generation) — music model configuration
256258
- [Video Generation](/tools/video-generation) — video model configuration
257259
- [Configuration Reference](/gateway/configuration-reference#agent-defaults) — model config keys

‎docs/gateway/configuration-reference.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1026,6 +1026,11 @@ Time format in system prompt. Default: `auto` (OS preference).
10261026
- Typical values: `google/gemini-3.1-flash-image-preview` for native Gemini image generation, `fal/fal-ai/flux/dev` for fal, or `openai/gpt-image-1` for OpenAI Images.
10271027
- If you select a provider/model directly, configure the matching provider auth/API key too (for example `GEMINI_API_KEY` or `GOOGLE_API_KEY` for `google/*`, `OPENAI_API_KEY` for `openai/*`, `FAL_KEY` for `fal/*`).
10281028
- If omitted, `image_generate` can still infer an auth-backed provider default. It tries the current default provider first, then the remaining registered image-generation providers in provider-id order.
1029+
- `musicGenerationModel`: accepts either a string (`"provider/model"`) or an object (`{ primary, fallbacks }`).
1030+
- Used by the shared music-generation capability and the built-in `music_generate` tool.
1031+
- Typical values: `google/lyria-3-clip-preview`, `google/lyria-3-pro-preview`, or `minimax/music-2.5+`.
1032+
- If omitted, `music_generate` can still infer an auth-backed provider default. It tries the current default provider first, then the remaining registered music-generation providers in provider-id order.
1033+
- If you select a provider/model directly, configure the matching provider auth/API key too.
10291034
- `videoGenerationModel`: accepts either a string (`"provider/model"`) or an object (`{ primary, fallbacks }`).
10301035
- Used by the shared video-generation capability and the built-in `video_generate` tool.
10311036
- Typical values: `qwen/wan2.6-t2v`, `qwen/wan2.6-i2v`, `qwen/wan2.6-r2v`, `qwen/wan2.6-r2v-flash`, or `qwen/wan2.7-r2v`.

‎docs/plugins/architecture.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ native OpenClaw plugin registers against one or more capability types:
3535
| Realtime voice | `api.registerRealtimeVoiceProvider(...)` | `openai` |
3636
| Media understanding | `api.registerMediaUnderstandingProvider(...)` | `openai`, `google` |
3737
| Image generation | `api.registerImageGenerationProvider(...)` | `openai`, `google`, `fal`, `minimax` |
38+
| Music generation | `api.registerMusicGenerationProvider(...)` | `google`, `minimax` |
3839
| Video generation | `api.registerVideoGenerationProvider(...)` | `qwen` |
3940
| Web fetch | `api.registerWebFetchProvider(...)` | `firecrawl` |
4041
| Web search | `api.registerWebSearchProvider(...)` | `google` |

‎docs/plugins/building-plugins.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,7 @@ A single plugin can register any number of capabilities via the `api` object:
157157
| Realtime voice | `api.registerRealtimeVoiceProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |
158158
| Media understanding | `api.registerMediaUnderstandingProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |
159159
| Image generation | `api.registerImageGenerationProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |
160+
| Music generation | `api.registerMusicGenerationProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |
160161
| Video generation | `api.registerVideoGenerationProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |
161162
| Web fetch | `api.registerWebFetchProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |
162163
| Web search | `api.registerWebSearchProvider(...)` | [Provider Plugins](/plugins/sdk-provider-plugins#step-5-add-extra-capabilities) |

‎docs/plugins/manifest.md‎

Lines changed: 20 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -128,26 +128,26 @@ Those belong in your plugin code and `package.json`.
128128

129129
## Top-level field reference
130130

131-
| Field | Required | Type | What it means |
132-
| ----------------------------------- | -------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
133-
| `id` | Yes | `string` | Canonical plugin id. This is the id used in `plugins.entries.<id>`. |
134-
| `configSchema` | Yes | `object` | Inline JSON Schema for this plugin's config. |
135-
| `enabledByDefault` | No | `true` | Marks a bundled plugin as enabled by default. Omit it, or set any non-`true` value, to leave the plugin disabled by default. |
136-
| `legacyPluginIds` | No | `string[]` | Legacy ids that normalize to this canonical plugin id. |
137-
| `autoEnableWhenConfiguredProviders` | No | `string[]` | Provider ids that should auto-enable this plugin when auth, config, or model refs mention them. |
138-
| `kind` | No | `"memory"` \| `"context-engine"` | Declares an exclusive plugin kind used by `plugins.slots.*`. |
139-
| `channels` | No | `string[]` | Channel ids owned by this plugin. Used for discovery and config validation. |
140-
| `providers` | No | `string[]` | Provider ids owned by this plugin. |
141-
| `modelSupport` | No | `object` | Manifest-owned shorthand model-family metadata used to auto-load the plugin before runtime. |
142-
| `providerAuthEnvVars` | No | `Record<string, string[]>` | Cheap provider-auth env metadata that OpenClaw can inspect without loading plugin code. |
143-
| `providerAuthChoices` | No | `object[]` | Cheap auth-choice metadata for onboarding pickers, preferred-provider resolution, and simple CLI flag wiring. |
144-
| `contracts` | No | `object` | Static bundled capability snapshot for speech, realtime transcription, realtime voice, media-understanding, image-generation, video-generation, web-fetch, web search, and tool ownership. |
145-
| `channelConfigs` | No | `Record<string, object>` | Manifest-owned channel config metadata merged into discovery and validation surfaces before runtime loads. |
146-
| `skills` | No | `string[]` | Skill directories to load, relative to the plugin root. |
147-
| `name` | No | `string` | Human-readable plugin name. |
148-
| `description` | No | `string` | Short summary shown in plugin surfaces. |
149-
| `version` | No | `string` | Informational plugin version. |
150-
| `uiHints` | No | `Record<string, object>` | UI labels, placeholders, and sensitivity hints for config fields. |
131+
| Field | Required | Type | What it means |
132+
| ----------------------------------- | -------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
133+
| `id` | Yes | `string` | Canonical plugin id. This is the id used in `plugins.entries.<id>`. |
134+
| `configSchema` | Yes | `object` | Inline JSON Schema for this plugin's config. |
135+
| `enabledByDefault` | No | `true` | Marks a bundled plugin as enabled by default. Omit it, or set any non-`true` value, to leave the plugin disabled by default. |
136+
| `legacyPluginIds` | No | `string[]` | Legacy ids that normalize to this canonical plugin id. |
137+
| `autoEnableWhenConfiguredProviders` | No | `string[]` | Provider ids that should auto-enable this plugin when auth, config, or model refs mention them. |
138+
| `kind` | No | `"memory"` \| `"context-engine"` | Declares an exclusive plugin kind used by `plugins.slots.*`. |
139+
| `channels` | No | `string[]` | Channel ids owned by this plugin. Used for discovery and config validation. |
140+
| `providers` | No | `string[]` | Provider ids owned by this plugin. |
141+
| `modelSupport` | No | `object` | Manifest-owned shorthand model-family metadata used to auto-load the plugin before runtime. |
142+
| `providerAuthEnvVars` | No | `Record<string, string[]>` | Cheap provider-auth env metadata that OpenClaw can inspect without loading plugin code. |
143+
| `providerAuthChoices` | No | `object[]` | Cheap auth-choice metadata for onboarding pickers, preferred-provider resolution, and simple CLI flag wiring. |
144+
| `contracts` | No | `object` | Static bundled capability snapshot for speech, realtime transcription, realtime voice, media-understanding, image-generation, music-generation, video-generation, web-fetch, web search, and tool ownership. |
145+
| `channelConfigs` | No | `Record<string, object>` | Manifest-owned channel config metadata merged into discovery and validation surfaces before runtime loads. |
146+
| `skills` | No | `string[]` | Skill directories to load, relative to the plugin root. |
147+
| `name` | No | `string` | Human-readable plugin name. |
148+
| `description` | No | `string` | Short summary shown in plugin surfaces. |
149+
| `version` | No | `string` | Informational plugin version. |
150+
| `uiHints` | No | `Record<string, object>` | UI labels, placeholders, and sensitivity hints for config fields. |
151151

152152
## providerAuthChoices reference
153153

‎docs/plugins/sdk-migration.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,8 @@ Current bundled provider examples:
263263
| `plugin-sdk/realtime-transcription` | Realtime transcription helpers | Provider types and registry helpers |
264264
| `plugin-sdk/realtime-voice` | Realtime voice helpers | Provider types and registry helpers |
265265
| `plugin-sdk/image-generation-core` | Shared image-generation core | Image-generation types, failover, auth, and registry helpers |
266+
| `plugin-sdk/music-generation` | Music-generation helpers | Music-generation provider/request/result types |
267+
| `plugin-sdk/music-generation-core` | Shared music-generation core | Music-generation types, failover helpers, provider lookup, and model-ref parsing |
266268
| `plugin-sdk/video-generation` | Video-generation helpers | Video-generation provider/request/result types |
267269
| `plugin-sdk/video-generation-core` | Shared video-generation core | Video-generation types, failover helpers, provider lookup, and model-ref parsing |
268270
| `plugin-sdk/interactive-runtime` | Interactive reply helpers | Interactive reply payload normalization/reduction |

‎docs/plugins/sdk-overview.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,8 @@ explicitly promotes one as public.
232232
| `plugin-sdk/realtime-voice` | Realtime voice provider types and registry helpers |
233233
| `plugin-sdk/image-generation` | Image generation provider types |
234234
| `plugin-sdk/image-generation-core` | Shared image-generation types, failover, auth, and registry helpers |
235+
| `plugin-sdk/music-generation` | Music generation provider/request/result types |
236+
| `plugin-sdk/music-generation-core` | Shared music-generation types, failover helpers, provider lookup, and model-ref parsing |
235237
| `plugin-sdk/video-generation` | Video generation provider/request/result types |
236238
| `plugin-sdk/video-generation-core` | Shared video-generation types, failover helpers, provider lookup, and model-ref parsing |
237239
| `plugin-sdk/webhook-targets` | Webhook target registry and route-install helpers |
@@ -288,6 +290,7 @@ methods:
288290
| `api.registerRealtimeVoiceProvider(...)` | Duplex realtime voice sessions |
289291
| `api.registerMediaUnderstandingProvider(...)` | Image/audio/video analysis |
290292
| `api.registerImageGenerationProvider(...)` | Image generation |
293+
| `api.registerMusicGenerationProvider(...)` | Music generation |
291294
| `api.registerVideoGenerationProvider(...)` | Video generation |
292295
| `api.registerWebFetchProvider(...)` | Web fetch / scrape provider |
293296
| `api.registerWebSearchProvider(...)` | Web search |

0 commit comments

Comments
 (0)