Skip to content

Commit 9b2b22f

Browse files
committed
feat: add vydra media provider
1 parent 7d2dc7a commit 9b2b22f

21 files changed

Lines changed: 1358 additions & 11 deletions

docs/docs.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1272,6 +1272,7 @@
12721272
"providers/together",
12731273
"providers/venice",
12741274
"providers/vercel-ai-gateway",
1275+
"providers/vydra",
12751276
"providers/vllm",
12761277
"providers/volcengine",
12771278
"providers/xai",

docs/providers/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ Looking for chat channel docs (WhatsApp/Telegram/Discord/Slack/Mattermost (plugi
6262
- [Together AI](/providers/together)
6363
- [Venice (Venice AI, privacy-focused)](/providers/venice)
6464
- [Vercel AI Gateway](/providers/vercel-ai-gateway)
65+
- [Vydra](/providers/vydra)
6566
- [vLLM (local models)](/providers/vllm)
6667
- [Volcengine (Doubao)](/providers/volcengine)
6768
- [xAI](/providers/xai)

docs/providers/vydra.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
---
2+
summary: "Use Vydra image, video, and speech in OpenClaw"
3+
read_when:
4+
- You want Vydra media generation in OpenClaw
5+
- You need Vydra API key setup guidance
6+
title: "Vydra"
7+
---
8+
9+
# Vydra
10+
11+
The bundled Vydra plugin adds:
12+
13+
- image generation via `vydra/grok-imagine`
14+
- video generation via `vydra/veo3` and `vydra/kling`
15+
- speech synthesis via Vydra's ElevenLabs-backed TTS route
16+
17+
OpenClaw uses the same `VYDRA_API_KEY` for all three capabilities.
18+
19+
## Important base URL
20+
21+
Use `https://www.vydra.ai/api/v1`.
22+
23+
Vydra's apex host (`https://vydra.ai/api/v1`) currently redirects to `www`. Some HTTP clients drop `Authorization` on that cross-host redirect, which turns a valid API key into a misleading auth failure. The bundled plugin uses the `www` base URL directly to avoid that.
24+
25+
## Setup
26+
27+
Interactive onboarding:
28+
29+
```bash
30+
openclaw onboard --auth-choice vydra-api-key
31+
```
32+
33+
Or set the env var directly:
34+
35+
```bash
36+
export VYDRA_API_KEY="vydra_live_..."
37+
```
38+
39+
## Image generation
40+
41+
Default image model:
42+
43+
- `vydra/grok-imagine`
44+
45+
Set it as the default image provider:
46+
47+
```json5
48+
{
49+
agents: {
50+
defaults: {
51+
imageGenerationModel: {
52+
primary: "vydra/grok-imagine",
53+
},
54+
},
55+
},
56+
}
57+
```
58+
59+
Current bundled support is text-to-image only. Vydra's hosted edit routes expect remote image URLs, and OpenClaw does not add a Vydra-specific upload bridge in the bundled plugin yet.
60+
61+
See [Image Generation](/tools/image-generation) for shared tool behavior.
62+
63+
## Video generation
64+
65+
Registered video models:
66+
67+
- `vydra/veo3` for text-to-video
68+
- `vydra/kling` for image-to-video
69+
70+
Set Vydra as the default video provider:
71+
72+
```json5
73+
{
74+
agents: {
75+
defaults: {
76+
videoGenerationModel: {
77+
primary: "vydra/veo3",
78+
},
79+
},
80+
},
81+
}
82+
```
83+
84+
Notes:
85+
86+
- `vydra/veo3` is bundled as text-to-video only.
87+
- `vydra/kling` currently requires a remote image URL reference. Local file uploads are rejected up front.
88+
- The bundled plugin stays conservative and does not forward undocumented style knobs such as aspect ratio, resolution, watermark, or generated audio.
89+
90+
See [Video Generation](/tools/video-generation) for shared tool behavior.
91+
92+
## Speech synthesis
93+
94+
Set Vydra as the speech provider:
95+
96+
```json5
97+
{
98+
messages: {
99+
tts: {
100+
provider: "vydra",
101+
providers: {
102+
vydra: {
103+
apiKey: "${VYDRA_API_KEY}",
104+
voiceId: "21m00Tcm4TlvDq8ikWAM",
105+
},
106+
},
107+
},
108+
},
109+
}
110+
```
111+
112+
Defaults:
113+
114+
- model: `elevenlabs/tts`
115+
- voice id: `21m00Tcm4TlvDq8ikWAM`
116+
117+
The bundled plugin currently exposes one known-good default voice and returns MP3 audio files.
118+
119+
## Related
120+
121+
- [Provider Directory](/providers/index)
122+
- [Image Generation](/tools/image-generation)
123+
- [Video Generation](/tools/video-generation)

docs/tools/image-generation.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
summary: "Generate and edit images using configured providers (OpenAI, Google Gemini, fal, MiniMax, ComfyUI)"
2+
summary: "Generate and edit images using configured providers (OpenAI, Google Gemini, fal, MiniMax, ComfyUI, Vydra)"
33
read_when:
44
- Generating images via the agent
55
- Configuring image generation providers and models
@@ -45,6 +45,7 @@ The agent calls `image_generate` automatically. No tool allow-listing needed —
4545
| fal | `fal-ai/flux/dev` | Yes | `FAL_KEY` |
4646
| MiniMax | `image-01` | Yes (subject reference) | `MINIMAX_API_KEY` or MiniMax OAuth (`minimax-portal`) |
4747
| ComfyUI | `workflow` | Yes (1 image, workflow-configured) | `COMFY_API_KEY` or `COMFY_CLOUD_API_KEY` for cloud |
48+
| Vydra | `grok-imagine` | No | `VYDRA_API_KEY` |
4849

4950
Use `action: "list"` to inspect available providers and models at runtime:
5051

@@ -123,13 +124,13 @@ MiniMax image generation is available through both bundled MiniMax auth paths:
123124

124125
## Provider capabilities
125126

126-
| Capability | OpenAI | Google | fal | MiniMax | ComfyUI |
127-
| --------------------- | -------------------- | -------------------- | ------------------- | -------------------------- | ---------------------------------- |
128-
| Generate | Yes (up to 4) | Yes (up to 4) | Yes (up to 4) | Yes (up to 9) | Yes (workflow-defined outputs) |
129-
| Edit/reference | Yes (up to 5 images) | Yes (up to 5 images) | Yes (1 image) | Yes (1 image, subject ref) | Yes (1 image, workflow-configured) |
130-
| Size control | Yes | Yes | Yes | No | No |
131-
| Aspect ratio | No | Yes | Yes (generate only) | Yes | No |
132-
| Resolution (1K/2K/4K) | No | Yes | Yes | No | No |
127+
| Capability | OpenAI | Google | fal | MiniMax | ComfyUI | Vydra |
128+
| --------------------- | -------------------- | -------------------- | ------------------- | -------------------------- | ---------------------------------- | ----- |
129+
| Generate | Yes (up to 4) | Yes (up to 4) | Yes (up to 4) | Yes (up to 9) | Yes (workflow-defined outputs) | Yes (1) |
130+
| Edit/reference | Yes (up to 5 images) | Yes (up to 5 images) | Yes (1 image) | Yes (1 image, subject ref) | Yes (1 image, workflow-configured) | No |
131+
| Size control | Yes | Yes | Yes | No | No | No |
132+
| Aspect ratio | No | Yes | Yes (generate only) | Yes | No | No |
133+
| Resolution (1K/2K/4K) | No | Yes | Yes | No | No | No |
133134

134135
## Related
135136

@@ -139,5 +140,6 @@ MiniMax image generation is available through both bundled MiniMax auth paths:
139140
- [Google (Gemini)](/providers/google) — Gemini image provider setup
140141
- [MiniMax](/providers/minimax) — MiniMax image provider setup
141142
- [OpenAI](/providers/openai) — OpenAI Images provider setup
143+
- [Vydra](/providers/vydra) — Vydra image, video, and speech setup
142144
- [Configuration Reference](/gateway/configuration-reference#agent-defaults)`imageGenerationModel` config
143145
- [Models](/concepts/models) — model configuration and failover

docs/tools/video-generation.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
summary: "Generate videos from text, images, or existing videos using 11 provider backends"
2+
summary: "Generate videos from text, images, or existing videos using 12 provider backends"
33
read_when:
44
- Generating videos via the agent
55
- Configuring video generation providers and models
@@ -9,7 +9,7 @@ title: "Video Generation"
99

1010
# Video Generation
1111

12-
OpenClaw agents can generate videos from text prompts, reference images, or existing videos. Eleven provider backends are supported, each with different model options, input modes, and feature sets. The agent picks the right provider automatically based on your configuration and available API keys.
12+
OpenClaw agents can generate videos from text prompts, reference images, or existing videos. Twelve provider backends are supported, each with different model options, input modes, and feature sets. The agent picks the right provider automatically based on your configuration and available API keys.
1313

1414
<Note>
1515
The `video_generate` tool only appears when at least one video-generation provider is available. If you do not see it in your agent tools, set a provider API key or configure `agents.defaults.videoGenerationModel`.
@@ -62,6 +62,7 @@ Outside of session-backed agent runs (for example, direct tool invocations), the
6262
| Qwen | `wan2.6-t2v` | Yes | Yes (remote URL) | Yes (remote URL) | `QWEN_API_KEY` |
6363
| Runway | `gen4.5` | Yes | 1 image | 1 video | `RUNWAYML_API_SECRET` |
6464
| Together | `Wan-AI/Wan2.2-T2V-A14B` | Yes | 1 image | No | `TOGETHER_API_KEY` |
65+
| Vydra | `veo3` | Yes | 1 image (`kling`) | No | `VYDRA_API_KEY` |
6566
| xAI | `grok-imagine-video` | Yes | 1 image | 1 video | `XAI_API_KEY` |
6667

6768
Some providers accept additional or alternate API key env vars. See individual [provider pages](#related) for details.
@@ -109,7 +110,7 @@ Not all providers support all parameters. Unsupported overrides are ignored on a
109110
## Actions
110111

111112
- **generate** (default) -- create a video from the given prompt and optional reference inputs.
112-
- **status** -- check the state of the in-flight video task for the current session without starting a new one.
113+
- **status** -- check the state of the in-flight video task for the current session without starting another generation.
113114
- **list** -- show available providers, models, and their capabilities.
114115

115116
## Model selection
@@ -150,6 +151,7 @@ If a provider fails, the next candidate is tried automatically. If all candidate
150151
| Qwen | Same DashScope backend as Alibaba. Reference inputs must be remote `http(s)` URLs; local files are rejected upfront. |
151152
| Runway | Supports local files via data URIs. Video-to-video requires `runway/gen4_aleph`. Text-only runs expose `16:9` and `9:16` aspect ratios. |
152153
| Together | Single image reference only. |
154+
| Vydra | Uses `https://www.vydra.ai/api/v1` directly to avoid auth-dropping redirects. `veo3` is bundled as text-to-video only; `kling` requires a remote image URL. |
153155
| xAI | Supports text-to-video, image-to-video, and remote video edit/extend flows. |
154156

155157
## Configuration
@@ -189,6 +191,7 @@ openclaw config set agents.defaults.videoGenerationModel.primary "qwen/wan2.6-t2
189191
- [Qwen](/providers/qwen)
190192
- [Runway](/providers/runway)
191193
- [Together AI](/providers/together)
194+
- [Vydra](/providers/vydra)
192195
- [xAI](/providers/xai)
193196
- [Configuration Reference](/gateway/configuration-reference#agent-defaults)
194197
- [Models](/concepts/models)

extensions/video-generation-providers.live.test.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import openaiPlugin from "./openai/index.js";
2222
import qwenPlugin from "./qwen/index.js";
2323
import runwayPlugin from "./runway/index.js";
2424
import togetherPlugin from "./together/index.js";
25+
import vydraPlugin from "./vydra/index.js";
2526
import xaiPlugin from "./xai/index.js";
2627

2728
const LIVE = isLiveTestEnabled();
@@ -65,6 +66,7 @@ const CASES: LiveProviderCase[] = [
6566
pluginName: "Together Provider",
6667
providerId: "together",
6768
},
69+
{ plugin: vydraPlugin, pluginId: "vydra", pluginName: "Vydra Provider", providerId: "vydra" },
6870
{ plugin: xaiPlugin, pluginId: "xai", pluginName: "xAI Plugin", providerId: "xai" },
6971
]
7072
.filter((entry) => (providerFilter ? providerFilter.has(entry.providerId) : true))
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
import * as providerAuth from "openclaw/plugin-sdk/provider-auth-runtime";
2+
import { afterEach, describe, expect, it, vi } from "vitest";
3+
import { buildVydraImageGenerationProvider } from "./image-generation-provider.js";
4+
5+
describe("vydra image-generation provider", () => {
6+
afterEach(() => {
7+
vi.unstubAllGlobals();
8+
vi.restoreAllMocks();
9+
});
10+
11+
it("posts to the www api and downloads the generated image", async () => {
12+
vi.spyOn(providerAuth, "resolveApiKeyForProvider").mockResolvedValue({
13+
apiKey: "vydra-test-key",
14+
source: "env",
15+
mode: "api-key",
16+
});
17+
const fetchMock = vi
18+
.fn()
19+
.mockResolvedValueOnce(
20+
new Response(
21+
JSON.stringify({
22+
jobId: "job-123",
23+
status: "completed",
24+
imageUrl: "https://cdn.vydra.ai/generated/test.png",
25+
}),
26+
{
27+
status: 200,
28+
headers: { "Content-Type": "application/json" },
29+
},
30+
),
31+
)
32+
.mockResolvedValueOnce(
33+
new Response(Buffer.from("png-data"), {
34+
status: 200,
35+
headers: { "Content-Type": "image/png" },
36+
}),
37+
);
38+
vi.stubGlobal("fetch", fetchMock);
39+
40+
const provider = buildVydraImageGenerationProvider();
41+
const result = await provider.generateImage({
42+
provider: "vydra",
43+
model: "grok-imagine",
44+
prompt: "draw a cat",
45+
cfg: {},
46+
});
47+
48+
expect(fetchMock).toHaveBeenNthCalledWith(
49+
1,
50+
"https://www.vydra.ai/api/v1/models/grok-imagine",
51+
expect.objectContaining({
52+
method: "POST",
53+
body: JSON.stringify({
54+
prompt: "draw a cat",
55+
model: "text-to-image",
56+
}),
57+
}),
58+
);
59+
const [, init] = fetchMock.mock.calls[0] as [string, RequestInit];
60+
const headers = new Headers(init.headers);
61+
expect(headers.get("authorization")).toBe("Bearer vydra-test-key");
62+
expect(result).toEqual({
63+
images: [
64+
{
65+
buffer: Buffer.from("png-data"),
66+
mimeType: "image/png",
67+
fileName: "image-1.png",
68+
},
69+
],
70+
model: "grok-imagine",
71+
metadata: {
72+
jobId: "job-123",
73+
imageUrl: "https://cdn.vydra.ai/generated/test.png",
74+
status: "completed",
75+
},
76+
});
77+
});
78+
79+
it("polls jobs when the create response is not completed yet", async () => {
80+
vi.spyOn(providerAuth, "resolveApiKeyForProvider").mockResolvedValue({
81+
apiKey: "vydra-test-key",
82+
source: "env",
83+
mode: "api-key",
84+
});
85+
const fetchMock = vi
86+
.fn()
87+
.mockResolvedValueOnce(
88+
new Response(JSON.stringify({ jobId: "job-456", status: "queued" }), {
89+
status: 200,
90+
headers: { "Content-Type": "application/json" },
91+
}),
92+
)
93+
.mockResolvedValueOnce(
94+
new Response(
95+
JSON.stringify({
96+
jobId: "job-456",
97+
status: "completed",
98+
resultUrls: ["https://cdn.vydra.ai/generated/polled.png"],
99+
}),
100+
{
101+
status: 200,
102+
headers: { "Content-Type": "application/json" },
103+
},
104+
),
105+
)
106+
.mockResolvedValueOnce(
107+
new Response(Buffer.from("png-data"), {
108+
status: 200,
109+
headers: { "Content-Type": "image/png" },
110+
}),
111+
);
112+
vi.stubGlobal("fetch", fetchMock);
113+
114+
const provider = buildVydraImageGenerationProvider();
115+
await provider.generateImage({
116+
provider: "vydra",
117+
model: "grok-imagine",
118+
prompt: "draw a cat",
119+
cfg: {},
120+
});
121+
122+
expect(fetchMock).toHaveBeenNthCalledWith(
123+
2,
124+
"https://www.vydra.ai/api/v1/jobs/job-456",
125+
expect.objectContaining({ method: "GET" }),
126+
);
127+
});
128+
});

0 commit comments

Comments
 (0)