Skip to content

Commit d3cb8ab

Browse files
docs: Add video generation support to Vercel AI SDK documentation (#571)
2 parents 5eb8e48 + 34b0a2b commit d3cb8ab

1 file changed

Lines changed: 107 additions & 2 deletions

File tree

public-endpoints/ai-sdk.mdx

Lines changed: 107 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,13 @@ description: "Use the @runpod/ai-sdk-provider package to integrate Public Endpoi
55
tag: "NEW"
66
---
77

8-
The `@runpod/ai-sdk-provider` package integrates Runpod Public Endpoints with the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). This gives you a streamlined, type-safe interface for text generation, streaming, and image generation in JavaScript and TypeScript projects.
8+
The `@runpod/ai-sdk-provider` package integrates Runpod Public Endpoints with the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). This gives you a streamlined, type-safe interface for text generation, streaming, image generation, and video generation in JavaScript and TypeScript projects.
99

1010
The Vercel AI SDK is a popular open-source library for building AI applications. By using the Runpod provider, you can access Runpod's Public Endpoints using the same patterns and APIs you'd use with other AI providers like OpenAI or Anthropic.
1111

1212
## Why use the Vercel AI SDK?
1313

14-
- **Unified interface**: Use the same `generateText`, `streamText`, and `generateImage` functions regardless of which AI provider you're using.
14+
- **Unified interface**: Use the same `generateText`, `streamText`, `generateImage`, and `generateVideo` functions regardless of which AI provider you're using.
1515
- **Type safety**: Full TypeScript support with typed responses and parameters.
1616
- **Streaming built-in**: First-class support for streaming text responses.
1717
- **Framework integrations**: Works seamlessly with Next.js, React, Svelte, and other frameworks.
@@ -261,6 +261,91 @@ const { image } = await generateImage({
261261
| `maxPollAttempts` | Max polling attempts for async generation |
262262
| `pollIntervalMillis` | Milliseconds between status polls |
263263

264+
## Video generation
265+
266+
Use `experimental_generateVideo` to generate videos from text prompts or images. The Runpod provider supports 15 video models, including Sora, Wan, Seedance, and Kling.
267+
268+
Video generation is asynchronous—the SDK submits a job, polls for completion, and returns the video URL when ready.
269+
270+
### Text-to-video
271+
272+
Generate videos from text prompts:
273+
274+
```typescript
275+
import { runpod } from "@runpod/ai-sdk-provider";
276+
import { experimental_generateVideo as generateVideo } from "ai";
277+
278+
const { video } = await generateVideo({
279+
model: runpod.video("alibaba/wan-2.6-t2v"),
280+
prompt: "A golden retriever running on a sunny beach, cinematic, 4k",
281+
});
282+
283+
console.log(video.url);
284+
```
285+
286+
The response includes:
287+
- `video.url`: URL to the generated video
288+
- `video.mediaType`: Video MIME type (`video/mp4`)
289+
290+
### Image-to-video
291+
292+
Animate an existing image:
293+
294+
```typescript
295+
import { runpod } from "@runpod/ai-sdk-provider";
296+
import { experimental_generateVideo as generateVideo } from "ai";
297+
298+
const { video } = await generateVideo({
299+
model: runpod.video("alibaba/wan-2.6-i2v"),
300+
prompt: "Animate this scene with gentle camera movement",
301+
image: new URL("https://example.com/image.png"),
302+
});
303+
304+
console.log(video.url);
305+
```
306+
307+
### Video generation parameters
308+
309+
Control the video generation with additional parameters:
310+
311+
```typescript
312+
const { video } = await generateVideo({
313+
model: runpod.video("alibaba/wan-2.6-t2v"),
314+
prompt: "A serene mountain landscape with flowing water",
315+
duration: 5,
316+
aspectRatio: "16:9",
317+
seed: 42,
318+
});
319+
```
320+
321+
### Video provider options
322+
323+
Pass model-specific parameters using `providerOptions`:
324+
325+
```typescript
326+
const { video } = await generateVideo({
327+
model: runpod.video("alibaba/wan-2.6-t2v"),
328+
prompt: "A serene mountain landscape with flowing water",
329+
duration: 5,
330+
aspectRatio: "16:9",
331+
providerOptions: {
332+
runpod: {
333+
negative_prompt: "blurry, low quality",
334+
guidance_scale: 7.5,
335+
},
336+
},
337+
});
338+
```
339+
340+
| Option | Description |
341+
|--------|-------------|
342+
| `negative_prompt` | Elements to exclude from the video |
343+
| `guidance_scale` | How closely to follow the prompt |
344+
| `num_inference_steps` | Number of inference steps |
345+
| `style` | Style preset (model-specific) |
346+
| `maxPollAttempts` | Max polling attempts (default: 120) |
347+
| `pollIntervalMillis` | Milliseconds between status polls (default: 5000) |
348+
264349
## Supported models
265350

266351
### Text models
@@ -281,6 +366,26 @@ const { image } = await generateImage({
281366
| `bytedance-seedream-4-0-t2i` | [Seedream 4.0](/public-endpoints/models/seedream-4-t2i). Text-to-image with good prompt adherence. |
282367
| `tongyi-mai/z-image-turbo` | [Z-Image Turbo](/public-endpoints/models/z-image-turbo). Fast 6B parameter model with text-to-image support. |
283368

369+
### Video models
370+
371+
| Model ID | Type | Resolution | Aspect Ratios | Duration |
372+
|----------|------|------------|---------------|----------|
373+
| `pruna/p-video` | t2v | 720p, 1080p | 16:9, 9:16 | 5s |
374+
| `vidu/q3-t2v` | t2v | 720p, 1080p | 16:9, 9:16, 1:1 | 5, 10s |
375+
| `vidu/q3-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 5, 10s |
376+
| `kwaivgi/kling-v2.6-std-motion-control` | i2v + video | 720p | 16:9, 9:16, 1:1 | 5, 10s |
377+
| `kwaivgi/kling-video-o1-r2v` | i2v | 720p | 16:9, 9:16, 1:1 | 3–10s |
378+
| `kwaivgi/kling-v2.1-i2v-pro` | i2v | 720p | 16:9, 9:16, 1:1 | 5, 10s |
379+
| `alibaba/wan-2.6-t2v` | t2v | 720p, 1080p | 16:9, 9:16 | 5, 10, 15s |
380+
| `alibaba/wan-2.6-i2v` | i2v | 720p, 1080p | 16:9, 9:16 | 5, 10, 15s |
381+
| `alibaba/wan-2.5` | i2v | 480p, 720p, 1080p | 16:9, 9:16 | 5, 10s |
382+
| `alibaba/wan-2.2-t2v-720-lora` | i2v | 720p | 16:9 | 5, 8s |
383+
| `alibaba/wan-2.2-i2v-720` | i2v | 720p | 16:9 | 5, 8s |
384+
| `alibaba/wan-2.1-i2v-720` | i2v | 720p | 16:9 | 5s |
385+
| `bytedance/seedance-v1.5-pro-i2v` | i2v | 480p, 720p | 21:9, 16:9, 9:16, 1:1, 4:3, 3:4 | 4–12s |
386+
| `openai/sora-2-pro-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 4, 8, 12s |
387+
| `openai/sora-2-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 4, 8, 12s |
388+
284389
For a complete list of available models and their parameters, see the [model reference](/public-endpoints/reference).
285390

286391
## Example: Chat application

0 commit comments

Comments
 (0)