Skip to content

Commit aadb91d

Browse files
mohit-twelvelabsNash0x7E2
mohit-twelvelabs
and
Nash0x7E2
authored
Add TwelveLabs Pegasus video understanding plugin (#607)
* Add TwelveLabs Pegasus video understanding plugin Adds an opt-in vision-agents-plugins-twelvelabs package exposing PegasusVLM, a VideoLLM backed by TwelveLabs Pegasus. Unlike frame-by-frame VLMs it buffers recent frames, encodes a short MP4 clip, uploads it to the TwelveLabs Assets API, and streams the analysis back so the agent can reason about motion and events over time. Wired into the workspace and the agents-core optional-dependency extras the same way as existing plugins. Includes no-network unit tests and an integration test gated on TWELVELABS_API_KEY. * fix(twelvelabs): address CodeRabbit review on Pegasus VLM - Enforce >=4s clip duration and use fractions.Fraction for the encode rate so non-integer fps (e.g. 29.97) is not truncated; pad short clips by repeating the last frame. - Remove the previously-registered frame handler before reassigning to a shared VideoForwarder, and detach (rather than stop) on teardown so a shared forwarder keeps running for other subscribers. - Delete the uploaded TwelveLabs asset after analysis (best-effort in a finally block, force=True) so clips don't persist indefinitely. - Log a response length at debug level instead of the full generated text. - Drop the skipif on the opt-in integration test so it fails with a clear missing-secret message instead of silently skipping; add __init__ -> None. * Ignore mypy raises from TwelveLabs * docs(twelvelabs): add Pegasus example with run instructions Adds a runnable agent example (Pegasus VLM + Deepgram STT + ElevenLabs TTS joined to a Stream call), a README with setup/run steps, and an .env.example template. Verified end-to-end against the TwelveLabs API. * docs(twelvelabs): use "Vision Agents" branding in plugin README * ci(twelvelabs): pass TWELVELABS_API_KEY to integration tests * docs: list TwelveLabs in root README integrations * docs: add language specifier to .env fenced block in example README Addresses CodeRabbit markdownlint (MD040) note on the example README. --------- Co-authored-by: Neevash Ramdial (Nash) <mail@neevash.com>
1 parent d93f0b4 commit aadb91d

14 files changed

Lines changed: 761 additions & 3 deletions

File tree

‎.github/workflows/run_tests.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ jobs:
8686
MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
8787
SARVAM_API_KEY: ${{ secrets.SARVAM_API_KEY }}
8888
INWORLD_API_KEY: ${{ secrets.INWORLD_API_KEY }}
89+
TWELVELABS_API_KEY: ${{ secrets.TWELVELABS_API_KEY }}
8990
timeout-minutes: 30
9091
steps:
9192
- name: Checkout

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ agent = Agent(
8989

9090
**TTS:** [ElevenLabs](https://visionagents.ai/integrations/elevenlabs) · [Cartesia](https://visionagents.ai/integrations/cartesia) · [Deepgram](https://visionagents.ai/integrations/deepgram) · [AWS Polly](https://visionagents.ai/integrations/aws-polly) · [Pocket](https://visionagents.ai/integrations/pocket) · [Kokoro](https://visionagents.ai/integrations/kokoro) · [Inworld](https://visionagents.ai/integrations/inworld) · [Fish Audio](https://visionagents.ai/integrations/fish)
9191

92-
**Vision:** [Ultralytics](https://visionagents.ai/integrations/ultralytics) · [Roboflow](https://visionagents.ai/integrations/roboflow) · [Moondream](https://visionagents.ai/integrations/moondream) · [NVIDIA Cosmos](https://visionagents.ai/integrations/nvidia) · [Decart](https://visionagents.ai/integrations/decart)
92+
**Vision:** [Ultralytics](https://visionagents.ai/integrations/ultralytics) · [Roboflow](https://visionagents.ai/integrations/roboflow) · [Moondream](https://visionagents.ai/integrations/moondream) · [TwelveLabs](https://github.com/GetStream/Vision-Agents/tree/main/plugins/twelvelabs) · [NVIDIA Cosmos](https://visionagents.ai/integrations/nvidia) · [Decart](https://visionagents.ai/integrations/decart)
9393

9494
**Avatars:** [LemonSlice](https://visionagents.ai/integrations/lemonslice)
9595

‎agents-core/pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ sarvam = ["vision-agents-plugins-sarvam"]
8888
liveavatar = ["vision-agents-plugins-liveavatar"]
8989
tencent = ["vision-agents-plugins-tencent; sys_platform == 'linux'"]
9090
minimax = ["vision-agents-plugins-minimax"]
91+
twelvelabs = ["vision-agents-plugins-twelvelabs"]
9192

9293

9394
[tool.hatch.metadata]

‎plugins/twelvelabs/README.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# TwelveLabs Plugin
2+
3+
This plugin brings [TwelveLabs](https://twelvelabs.io) **Pegasus** video
4+
understanding to Vision Agents as a first-class `VideoLLM`.
5+
6+
Unlike frame-by-frame VLMs, Pegasus analyzes a short **video clip**, so it can
7+
reason about motion and events over time ("what just happened?") rather than a
8+
single still frame. Recent frames from the watched track are buffered, encoded
9+
into a short MP4 clip on demand, uploaded to the TwelveLabs Assets API, and
10+
analyzed with your prompt. The streamed answer is vocalized by the agent's TTS
11+
service.
12+
13+
## Installation
14+
15+
```bash
16+
uv add "vision-agents[twelvelabs]"
17+
# or directly
18+
uv add vision-agents-plugins-twelvelabs
19+
```
20+
21+
You can grab a free API key at https://twelvelabs.io — there's a generous free
22+
tier.
23+
24+
## Quick Start
25+
26+
```python
27+
import asyncio
28+
import os
29+
from dotenv import load_dotenv
30+
from vision_agents.core import User, Agent, Runner
31+
from vision_agents.core.agents import AgentLauncher
32+
from vision_agents.plugins import deepgram, getstream, elevenlabs, twelvelabs
33+
from vision_agents.plugins.getstream import CallSessionParticipantJoinedEvent
34+
35+
load_dotenv()
36+
37+
38+
async def create_agent(**kwargs) -> Agent:
39+
llm = twelvelabs.PegasusVLM(
40+
api_key=os.getenv("TWELVELABS_API_KEY"), # or set TWELVELABS_API_KEY
41+
)
42+
43+
agent = Agent(
44+
edge=getstream.Edge(),
45+
agent_user=User(name="My happy AI friend", id="agent"),
46+
llm=llm,
47+
tts=elevenlabs.TTS(),
48+
stt=deepgram.STT(),
49+
)
50+
return agent
51+
52+
53+
async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
54+
call = await agent.create_call(call_type, call_id)
55+
56+
@agent.events.subscribe
57+
async def on_participant_joined(event: CallSessionParticipantJoinedEvent):
58+
if event.participant.user.id != "agent":
59+
await asyncio.sleep(5) # let a few seconds of video buffer
60+
await agent.simple_response("Describe what just happened in the video")
61+
62+
async with agent.join(call):
63+
await agent.finish()
64+
65+
66+
if __name__ == "__main__":
67+
Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()
68+
```
69+
70+
## Configuration
71+
72+
### PegasusVLM Parameters
73+
74+
- `api_key`: str - TwelveLabs API key. If not provided, read from the
75+
`TWELVELABS_API_KEY` environment variable.
76+
- `model_name`: str - Pegasus model identifier (default: `"pegasus1.5"`).
77+
- `fps`: float - Frame sampling rate for the buffered clip (default: `1.0`).
78+
- `clip_seconds`: int - Length of the clip analyzed per request. Pegasus
79+
requires at least 4 seconds of video (default: `5`).
80+
- `max_tokens`: int - Maximum tokens in the response. Pegasus requires at least
81+
`512` (default: `512`).
82+
83+
## Notes
84+
85+
- Pegasus requires a minimum resolution of 360x360; lower-resolution frames are
86+
scaled up to that floor on encode.
87+
- Pegasus requires the analyzed clip to be at least 4 seconds long, so
88+
`clip_seconds` must be `>= 4`.
89+
- Each request uploads a clip and runs server-side analysis, so latency is
90+
higher than single-frame VLMs. Tune `fps` and `clip_seconds` for your use
91+
case.
92+
93+
## Testing
94+
95+
```bash
96+
# Unit tests (no API key needed)
97+
uv run pytest plugins/twelvelabs/tests -m "not integration"
98+
99+
# Integration tests (needs TWELVELABS_API_KEY)
100+
export TWELVELABS_API_KEY="your-key-here"
101+
uv run pytest plugins/twelvelabs/tests -m integration
102+
```
103+
104+
## Links
105+
106+
- [TwelveLabs Documentation](https://docs.twelvelabs.io/)
107+
- [Vision Agents Documentation](https://visionagents.ai/)
108+
- [GitHub Repository](https://github.com/GetStream/Vision-Agents)

‎plugins/twelvelabs/example/.env.example‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# TwelveLabs API key — get a free key at https://twelvelabs.io
2+
TWELVELABS_API_KEY=...
3+
4+
# Stream API credentials — used for the low-latency video edge
5+
# Free key at https://getstream.io/try-for-free/
6+
STREAM_API_KEY=...
7+
STREAM_API_SECRET=...
8+
EXAMPLE_BASE_URL=https://demo.visionagents.ai
9+
10+
# Speech-to-text and text-to-speech for the agent's voice
11+
DEEPGRAM_API_KEY=...
12+
ELEVENLABS_API_KEY=...

‎plugins/twelvelabs/example/README.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# TwelveLabs Pegasus Example
2+
3+
A small, runnable agent that demonstrates the TwelveLabs **Pegasus** `VideoLLM`.
4+
5+
Unlike frame-by-frame VLMs, Pegasus analyzes a short **video clip**, so it can
6+
reason about motion and events over time ("what just happened?"). This example
7+
joins a Stream call, buffers a few seconds of your webcam, and has the agent
8+
describe what it sees out loud.
9+
10+
- `twelvelabs_pegasus_example.py` — Pegasus VLM + Deepgram STT + ElevenLabs TTS,
11+
joined to a Stream video call.
12+
13+
## Prerequisites
14+
15+
- Python 3.10+
16+
- API keys for:
17+
- [TwelveLabs](https://twelvelabs.io) — Pegasus video understanding (generous free tier)
18+
- [Stream](https://getstream.io/try-for-free/) — low-latency video edge
19+
- [Deepgram](https://deepgram.com/) — speech-to-text
20+
- [ElevenLabs](https://elevenlabs.io/) — text-to-speech
21+
22+
## Setup
23+
24+
1. From the workspace root, install dependencies:
25+
```bash
26+
uv sync
27+
```
28+
29+
2. Create a `.env` file in the workspace root with your keys (see
30+
[`.env.example`](./.env.example)):
31+
```bash
32+
TWELVELABS_API_KEY=your_twelvelabs_key
33+
STREAM_API_KEY=your_stream_key
34+
STREAM_API_SECRET=your_stream_secret
35+
DEEPGRAM_API_KEY=your_deepgram_key
36+
ELEVENLABS_API_KEY=your_elevenlabs_key
37+
```
38+
39+
## Running
40+
41+
From the workspace root:
42+
43+
```bash
44+
uv run plugins/twelvelabs/example/twelvelabs_pegasus_example.py
45+
```
46+
47+
Open the join URL printed in the logs in your browser, turn on your camera, and
48+
wait a few seconds — the agent buffers a short clip, sends it to Pegasus, and
49+
speaks a description of what just happened.
50+
51+
## Tuning
52+
53+
Pegasus analyzes a clip rather than a single frame, so a couple of knobs matter
54+
for latency and accuracy:
55+
56+
```python
57+
twelvelabs.PegasusVLM(
58+
clip_seconds=5, # length of the analyzed clip (must be >= 4)
59+
fps=1.0, # frame sampling rate for the buffer
60+
max_tokens=512, # response length (must be >= 512)
61+
)
62+
```
63+
64+
Each request uploads a clip and runs server-side analysis, so latency is higher
65+
than single-frame VLMs. See the [plugin README](../README.md) for the full
66+
parameter list.
67+
68+
## Learn More
69+
70+
- [TwelveLabs Documentation](https://docs.twelvelabs.io/)
71+
- [Vision Agents Documentation](https://visionagents.ai/)

‎plugins/twelvelabs/example/twelvelabs_pegasus_example.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
"""TwelveLabs Pegasus video understanding demo.
2+
3+
Runs a Vision Agent that joins a Stream call, buffers a few seconds of the
4+
caller's video, and uses Pegasus to answer questions about what just happened.
5+
6+
Usage:
7+
uv run plugins/twelvelabs/example/twelvelabs_pegasus_example.py
8+
9+
Then open the join URL printed in the logs in your browser and turn on your
10+
camera; the agent will describe what it sees.
11+
"""
12+
13+
import asyncio
14+
import logging
15+
import os
16+
17+
from dotenv import load_dotenv
18+
from vision_agents.core import Agent, Runner, User
19+
from vision_agents.core.agents import AgentLauncher
20+
from vision_agents.plugins import deepgram, elevenlabs, getstream, twelvelabs
21+
from vision_agents.plugins.getstream import CallSessionParticipantJoinedEvent
22+
23+
logging.basicConfig(level=logging.INFO)
24+
logger = logging.getLogger(__name__)
25+
26+
load_dotenv()
27+
28+
29+
async def create_agent(**kwargs) -> Agent:
30+
llm = twelvelabs.PegasusVLM(
31+
api_key=os.getenv("TWELVELABS_API_KEY"),
32+
)
33+
34+
agent = Agent(
35+
edge=getstream.Edge(), # low latency edge. clients for React, iOS, Android, RN, Flutter etc.
36+
agent_user=User(name="Pegasus Vision Agent", id="agent"),
37+
instructions="Describe what just happened in the video.",
38+
llm=llm,
39+
tts=elevenlabs.TTS(),
40+
stt=deepgram.STT(),
41+
)
42+
return agent
43+
44+
45+
async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
46+
call = await agent.create_call(call_type, call_id)
47+
48+
@agent.events.subscribe
49+
async def on_participant_joined(event: CallSessionParticipantJoinedEvent):
50+
if event.participant.user.id != "agent":
51+
await asyncio.sleep(5) # let a few seconds of video buffer
52+
await agent.simple_response("Describe what just happened in the video")
53+
54+
async with agent.join(call):
55+
await agent.finish()
56+
57+
58+
if __name__ == "__main__":
59+
Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

‎plugins/twelvelabs/py.typed‎

Whitespace-only changes.

‎plugins/twelvelabs/pyproject.toml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
[build-system]
2+
requires = ["hatchling", "hatch-vcs"]
3+
build-backend = "hatchling.build"
4+
5+
[project]
6+
name = "vision-agents-plugins-twelvelabs"
7+
dynamic = ["version"]
8+
description = "TwelveLabs Pegasus video understanding plugin for Vision Agents"
9+
readme = "README.md"
10+
requires-python = ">=3.10"
11+
license = "MIT"
12+
dependencies = [
13+
"vision-agents",
14+
"twelvelabs>=1.2.8,<2",
15+
"av>=12.0.0",
16+
]
17+
18+
[project.urls]
19+
Documentation = "https://visionagents.ai/"
20+
Website = "https://visionagents.ai/"
21+
Source = "https://github.com/GetStream/Vision-Agents"
22+
23+
[tool.hatch.version]
24+
source = "vcs"
25+
raw-options = { root = "..", search_parent_directories = true, fallback_version = "0.0.0" }
26+
27+
[tool.hatch.build.targets.wheel]
28+
packages = ["vision_agents"]
29+
30+
[tool.uv.sources]
31+
vision-agents = { workspace = true }
32+
33+
[dependency-groups]
34+
dev = [
35+
"pytest>=8.4.1",
36+
"pytest-asyncio>=1.0.0",
37+
"numpy>=1.26.0",
38+
]

0 commit comments

Comments
 (0)