Add Telnyx media streaming plugin (#594)

* Add Telnyx media streaming plugin

* Address Telnyx media stream review

* docs(telnyx): record example implementation plan

* feat(telnyx): add runnable phone examples

* test(telnyx): cover example setup helpers

* docs(telnyx): document phone example setup

* docs: surface Telnyx phone examples

* docs(telnyx): remove implementation plan artifact

* fix(telnyx): address CodeRabbit review on phone examples

Move example helpers into the plugin package, verify Telnyx webhook
signatures, redact PII from logs, and harden async cleanup in the
runnable inbound/outbound examples.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(telnyx): treat malformed webhook input as verification failures

Map invalid timestamps, payloads, and key material to
TelnyxWebhookVerificationError instead of leaking parse errors.

Co-authored-by: Cursor <cursoragent@cursor.com>

* style(telnyx): apply ruff formatting to example helpers and tests

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: a692570 <a692570@users.noreply.github.com>
Co-authored-by: Neevash Ramdial (Nash) <mail@neevash.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 3 people

README.md

@@ -73,7 +73,7 @@ agent = Agent(
 | **Video Processing**     | Pluggable processor pipeline for YOLO, Roboflow, or custom PyTorch/ONNX models before/after LLM calls. |
 | **Turn Detection**       | Natural conversation flow with VAD, diarization, and smart turn-taking.                                 |
 | **Tool Calling & MCP**   | Execute code and APIs mid-conversation — Linear issues, weather, telephony, or any MCP server.          |
-| **Phone Integration**    | Inbound and outbound voice calls via Twilio with bidirectional audio streaming.                         |
+| **Phone Integration**    | Inbound and outbound voice calls via Twilio or Telnyx with bidirectional audio streaming.               |
 | **RAG**                  | Retrieval-augmented generation with TurboPuffer/Qdrant vector search or Gemini FileSearch.                     |
 | **Memory**               | Agents recall context across turns and sessions via Stream Chat.                                        |
 | **Text Back-channel**    | Message the agent silently during a call — coaching overlays, silent instructions, etc.                 |
@@ -95,7 +95,7 @@ agent = Agent(
 
 **Turn Detection:** [Vogent](https://visionagents.ai/integrations/vogent) · [Smart Turn](https://visionagents.ai/integrations/smart-turn)
 
-**Other:** [Twilio](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [TurboPuffer](https://visionagents.ai/guides/rag)
+**Other:** [Twilio](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [Telnyx](https://github.com/GetStream/Vision-Agents/tree/main/plugins/telnyx/examples) · [TurboPuffer](https://visionagents.ai/guides/rag)
 
 ## Documentation
 
@@ -115,7 +115,7 @@ Check out the full docs at [VisionAgents.ai](https://visionagents.ai/).
 | <br><h3>Realtime Coaching and Video Understanding</h3>Power interactive coaching flows with live pose tracking and processor pipelines for frame-by-frame understanding.<br><br>• Real-time pose tracking<br>• Actionable coaching feedback<br>• Video processor pipeline support<br><br> [>Source Code and tutorial](https://github.com/GetStream/Vision-Agents/tree/main/examples/02_golf_coach_example)                                                     | <img src="assets/demo_gifs/golf.gif" width="320" alt="Realtime Coaching Demo">                 |
 | <br><h3>Video Restyling and Avatars</h3>Use models like Decart Lucy to build virtual try-ons, stylized scenes, or give your agents a visual identity.<br><br>• Real-time video restyling<br>• Virtual try-on experiences<br>• Avatar-like visual presence<br><br> [>Source Code and tutorial](https://github.com/GetStream/Vision-Agents/tree/main/plugins/decart/example)                                                                                                    | <img src="assets/demo_gifs/mirage.gif" width="320" alt="Video Restyling Demo">           |
 | <br><h3>Custom Video Models (Roboflow, YOLO, and More)</h3>Train and run custom computer vision models for security monitoring, moderation, and other domain-specific workflows.<br><br>• Bring your own CV models<br>• Real-time moderation pipelines<br>• Security and detection use cases<br><br> [>Source Code and tutorial](https://github.com/GetStream/Vision-Agents/tree/main/examples/11_moderation_example) | <img src="assets/demo_gifs/security_camera.gif" width="320" alt="Custom Video Models Demo">          |
-| <br><h3>Tools, MCP, and Phone Calling</h3>Connect external APIs and services so agents can validate data and take real-world actions during live conversations.<br><br>• MCP and function calling support<br>• Twilio-based phone workflows<br>• Real-time fraud response automation<br><br> [>Phone + RAG example](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [>Fraud workflow example](https://github.com/GetStream/Vision-Agents/tree/main/plugins/openai/examples/nemotron_example) | <img src="assets/demo_gifs/fraud_detection.gif" width="320" alt="Tools and Phone Demo"> |
+| <br><h3>Tools, MCP, and Phone Calling</h3>Connect external APIs and services so agents can validate data and take real-world actions during live conversations.<br><br>• MCP and function calling support<br>• Twilio and Telnyx phone workflows<br>• Real-time fraud response automation<br><br> [>Phone + RAG example](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [>Telnyx phone examples](https://github.com/GetStream/Vision-Agents/tree/main/plugins/telnyx/examples) · [>Fraud workflow example](https://github.com/GetStream/Vision-Agents/tree/main/plugins/openai/examples/nemotron_example) | <img src="assets/demo_gifs/fraud_detection.gif" width="320" alt="Tools and Phone Demo"> |
 
 ## Community Highlights
 

agents-core/pyproject.toml

@@ -62,6 +62,7 @@ nvidia = ["vision-agents-plugins-nvidia"]
 openai = ["vision-agents-plugins-openai"]
 roboflow = ["vision-agents-plugins-roboflow"]
 smart_turn = ["vision-agents-plugins-smart-turn"]
+telnyx = ["vision-agents-plugins-telnyx"]
 ultralytics = ["vision-agents-plugins-ultralytics"]
 wizper = ["vision-agents-plugins-wizper"]
 xai = ["vision-agents-plugins-xai"]

plugins/telnyx/README.md

@@ -0,0 +1,193 @@
+# Telnyx Plugin
+
+Telnyx plugin for Vision Agents enabling inbound and outbound phone calls with
+real-time bidirectional media streaming.
+
+## Features
+
+- **Media Streaming**: Bidirectional audio streaming via Telnyx Media Streaming
+- **Call Control**: Support for programmable inbound and outbound phone calls
+- **Call Registry**: Track active calls with metadata, stream objects, and
+  validation tokens
+- **Audio Conversion**: PCMU, PCMA, and L16 RTP payload conversion
+- **WebSocket Management**: Handle Telnyx WebSocket media events
+- **Stream Bridge**: Attach a Telnyx phone participant to a Stream call
+
+## Installation
+
+```bash
+uv add "vision-agents[telnyx]"
+# or directly
+uv add vision-agents-plugins-telnyx
+```
+
+## Usage
+
+```python
+from vision_agents.plugins import telnyx
+
+# Create a call registry to track active calls
+registry = telnyx.CallRegistry()
+
+# Register a call from your Telnyx webhook handler
+call = registry.create(
+    call_control_id="v2:abc123",
+    webhook_data={"data": {"payload": {"from": "+15551234567"}}},
+)
+
+# Create a media stream for the WebSocket connection
+stream = telnyx.MediaStream(websocket)
+await stream.accept()
+
+# Associate stream with call
+call.telnyx_stream = stream
+
+# Run the stream until Telnyx sends a stop event
+await stream.run()
+```
+
+## Examples
+
+See [examples/](examples/) for minimal inbound and outbound Telnyx phone
+examples.
+
+```bash
+# Outbound call
+uv run plugins/telnyx/examples/outbound_call.py \
+  --setup-telnyx \
+  --from +15551234567 \
+  --to +15557654321
+
+# Inbound call server
+uv run plugins/telnyx/examples/inbound_call.py \
+  --setup-telnyx \
+  --phone-number +15551234567
+```
+
+Telnyx phone calls require a Call Control App. The Call Control App is where
+Telnyx sends call webhooks such as `call.initiated`, `call.answered`, and
+`call.hangup`. It is also the `connection_id` used by the outbound Dial API. A
+forwarding-only phone-number connection is not enough for media streaming through
+this plugin.
+
+With `--setup-telnyx`, the examples create a temporary Call Control App and
+delete it on normal shutdown. The inbound example also routes the Telnyx number
+to the temporary app and restores the previous routing on shutdown.
+
+Without `--setup-telnyx`, the examples validate the common setup requirements:
+
+- `TELNYX_CALL_CONTROL_APP_ID` exists and is active
+- the Call Control App webhook URL matches `https://<NGROK_URL>/telnyx/events`
+- inbound phone numbers are routed to the Call Control App
+- restricted accounts verify outbound destination numbers before dialing
+
+## Components
+
+### TelnyxCall
+
+Dataclass representing an active call session:
+
+```python
+@dataclass
+class TelnyxCall:
+    call_control_id: str
+    token: str
+    webhook_data: Optional[dict[str, Any]]
+    telnyx_stream: Optional[TelnyxMediaStream]
+    stream_call: Optional[Any]
+    started_at: datetime
+    ended_at: Optional[datetime]
+
+    # Convenience properties from Telnyx webhook payloads
+    from_number: Optional[str]
+    to_number: Optional[str]
+    call_status: Optional[str]
+```
+
+### TelnyxCallRegistry
+
+In-memory registry for managing active calls:
+
+```python
+registry = telnyx.CallRegistry()
+registry.create(call_control_id, webhook_data=webhook_data)  # Register new call
+registry.get(call_control_id)                                # Look up call
+registry.require(call_control_id)                            # Look up or raise
+registry.validate(call_control_id, token)                    # Validate media URL token
+registry.remove(call_control_id)                             # Remove and mark ended
+registry.list_active()                                       # List active calls
+```
+
+### TelnyxMediaStream
+
+Manages Telnyx Media Streaming WebSocket connections:
+
+```python
+stream = telnyx.MediaStream(websocket)
+await stream.accept()
+
+# Access the audio track for publishing
+stream.audio_track  # AudioStreamTrack matching the Telnyx media format
+
+# Send audio back to Telnyx when bidirectional RTP streaming is enabled
+await stream.send_audio(pcm_data)
+
+# Run until the stream ends
+await stream.run()
+```
+
+To send audio back to the call, start Telnyx streaming with
+`stream_bidirectional_mode=rtp`. The plugin supports PCMU and PCMA at 8 kHz, and
+L16 at 16 kHz.
+
+Use `attach_phone_to_call` to bridge audio between a Telnyx media stream and a
+Stream call:
+
+```python
+await telnyx.attach_phone_to_call(stream_call, stream, user_id="phone-user")
+```
+
+## Audio Utilities
+
+```python
+from vision_agents.plugins.telnyx import (
+    TELNYX_DEFAULT_SAMPLE_RATE,
+    TELNYX_L16_SAMPLE_RATE,
+    l16_to_pcm,
+    pcma_to_pcm,
+    pcm_to_l16,
+    pcm_to_pcma,
+    pcm_to_pcmu,
+    pcm_to_telnyx_payload,
+    pcmu_to_pcm,
+    telnyx_payload_to_pcm,
+)
+
+pcm = pcmu_to_pcm(payload)
+payload = pcm_to_pcmu(pcm)
+```
+
+## Configuration
+
+| Parameter                    | Description                          | Default |
+|------------------------------|--------------------------------------|---------|
+| `TELNYX_DEFAULT_SAMPLE_RATE` | Telnyx PCMU and PCMA sample rate     | `8000`  |
+| `TELNYX_L16_SAMPLE_RATE`     | Telnyx L16 bidirectional sample rate | `16000` |
+
+## Environment Variables
+
+- `TELNYX_API_KEY`: Your Telnyx API key for Call Control API requests.
+- `TELNYX_PHONE_NUMBER`: Telnyx caller ID or inbound number, in E.164 format.
+  You can also pass this as `--from` or `--phone-number`.
+- `NGROK_URL`: Public HTTPS hostname that forwards to your local example server.
+  The examples can also auto-detect a local ngrok tunnel.
+- `TELNYX_CALL_CONTROL_APP_ID`: Existing Call Control App ID. Required only when
+  running without `--setup-telnyx`.
+- `TELNYX_PHONE_NUMBER_ID`: Telnyx phone number resource ID. Required for
+  inbound only when running without `--setup-telnyx`.
+
+## Dependencies
+
+- vision-agents
+- numpy
+- fastapi

plugins/telnyx/examples/README.md

@@ -0,0 +1,141 @@
+# Telnyx Phone Examples
+
+Minimal inbound and outbound phone examples for the Telnyx plugin. These examples
+use Telnyx Call Control, Telnyx Media Streaming, Stream, and Gemini Realtime.
+
+## Requirements
+
+Create a `.env` file at the repo root or export these variables:
+
+```bash
+STREAM_API_KEY=
+STREAM_API_SECRET=
+GOOGLE_API_KEY=
+TELNYX_API_KEY=
+TELNYX_PUBLIC_KEY=
+```
+
+`TELNYX_PUBLIC_KEY` is the Base64 Ed25519 public key from the Telnyx Mission
+Control Portal. The examples verify webhook signatures before handling events.
+
+You also need a Telnyx phone number. You can pass it to the examples with
+`--from` or `--phone-number`, or set:
+
+```bash
+TELNYX_PHONE_NUMBER=+15551234567
+```
+
+Start ngrok before running either example:
+
+```bash
+ngrok http 8000
+```
+
+The examples auto-detect the local ngrok HTTPS tunnel. You can also set
+`NGROK_URL=example.ngrok-free.app` or pass `--ngrok-url`.
+
+## Quick Start
+
+Use `--setup-telnyx` for the most direct local development flow. The example
+creates a temporary Telnyx Call Control App with webhook URL
+`https://<NGROK_URL>/telnyx/events` and deletes it on normal shutdown.
+
+Outbound:
+
+```bash
+uv run plugins/telnyx/examples/outbound_call.py \
+  --setup-telnyx \
+  --from +15551234567 \
+  --to +15557654321
+```
+
+Inbound:
+
+```bash
+uv run plugins/telnyx/examples/inbound_call.py \
+  --setup-telnyx \
+  --phone-number +15551234567
+```
+
+For inbound calls, `--setup-telnyx` also routes the Telnyx number to the
+temporary Call Control App and restores the previous routing on normal shutdown.
+
+Restricted Telnyx accounts may only call verified destination numbers. The
+outbound example checks that by default. If your account is unrestricted, you can
+skip that preflight:
+
+```bash
+uv run plugins/telnyx/examples/outbound_call.py \
+  --setup-telnyx \
+  --from +15551234567 \
+  --to +15557654321 \
+  --skip-verified-destination-check
+```
+
+## Manual Telnyx Setup
+
+If you do not want the examples to create or route Telnyx resources, configure
+Telnyx yourself and omit `--setup-telnyx`.
+
+1. Create a Telnyx Call Control App.
+2. Set the app webhook URL to:
+
+   ```text
+   https://<NGROK_URL>/telnyx/events
+   ```
+
+3. For inbound calls, route your Telnyx phone number to that Call Control App.
+4. For outbound calls, make sure the Call Control App has an outbound voice
+   profile that can call your target country.
+5. If your Telnyx account is restricted, verify the destination phone number in
+   Telnyx before running the outbound example.
+
+Then set:
+
+```bash
+TELNYX_CALL_CONTROL_APP_ID=
+TELNYX_PHONE_NUMBER=+15551234567
+NGROK_URL=example.ngrok-free.app
+```
+
+Inbound manual setup also needs the Telnyx phone number resource ID:
+
+```bash
+TELNYX_PHONE_NUMBER_ID=
+```
+
+Run manually configured outbound:
+
+```bash
+uv run plugins/telnyx/examples/outbound_call.py --to +15557654321
+```
+
+Run manually configured inbound:
+
+```bash
+uv run plugins/telnyx/examples/inbound_call.py
+```
+
+## Why Call Control Is Required
+
+Telnyx media streaming for programmable calls is configured through a Call
+Control App. A regular phone-number connection, including a forwarding-only
+connection, is not enough for these examples because the app needs a webhook URL
+where Telnyx can send call events and receive answer/dial commands.
+
+For outbound calls, the Call Control App ID is passed to the Telnyx Dial API as
+`connection_id`.
+
+For inbound calls, the Telnyx phone number must be routed to the same Call
+Control App so Telnyx sends `call.initiated` webhooks to this example.
+
+## Common Setup Errors
+
+- `Invalid value for connection_id`: the Call Control App ID is missing,
+  invalid, or inactive.
+- Webhook URL mismatch: update the Call Control App webhook URL to the current
+  ngrok URL, or use `--setup-telnyx`.
+- Inbound number not routed: assign the Telnyx phone number to the Call Control
+  App, or use `--setup-telnyx`.
+- Destination not verified: verify the `--to` number in Telnyx or use an
+  unrestricted account.

plugins/telnyx/examples/__init__.py

@@ -0,0 +1 @@
+"""Minimal Telnyx phone examples for Vision Agents."""