diff --git a/README.md b/README.md index 43a2722e..da05fe73 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ agent = Agent( | **Video Processing** | Pluggable processor pipeline for YOLO, Roboflow, or custom PyTorch/ONNX models before/after LLM calls. | | **Turn Detection** | Natural conversation flow with VAD, diarization, and smart turn-taking. | | **Tool Calling & MCP** | Execute code and APIs mid-conversation — Linear issues, weather, telephony, or any MCP server. | -| **Phone Integration** | Inbound and outbound voice calls via Twilio with bidirectional audio streaming. | +| **Phone Integration** | Inbound and outbound voice calls via Twilio or Telnyx with bidirectional audio streaming. | | **RAG** | Retrieval-augmented generation with TurboPuffer/Qdrant vector search or Gemini FileSearch. | | **Memory** | Agents recall context across turns and sessions via Stream Chat. | | **Text Back-channel** | Message the agent silently during a call — coaching overlays, silent instructions, etc. | @@ -95,7 +95,7 @@ agent = Agent( **Turn Detection:** [Vogent](https://visionagents.ai/integrations/vogent) · [Smart Turn](https://visionagents.ai/integrations/smart-turn) -**Other:** [Twilio](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [TurboPuffer](https://visionagents.ai/guides/rag) +**Other:** [Twilio](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [Telnyx](https://github.com/GetStream/Vision-Agents/tree/main/plugins/telnyx/examples) · [TurboPuffer](https://visionagents.ai/guides/rag) ## Documentation @@ -115,7 +115,7 @@ Check out the full docs at [VisionAgents.ai](https://visionagents.ai/). |

Realtime Coaching and Video Understanding

Power interactive coaching flows with live pose tracking and processor pipelines for frame-by-frame understanding.

• Real-time pose tracking
• Actionable coaching feedback
• Video processor pipeline support

[>Source Code and tutorial](https://github.com/GetStream/Vision-Agents/tree/main/examples/02_golf_coach_example) | Realtime Coaching Demo | |

Video Restyling and Avatars

Use models like Decart Lucy to build virtual try-ons, stylized scenes, or give your agents a visual identity.

• Real-time video restyling
• Virtual try-on experiences
• Avatar-like visual presence

[>Source Code and tutorial](https://github.com/GetStream/Vision-Agents/tree/main/plugins/decart/example) | Video Restyling Demo | |

Custom Video Models (Roboflow, YOLO, and More)

Train and run custom computer vision models for security monitoring, moderation, and other domain-specific workflows.

• Bring your own CV models
• Real-time moderation pipelines
• Security and detection use cases

[>Source Code and tutorial](https://github.com/GetStream/Vision-Agents/tree/main/examples/11_moderation_example) | Custom Video Models Demo | -|

Tools, MCP, and Phone Calling

Connect external APIs and services so agents can validate data and take real-world actions during live conversations.

• MCP and function calling support
• Twilio-based phone workflows
• Real-time fraud response automation

[>Phone + RAG example](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [>Fraud workflow example](https://github.com/GetStream/Vision-Agents/tree/main/plugins/openai/examples/nemotron_example) | Tools and Phone Demo | +|

Tools, MCP, and Phone Calling

Connect external APIs and services so agents can validate data and take real-world actions during live conversations.

• MCP and function calling support
• Twilio and Telnyx phone workflows
• Real-time fraud response automation

[>Phone + RAG example](https://github.com/GetStream/Vision-Agents/tree/main/examples/03_phone_and_rag_example) · [>Telnyx phone examples](https://github.com/GetStream/Vision-Agents/tree/main/plugins/telnyx/examples) · [>Fraud workflow example](https://github.com/GetStream/Vision-Agents/tree/main/plugins/openai/examples/nemotron_example) | Tools and Phone Demo | ## Community Highlights diff --git a/agents-core/pyproject.toml b/agents-core/pyproject.toml index 985b949d..d1c607ac 100644 --- a/agents-core/pyproject.toml +++ b/agents-core/pyproject.toml @@ -62,6 +62,7 @@ nvidia = ["vision-agents-plugins-nvidia"] openai = ["vision-agents-plugins-openai"] roboflow = ["vision-agents-plugins-roboflow"] smart_turn = ["vision-agents-plugins-smart-turn"] +telnyx = ["vision-agents-plugins-telnyx"] ultralytics = ["vision-agents-plugins-ultralytics"] wizper = ["vision-agents-plugins-wizper"] xai = ["vision-agents-plugins-xai"] diff --git a/plugins/telnyx/README.md b/plugins/telnyx/README.md new file mode 100644 index 00000000..dd4564e8 --- /dev/null +++ b/plugins/telnyx/README.md @@ -0,0 +1,193 @@ +# Telnyx Plugin + +Telnyx plugin for Vision Agents enabling inbound and outbound phone calls with +real-time bidirectional media streaming. + +## Features + +- **Media Streaming**: Bidirectional audio streaming via Telnyx Media Streaming +- **Call Control**: Support for programmable inbound and outbound phone calls +- **Call Registry**: Track active calls with metadata, stream objects, and + validation tokens +- **Audio Conversion**: PCMU, PCMA, and L16 RTP payload conversion +- **WebSocket Management**: Handle Telnyx WebSocket media events +- **Stream Bridge**: Attach a Telnyx phone participant to a Stream call + +## Installation + +```bash +uv add "vision-agents[telnyx]" +# or directly +uv add vision-agents-plugins-telnyx +``` + +## Usage + +```python +from vision_agents.plugins import telnyx + +# Create a call registry to track active calls +registry = telnyx.CallRegistry() + +# Register a call from your Telnyx webhook handler +call = registry.create( + call_control_id="v2:abc123", + webhook_data={"data": {"payload": {"from": "+15551234567"}}}, +) + +# Create a media stream for the WebSocket connection +stream = telnyx.MediaStream(websocket) +await stream.accept() + +# Associate stream with call +call.telnyx_stream = stream + +# Run the stream until Telnyx sends a stop event +await stream.run() +``` + +## Examples + +See [examples/](examples/) for minimal inbound and outbound Telnyx phone +examples. + +```bash +# Outbound call +uv run plugins/telnyx/examples/outbound_call.py \ + --setup-telnyx \ + --from +15551234567 \ + --to +15557654321 + +# Inbound call server +uv run plugins/telnyx/examples/inbound_call.py \ + --setup-telnyx \ + --phone-number +15551234567 +``` + +Telnyx phone calls require a Call Control App. The Call Control App is where +Telnyx sends call webhooks such as `call.initiated`, `call.answered`, and +`call.hangup`. It is also the `connection_id` used by the outbound Dial API. A +forwarding-only phone-number connection is not enough for media streaming through +this plugin. + +With `--setup-telnyx`, the examples create a temporary Call Control App and +delete it on normal shutdown. The inbound example also routes the Telnyx number +to the temporary app and restores the previous routing on shutdown. + +Without `--setup-telnyx`, the examples validate the common setup requirements: + +- `TELNYX_CALL_CONTROL_APP_ID` exists and is active +- the Call Control App webhook URL matches `https:///telnyx/events` +- inbound phone numbers are routed to the Call Control App +- restricted accounts verify outbound destination numbers before dialing + +## Components + +### TelnyxCall + +Dataclass representing an active call session: + +```python +@dataclass +class TelnyxCall: + call_control_id: str + token: str + webhook_data: Optional[dict[str, Any]] + telnyx_stream: Optional[TelnyxMediaStream] + stream_call: Optional[Any] + started_at: datetime + ended_at: Optional[datetime] + + # Convenience properties from Telnyx webhook payloads + from_number: Optional[str] + to_number: Optional[str] + call_status: Optional[str] +``` + +### TelnyxCallRegistry + +In-memory registry for managing active calls: + +```python +registry = telnyx.CallRegistry() +registry.create(call_control_id, webhook_data=webhook_data) # Register new call +registry.get(call_control_id) # Look up call +registry.require(call_control_id) # Look up or raise +registry.validate(call_control_id, token) # Validate media URL token +registry.remove(call_control_id) # Remove and mark ended +registry.list_active() # List active calls +``` + +### TelnyxMediaStream + +Manages Telnyx Media Streaming WebSocket connections: + +```python +stream = telnyx.MediaStream(websocket) +await stream.accept() + +# Access the audio track for publishing +stream.audio_track # AudioStreamTrack matching the Telnyx media format + +# Send audio back to Telnyx when bidirectional RTP streaming is enabled +await stream.send_audio(pcm_data) + +# Run until the stream ends +await stream.run() +``` + +To send audio back to the call, start Telnyx streaming with +`stream_bidirectional_mode=rtp`. The plugin supports PCMU and PCMA at 8 kHz, and +L16 at 16 kHz. + +Use `attach_phone_to_call` to bridge audio between a Telnyx media stream and a +Stream call: + +```python +await telnyx.attach_phone_to_call(stream_call, stream, user_id="phone-user") +``` + +## Audio Utilities + +```python +from vision_agents.plugins.telnyx import ( + TELNYX_DEFAULT_SAMPLE_RATE, + TELNYX_L16_SAMPLE_RATE, + l16_to_pcm, + pcma_to_pcm, + pcm_to_l16, + pcm_to_pcma, + pcm_to_pcmu, + pcm_to_telnyx_payload, + pcmu_to_pcm, + telnyx_payload_to_pcm, +) + +pcm = pcmu_to_pcm(payload) +payload = pcm_to_pcmu(pcm) +``` + +## Configuration + +| Parameter | Description | Default | +|------------------------------|--------------------------------------|---------| +| `TELNYX_DEFAULT_SAMPLE_RATE` | Telnyx PCMU and PCMA sample rate | `8000` | +| `TELNYX_L16_SAMPLE_RATE` | Telnyx L16 bidirectional sample rate | `16000` | + +## Environment Variables + +- `TELNYX_API_KEY`: Your Telnyx API key for Call Control API requests. +- `TELNYX_PHONE_NUMBER`: Telnyx caller ID or inbound number, in E.164 format. + You can also pass this as `--from` or `--phone-number`. +- `NGROK_URL`: Public HTTPS hostname that forwards to your local example server. + The examples can also auto-detect a local ngrok tunnel. +- `TELNYX_CALL_CONTROL_APP_ID`: Existing Call Control App ID. Required only when + running without `--setup-telnyx`. +- `TELNYX_PHONE_NUMBER_ID`: Telnyx phone number resource ID. Required for + inbound only when running without `--setup-telnyx`. + +## Dependencies + +- vision-agents +- numpy +- fastapi diff --git a/plugins/telnyx/examples/README.md b/plugins/telnyx/examples/README.md new file mode 100644 index 00000000..c95826e1 --- /dev/null +++ b/plugins/telnyx/examples/README.md @@ -0,0 +1,141 @@ +# Telnyx Phone Examples + +Minimal inbound and outbound phone examples for the Telnyx plugin. These examples +use Telnyx Call Control, Telnyx Media Streaming, Stream, and Gemini Realtime. + +## Requirements + +Create a `.env` file at the repo root or export these variables: + +```bash +STREAM_API_KEY= +STREAM_API_SECRET= +GOOGLE_API_KEY= +TELNYX_API_KEY= +TELNYX_PUBLIC_KEY= +``` + +`TELNYX_PUBLIC_KEY` is the Base64 Ed25519 public key from the Telnyx Mission +Control Portal. The examples verify webhook signatures before handling events. + +You also need a Telnyx phone number. You can pass it to the examples with +`--from` or `--phone-number`, or set: + +```bash +TELNYX_PHONE_NUMBER=+15551234567 +``` + +Start ngrok before running either example: + +```bash +ngrok http 8000 +``` + +The examples auto-detect the local ngrok HTTPS tunnel. You can also set +`NGROK_URL=example.ngrok-free.app` or pass `--ngrok-url`. + +## Quick Start + +Use `--setup-telnyx` for the most direct local development flow. The example +creates a temporary Telnyx Call Control App with webhook URL +`https:///telnyx/events` and deletes it on normal shutdown. + +Outbound: + +```bash +uv run plugins/telnyx/examples/outbound_call.py \ + --setup-telnyx \ + --from +15551234567 \ + --to +15557654321 +``` + +Inbound: + +```bash +uv run plugins/telnyx/examples/inbound_call.py \ + --setup-telnyx \ + --phone-number +15551234567 +``` + +For inbound calls, `--setup-telnyx` also routes the Telnyx number to the +temporary Call Control App and restores the previous routing on normal shutdown. + +Restricted Telnyx accounts may only call verified destination numbers. The +outbound example checks that by default. If your account is unrestricted, you can +skip that preflight: + +```bash +uv run plugins/telnyx/examples/outbound_call.py \ + --setup-telnyx \ + --from +15551234567 \ + --to +15557654321 \ + --skip-verified-destination-check +``` + +## Manual Telnyx Setup + +If you do not want the examples to create or route Telnyx resources, configure +Telnyx yourself and omit `--setup-telnyx`. + +1. Create a Telnyx Call Control App. +2. Set the app webhook URL to: + + ```text + https:///telnyx/events + ``` + +3. For inbound calls, route your Telnyx phone number to that Call Control App. +4. For outbound calls, make sure the Call Control App has an outbound voice + profile that can call your target country. +5. If your Telnyx account is restricted, verify the destination phone number in + Telnyx before running the outbound example. + +Then set: + +```bash +TELNYX_CALL_CONTROL_APP_ID= +TELNYX_PHONE_NUMBER=+15551234567 +NGROK_URL=example.ngrok-free.app +``` + +Inbound manual setup also needs the Telnyx phone number resource ID: + +```bash +TELNYX_PHONE_NUMBER_ID= +``` + +Run manually configured outbound: + +```bash +uv run plugins/telnyx/examples/outbound_call.py --to +15557654321 +``` + +Run manually configured inbound: + +```bash +uv run plugins/telnyx/examples/inbound_call.py +``` + +## Why Call Control Is Required + +Telnyx media streaming for programmable calls is configured through a Call +Control App. A regular phone-number connection, including a forwarding-only +connection, is not enough for these examples because the app needs a webhook URL +where Telnyx can send call events and receive answer/dial commands. + +For outbound calls, the Call Control App ID is passed to the Telnyx Dial API as +`connection_id`. + +For inbound calls, the Telnyx phone number must be routed to the same Call +Control App so Telnyx sends `call.initiated` webhooks to this example. + +## Common Setup Errors + +- `Invalid value for connection_id`: the Call Control App ID is missing, + invalid, or inactive. +- Webhook URL mismatch: update the Call Control App webhook URL to the current + ngrok URL, or use `--setup-telnyx`. +- Inbound number not routed: assign the Telnyx phone number to the Call Control + App, or use `--setup-telnyx`. +- Destination not verified: verify the `--to` number in Telnyx or use an + unrestricted account. diff --git a/plugins/telnyx/examples/__init__.py b/plugins/telnyx/examples/__init__.py new file mode 100644 index 00000000..813d7387 --- /dev/null +++ b/plugins/telnyx/examples/__init__.py @@ -0,0 +1 @@ +"""Minimal Telnyx phone examples for Vision Agents.""" diff --git a/plugins/telnyx/examples/inbound_call.py b/plugins/telnyx/examples/inbound_call.py new file mode 100644 index 00000000..a0f9a7c9 --- /dev/null +++ b/plugins/telnyx/examples/inbound_call.py @@ -0,0 +1,236 @@ +"""Minimal inbound Telnyx phone example. + +Run after starting ngrok and routing your Telnyx number to a Call Control App: + + NGROK_URL=example.ngrok-free.app uv run plugins/telnyx/examples/inbound_call.py +""" + +import argparse +import asyncio +import contextlib +import logging +import os +import uuid + +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, Request, WebSocket +from fastapi.responses import JSONResponse +from uvicorn.middleware.proxy_headers import ProxyHeadersMiddleware +from vision_agents.core import Agent, User +from vision_agents.plugins import gemini, getstream, telnyx +from vision_agents.plugins.getstream.stream_edge_transport import StreamEdge +from vision_agents.plugins.telnyx.example_helpers import ( + TelnyxClient, + TelnyxConfig, + TelnyxSetupError, + cleanup_telnyx_example_setup, + media_stream_url, + parse_verified_telnyx_webhook, + preflight_inbound, + prepare_telnyx_example_setup, + require_env, + require_telnyx_public_key, +) + + +logger = logging.getLogger(__name__) +logging.basicConfig(level=logging.INFO) + +load_dotenv() + +app = FastAPI() +app.add_middleware(ProxyHeadersMiddleware, trusted_hosts=["*"]) +call_registry = telnyx.TelnyxCallRegistry() +telnyx_client: TelnyxClient | None = None +telnyx_config: TelnyxConfig | None = None +telnyx_public_key: str | None = None + + +@app.exception_handler(Exception) +async def global_exception_handler(_request: Request, exc: Exception): + logger.exception("Unhandled exception: %s", exc) + return JSONResponse(status_code=500, content={"detail": "Internal server error"}) + + +async def create_agent() -> tuple[Agent, StreamEdge]: + edge = getstream.Edge() + agent = Agent( + edge=edge, + agent_user=User(id="ai-agent", name="AI Assistant"), + instructions=( + "Speak English. Keep replies short and natural. You are answering " + "an inbound Telnyx test call through Vision Agents. Start by saying " + "this is a quick inbound Telnyx bridge test and ask whether the " + "audio is clear." + ), + llm=gemini.Realtime(), + ) + return agent, edge + + +async def prepare_call(call_id: str): + agent, edge = await create_agent() + phone_user = User( + name=f"Inbound Telnyx call {call_id[:8]}", + id=f"phone-{call_id}", + ) + await edge.create_users([agent.agent_user, phone_user]) + stream_call = await agent.create_call("default", call_id) + logger.info("Prepared Stream call %s", call_id) + return agent, phone_user, stream_call + + +async def wait_for_start( + stream: telnyx.TelnyxMediaStream, timeout: float = 10.0 +) -> None: + deadline = asyncio.get_running_loop().time() + timeout + while not stream.has_started: + if asyncio.get_running_loop().time() >= deadline: + raise TimeoutError("Telnyx media stream did not start in time") + await asyncio.sleep(0.05) + + +@app.post("/telnyx/events") +async def telnyx_events(request: Request): + if telnyx_client is None or telnyx_config is None or telnyx_public_key is None: + raise RuntimeError("Telnyx example was not initialized") + + data = await parse_verified_telnyx_webhook(request, telnyx_public_key) + event_type = data.get("data", {}).get("event_type") + payload = data.get("data", {}).get("payload", {}) + logger.info("Telnyx webhook event: %s", event_type) + + if event_type == "call.initiated" and payload.get("direction") == "incoming": + call_control_id = payload["call_control_id"] + call_id = str(uuid.uuid4()) + telnyx_call = call_registry.create( + call_id, + webhook_data=data, + prepare=lambda: prepare_call(call_id), + ) + stream_url = media_stream_url( + telnyx_config.ngrok_url, + call_id, + telnyx_call.token, + ) + await asyncio.to_thread( + telnyx_client.answer_call, + call_control_id, + stream_url=stream_url, + ) + logger.info("Answered inbound Telnyx call %s", call_id) + + return {"ok": True} + + +@app.websocket("/telnyx/media/{call_id}/{token}") +async def media_stream(websocket: WebSocket, call_id: str, token: str): + telnyx_call = call_registry.validate(call_id, token) + logger.info("Media stream connected for inbound call %s", call_id) + + telnyx_stream = telnyx.TelnyxMediaStream(websocket) + await telnyx_stream.accept() + telnyx_call.telnyx_stream = telnyx_stream + stream_task = asyncio.create_task(telnyx_stream.run()) + + try: + agent, phone_user, stream_call = await telnyx_call.await_prepare() + telnyx_call.stream_call = stream_call + + await telnyx.attach_phone_to_call(stream_call, telnyx_stream, phone_user.id) + await wait_for_start(telnyx_stream) + + async with agent.join(stream_call, participant_wait_timeout=0): + await agent.simple_response( + text="Greet the caller and ask whether the inbound Telnyx audio is clear." + ) + await stream_task + finally: + if not stream_task.done(): + stream_task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await stream_task + call_registry.remove(call_id) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Run a minimal inbound Telnyx call.") + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", default=8000, type=int) + parser.add_argument( + "--phone-number-id", + default=None, + help="Telnyx phone number resource ID. Defaults to TELNYX_PHONE_NUMBER_ID.", + ) + parser.add_argument( + "--phone-number", + default=None, + help="Telnyx inbound number. Defaults to TELNYX_PHONE_NUMBER.", + ) + parser.add_argument( + "--call-control-app-id", + default=None, + help="Existing Telnyx Call Control App ID. Defaults to TELNYX_CALL_CONTROL_APP_ID.", + ) + parser.add_argument( + "--ngrok-url", + default=None, + help="Public ngrok hostname. Defaults to NGROK_URL or local ngrok autodetection.", + ) + parser.add_argument( + "--setup-telnyx", + action="store_true", + help="Create a temporary Call Control App, route the phone number, and restore it on exit.", + ) + return parser.parse_args() + + +def main() -> None: + global telnyx_client, telnyx_config, telnyx_public_key + + args = parse_args() + values = require_env( + ["STREAM_API_KEY", "STREAM_API_SECRET", "GOOGLE_API_KEY", "TELNYX_API_KEY"] + ) + telnyx_public_key = require_telnyx_public_key() + telnyx_client = TelnyxClient(values["TELNYX_API_KEY"]) + setup = prepare_telnyx_example_setup( + telnyx_client, + api_key=values["TELNYX_API_KEY"], + phone_number=args.phone_number or os.environ.get("TELNYX_PHONE_NUMBER"), + ngrok_url=args.ngrok_url or os.environ.get("NGROK_URL"), + call_control_app_id=( + args.call_control_app_id or os.environ.get("TELNYX_CALL_CONTROL_APP_ID") + ), + phone_number_id=( + args.phone_number_id or os.environ.get("TELNYX_PHONE_NUMBER_ID") + ), + setup_telnyx=args.setup_telnyx, + route_phone_number=True, + ) + telnyx_config = setup.config + resolved_phone_number_id = setup.phone_number_id or ( + args.phone_number_id or os.environ.get("TELNYX_PHONE_NUMBER_ID") + ) + if not resolved_phone_number_id: + raise TelnyxSetupError( + "Missing TELNYX_PHONE_NUMBER_ID. Pass `--setup-telnyx` to discover " + "and route the Telnyx number automatically." + ) + + try: + preflight_inbound( + telnyx_client, + config=telnyx_config, + telnyx_phone_number_id=resolved_phone_number_id, + ) + + logger.info("Inbound Telnyx runner ready for call %s", resolved_phone_number_id) + uvicorn.run(app, host=args.host, port=args.port) + finally: + cleanup_telnyx_example_setup(telnyx_client, setup) + + +if __name__ == "__main__": + main() diff --git a/plugins/telnyx/examples/outbound_call.py b/plugins/telnyx/examples/outbound_call.py new file mode 100644 index 00000000..0369eb40 --- /dev/null +++ b/plugins/telnyx/examples/outbound_call.py @@ -0,0 +1,239 @@ +"""Minimal outbound Telnyx phone example. + +Run from the repository root or this directory after starting ngrok: + + NGROK_URL=example.ngrok-free.app uv run plugins/telnyx/examples/outbound_call.py --to +15551234567 +""" + +import argparse +import asyncio +import contextlib +import logging +import os +import uuid + +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, Request, WebSocket +from vision_agents.core import Agent, User +from vision_agents.plugins import gemini, getstream, telnyx +from vision_agents.plugins.getstream.stream_edge_transport import StreamEdge +from vision_agents.plugins.telnyx.example_helpers import ( + TelnyxClient, + TelnyxConfig, + cleanup_telnyx_example_setup, + media_stream_url, + parse_verified_telnyx_webhook, + preflight_outbound, + prepare_telnyx_example_setup, + require_env, + require_telnyx_public_key, +) + + +logger = logging.getLogger(__name__) +logging.basicConfig(level=logging.INFO) + +load_dotenv() + +app = FastAPI() +call_registry = telnyx.TelnyxCallRegistry() +call_done: asyncio.Event | None = None +telnyx_public_key: str | None = None + + +@app.post("/telnyx/events") +async def telnyx_events(request: Request): + if telnyx_public_key is None: + raise RuntimeError("Telnyx example was not initialized") + + data = await parse_verified_telnyx_webhook(request, telnyx_public_key) + event_type = data.get("data", {}).get("event_type") + logger.info("Telnyx webhook event: %s", event_type) + return {"ok": True} + + +async def create_agent() -> tuple[Agent, StreamEdge]: + edge = getstream.Edge() + agent = Agent( + edge=edge, + agent_user=User(id="ai-agent", name="AI Assistant"), + instructions=( + "Speak English. Keep replies short and natural. You are making a " + "Telnyx test call through Vision Agents. Start by saying this is a " + "quick Telnyx media streaming test and ask whether the audio is clear." + ), + llm=gemini.Realtime(), + ) + return agent, edge + + +async def prepare_call(call_id: str): + agent, edge = await create_agent() + phone_user = User(name=f"Telnyx outbound {call_id[:8]}", id=f"phone-{call_id}") + await edge.create_users([agent.agent_user, phone_user]) + stream_call = await agent.create_call("default", call_id) + logger.info("Prepared Stream call %s", call_id) + return agent, phone_user, stream_call + + +async def wait_for_start( + stream: telnyx.TelnyxMediaStream, timeout: float = 10.0 +) -> None: + deadline = asyncio.get_running_loop().time() + timeout + while not stream.has_started: + if asyncio.get_running_loop().time() >= deadline: + raise TimeoutError("Telnyx media stream did not start in time") + await asyncio.sleep(0.05) + + +@app.websocket("/telnyx/media/{call_id}/{token}") +async def media_stream(websocket: WebSocket, call_id: str, token: str): + telnyx_call = call_registry.validate(call_id, token) + logger.info("Media stream connected for outbound call %s", call_id) + + telnyx_stream = telnyx.TelnyxMediaStream(websocket) + await telnyx_stream.accept() + telnyx_call.telnyx_stream = telnyx_stream + stream_task = asyncio.create_task(telnyx_stream.run()) + + try: + agent, phone_user, stream_call = await telnyx_call.await_prepare() + telnyx_call.stream_call = stream_call + + await telnyx.attach_phone_to_call(stream_call, telnyx_stream, phone_user.id) + await wait_for_start(telnyx_stream) + + async with agent.join(stream_call, participant_wait_timeout=0): + await agent.simple_response( + text="Start the call and ask whether the Telnyx audio is clear." + ) + await stream_task + finally: + if not stream_task.done(): + stream_task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await stream_task + call_registry.remove(call_id) + if call_done is not None: + call_done.set() + + +async def run_with_server( + *, + client: TelnyxClient, + config: TelnyxConfig, + to_number: str, + from_number: str, + host: str, + port: int, +) -> None: + global call_done + + call_done = asyncio.Event() + call_id = str(uuid.uuid4()) + telnyx_call = call_registry.create(call_id, prepare=lambda: prepare_call(call_id)) + stream_url = media_stream_url(config.ngrok_url, call_id, telnyx_call.token) + + server = uvicorn.Server(uvicorn.Config(app, host=host, port=port, log_level="info")) + server_task = asyncio.create_task(server.serve()) + try: + deadline = asyncio.get_running_loop().time() + 30.0 + while not server.started: + if server_task.done(): + raise RuntimeError("Uvicorn server failed to start") + if asyncio.get_running_loop().time() >= deadline: + raise TimeoutError("Uvicorn server did not start in time") + await asyncio.sleep(0.1) + + logger.info("Dialing outbound call %s", call_id) + await asyncio.to_thread( + client.dial_call, + connection_id=config.call_control_app_id, + from_number=from_number, + to_number=to_number, + stream_url=stream_url, + ) + logger.info("Telnyx dial accepted for call %s", call_id) + await call_done.wait() + finally: + server.should_exit = True + await server_task + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Run a minimal outbound Telnyx call.") + parser.add_argument("--to", dest="to_number", required=True) + parser.add_argument( + "--from", + dest="from_number", + default=None, + help="Telnyx caller ID. Defaults to TELNYX_PHONE_NUMBER.", + ) + parser.add_argument( + "--call-control-app-id", + default=None, + help="Existing Telnyx Call Control App ID. Defaults to TELNYX_CALL_CONTROL_APP_ID.", + ) + parser.add_argument( + "--ngrok-url", + default=None, + help="Public ngrok hostname. Defaults to NGROK_URL or local ngrok autodetection.", + ) + parser.add_argument( + "--setup-telnyx", + action="store_true", + help="Create a temporary Call Control App for this run and delete it on exit.", + ) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", default=8000, type=int) + parser.add_argument( + "--skip-verified-destination-check", + action="store_true", + help="Skip Telnyx verified-number preflight for unrestricted accounts.", + ) + return parser.parse_args() + + +def main() -> None: + global telnyx_public_key + + args = parse_args() + values = require_env( + ["STREAM_API_KEY", "STREAM_API_SECRET", "GOOGLE_API_KEY", "TELNYX_API_KEY"] + ) + telnyx_public_key = require_telnyx_public_key() + client = TelnyxClient(values["TELNYX_API_KEY"]) + setup = prepare_telnyx_example_setup( + client, + api_key=values["TELNYX_API_KEY"], + phone_number=args.from_number or os.environ.get("TELNYX_PHONE_NUMBER"), + ngrok_url=args.ngrok_url or os.environ.get("NGROK_URL"), + call_control_app_id=( + args.call_control_app_id or os.environ.get("TELNYX_CALL_CONTROL_APP_ID") + ), + setup_telnyx=args.setup_telnyx, + ) + try: + preflight_outbound( + client, + config=setup.config, + to_number=args.to_number, + check_verified_destination=not args.skip_verified_destination_check, + ) + asyncio.run( + run_with_server( + client=client, + config=setup.config, + to_number=args.to_number, + from_number=setup.config.phone_number, + host=args.host, + port=args.port, + ) + ) + finally: + cleanup_telnyx_example_setup(client, setup) + + +if __name__ == "__main__": + main() diff --git a/plugins/telnyx/py.typed b/plugins/telnyx/py.typed new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/plugins/telnyx/py.typed @@ -0,0 +1 @@ + diff --git a/plugins/telnyx/pyproject.toml b/plugins/telnyx/pyproject.toml new file mode 100644 index 00000000..961aedd0 --- /dev/null +++ b/plugins/telnyx/pyproject.toml @@ -0,0 +1,41 @@ +[build-system] +requires = ["hatchling", "hatch-vcs"] +build-backend = "hatchling.build" + +[project] +name = "vision-agents-plugins-telnyx" +dynamic = ["version"] +description = "Telnyx plugin for Vision Agents - Voice call integration with media streaming" +readme = "README.md" +requires-python = ">=3.10" +license = "MIT" +dependencies = [ + "vision-agents", + "numpy>=1.24.0", # capped at <2.0 via workspace override in root pyproject.toml + "cryptography>=44.0.0", + "fastapi>=0.135.1", +] + +[project.urls] +Documentation = "https://visionagents.ai/" +Website = "https://visionagents.ai/" +Source = "https://github.com/GetStream/Vision-Agents" + +[tool.hatch.version] +source = "vcs" +raw-options = { root = "..", search_parent_directories = true, fallback_version = "0.0.0" } + +[tool.hatch.build.targets.wheel] +packages = ["vision_agents"] + +[tool.hatch.build.targets.sdist] +include = ["/vision_agents"] + +[tool.uv.sources] +vision-agents = { workspace = true } + +[dependency-groups] +dev = [ + "pytest>=8.4.1", + "pytest-asyncio>=1.0.0", +] diff --git a/plugins/telnyx/tests/test_audio.py b/plugins/telnyx/tests/test_audio.py new file mode 100644 index 00000000..d15921b7 --- /dev/null +++ b/plugins/telnyx/tests/test_audio.py @@ -0,0 +1,88 @@ +"""Tests for Telnyx audio conversion.""" + +import numpy as np +from getstream.video.rtc.track_util import AudioFormat, PcmData + +from vision_agents.plugins.telnyx.audio import ( + TELNYX_DEFAULT_SAMPLE_RATE, + TELNYX_L16_SAMPLE_RATE, + l16_to_pcm, + pcma_to_pcm, + pcm_to_l16, + pcm_to_pcma, + pcm_to_pcmu, + pcm_to_telnyx_payload, + pcmu_to_pcm, + telnyx_payload_to_pcm, +) + + +def test_pcmu_to_pcm(): + pcm = pcmu_to_pcm(bytes([0xFF, 0x7F, 0x00, 0x80])) + + assert pcm.sample_rate == TELNYX_DEFAULT_SAMPLE_RATE + assert pcm.channels == 1 + assert len(pcm.samples) == 4 + + +def test_pcm_to_pcmu(): + pcm = PcmData( + samples=np.array([0, 1000, -1000, 16000], dtype=np.int16), + sample_rate=TELNYX_DEFAULT_SAMPLE_RATE, + channels=1, + format=AudioFormat.S16, + ) + + payload = pcm_to_pcmu(pcm) + + assert isinstance(payload, bytes) + assert len(payload) == 4 + + +def test_pcma_to_pcm(): + pcm = pcma_to_pcm(bytes([0xD5, 0x55, 0x00, 0x80])) + + assert pcm.sample_rate == TELNYX_DEFAULT_SAMPLE_RATE + assert pcm.channels == 1 + assert len(pcm.samples) == 4 + + +def test_pcm_to_pcma(): + pcm = PcmData( + samples=np.array([0, 1000, -1000, 16000], dtype=np.int16), + sample_rate=TELNYX_DEFAULT_SAMPLE_RATE, + channels=1, + format=AudioFormat.S16, + ) + + payload = pcm_to_pcma(pcm) + + assert isinstance(payload, bytes) + assert len(payload) == 4 + + +def test_l16_to_pcm(): + pcm = l16_to_pcm(b"\x00\x00\x7f\xff\x80\x00") + + assert pcm.sample_rate == TELNYX_L16_SAMPLE_RATE + assert pcm.channels == 1 + assert pcm.samples.tolist() == [0, 32767, -32768] + + +def test_pcm_to_l16(): + pcm = PcmData( + samples=np.array([0, 32767, -32768], dtype=np.int16), + sample_rate=TELNYX_L16_SAMPLE_RATE, + channels=1, + format=AudioFormat.S16, + ) + + assert pcm_to_l16(pcm) == b"\x00\x00\x7f\xff\x80\x00" + + +def test_payload_helpers(): + pcm = telnyx_payload_to_pcm(bytes([0xFF]), "PCMU") + payload = pcm_to_telnyx_payload(pcm, "PCMU") + + assert isinstance(payload, bytes) + assert len(payload) == 1 diff --git a/plugins/telnyx/tests/test_examples.py b/plugins/telnyx/tests/test_examples.py new file mode 100644 index 00000000..13ac3c99 --- /dev/null +++ b/plugins/telnyx/tests/test_examples.py @@ -0,0 +1,326 @@ +"""Tests for the minimal Telnyx example helpers.""" + +import base64 +import time + +import pytest +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from vision_agents.plugins.telnyx.example_helpers import ( + TelnyxConfig, + TelnyxSetupError, + TelnyxWebhookVerificationError, + cleanup_telnyx_example_setup, + load_config, + media_stream_url, + prepare_telnyx_example_setup, + preflight_inbound, + preflight_outbound, + require_env, + validate_call_control_app, + validate_phone_number_routing, + validate_verified_destination, + verify_telnyx_webhook, + webhook_url, +) + + +class FakeTelnyxClient: + def __init__( + self, + *, + app=None, + phone_number=None, + verified_number=None, + outbound_voice_profile_id="profile-id", + ): + self.app = app + self.phone_number = phone_number + self.verified_number = verified_number + self.outbound_voice_profile_id = outbound_voice_profile_id + self.call_control_apps: dict[str, dict] = {} + self.deleted_app_ids: list[str] = [] + self.phone_connection_id = ( + phone_number.get("connection_id") if phone_number else None + ) + + def retrieve_call_control_app(self, _app_id): + return self.app + + def retrieve_phone_number(self, _phone_number_id): + return self.phone_number + + def find_phone_number(self, _phone_number): + return self.phone_number + + def get_first_outbound_voice_profile_id(self): + return self.outbound_voice_profile_id + + def create_call_control_app( + self, + *, + application_name, + webhook_event_url, + outbound_voice_profile_id, + ): + app = { + "id": "created-app-id", + "application_name": application_name, + "webhook_event_url": webhook_event_url, + "outbound": {"outbound_voice_profile_id": outbound_voice_profile_id}, + } + self.call_control_apps[app["id"]] = app + return {"data": app} + + def delete_call_control_app(self, app_id): + self.deleted_app_ids.append(app_id) + self.call_control_apps.pop(app_id, None) + + def update_phone_number_connection(self, phone_number_id, connection_id): + self.phone_connection_id = connection_id + if self.phone_number is not None: + self.phone_number["connection_id"] = connection_id + self.phone_number["id"] = phone_number_id + + def get_verified_number(self, _phone_number): + return self.verified_number + + +def test_url_builders_normalize_scheme_and_trailing_slash(): + assert webhook_url("https://example.ngrok-free.app/") == ( + "https://example.ngrok-free.app/telnyx/events" + ) + assert media_stream_url("http://example.ngrok-free.app/", "call-1", "token-1") == ( + "wss://example.ngrok-free.app/telnyx/media/call-1/token-1" + ) + + +def test_require_env_raises_for_missing_values(): + with pytest.raises(TelnyxSetupError, match="TELNYX_API_KEY"): + require_env(["TELNYX_API_KEY"], env={}) + + +def test_load_config_reads_required_values_from_mapping(): + config = load_config( + { + "TELNYX_API_KEY": "key", + "TELNYX_CALL_CONTROL_APP_ID": "app-id", + "TELNYX_PHONE_NUMBER": "+15551234567", + "NGROK_URL": "example.ngrok-free.app", + } + ) + + assert config.api_key == "key" + assert config.call_control_app_id == "app-id" + assert config.phone_number == "+15551234567" + assert config.ngrok_url == "example.ngrok-free.app" + + +def test_validate_call_control_app_requires_matching_webhook(): + with pytest.raises(TelnyxSetupError, match="webhook URL mismatch"): + validate_call_control_app( + { + "data": { + "id": "app-id", + "record_type": "call_control_application", + "active": True, + "webhook_event_url": "https://old.example/telnyx/events", + } + }, + app_id="app-id", + expected_webhook_url="https://new.example/telnyx/events", + ) + + +def test_validate_call_control_app_rejects_inactive_app(): + with pytest.raises(TelnyxSetupError, match="inactive"): + validate_call_control_app( + { + "data": { + "id": "app-id", + "record_type": "call_control_application", + "active": False, + "webhook_event_url": "https://example/telnyx/events", + } + }, + app_id="app-id", + expected_webhook_url="https://example/telnyx/events", + ) + + +def test_validate_phone_number_routing_requires_call_control_app_connection(): + with pytest.raises(TelnyxSetupError, match="not routed"): + validate_phone_number_routing( + { + "data": { + "id": "phone-id", + "phone_number": "+15551234567", + "connection_id": "forward-only-id", + } + }, + phone_number_id="phone-id", + expected_connection_id="call-control-app-id", + ) + + +def test_validate_verified_destination_rejects_missing_number(): + with pytest.raises(TelnyxSetupError, match="not verified"): + validate_verified_destination(None, to_number="+15557654321") + + +def test_verify_telnyx_webhook_accepts_valid_signature(): + private_key = Ed25519PrivateKey.generate() + public_key = base64.b64encode(private_key.public_key().public_bytes_raw()).decode( + "ascii" + ) + payload = b'{"data":{"event_type":"call.initiated"}}' + timestamp = str(int(time.time())) + signed_payload = f"{timestamp}|{payload.decode('utf-8')}".encode("utf-8") + signature = base64.b64encode(private_key.sign(signed_payload)).decode("ascii") + + verify_telnyx_webhook(payload, signature, timestamp, public_key) + + +def test_verify_telnyx_webhook_rejects_invalid_signature(): + private_key = Ed25519PrivateKey.generate() + public_key = base64.b64encode(private_key.public_key().public_bytes_raw()).decode( + "ascii" + ) + payload = b'{"data":{"event_type":"call.initiated"}}' + timestamp = str(int(time.time())) + + with pytest.raises(TelnyxWebhookVerificationError, match="Invalid Telnyx"): + verify_telnyx_webhook(payload, "invalid", timestamp, public_key) + + +def test_verify_telnyx_webhook_rejects_malformed_timestamp(): + private_key = Ed25519PrivateKey.generate() + public_key = base64.b64encode(private_key.public_key().public_bytes_raw()).decode( + "ascii" + ) + payload = b'{"data":{"event_type":"call.initiated"}}' + + with pytest.raises(TelnyxWebhookVerificationError, match="Invalid Telnyx"): + verify_telnyx_webhook(payload, "invalid", "not-a-timestamp", public_key) + + +def test_prepare_telnyx_example_setup_requires_app_or_setup_flag(): + client = FakeTelnyxClient() + + with pytest.raises(TelnyxSetupError, match="--setup-telnyx"): + prepare_telnyx_example_setup( + client, + api_key="key", + phone_number="+15551234567", + ngrok_url="example.ngrok-free.app", + ) + + +def test_prepare_telnyx_example_setup_creates_temp_outbound_app(): + client = FakeTelnyxClient( + phone_number={ + "id": "phone-id", + "phone_number": "+15551234567", + "connection_id": "original-app-id", + }, + ) + + setup = prepare_telnyx_example_setup( + client, + api_key="key", + phone_number="+15551234567", + ngrok_url="example.ngrok-free.app", + setup_telnyx=True, + ) + + assert setup.config.call_control_app_id == "created-app-id" + assert setup.created_call_control_app_id == "created-app-id" + assert setup.phone_number_id == "phone-id" + assert setup.original_connection_id is None + assert client.call_control_apps["created-app-id"]["webhook_event_url"] == ( + "https://example.ngrok-free.app/telnyx/events" + ) + assert client.phone_connection_id == "original-app-id" + + cleanup_telnyx_example_setup(client, setup) + + assert client.deleted_app_ids == ["created-app-id"] + assert "created-app-id" not in client.call_control_apps + + +def test_prepare_telnyx_example_setup_routes_and_restores_inbound_number(): + client = FakeTelnyxClient( + phone_number={ + "id": "phone-id", + "phone_number": "+15551234567", + "connection_id": "original-app-id", + }, + ) + + setup = prepare_telnyx_example_setup( + client, + api_key="key", + phone_number="+15551234567", + ngrok_url="example.ngrok-free.app", + setup_telnyx=True, + route_phone_number=True, + ) + + assert setup.phone_number_id == "phone-id" + assert setup.original_connection_id == "original-app-id" + assert client.phone_connection_id == "created-app-id" + + cleanup_telnyx_example_setup(client, setup) + + assert client.phone_connection_id == "original-app-id" + assert client.deleted_app_ids == ["created-app-id"] + + +def test_preflight_outbound_passes_with_matching_app_and_verified_destination(): + config = TelnyxConfig( + api_key="key", + call_control_app_id="app-id", + phone_number="+15551234567", + ngrok_url="example.ngrok-free.app", + ) + client = FakeTelnyxClient( + app={ + "data": { + "id": "app-id", + "record_type": "call_control_application", + "active": True, + "webhook_event_url": "https://example.ngrok-free.app/telnyx/events", + } + }, + verified_number={"data": {"phone_number": "+15557654321"}}, + ) + + preflight_outbound(client, config=config, to_number="+15557654321") + + +def test_preflight_inbound_requires_phone_number_to_route_to_app(): + config = TelnyxConfig( + api_key="key", + call_control_app_id="app-id", + phone_number="+15551234567", + ngrok_url="example.ngrok-free.app", + ) + client = FakeTelnyxClient( + app={ + "data": { + "id": "app-id", + "record_type": "call_control_application", + "active": True, + "webhook_event_url": "https://example.ngrok-free.app/telnyx/events", + } + }, + phone_number={ + "data": { + "id": "phone-id", + "phone_number": "+15551234567", + "connection_id": "other-app-id", + } + }, + ) + + with pytest.raises(TelnyxSetupError, match="not routed"): + preflight_inbound(client, config=config, telnyx_phone_number_id="phone-id") diff --git a/plugins/telnyx/tests/test_telnyx.py b/plugins/telnyx/tests/test_telnyx.py new file mode 100644 index 00000000..8a0e49bd --- /dev/null +++ b/plugins/telnyx/tests/test_telnyx.py @@ -0,0 +1,211 @@ +"""Tests for the Telnyx plugin.""" + +import asyncio +import base64 +import json +from datetime import datetime + +from vision_agents.plugins import telnyx + + +class FakeWebSocket: + def __init__(self, messages): + self.messages = list(messages) + self.accepted = False + self.sent = [] + + async def accept(self): + self.accepted = True + + async def receive_text(self): + if not self.messages: + raise RuntimeError("no more messages") + return self.messages.pop(0) + + async def send_json(self, data): + self.sent.append(data) + + +class DummyAudioTrack: + def __init__(self): + self.writes = [] + + async def write(self, pcm): + self.writes.append(pcm) + + +def test_import(): + assert telnyx.TelnyxCall is not None + assert telnyx.TelnyxCallRegistry is not None + assert telnyx.TelnyxMediaStream is not None + + +def test_create_call(): + webhook_data = { + "data": { + "payload": { + "call_control_id": "v2:abc123", + "from": "+1234567890", + "to": "+10987654321", + "state": "answered", + } + } + } + + call = telnyx.TelnyxCall( + call_control_id="v2:abc123", + webhook_data=webhook_data, + ) + + assert call.call_control_id == "v2:abc123" + assert call.from_number == "+1234567890" + assert call.to_number == "+10987654321" + assert call.call_status == "answered" + assert call.ended_at is None + + +def test_end_call(): + call = telnyx.TelnyxCall(call_control_id="v2:abc123") + assert call.ended_at is None + + call.end() + assert call.ended_at is not None + assert isinstance(call.ended_at, datetime) + + +def test_registry_create_get_remove(): + registry = telnyx.TelnyxCallRegistry() + call = registry.create("v2:abc123") + + assert registry.get("v2:abc123") is call + assert registry.validate("v2:abc123", call.token) is call + + removed = registry.remove("v2:abc123") + assert removed is call + assert removed.ended_at is not None + assert registry.get("v2:abc123") is None + + +def test_registry_list_active(): + registry = telnyx.TelnyxCallRegistry() + registry.create("v2:one") + registry.create("v2:two") + registry.remove("v2:two") + + active = registry.list_active() + + assert len(active) == 1 + assert active[0].call_control_id == "v2:one" + + +def test_media_stream_run_writes_audio(): + payload = base64.b64encode(bytes([0xFF] * 160)).decode("ascii") + websocket = FakeWebSocket( + [ + json.dumps({"event": "connected", "version": "1.0.0"}), + json.dumps( + { + "event": "start", + "stream_id": "stream-1", + "start": { + "call_control_id": "v2:abc123", + "media_format": { + "encoding": "PCMU", + "sample_rate": 8000, + "channels": 1, + }, + }, + } + ), + json.dumps( + { + "event": "media", + "stream_id": "stream-1", + "media": { + "track": "inbound", + "payload": payload, + }, + } + ), + json.dumps({"event": "stop", "stream_id": "stream-1"}), + ] + ) + stream = telnyx.TelnyxMediaStream(websocket) + dummy_track = DummyAudioTrack() + stream.audio_track = dummy_track + + asyncio.run(stream.accept()) + asyncio.run(stream.run()) + + assert websocket.accepted is True + assert stream.stream_id == "stream-1" + assert stream.call_control_id == "v2:abc123" + assert stream.is_connected is False + assert len(dummy_track.writes) == 1 + assert dummy_track.writes[0].sample_rate == 8000 + + +def test_media_stream_send_audio(): + websocket = FakeWebSocket([]) + stream = telnyx.TelnyxMediaStream(websocket) + pcm = telnyx.pcmu_to_pcm(bytes([0xFF] * 160)) + + asyncio.run(stream.accept()) + asyncio.run(stream.send_audio(pcm)) + + assert websocket.sent == [] + + stream.stream_id = "stream-1" + stream._started = True + asyncio.run(stream.send_audio(pcm)) + + assert websocket.sent[0]["event"] == "media" + assert "payload" in websocket.sent[0]["media"] + + +def test_media_stream_start_recreates_audio_track_for_negotiated_format(): + websocket = FakeWebSocket( + [ + json.dumps( + { + "event": "start", + "stream_id": "stream-1", + "start": { + "call_control_id": "v2:abc123", + "media_format": { + "encoding": "L16", + "sample_rate": 16000, + "channels": 1, + }, + }, + } + ), + json.dumps({"event": "stop", "stream_id": "stream-1"}), + ] + ) + stream = telnyx.TelnyxMediaStream(websocket) + original_track = stream.audio_track + + asyncio.run(stream.accept()) + asyncio.run(stream.run()) + + assert stream.media_format.encoding == "L16" + assert stream.media_format.sample_rate == 16000 + assert stream.audio_track is not original_track + + +def test_media_stream_runs_cleanup_callbacks_on_close(): + cleanup_ran = False + + async def cleanup(): + nonlocal cleanup_ran + cleanup_ran = True + + websocket = FakeWebSocket([json.dumps({"event": "stop", "stream_id": "stream-1"})]) + stream = telnyx.TelnyxMediaStream(websocket) + stream.add_cleanup_callback(cleanup) + + asyncio.run(stream.accept()) + asyncio.run(stream.run()) + + assert cleanup_ran diff --git a/plugins/telnyx/vision_agents/plugins/telnyx/__init__.py b/plugins/telnyx/vision_agents/plugins/telnyx/__init__.py new file mode 100644 index 00000000..cf5b476c --- /dev/null +++ b/plugins/telnyx/vision_agents/plugins/telnyx/__init__.py @@ -0,0 +1,39 @@ +"""Telnyx plugin for Vision Agents.""" + +from .audio import ( + TELNYX_DEFAULT_SAMPLE_RATE, + TELNYX_L16_SAMPLE_RATE, + l16_to_pcm, + pcma_to_pcm, + pcm_to_l16, + pcm_to_pcma, + pcm_to_pcmu, + pcm_to_telnyx_payload, + pcmu_to_pcm, + telnyx_payload_to_pcm, +) +from .call_registry import TelnyxCall, TelnyxCallRegistry +from .media_stream import TelnyxMediaFormat, TelnyxMediaStream, attach_phone_to_call + +CallRegistry = TelnyxCallRegistry +MediaStream = TelnyxMediaStream + +__all__ = [ + "CallRegistry", + "MediaStream", + "TELNYX_DEFAULT_SAMPLE_RATE", + "TELNYX_L16_SAMPLE_RATE", + "TelnyxCall", + "TelnyxCallRegistry", + "TelnyxMediaFormat", + "TelnyxMediaStream", + "attach_phone_to_call", + "l16_to_pcm", + "pcma_to_pcm", + "pcm_to_l16", + "pcm_to_pcma", + "pcm_to_pcmu", + "pcm_to_telnyx_payload", + "pcmu_to_pcm", + "telnyx_payload_to_pcm", +] diff --git a/plugins/telnyx/vision_agents/plugins/telnyx/audio.py b/plugins/telnyx/vision_agents/plugins/telnyx/audio.py new file mode 100644 index 00000000..4f2f7f05 --- /dev/null +++ b/plugins/telnyx/vision_agents/plugins/telnyx/audio.py @@ -0,0 +1,202 @@ +"""Audio conversion utilities for Telnyx media streaming.""" + +import numpy as np +from getstream.video.rtc.track_util import AudioFormat, PcmData + +TELNYX_DEFAULT_SAMPLE_RATE = 8000 +TELNYX_L16_SAMPLE_RATE = 16000 + +_ULAW_BIAS = 0x84 +_ULAW_CLIP = 32635 +_SEG_END = (0xFF, 0x1FF, 0x3FF, 0x7FF, 0xFFF, 0x1FFF, 0x3FFF, 0x7FFF) + + +def pcmu_to_pcm(payload: bytes) -> PcmData: + """ + Convert Telnyx PCMU RTP payload bytes to PcmData. + + Args: + payload: Base64-decoded PCMU RTP payload bytes from Telnyx. + + Returns: + PcmData at 8 kHz mono. + """ + samples = np.array([_decode_ulaw_byte(value) for value in payload], dtype=np.int16) + return PcmData( + samples=samples, + sample_rate=TELNYX_DEFAULT_SAMPLE_RATE, + channels=1, + format=AudioFormat.S16, + ) + + +def pcma_to_pcm(payload: bytes) -> PcmData: + """ + Convert Telnyx PCMA RTP payload bytes to PcmData. + + Args: + payload: Base64-decoded PCMA RTP payload bytes from Telnyx. + + Returns: + PcmData at 8 kHz mono. + """ + samples = np.array([_decode_alaw_byte(value) for value in payload], dtype=np.int16) + return PcmData( + samples=samples, + sample_rate=TELNYX_DEFAULT_SAMPLE_RATE, + channels=1, + format=AudioFormat.S16, + ) + + +def l16_to_pcm(payload: bytes, sample_rate: int = TELNYX_L16_SAMPLE_RATE) -> PcmData: + """ + Convert Telnyx L16 RTP payload bytes to PcmData. + + RTP L16 is network byte order. + """ + if len(payload) % 2: + payload = payload[:-1] + samples = np.frombuffer(payload, dtype=">i2").astype(np.int16) + return PcmData( + samples=samples, + sample_rate=sample_rate, + channels=1, + format=AudioFormat.S16, + ) + + +def pcm_to_pcmu(pcm: PcmData) -> bytes: + """ + Convert PcmData to PCMU bytes for Telnyx bidirectional RTP streaming. + """ + pcm = _as_mono_s16(pcm, TELNYX_DEFAULT_SAMPLE_RATE) + return bytes(_encode_ulaw_sample(sample) for sample in pcm.samples) + + +def pcm_to_pcma(pcm: PcmData) -> bytes: + """ + Convert PcmData to PCMA bytes for Telnyx bidirectional RTP streaming. + """ + pcm = _as_mono_s16(pcm, TELNYX_DEFAULT_SAMPLE_RATE) + return bytes(_encode_alaw_sample(sample) for sample in pcm.samples) + + +def pcm_to_l16(pcm: PcmData, sample_rate: int = TELNYX_L16_SAMPLE_RATE) -> bytes: + """ + Convert PcmData to big-endian L16 bytes for Telnyx bidirectional RTP. + """ + pcm = _as_mono_s16(pcm, sample_rate) + return pcm.samples.astype(">i2").tobytes() + + +def telnyx_payload_to_pcm( + payload: bytes, + encoding: str, + sample_rate: int = TELNYX_DEFAULT_SAMPLE_RATE, +) -> PcmData: + """ + Decode a Telnyx RTP payload using the stream media format. + """ + encoding = encoding.upper() + if encoding == "PCMU": + return pcmu_to_pcm(payload) + if encoding == "PCMA": + return pcma_to_pcm(payload) + if encoding == "L16": + return l16_to_pcm(payload, sample_rate=sample_rate) + raise ValueError(f"Unsupported Telnyx media encoding: {encoding}") + + +def pcm_to_telnyx_payload( + pcm: PcmData, + encoding: str, + sample_rate: int = TELNYX_DEFAULT_SAMPLE_RATE, +) -> bytes: + """ + Encode PcmData for a Telnyx bidirectional RTP media frame. + """ + encoding = encoding.upper() + if encoding == "PCMU": + return pcm_to_pcmu(pcm) + if encoding == "PCMA": + return pcm_to_pcma(pcm) + if encoding == "L16": + return pcm_to_l16(pcm, sample_rate=sample_rate) + raise ValueError(f"Unsupported Telnyx media encoding: {encoding}") + + +def _as_mono_s16(pcm: PcmData, sample_rate: int) -> PcmData: + if pcm.sample_rate != sample_rate or pcm.channels != 1: + pcm = pcm.resample(target_sample_rate=sample_rate, target_channels=1) + if pcm.samples.dtype != np.int16: + pcm = PcmData( + samples=pcm.samples.astype(np.int16), + sample_rate=pcm.sample_rate, + channels=pcm.channels, + format=AudioFormat.S16, + ) + return pcm + + +def _decode_ulaw_byte(value: int) -> int: + value = ~value & 0xFF + sample = ((value & 0x0F) << 3) + _ULAW_BIAS + sample <<= (value & 0x70) >> 4 + sample -= _ULAW_BIAS + return -sample if value & 0x80 else sample + + +def _encode_ulaw_sample(sample: int) -> int: + sample = int(sample) + if sample < 0: + sample = _ULAW_BIAS - sample + mask = 0x7F + else: + sample += _ULAW_BIAS + mask = 0xFF + + sample = min(sample, _ULAW_CLIP) + segment = _search_segment(sample) + if segment >= 8: + return 0x7F ^ mask + + encoded = (segment << 4) | ((sample >> (segment + 3)) & 0x0F) + return encoded ^ mask + + +def _decode_alaw_byte(value: int) -> int: + value ^= 0x55 + sign = value & 0x80 + exponent = (value & 0x70) >> 4 + sample = (value & 0x0F) << 4 + if exponent == 0: + sample += 8 + else: + sample += 0x108 + sample <<= exponent - 1 + return sample if sign else -sample + + +def _encode_alaw_sample(sample: int) -> int: + sample = int(sample) + if sample >= 0: + mask = 0xD5 + else: + mask = 0x55 + sample = -sample - 8 + + sample = max(0, min(sample, 32635)) + if sample < 256: + encoded = sample >> 4 + else: + segment = _search_segment(sample) + encoded = (segment << 4) | ((sample >> (segment + 3)) & 0x0F) + return encoded ^ mask + + +def _search_segment(sample: int) -> int: + for index, end in enumerate(_SEG_END): + if sample <= end: + return index + return 8 diff --git a/plugins/telnyx/vision_agents/plugins/telnyx/call_registry.py b/plugins/telnyx/vision_agents/plugins/telnyx/call_registry.py new file mode 100644 index 00000000..a0997ee0 --- /dev/null +++ b/plugins/telnyx/vision_agents/plugins/telnyx/call_registry.py @@ -0,0 +1,130 @@ +"""Telnyx call registry for tracking active calls.""" + +import asyncio +import logging +import secrets +from dataclasses import dataclass, field +from datetime import datetime +from typing import Any, Callable, Coroutine, Optional, TYPE_CHECKING + +if TYPE_CHECKING: + from .media_stream import TelnyxMediaStream + +logger = logging.getLogger(__name__) + + +@dataclass +class TelnyxCall: + """ + Represents an active Telnyx call session. + """ + + call_control_id: str + token: str = field(default_factory=lambda: secrets.token_urlsafe(32)) + webhook_data: Optional[dict[str, Any]] = None + telnyx_stream: Optional["TelnyxMediaStream"] = None + stream_call: Optional[Any] = None + started_at: datetime = field(default_factory=datetime.utcnow) + ended_at: Optional[datetime] = None + _prepare_task: Optional[asyncio.Task] = field(default=None, repr=False) + + @property + def payload(self) -> dict[str, Any]: + return ( + self.webhook_data.get("data", {}).get("payload", {}) + if self.webhook_data + else {} + ) + + @property + def from_number(self) -> Optional[str]: + return self.payload.get("from") + + @property + def to_number(self) -> Optional[str]: + return self.payload.get("to") + + @property + def call_status(self) -> Optional[str]: + return self.payload.get("state") + + def end(self): + """Mark the call as ended.""" + self.ended_at = datetime.utcnow() + + @property + def duration_seconds(self) -> Optional[float]: + """Get call duration in seconds, or None if still active.""" + if self.ended_at is None: + return None + return (self.ended_at - self.started_at).total_seconds() + + async def await_prepare(self) -> Any: + """ + Wait for the prepare task to complete and return its result. + """ + if self._prepare_task is None: + return None + return await self._prepare_task + + +class TelnyxCallRegistry: + """ + In-memory registry for active Telnyx calls. + """ + + def __init__(self): + self._calls: dict[str, TelnyxCall] = {} + + def create( + self, + call_control_id: str, + webhook_data: Optional[dict[str, Any]] = None, + prepare: Optional[Callable[[], Coroutine[Any, Any, Any]]] = None, + ) -> TelnyxCall: + call = TelnyxCall(call_control_id=call_control_id, webhook_data=webhook_data) + + if prepare is not None: + call._prepare_task = asyncio.create_task(prepare()) + + self._calls[call_control_id] = call + logger.info( + "TelnyxCallRegistry: Created call %s", + call.call_control_id, + ) + return call + + def get(self, call_control_id: str) -> Optional[TelnyxCall]: + return self._calls.get(call_control_id) + + def require(self, call_control_id: str) -> TelnyxCall: + call = self._calls.get(call_control_id) + if not call: + raise ValueError(f"Unknown call_control_id: {call_control_id}") + return call + + def validate(self, call_control_id: str, token: str) -> TelnyxCall: + call = self._calls.get(call_control_id) + if not call: + raise ValueError(f"Unknown call_control_id: {call_control_id}") + if not secrets.compare_digest(call.token, token): + raise ValueError(f"Invalid token for call_control_id: {call_control_id}") + return call + + def remove(self, call_control_id: str) -> Optional[TelnyxCall]: + call = self._calls.pop(call_control_id, None) + if call: + call.end() + duration = call.duration_seconds + logger.info( + "TelnyxCallRegistry: Removed call %s (duration: %.1fs)", + call_control_id, + duration, + ) + return call + + def list_active(self) -> list[TelnyxCall]: + return [call for call in self._calls.values() if call.ended_at is None] + + def __len__(self) -> int: + return len(self._calls) diff --git a/plugins/telnyx/vision_agents/plugins/telnyx/example_helpers.py b/plugins/telnyx/vision_agents/plugins/telnyx/example_helpers.py new file mode 100644 index 00000000..9695f9e8 --- /dev/null +++ b/plugins/telnyx/vision_agents/plugins/telnyx/example_helpers.py @@ -0,0 +1,542 @@ +"""Helpers shared by the minimal Telnyx phone examples.""" + +import base64 +import binascii +import json +import os +import signal +import time +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass +from typing import Any, Mapping + +from cryptography.exceptions import InvalidSignature +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey +from fastapi import HTTPException, Request + + +TELNYX_API_BASE_URL = "https://api.telnyx.com/v2" +TELNYX_MEDIA_PATH = "/telnyx/media" +TELNYX_EVENTS_PATH = "/telnyx/events" + + +class TelnyxSetupError(RuntimeError): + """Raised when local or Telnyx account setup is not ready for the example.""" + + +class TelnyxWebhookVerificationError(ValueError): + """Raised when a Telnyx webhook signature fails verification.""" + + +class TelnyxAPIError(RuntimeError): + """Raised when a Telnyx API request fails.""" + + def __init__(self, status_code: int, detail: str): + self.status_code = status_code + self.detail = detail + super().__init__(f"Telnyx API failed with HTTP {status_code}: {detail}") + + +@dataclass(frozen=True) +class TelnyxConfig: + api_key: str + call_control_app_id: str + phone_number: str + ngrok_url: str + + +@dataclass(frozen=True) +class TelnyxExampleSetup: + config: TelnyxConfig + phone_number_id: str | None = None + created_call_control_app_id: str | None = None + original_connection_id: str | None = None + + +class TelnyxClient: + def __init__( + self, + api_key: str, + *, + base_url: str = TELNYX_API_BASE_URL, + timeout: float = 20.0, + ): + self.api_key = api_key + self.base_url = base_url.rstrip("/") + self.timeout = timeout + + def request( + self, + method: str, + path: str, + body: Mapping[str, Any] | None = None, + ) -> dict[str, Any]: + data = json.dumps(body).encode("utf-8") if body is not None else None + request = urllib.request.Request( + f"{self.base_url}{path}", + data=data, + headers={ + "Authorization": f"Bearer {self.api_key}", + "Content-Type": "application/json", + "Accept": "application/json", + }, + method=method, + ) + + try: + with urllib.request.urlopen(request, timeout=self.timeout) as response: + content = response.read().decode("utf-8") + return json.loads(content) if content else {} + except urllib.error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace") + raise TelnyxAPIError(exc.code, detail) from exc + + def retrieve_call_control_app(self, app_id: str) -> dict[str, Any]: + return self.request("GET", f"/call_control_applications/{quote_path(app_id)}") + + def retrieve_phone_number(self, phone_number_id: str) -> dict[str, Any]: + return self.request("GET", f"/phone_numbers/{quote_path(phone_number_id)}") + + def find_phone_number(self, phone_number: str) -> dict[str, Any]: + query = urllib.parse.urlencode({"filter[phone_number]": phone_number}) + response = self.request("GET", f"/phone_numbers?{query}") + phone_numbers = response.get("data", []) + if not phone_numbers: + raise TelnyxSetupError(f"Telnyx phone number {phone_number} was not found.") + return phone_numbers[0] + + def get_first_outbound_voice_profile_id(self) -> str: + response = self.request("GET", "/outbound_voice_profiles") + profiles = response.get("data", []) + if not profiles: + raise TelnyxSetupError("No Telnyx outbound voice profiles were found.") + return profiles[0]["id"] + + def create_call_control_app( + self, + *, + application_name: str, + webhook_event_url: str, + outbound_voice_profile_id: str, + ) -> dict[str, Any]: + return self.request( + "POST", + "/call_control_applications", + { + "application_name": application_name, + "webhook_event_url": webhook_event_url, + "active": True, + "outbound": { + "outbound_voice_profile_id": outbound_voice_profile_id, + }, + }, + ) + + def delete_call_control_app(self, app_id: str) -> None: + self.request("DELETE", f"/call_control_applications/{quote_path(app_id)}") + + def update_phone_number_connection( + self, + phone_number_id: str, + connection_id: str, + ) -> None: + self.request( + "PATCH", + f"/phone_numbers/{quote_path(phone_number_id)}", + {"connection_id": connection_id}, + ) + + def get_verified_number(self, phone_number: str) -> dict[str, Any] | None: + try: + return self.request("GET", f"/verified_numbers/{quote_path(phone_number)}") + except TelnyxAPIError as exc: + if exc.status_code == 404: + return None + raise + + def dial_call( + self, + *, + connection_id: str, + from_number: str, + to_number: str, + stream_url: str, + ) -> dict[str, Any]: + return self.request( + "POST", + "/calls", + { + "connection_id": connection_id, + "from": from_number, + "to": to_number, + "stream_url": stream_url, + "stream_track": "inbound_track", + "stream_bidirectional_mode": "rtp", + "stream_bidirectional_codec": "PCMU", + }, + ) + + def answer_call(self, call_control_id: str, *, stream_url: str) -> dict[str, Any]: + return self.request( + "POST", + f"/calls/{quote_path(call_control_id)}/actions/answer", + { + "stream_url": stream_url, + "stream_track": "inbound_track", + "stream_bidirectional_mode": "rtp", + "stream_bidirectional_codec": "PCMU", + }, + ) + + +def quote_path(value: str) -> str: + return urllib.parse.quote(value, safe="") + + +def normalize_public_host(value: str) -> str: + value = value.strip() + value = value.removeprefix("https://").removeprefix("http://") + return value.strip("/") + + +def webhook_url(public_host: str) -> str: + return f"https://{normalize_public_host(public_host)}{TELNYX_EVENTS_PATH}" + + +def media_stream_url(public_host: str, call_id: str, token: str) -> str: + return ( + f"wss://{normalize_public_host(public_host)}" + f"{TELNYX_MEDIA_PATH}/{call_id}/{token}" + ) + + +def detect_ngrok_url() -> str | None: + try: + with urllib.request.urlopen( + "http://127.0.0.1:4040/api/tunnels", timeout=2 + ) as response: + data = json.loads(response.read().decode("utf-8")) + except ( + urllib.error.URLError, + TimeoutError, + UnicodeDecodeError, + json.JSONDecodeError, + ): + return None + + for tunnel in data.get("tunnels", []): + if tunnel.get("proto") == "https" and tunnel.get("public_url"): + return normalize_public_host(tunnel["public_url"]) + return None + + +def require_env( + names: list[str], env: Mapping[str, str] | None = None +) -> dict[str, str]: + env = env if env is not None else os.environ + missing = [name for name in names if not env.get(name)] + if missing: + raise TelnyxSetupError( + "Missing required environment variables: " + ", ".join(missing) + ) + return {name: env[name] for name in names} + + +def load_config(env: Mapping[str, str] | None = None) -> TelnyxConfig: + values = require_env( + [ + "TELNYX_API_KEY", + "TELNYX_CALL_CONTROL_APP_ID", + "TELNYX_PHONE_NUMBER", + "NGROK_URL", + ], + env, + ) + return TelnyxConfig( + api_key=values["TELNYX_API_KEY"], + call_control_app_id=values["TELNYX_CALL_CONTROL_APP_ID"], + phone_number=values["TELNYX_PHONE_NUMBER"], + ngrok_url=values["NGROK_URL"], + ) + + +def resolve_ngrok_url(value: str | None) -> str: + if value: + return normalize_public_host(value) + + detected = detect_ngrok_url() + if detected: + return detected + + raise TelnyxSetupError( + "Missing NGROK_URL and no local ngrok tunnel was detected. " + "Start ngrok with `ngrok http 8000` or set NGROK_URL." + ) + + +def prepare_telnyx_example_setup( + client: TelnyxClient, + *, + api_key: str, + phone_number: str | None, + ngrok_url: str | None, + call_control_app_id: str | None = None, + phone_number_id: str | None = None, + setup_telnyx: bool = False, + route_phone_number: bool = False, +) -> TelnyxExampleSetup: + if not phone_number: + raise TelnyxSetupError( + "Missing Telnyx phone number. Set TELNYX_PHONE_NUMBER or pass " + "`--from`/`--phone-number`." + ) + + resolved_ngrok_url = resolve_ngrok_url(ngrok_url) + resolved_phone_number_id = phone_number_id + original_connection_id: str | None = None + created_app_id: str | None = None + resolved_app_id = call_control_app_id + + if setup_telnyx: + phone_number_data = client.find_phone_number(phone_number) + resolved_phone_number_id = resolved_phone_number_id or phone_number_data["id"] + + try: + if not resolved_app_id: + outbound_voice_profile_id = client.get_first_outbound_voice_profile_id() + response = client.create_call_control_app( + application_name=f"vision-agents-example-{int(time.time())}", + webhook_event_url=webhook_url(resolved_ngrok_url), + outbound_voice_profile_id=outbound_voice_profile_id, + ) + app = response.get("data", response) + resolved_app_id = app["id"] + created_app_id = resolved_app_id + + if route_phone_number: + original_connection_id = phone_number_data.get("connection_id") or "" + client.update_phone_number_connection( + resolved_phone_number_id, + resolved_app_id, + ) + except (TelnyxAPIError, TelnyxSetupError, urllib.error.URLError, TimeoutError): + if created_app_id: + client.delete_call_control_app(created_app_id) + raise + elif not resolved_app_id: + raise TelnyxSetupError( + "Missing TELNYX_CALL_CONTROL_APP_ID. Pass `--setup-telnyx` to create " + "a temporary Call Control App for this example." + ) + + return TelnyxExampleSetup( + config=TelnyxConfig( + api_key=api_key, + call_control_app_id=resolved_app_id, + phone_number=phone_number, + ngrok_url=resolved_ngrok_url, + ), + phone_number_id=resolved_phone_number_id, + created_call_control_app_id=created_app_id, + original_connection_id=original_connection_id, + ) + + +def cleanup_telnyx_example_setup( + client: TelnyxClient, + setup: TelnyxExampleSetup, +) -> None: + previous_sigint_handler = None + try: + previous_sigint_handler = signal.getsignal(signal.SIGINT) + signal.signal(signal.SIGINT, signal.SIG_IGN) + except ValueError: + pass + + errors: list[str] = [] + + try: + if setup.original_connection_id is not None and setup.phone_number_id: + try: + client.update_phone_number_connection( + setup.phone_number_id, + setup.original_connection_id, + ) + except (TelnyxAPIError, urllib.error.URLError, TimeoutError) as exc: + errors.append(f"restore phone number routing failed: {exc}") + + if setup.created_call_control_app_id: + try: + client.delete_call_control_app(setup.created_call_control_app_id) + except (TelnyxAPIError, urllib.error.URLError, TimeoutError) as exc: + errors.append(f"delete temporary Call Control App failed: {exc}") + + if errors: + raise TelnyxSetupError("; ".join(errors)) + finally: + if previous_sigint_handler is not None: + signal.signal(signal.SIGINT, previous_sigint_handler) + + +def validate_call_control_app( + app: Mapping[str, Any], + *, + app_id: str, + expected_webhook_url: str, +) -> None: + data = app.get("data", app) + if data.get("id") != app_id: + raise TelnyxSetupError( + f"Telnyx Call Control App {app_id} was not returned by the API." + ) + if data.get("record_type") != "call_control_application": + raise TelnyxSetupError(f"{app_id} is not a Telnyx Call Control App.") + if data.get("active") is False: + raise TelnyxSetupError(f"Telnyx Call Control App {app_id} is inactive.") + actual_webhook_url = data.get("webhook_event_url") + if actual_webhook_url != expected_webhook_url: + raise TelnyxSetupError( + "Telnyx Call Control App webhook URL mismatch. " + f"Expected {expected_webhook_url}, got {actual_webhook_url!r}." + ) + + +def validate_phone_number_routing( + phone_number: Mapping[str, Any], + *, + phone_number_id: str, + expected_connection_id: str, +) -> None: + data = phone_number.get("data", phone_number) + if data.get("id") != phone_number_id: + raise TelnyxSetupError( + f"Telnyx phone number {phone_number_id} was not returned by the API." + ) + actual_connection_id = data.get("connection_id") + if actual_connection_id != expected_connection_id: + raise TelnyxSetupError( + "Telnyx phone number is not routed to the Call Control App. " + f"Expected connection_id {expected_connection_id}, " + f"got {actual_connection_id!r}." + ) + + +def validate_verified_destination( + verified_number: Mapping[str, Any] | None, + *, + to_number: str, +) -> None: + if verified_number is None: + raise TelnyxSetupError( + f"{to_number} is not verified in Telnyx. Restricted accounts must " + "verify destination numbers before outbound dialing." + ) + + +def preflight_outbound( + client: TelnyxClient, + *, + config: TelnyxConfig, + to_number: str, + check_verified_destination: bool = True, +) -> None: + expected_webhook_url = webhook_url(config.ngrok_url) + app = client.retrieve_call_control_app(config.call_control_app_id) + validate_call_control_app( + app, + app_id=config.call_control_app_id, + expected_webhook_url=expected_webhook_url, + ) + if check_verified_destination: + validate_verified_destination( + client.get_verified_number(to_number), + to_number=to_number, + ) + + +def verify_telnyx_webhook( + payload: bytes, + signature: str | None, + timestamp: str | None, + public_key: str, + *, + tolerance_seconds: int = 300, +) -> None: + """Verify a Telnyx webhook using Ed25519 signature headers.""" + if not signature or not timestamp: + raise TelnyxWebhookVerificationError("Missing Telnyx webhook signature headers") + + try: + now = int(time.time()) + webhook_timestamp = int(timestamp) + payload_text = payload.decode("utf-8") + key = Ed25519PublicKey.from_public_bytes(base64.b64decode(public_key)) + signature_bytes = base64.b64decode(signature) + except (ValueError, UnicodeDecodeError, binascii.Error) as exc: + raise TelnyxWebhookVerificationError( + "Invalid Telnyx webhook signature" + ) from exc + + if abs(now - webhook_timestamp) > tolerance_seconds: + raise TelnyxWebhookVerificationError( + "Telnyx webhook timestamp outside tolerance window" + ) + + signed_payload = f"{timestamp}|{payload_text}".encode("utf-8") + try: + key.verify(signature_bytes, signed_payload) + except InvalidSignature as exc: + raise TelnyxWebhookVerificationError( + "Invalid Telnyx webhook signature" + ) from exc + + +async def parse_verified_telnyx_webhook( + request: Request, + public_key: str, +) -> dict[str, object]: + """Read and verify a Telnyx webhook request body.""" + payload = await request.body() + try: + verify_telnyx_webhook( + payload, + request.headers.get("telnyx-signature-ed25519"), + request.headers.get("telnyx-timestamp"), + public_key, + ) + except TelnyxWebhookVerificationError as exc: + raise HTTPException(status_code=403, detail=str(exc)) from exc + try: + return json.loads(payload) + except json.JSONDecodeError as exc: + raise HTTPException( + status_code=400, detail="Invalid Telnyx webhook payload" + ) from exc + + +def require_telnyx_public_key(env: Mapping[str, str] | None = None) -> str: + values = require_env(["TELNYX_PUBLIC_KEY"], env) + return values["TELNYX_PUBLIC_KEY"] + + +def preflight_inbound( + client: TelnyxClient, + *, + config: TelnyxConfig, + telnyx_phone_number_id: str, +) -> None: + expected_webhook_url = webhook_url(config.ngrok_url) + app = client.retrieve_call_control_app(config.call_control_app_id) + validate_call_control_app( + app, + app_id=config.call_control_app_id, + expected_webhook_url=expected_webhook_url, + ) + phone_number = client.retrieve_phone_number(telnyx_phone_number_id) + validate_phone_number_routing( + phone_number, + phone_number_id=telnyx_phone_number_id, + expected_connection_id=config.call_control_app_id, + ) diff --git a/plugins/telnyx/vision_agents/plugins/telnyx/media_stream.py b/plugins/telnyx/vision_agents/plugins/telnyx/media_stream.py new file mode 100644 index 00000000..f48e6aea --- /dev/null +++ b/plugins/telnyx/vision_agents/plugins/telnyx/media_stream.py @@ -0,0 +1,247 @@ +"""Telnyx Media Streaming WebSocket handler.""" + +import base64 +import json +import logging +from dataclasses import dataclass +from typing import Any, Awaitable, Callable, Protocol + +from getstream.video import rtc +from getstream.video.rtc.audio_track import AudioStreamTrack +from getstream.video.rtc.pb.stream.video.sfu.models.models_pb2 import TrackType +from getstream.video.rtc.track_util import PcmData +from getstream.video.rtc.tracks import SubscriptionConfig, TrackSubscriptionConfig + +from .audio import ( + TELNYX_DEFAULT_SAMPLE_RATE, + pcm_to_telnyx_payload, + telnyx_payload_to_pcm, +) + +logger = logging.getLogger(__name__) + + +class WebSocketProtocol(Protocol): + """Protocol for WebSocket connections compatible with FastAPI and Starlette.""" + + async def accept(self) -> None: ... + async def receive_text(self) -> str: ... + async def send_json(self, data: Any) -> None: ... + + +@dataclass +class TelnyxMediaFormat: + encoding: str = "PCMU" + sample_rate: int = TELNYX_DEFAULT_SAMPLE_RATE + channels: int = 1 + + @classmethod + def from_start_event(cls, data: dict[str, Any]) -> "TelnyxMediaFormat": + media_format = data.get("start", {}).get("media_format", {}) + return cls( + encoding=str(media_format.get("encoding", "PCMU")).upper(), + sample_rate=int( + media_format.get("sample_rate", TELNYX_DEFAULT_SAMPLE_RATE) + ), + channels=int(media_format.get("channels", 1)), + ) + + +class TelnyxMediaStream: + """ + Manages a Telnyx Media Streaming WebSocket connection. + """ + + def __init__( + self, + websocket: WebSocketProtocol, + *, + track: str = "inbound", + media_format: TelnyxMediaFormat | None = None, + ): + self.websocket = websocket + self.track = track + self.stream_id: str | None = None + self.call_control_id: str | None = None + self.media_format = media_format or TelnyxMediaFormat() + self.audio_track = self._create_audio_track() + self._connected = False + self._started = False + self._start_callbacks: list[Callable[[], Awaitable[None]]] = [] + self._cleanup_callbacks: list[Callable[[], Awaitable[None]]] = [] + self._cleanup_done = False + + async def accept(self) -> None: + await self.websocket.accept() + self._connected = True + logger.info("TelnyxMediaStream: WebSocket connection accepted") + + async def run(self) -> None: + """ + Process incoming Telnyx WebSocket messages until the stream ends. + """ + has_seen_media = False + message_count = 0 + + try: + while True: + message = await self.websocket.receive_text() + data = json.loads(message) + + match data["event"]: + case "connected": + logger.info("TelnyxMediaStream: Connected: %s", data) + case "start": + self.stream_id = data.get("stream_id") + self.call_control_id = data.get("start", {}).get( + "call_control_id" + ) + await self._handle_start( + TelnyxMediaFormat.from_start_event(data) + ) + logger.info( + "TelnyxMediaStream: Stream started, stream_id=%s, encoding=%s", + self.stream_id, + self.media_format.encoding, + ) + case "media": + media = data["media"] + if not self._track_matches(str(media.get("track", ""))): + continue + + payload = base64.b64decode(media["payload"]) + pcm = telnyx_payload_to_pcm( + payload, + self.media_format.encoding, + sample_rate=self.media_format.sample_rate, + ) + await self.audio_track.write(pcm) + + if not has_seen_media: + logger.info( + "TelnyxMediaStream: Receiving audio: %s bytes/chunk", + len(payload), + ) + has_seen_media = True + case "stop": + logger.info("TelnyxMediaStream: Stream stopped") + break + case "error": + logger.warning("TelnyxMediaStream: Error frame: %s", data) + break + case "mark" | "dtmf": + logger.debug("TelnyxMediaStream: Event: %s", data) + + message_count += 1 + except Exception: + logger.exception("TelnyxMediaStream: WebSocket processing failed") + finally: + await self.close() + + logger.info( + "TelnyxMediaStream: Connection closed. Received %s messages", + message_count, + ) + + async def send_audio(self, pcm: PcmData) -> None: + """ + Send PCM audio back to Telnyx as a bidirectional RTP media frame. + """ + if not self._connected or not self._started or self.stream_id is None: + return + + payload = pcm_to_telnyx_payload( + pcm, + self.media_format.encoding, + sample_rate=self.media_format.sample_rate, + ) + await self.websocket.send_json( + { + "event": "media", + "media": {"payload": base64.b64encode(payload).decode("ascii")}, + } + ) + + @property + def is_connected(self) -> bool: + return self._connected + + @property + def has_started(self) -> bool: + return self._started + + def add_start_callback(self, callback: Callable[[], Awaitable[None]]) -> None: + self._start_callbacks.append(callback) + + def add_cleanup_callback(self, callback: Callable[[], Awaitable[None]]) -> None: + self._cleanup_callbacks.append(callback) + + async def close(self) -> None: + self._connected = False + if self._cleanup_done: + return + self._cleanup_done = True + + callbacks = self._cleanup_callbacks + self._cleanup_callbacks = [] + for callback in callbacks: + try: + await callback() + except Exception: + logger.exception("TelnyxMediaStream: Cleanup callback failed") + + def _track_matches(self, received: str) -> bool: + if self.track == "both": + return True + if received.endswith("_track"): + received = received[: -len("_track")] + return received == self.track + + async def _handle_start(self, media_format: TelnyxMediaFormat) -> None: + if media_format != self.media_format: + self.media_format = media_format + self.audio_track = self._create_audio_track() + else: + self.media_format = media_format + + self._started = True + callbacks = self._start_callbacks + self._start_callbacks = [] + for callback in callbacks: + await callback() + + def _create_audio_track(self) -> AudioStreamTrack: + return AudioStreamTrack( + sample_rate=self.media_format.sample_rate, + channels=self.media_format.channels, + format="s16", + ) + + +async def attach_phone_to_call( + call, telnyx_stream: TelnyxMediaStream, user_id: str +) -> None: + """ + Attach a phone user to a Stream call, bridging audio between Telnyx and Stream. + """ + subscription_config = SubscriptionConfig( + default=TrackSubscriptionConfig(track_types=[TrackType.TRACK_TYPE_AUDIO]) + ) + + connection = await rtc.join(call, user_id, subscription_config=subscription_config) + + @connection.on("audio") + async def on_audio_received(pcm: PcmData): + await telnyx_stream.send_audio(pcm) + + async def add_telnyx_track() -> None: + await connection.add_tracks(audio=telnyx_stream.audio_track, video=None) + + await connection.__aenter__() + telnyx_stream.add_cleanup_callback(lambda: connection.__aexit__(None, None, None)) + if telnyx_stream.has_started: + await add_telnyx_track() + else: + telnyx_stream.add_start_callback(add_telnyx_track) + + logger.info("Phone user %s attached to call", user_id) diff --git a/pyproject.toml b/pyproject.toml index 13429e33..148c66fa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,6 +30,7 @@ vision-agents-plugins-vogent = { workspace = true } vision-agents-plugins-fast-whisper = { workspace = true } vision-agents-plugins-roboflow = { workspace = true } vision-agents-plugins-decart = { workspace = true } +vision-agents-plugins-telnyx = { workspace = true } vision-agents-plugins-twilio = { workspace = true } vision-agents-plugins-turbopuffer = { workspace = true } vision-agents-plugins-qdrant = { workspace = true } @@ -79,6 +80,7 @@ members = [ "plugins/fast_whisper", "plugins/roboflow", "plugins/decart", + "plugins/telnyx", "plugins/twilio", "plugins/turbopuffer", "plugins/qdrant", diff --git a/uv.lock b/uv.lock index 7f963067..2d0d478d 100644 --- a/uv.lock +++ b/uv.lock @@ -41,6 +41,7 @@ members = [ "vision-agents-plugins-roboflow", "vision-agents-plugins-sarvam", "vision-agents-plugins-smart-turn", + "vision-agents-plugins-telnyx", "vision-agents-plugins-tencent", "vision-agents-plugins-turbopuffer", "vision-agents-plugins-twilio", @@ -6566,6 +6567,9 @@ sarvam = [ smart-turn = [ { name = "vision-agents-plugins-smart-turn" }, ] +telnyx = [ + { name = "vision-agents-plugins-telnyx" }, +] tencent = [ { name = "vision-agents-plugins-tencent", marker = "sys_platform == 'linux'" }, ] @@ -6640,6 +6644,7 @@ requires-dist = [ { name = "vision-agents-plugins-roboflow", marker = "extra == 'roboflow'", editable = "plugins/roboflow" }, { name = "vision-agents-plugins-sarvam", marker = "extra == 'sarvam'", editable = "plugins/sarvam" }, { name = "vision-agents-plugins-smart-turn", marker = "extra == 'smart-turn'", editable = "plugins/smart_turn" }, + { name = "vision-agents-plugins-telnyx", marker = "extra == 'telnyx'", editable = "plugins/telnyx" }, { name = "vision-agents-plugins-tencent", marker = "sys_platform == 'linux' and extra == 'tencent'", editable = "plugins/tencent" }, { name = "vision-agents-plugins-turbopuffer", marker = "extra == 'turbopuffer'", editable = "plugins/turbopuffer" }, { name = "vision-agents-plugins-twilio", marker = "extra == 'twilio'", editable = "plugins/twilio" }, @@ -6648,7 +6653,7 @@ requires-dist = [ { name = "vision-agents-plugins-wizper", marker = "extra == 'wizper'", editable = "plugins/wizper" }, { name = "vision-agents-plugins-xai", marker = "extra == 'xai'", editable = "plugins/xai" }, ] -provides-extras = ["anam", "anthropic", "assemblyai", "aws", "cartesia", "decart", "deepgram", "dev", "elevenlabs", "fast-whisper", "fish", "gemini", "getstream", "huggingface", "inworld", "kokoro", "lemonslice", "liveavatar", "local", "minimax", "mistral", "moondream", "moonshine", "nvidia", "openai", "openrouter", "pocket", "qdrant", "qwen", "redis", "roboflow", "sarvam", "smart-turn", "tencent", "turbopuffer", "twilio", "ultralytics", "vogent", "wizper", "xai"] +provides-extras = ["anam", "anthropic", "assemblyai", "aws", "cartesia", "decart", "deepgram", "dev", "elevenlabs", "fast-whisper", "fish", "gemini", "getstream", "huggingface", "inworld", "kokoro", "lemonslice", "liveavatar", "local", "minimax", "mistral", "moondream", "moonshine", "nvidia", "openai", "openrouter", "pocket", "qdrant", "qwen", "redis", "roboflow", "sarvam", "smart-turn", "telnyx", "tencent", "turbopuffer", "twilio", "ultralytics", "vogent", "wizper", "xai"] [[package]] name = "vision-agents-plugins-anam" @@ -7543,6 +7548,36 @@ dev = [ { name = "pytest-asyncio", specifier = ">=1.0.0" }, ] +[[package]] +name = "vision-agents-plugins-telnyx" +source = { editable = "plugins/telnyx" } +dependencies = [ + { name = "cryptography" }, + { name = "fastapi" }, + { name = "numpy" }, + { name = "vision-agents" }, +] + +[package.dev-dependencies] +dev = [ + { name = "pytest" }, + { name = "pytest-asyncio" }, +] + +[package.metadata] +requires-dist = [ + { name = "cryptography", specifier = ">=44.0.0" }, + { name = "fastapi", specifier = ">=0.135.1" }, + { name = "numpy", specifier = ">=1.24.0" }, + { name = "vision-agents", editable = "agents-core" }, +] + +[package.metadata.requires-dev] +dev = [ + { name = "pytest", specifier = ">=8.4.1" }, + { name = "pytest-asyncio", specifier = ">=1.0.0" }, +] + [[package]] name = "vision-agents-plugins-tencent" source = { editable = "plugins/tencent" }