|
| 1 | +--- |
| 2 | +title: "OrcaRouterChatGenerator" |
| 3 | +id: orcarouterchatgenerator |
| 4 | +slug: "/orcarouterchatgenerator" |
| 5 | +description: "This component enables chat completion through [OrcaRouter](https://www.orcarouter.ai/), an OpenAI-compatible model routing gateway." |
| 6 | +--- |
| 7 | + |
| 8 | +# OrcaRouterChatGenerator |
| 9 | + |
| 10 | +This component enables chat completion through [OrcaRouter](https://www.orcarouter.ai/), an OpenAI-compatible model routing gateway. |
| 11 | + |
| 12 | +<div className="key-value-table"> |
| 13 | + |
| 14 | +| | | |
| 15 | +| --- | --- | |
| 16 | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | |
| 17 | +| **Mandatory init variables** | `api_key`: An OrcaRouter API key. Can be set with `ORCAROUTER_API_KEY` env variable or passed to `init()` method. | |
| 18 | +| **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects | |
| 19 | +| **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects | |
| 20 | +| **API reference** | [OrcaRouter](/reference/integrations-orcarouter) | |
| 21 | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/orcarouter | |
| 22 | +| **Package name** | `orcarouter-haystack` | |
| 23 | + |
| 24 | +</div> |
| 25 | + |
| 26 | +## Overview |
| 27 | + |
| 28 | +The `OrcaRouterChatGenerator` enables you to use models from multiple providers (such as `openai/gpt-4o-mini`, `anthropic/claude-opus-4.8`, and `google/gemini-2.5-flash`) by making chat completion calls to the [OrcaRouter API](https://docs.orcarouter.ai). Models are addressed with a `provider/model` namespace, and you can browse the available models in the [OrcaRouter model catalog](https://www.orcarouter.ai/models). |
| 29 | + |
| 30 | +This generator also supports OrcaRouter-specific features such as: |
| 31 | + |
| 32 | +- Automatic routing with the `orcarouter/auto` model, which lets OrcaRouter pick a live upstream model per request based on the policy configured in your OrcaRouter console. |
| 33 | +- Provider routing and model fallback that are configurable with the `generation_kwargs` parameter during initialization or runtime. OrcaRouter-specific routing options are forwarded to the gateway through `extra_body`. |
| 34 | + |
| 35 | +This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx). |
| 36 | + |
| 37 | +### Tool Support |
| 38 | + |
| 39 | +`OrcaRouterChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations: |
| 40 | + |
| 41 | +- **A list of Tool objects**: Pass individual tools as a list |
| 42 | +- **A single Toolset**: Pass an entire Toolset directly |
| 43 | +- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list |
| 44 | + |
| 45 | +This allows you to organize related tools into logical groups while also including standalone tools as needed. |
| 46 | + |
| 47 | +```python |
| 48 | +from haystack.tools import Tool, Toolset |
| 49 | +from haystack_integrations.components.generators.orcarouter import OrcaRouterChatGenerator |
| 50 | + |
| 51 | +# Create individual tools |
| 52 | +weather_tool = Tool(name="weather", description="Get weather info", ...) |
| 53 | +news_tool = Tool(name="news", description="Get latest news", ...) |
| 54 | + |
| 55 | +# Group related tools into a toolset |
| 56 | +math_toolset = Toolset([add_tool, subtract_tool, multiply_tool]) |
| 57 | + |
| 58 | +# Pass mixed tools and toolsets to the generator |
| 59 | +generator = OrcaRouterChatGenerator( |
| 60 | + tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects |
| 61 | +) |
| 62 | +``` |
| 63 | + |
| 64 | +For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation. |
| 65 | + |
| 66 | +### Initialization |
| 67 | + |
| 68 | +To use this integration, you need an OrcaRouter API key. You can provide it with the `ORCAROUTER_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx). |
| 69 | + |
| 70 | +Then, install the `orcarouter-haystack` integration: |
| 71 | + |
| 72 | +```shell |
| 73 | +pip install orcarouter-haystack |
| 74 | +``` |
| 75 | + |
| 76 | +### Streaming |
| 77 | + |
| 78 | +`OrcaRouterChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization. |
| 79 | + |
| 80 | +## Usage |
| 81 | + |
| 82 | +### On its own |
| 83 | + |
| 84 | +```python |
| 85 | +from haystack.dataclasses import ChatMessage |
| 86 | +from haystack_integrations.components.generators.orcarouter import ( |
| 87 | + OrcaRouterChatGenerator, |
| 88 | +) |
| 89 | + |
| 90 | +client = OrcaRouterChatGenerator(model="openai/gpt-4o-mini") |
| 91 | +response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]) |
| 92 | +print(response["replies"][0].text) |
| 93 | +``` |
| 94 | + |
| 95 | +With automatic routing and streaming: |
| 96 | + |
| 97 | +```python |
| 98 | +from haystack.dataclasses import ChatMessage |
| 99 | +from haystack_integrations.components.generators.orcarouter import ( |
| 100 | + OrcaRouterChatGenerator, |
| 101 | +) |
| 102 | + |
| 103 | +client = OrcaRouterChatGenerator( |
| 104 | + model="orcarouter/auto", |
| 105 | + streaming_callback=lambda chunk: print(chunk.content, end="", flush=True), |
| 106 | +) |
| 107 | + |
| 108 | +response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]) |
| 109 | + |
| 110 | +# check the model used for the response |
| 111 | +print("\n\n Model used: ", response["replies"][0].meta["model"]) |
| 112 | +``` |
| 113 | + |
| 114 | +With a fallback chain: |
| 115 | + |
| 116 | +```python |
| 117 | +from haystack.dataclasses import ChatMessage |
| 118 | +from haystack_integrations.components.generators.orcarouter import ( |
| 119 | + OrcaRouterChatGenerator, |
| 120 | +) |
| 121 | + |
| 122 | +client = OrcaRouterChatGenerator( |
| 123 | + model="openai/gpt-4o-mini", |
| 124 | + generation_kwargs={ |
| 125 | + "extra_body": { |
| 126 | + "route": "fallback", |
| 127 | + "models": [ |
| 128 | + "openai/gpt-4o-mini", |
| 129 | + "anthropic/claude-haiku-4.5", |
| 130 | + "google/gemini-2.5-flash", |
| 131 | + ], |
| 132 | + } |
| 133 | + }, |
| 134 | +) |
| 135 | + |
| 136 | +response = client.run([ChatMessage.from_user("What is Haystack?")]) |
| 137 | +print(response["replies"][0].text) |
| 138 | +``` |
| 139 | + |
| 140 | +### In a pipeline |
| 141 | + |
| 142 | +```python |
| 143 | +from haystack import Pipeline |
| 144 | +from haystack.components.builders import ChatPromptBuilder |
| 145 | +from haystack.dataclasses import ChatMessage |
| 146 | +from haystack_integrations.components.generators.orcarouter import ( |
| 147 | + OrcaRouterChatGenerator, |
| 148 | +) |
| 149 | + |
| 150 | +prompt_builder = ChatPromptBuilder() |
| 151 | +llm = OrcaRouterChatGenerator(model="openai/gpt-4o-mini") |
| 152 | + |
| 153 | +pipe = Pipeline() |
| 154 | +pipe.add_component("builder", prompt_builder) |
| 155 | +pipe.add_component("llm", llm) |
| 156 | +pipe.connect("builder.prompt", "llm.messages") |
| 157 | + |
| 158 | +messages = [ |
| 159 | + ChatMessage.from_system("Give brief answers."), |
| 160 | + ChatMessage.from_user("Tell me about {{city}}"), |
| 161 | +] |
| 162 | + |
| 163 | +response = pipe.run( |
| 164 | + data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}}, |
| 165 | +) |
| 166 | +print(response) |
| 167 | +``` |
0 commit comments