Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs-website/docs/pipeline-components/generators.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Generators are responsible for generating text after you give them a prompt. The
| [OpenAIImageGenerator](generators/openaiimagegenerator.mdx) | Generate images using OpenAI's image generation models such as `gpt-image-2`. | ❌ |
| [OpenAIResponsesChatGenerator](generators/openairesponseschatgenerator.mdx) | Enables chat completion using OpenAI's Responses API with support for reasoning models. | ✅ |
| [OpenRouterChatGenerator](generators/openrouterchatgenerator.mdx) | Enables chat completion with any model hosted on OpenRouter. | ✅ |
| [OrcaRouterChatGenerator](generators/orcarouterchatgenerator.mdx) | Enables chat completion using models routed through OrcaRouter. | ✅ |
| [PerplexityChatGenerator](generators/perplexitychatgenerator.mdx) | Enables chat completion using models via the Perplexity Agent API. | ✅ |
| [SagemakerGenerator](generators/sagemakergenerator.mdx) | Enables text generation using LLMs deployed on Amazon Sagemaker. | ❌ |
| [STACKITChatGenerator](generators/stackitchatgenerator.mdx) | Enables chat completions using the STACKIT API. | ✅ |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
title: "OrcaRouterChatGenerator"
id: orcarouterchatgenerator
slug: "/orcarouterchatgenerator"
description: "This component enables chat completion through [OrcaRouter](https://www.orcarouter.ai/), an OpenAI-compatible model routing gateway."
---

# OrcaRouterChatGenerator

This component enables chat completion through [OrcaRouter](https://www.orcarouter.ai/), an OpenAI-compatible model routing gateway.

<div className="key-value-table">

| | |
| --- | --- |
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
| **Mandatory init variables** | `api_key`: An OrcaRouter API key. Can be set with `ORCAROUTER_API_KEY` env variable or passed to `init()` method. |
| **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
| **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
| **API reference** | [OrcaRouter](/reference/integrations-orcarouter) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/orcarouter |
| **Package name** | `orcarouter-haystack` |

</div>

## Overview

The `OrcaRouterChatGenerator` enables you to use models from multiple providers (such as `openai/gpt-4o-mini`, `anthropic/claude-opus-4.8`, and `google/gemini-2.5-flash`) by making chat completion calls to the [OrcaRouter API](https://docs.orcarouter.ai). Models are addressed with a `provider/model` namespace, and you can browse the available models in the [OrcaRouter model catalog](https://www.orcarouter.ai/models).

This generator also supports OrcaRouter-specific features such as:

- Automatic routing with the `orcarouter/auto` model, which lets OrcaRouter pick a live upstream model per request based on the policy configured in your OrcaRouter console.
- Provider routing and model fallback that are configurable with the `generation_kwargs` parameter during initialization or runtime. OrcaRouter-specific routing options are forwarded to the gateway through `extra_body`.

This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).

### Tool Support

`OrcaRouterChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:

- **A list of Tool objects**: Pass individual tools as a list
- **A single Toolset**: Pass an entire Toolset directly
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list

This allows you to organize related tools into logical groups while also including standalone tools as needed.

```python
from haystack.tools import Tool, Toolset
from haystack_integrations.components.generators.orcarouter import OrcaRouterChatGenerator

# Create individual tools
weather_tool = Tool(name="weather", description="Get weather info", ...)
news_tool = Tool(name="news", description="Get latest news", ...)

# Group related tools into a toolset
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])

# Pass mixed tools and toolsets to the generator
generator = OrcaRouterChatGenerator(
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
)
```

For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.

### Initialization

To use this integration, you need an OrcaRouter API key. You can provide it with the `ORCAROUTER_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx).

Then, install the `orcarouter-haystack` integration:

```shell
pip install orcarouter-haystack
```

### Streaming

`OrcaRouterChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.

## Usage

### On its own

```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
OrcaRouterChatGenerator,
)

client = OrcaRouterChatGenerator(model="openai/gpt-4o-mini")
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
print(response["replies"][0].text)
```

With automatic routing and streaming:

```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
OrcaRouterChatGenerator,
)

client = OrcaRouterChatGenerator(
model="orcarouter/auto",
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
)

response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])

# check the model used for the response
print("\n\n Model used: ", response["replies"][0].meta["model"])
```

With a fallback chain:

```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
OrcaRouterChatGenerator,
)

client = OrcaRouterChatGenerator(
model="openai/gpt-4o-mini",
generation_kwargs={
"extra_body": {
"route": "fallback",
"models": [
"openai/gpt-4o-mini",
"anthropic/claude-haiku-4.5",
"google/gemini-2.5-flash",
],
}
},
)

response = client.run([ChatMessage.from_user("What is Haystack?")])
print(response["replies"][0].text)
```

### In a pipeline

```python
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
OrcaRouterChatGenerator,
)

prompt_builder = ChatPromptBuilder()
llm = OrcaRouterChatGenerator(model="openai/gpt-4o-mini")

pipe = Pipeline()
pipe.add_component("builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("builder.prompt", "llm.messages")

messages = [
ChatMessage.from_system("Give brief answers."),
ChatMessage.from_user("Tell me about {{city}}"),
]

response = pipe.run(
data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}},
)
print(response)
```
1 change: 1 addition & 0 deletions docs-website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,7 @@ export default {
'pipeline-components/generators/openaigenerator',
'pipeline-components/generators/openaiimagegenerator',
'pipeline-components/generators/openrouterchatgenerator',
'pipeline-components/generators/orcarouterchatgenerator',
'pipeline-components/generators/perplexitychatgenerator',
'pipeline-components/generators/sagemakergenerator',
'pipeline-components/generators/stackitchatgenerator',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Generators are responsible for generating text after you give them a prompt. The
| [OpenAIGenerator](generators/openaigenerator.mdx) | Enables text generation using OpenAI's large language models (LLMs). | ✅ |
| [OpenAIResponsesChatGenerator](generators/openairesponseschatgenerator.mdx) | Enables chat completion using OpenAI's Responses API with support for reasoning models. | ✅ |
| [OpenRouterChatGenerator](generators/openrouterchatgenerator.mdx) | Enables chat completion with any model hosted on OpenRouter. | ✅ |
| [OrcaRouterChatGenerator](generators/orcarouterchatgenerator.mdx) | Enables chat completion using models routed through OrcaRouter. | ✅ |
| [PerplexityChatGenerator](generators/perplexitychatgenerator.mdx) | Enables chat completion using models via the Perplexity Agent API. | ✅ |
| [SagemakerGenerator](generators/sagemakergenerator.mdx) | Enables text generation using LLMs deployed on Amazon Sagemaker. | ❌ |
| [STACKITChatGenerator](generators/stackitchatgenerator.mdx) | Enables chat completions using the STACKIT API. | ✅ |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,7 @@
"pipeline-components/generators/openairesponseschatgenerator",
"pipeline-components/generators/openaigenerator",
"pipeline-components/generators/openrouterchatgenerator",
"pipeline-components/generators/orcarouterchatgenerator",
"pipeline-components/generators/perplexitychatgenerator",
"pipeline-components/generators/sagemakergenerator",
"pipeline-components/generators/stackitchatgenerator",
Expand Down Expand Up @@ -799,4 +800,4 @@
]
}
]
}
}
Loading