Expose `timeout` and `max_retries` in `ChatGenerator`s to help work around rate limits

[mathislucka]

Is your feature request related to a problem? Please describe.
At the hackathon today we ran into quite a few rate limit issues with the OpenAI and Anthropic API. The main problem is that the number of tokens exceeds the rate limit for number of input tokens per minute. Because these agents might make many tool calls per minute, the number of input tokens accumulates quickly.

Describe the solution you'd like
We subclassed the AnthropicChatGenerator and overwrote the run method so that the call to Anthropic would be retried for rate limit errors after a 60 second waiting time.

This worked but I could imagine more sophisticated ways were maybe users could specify rate limits for the Agent and it could then wait once a request would hit the limit. The chat generators would benefit from simple retry mechanisms.

Update:
We've opted to use the builtin retry mechanisms of the underlying SDKs from the LLM providers.

Remaining ChatGenerators missing timeout and/or max_retries options:

• CohereChatGenerator
• GoogleGenAIChatGenerator
• MetaLlamaChatGenerator
• OllamaChatGenerator

[sjrl]

Just to provide some context. Depending on the provider you use it is already possible to have retries set up. For example, with the OpenAIChatGenerator

from haystack.components.generators.chat.openai import OpenAIChatGenerator

gen = OpenAIChatGenerator(max_retries=5)

This handles retries including with backoffs.

Similarly the AnthropicChatGenerator could also support this, but we would need to expose the max_retries parameter of the Anthropic client. By default it seems to be set to 2.

[sjrl]

As a follow up we have added this feature to AnthropicChatGenerator in this PR deepset-ai/haystack-core-integrations#1952

So the task for this would be to check if our other Chat Generators also expose parameters like max_retries and timeout

[vblagoje]

I ran into very similar rate limit issues while building itinerary-agent. When I had just one main orchestration agent doing everything the chat history was growing super fast and each call was burning tons of tokens for very little actual value. In fact, filling context with useless tokens, made things worse. Switching to a subagent design helped a lot. Subagents just branch off with only the needed context already prepared and they throw away the full chat history from the parent or orchestrator. This alone reduced token usage massively and basically stopped hitting rate limits. It could also be useful to have something like agent_history_depth parameter so chat generators do not always include the entire history by default. Anyway I do not think there is one magic knob you just turn and it solves it. Probably better to collect and share a few best practices like using subagents, trimming context, limiting history depth etc. Thoughts @sjrl ?

[sjrl]

Probably better to collect and share a few best practices like using subagents, trimming context, limiting history depth etc.

I think this sounds like a good idea! Some other things we could do:

• Check that all ChatGenerators expose max_retries and timeout so we can help navigate rate limit errors even when context windows are effectively managed.
• Completing this issue New ChatGenerator fallback component for quota limits/API errors #9786 would also be helpful by allowing users to fallback to other providers if an issue arises.
• And again what you say, provide best practices surrounding agent design and ways to limit history length, context window size, etc.

[vblagoje]

Exactly @sjrl and I’d just add that we should lean more into the definition and use of subagents in our docs. What we described above is actually referred to as “context preservation” in the Claude docs on sub-agents, and it’s a well documented pattern worth adopting. This Claude doc page basically talks about all these aspects in detail :-)

[yaowubarbara]

Hi, I'd like to take this one! I looked into the four remaining generators and here's my plan: MetaLlamaChatGenerator is the simplest since it inherits from OpenAIChatGenerator which already supports both parameters, so I just need to expose them in the subclass init and pass them through. OllamaChatGenerator already has timeout support so I'll check if the ollama SDK supports max_retries and add it if possible. For CohereChatGenerator and GoogleGenAIChatGenerator I'll follow the same pattern as the Anthropic implementation, only passing non-None values to respect SDK defaults. I'm planning to submit one PR per integration to keep reviews manageable, that approach works for you?