Gemma 4 - OpenAIResponseChatGenerator Tool Input error

[2-ulfhednar-2]

Describe the bug
When running the ResponseChatGenerator in an Agent WITHOUT the exit conditions being a tool (So the tool feeds back into the generator) you get the following.
haystack.core.errors.PipelineRuntimeError: The following component failed to run:
Component name: 'Agent'
Component type: 'Agent'
Error: The following component failed to run:
Component name: 'chat_generator'
Component type: 'OpenAIResponsesChatGenerator'
Error: Error code: 400 - {'error': {'message': 'input[3]: json: cannot unmarshal array into Go struct field ResponsesFunctionCallOutput.output of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}

This happens due to the responses endpoint expecting a string while the tool output is given as a list in openai_responses.py (around line 877)

Error message
haystack.core.errors.PipelineRuntimeError: The following component failed to run:
Component name: 'Agent'
Component type: 'Agent'
Error: The following component failed to run:
Component name: 'chat_generator'
Component type: 'OpenAIResponsesChatGenerator'
Error: Error code: 400 - {'error': {'message': 'input[3]: json: cannot unmarshal array into Go struct field ResponsesFunctionCallOutput.output of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}

Expected behavior
The tool output should be properly processed by the Agent.
This will become important in case you configure retrieval or generally searches as tools

Additional context
The Model was Gemma4 26B run locally through Ollama
A patch created by Opus could fix this issue

# --- Patch ---
import haystack.components.generators.chat.openai_responses as responses_mod
_original = responses_mod._convert_chat_message_to_responses_api_format

def _patched(message):
    results = _original(message)
    for item in results:
        if item.get("type") == "function_call_output" and not isinstance(item.get("output"), str):
            item["output"] = json.dumps(item["output"])
    return results

responses_mod._convert_chat_message_to_responses_api_format = _patched
# --- End Patch ---

To Reproduce
Run Ollama locally
Use a script similar to this

client = OpenAIResponsesChatGenerator(
    model="gemma4:26b",
    api_base_url="http://localhost:11434/v1",
    api_key=Secret.from_token("None"),
    streaming_callback=print_streaming_chunk,
)

def calculate(expression: str) -> str:
    try:
        result = eval(expression, {"__builtins__": {}})
        return json.dumps({"result": result})
    except Exception as e:
        return json.dumps({"error": str(e)})

calculator_tool = Tool(
    name="calculator",
    description="Evaluate basic math expressions.",
    parameters={
        "type": "object",
        "properties": {
            "expression": {
                "type": "string",
                "description": "Math expression to evaluate",
            },
        },
        "required": ["expression"],
    },
    function=calculate,
)

agent = Agent(
    chat_generator=client,
    tools=[calculator_tool],
)

response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
print(response["messages"])

You will get the error shown earlier.
Notice that exit condition for the Agent has to be dropped.
With exit condition you will just get the tool output essentially

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: Windows 11
• GPU/CPU: RX7900XTX / Ryzen 7700X
• Haystack version (commit or version number): 2.27.0
• DocumentStore: -
• Reader: -
• Retriever: -

Note
Ultimately this could also be caused by a bug in the Ollama Endpoint.
However applying the fix solved the issue.
I currently dont have any vLLM models running with the Responses endpoint so I can't currently check this properly

OpenAI reference
https://developers.openai.com/api/reference/resources/responses/methods/create#(resource)%20responses%20%3E%20(method)%20create

[anakin87]

For Ollama, we have a specific integration to be used.
I would not expect our OpenAIResponsesChatGenerator to work perfectly with non-OpenAI APIs/models.

The following works for me. I have used the E4B model, not the 26B.

from haystack_integrations.components.generators.ollama import OllamaChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.components.agents import Agent
from haystack.tools import Tool
import json
from haystack.components.generators.utils import print_streaming_chunk

client = OllamaChatGenerator(
    model="gemma4",
    streaming_callback=print_streaming_chunk,
)

def calculate(expression: str) -> str:
    try:
        result = eval(expression, {"__builtins__": {}})
        return json.dumps({"result": result})
    except Exception as e:
        return json.dumps({"error": str(e)})

calculator_tool = Tool(
    name="calculator",
    description="Evaluate basic math expressions.",
    parameters={
        "type": "object",
        "properties": {
            "expression": {
                "type": "string",
                "description": "Math expression to evaluate",
            },
        },
        "required": ["expression"],
    },
    function=calculate,
)

agent = Agent(
    chat_generator=client,
    tools=[calculator_tool],
)

response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
print(response["messages"])

@2-ulfhednar-2 let me know if this helps.

[2-ulfhednar-2]

For Ollama, we have a specific integration to be used. I would not expect our OpenAIResponsesChatGenerator to work perfectly with non-OpenAI APIs/models.

The following works for me. I have used the E4B model, not the 26B.

from haystack_integrations.components.generators.ollama import OllamaChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.components.agents import Agent
from haystack.tools import Tool
import json
from haystack.components.generators.utils import print_streaming_chunk

client = OllamaChatGenerator(
model="gemma4",
streaming_callback=print_streaming_chunk,
)

def calculate(expression: str) -> str:
try:
result = eval(expression, {"builtins": {}})
return json.dumps({"result": result})
except Exception as e:
return json.dumps({"error": str(e)})

calculator_tool = Tool(
name="calculator",
description="Evaluate basic math expressions.",
parameters={
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "Math expression to evaluate",
},
},
"required": ["expression"],
},
function=calculate,
)

agent = Agent(
chat_generator=client,
tools=[calculator_tool],
)

response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
print(response["messages"])
@2-ulfhednar-2 let me know if this helps.

This seemed to work properly.
I think should still be monitore incase a similar issue pops up with LiteLLM etc.
But it seems to be ultimately the OpenAI Endpoint implementation of Ollama.
The patch did work too tho. If I can get a Response Endpoint running model behind LiteLLM I'll check in again.

[2-ulfhednar-2]

Nvm it doesnt. At least not properly. The OllamaChatGenerator discards reasoning output like the regular chat generator.
Only the OpenAIResponsesChatGenerator currently doesnt dicard them silently.
Also regular OpenAIChatGenerator works with Ollama too since it exposes the regular OpenAI endpoint.
This seems to really be caused by v1/responses

[anakin87]

Nvm it doesnt. At least not properly. The OllamaChatGenerator discards reasoning output like the regular chat generator.
Only the OpenAIResponsesChatGenerator currently doesnt dicard them silently.

Could you better explain?

[2-ulfhednar-2]

Nvm it doesnt. At least not properly. The OllamaChatGenerator discards reasoning output like the regular chat generator.
Only the OpenAIResponsesChatGenerator currently doesnt dicard them silently.

Could you better explain?

OllamaChatGenerator output:

` instead. See https://docs.haystack.deepset.ai/docs/custom-components#requirements for details.
chunk.start = start
[TOOL CALL]
Tool: calculator
Arguments: {"expression": "7 * (4 + 2)"}

[ASSISTANT]
7 * (4 + 2) = 42

[ChatMessage(_role=<ChatRole.USER: 'user'>, _content=[TextContent(text='What is 7 * (4 + 2)?')], _name=None, _meta={}), ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[ToolCall(tool_name='calculator', arguments={'expression': '7 * (4 + 2)'}, id=None, extra=None)], _name=None, _meta={'model': 'gemma4:26b', 'done': True, 'total_duration': 1014082700, 'load_duration': 142173200, 'prompt_eval_duration': 213088200, 'eval_duration': 652214000, 'logprobs': None, 'role': 'assistant', 'finish_reason': 'stop', 'completion_start_time': '2026-04-07T08:20:05.2652337Z', 'usage': {'completion_tokens': 24, 'prompt_tokens': 74, 'total_tokens': 98}}), ChatMessage(_role=<ChatRole.TOOL: 'tool'>, _content=[ToolCallResult(result='{"result": 42}', origin=ToolCall(tool_name='calculator', arguments={'expression': '7 * (4 + 2)'}, id=None, extra=None), error=False)], _name=None, _meta={}), ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text='7 * (4 + 2) = 42')], _name=None, _meta={'model': 'gemma4:26b', 'done': True, 'total_duration': 677666600, 'load_duration': 158916000, 'prompt_eval_duration': 70036000, 'eval_duration': 444408900, 'logprobs': None, 'role': 'assistant', 'finish_reason': 'stop', 'completion_start_time': '2026-04-07T08:20:05.9450061Z', 'usage': {'completion_tokens': 17, 'prompt_tokens': 110, 'total_tokens': 127}})]

compared to OpenAIResponsesChatGenerator Output with the patch applied

[REASONING]
The user is asking for the result of a mathematical expression: 7 * (4 + 2).
I should follow the order of operations (PEMDAS/BODMAS):

1. Parentheses: (4 + 2) = 6
2. Multiplication: 7 * 6 = 42

I will use the calculator tool to confirm this.

[TOOL CALL]
Tool: calculator
Arguments: {"expression":"7 * (4 + 2)"}

[ASSISTANT]
The result of 7 * (4 + 2) is 42.

The regular OllamaChatGenerator has issues with the recieved reasoning chunks. They seem to get dicarded.

[anakin87]

Thank you... I'll investigate.

[2-ulfhednar-2]

Thank you... I'll investigate.

My suggestion would to also verify if OpenAIResponsesChatGenerator works with vLLM this should expose the same endpoint for smth like gemma4:26b Ollama exposes and iirc vllm should run with OpenAI generators through the invocation layer.
If that throws the same error this might ultimately not be Ollama specific.

For the reasoning all regular chat generators seem to discard that not just the Ollama one.

[reallyticsai]

This issue with OpenAIResponsesChatGenerator appears to stem from a type mismatch between what the component is outputting and what the API is expecting. When the tool outputs a list instead of a string, it creates a JSON parsing problem, which is highlighted in the error message.

In production, we've faced similar challenges when integrating AI responses into our pipelines. A couple of things you can try:

• Modify the Output: Before passing the output to the Agent, ensure that it is converted to the expected string format. You can use json.dumps() to serialize the list into a JSON string.
• Check the Tool Logic: Look into the logic around the openai_responses.py file around line 877. If the tool is consistently outputting a list, it might need an adjustment to handle single responses correctly.

Making these adjustments should help align the data types and avoid the unmarshal error.

[anakin87]

@2-ulfhednar-2 could you try setting the think parameter? It seems to work

client = OllamaChatGenerator(
    model="gemma4",
    streaming_callback=print_streaming_chunk,
    think=True,
)

[2-ulfhednar-2]

@2-ulfhednar-2 could you try setting the think parameter? It seems to work

client = OllamaChatGenerator(
model="gemma4",
streaming_callback=print_streaming_chunk,
think=True,
)

Yes this works.
I am more concerned now that this will not work on vLLM hosted version.
Can't verify if OpenAIResponsesChatGenerator works with those or if that too throws an error like
Error: Error code: 400 - {'error': {'message': 'input[3]: json: cannot unmarshal array into Go struct field ResponsesFunctionCallOutput.output of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}

[anakin87]

For vLLM, we are currently working on a targeted integration: deepset-ai/haystack-core-integrations#3099. I'll make sure that Gemma works properly.