We have expanded StreamingChunk to include new fields such as
content: str
meta: Dict[str, Any] = field(default_factory=dict, hash=False)
component_info: Optional[ComponentInfo] = field(default=None)
# All fields below here are new
index: Optional[int] = field(default=None)
tool_calls: Optional[List[ToolCallDelta]] = field(default=None)
tool_call_result: Optional[ToolCallResult] = field(default=None)
start: bool = field(default=False)
finish_reason: Optional[FinishReason] = field(default=None)
so we should update our existing ChatGenerator integrations to use these new fields. This will standardize the streaming in Haystack making it easy to use one streaming_callback like our print_streaming_chunk for all ChatGenerators.
This really emphasizes the interoperability of Haystack and will make it very straightforward to switch out to different ChatGenerators from different providers in a pipeline or when using a ChatGenerator within the Agent component.
When working on this I'd suggest:
- Testing with
haystack.components.generators.utils.print_streaming_chunk to make sure that the output looks clean. Ideally use the ChatGenerator in an Agent and try to get the Agent to output at least two tool calls in one round. You can refer to the PR description of this PR you can see example outputs of print_streaming_chunk
- Refer to the OpenAIChatGenerator implementation to get inspiration:
We have expanded
StreamingChunkto include new fields such asso we should update our existing ChatGenerator integrations to use these new fields. This will standardize the streaming in Haystack making it easy to use one
streaming_callbacklike ourprint_streaming_chunkfor all ChatGenerators.This really emphasizes the interoperability of Haystack and will make it very straightforward to switch out to different ChatGenerators from different providers in a pipeline or when using a ChatGenerator within the Agent component.
When working on this I'd suggest:
haystack.components.generators.utils.print_streaming_chunkto make sure that the output looks clean. Ideally use the ChatGenerator in an Agent and try to get the Agent to output at least two tool calls in one round. You can refer to the PR description of this PR you can see example outputs ofprint_streaming_chunk_handle_stream_responsewhich loops through each stream object from OpenAI and then converts it into aStreamingChunk_convert_chat_completion_chunk_to_streaming_chunkwhich converts each individual stream object from OpenAI and converts it into a singularStreamingChunk_convert_streaming_chunks_to_chat_messagewhich is a general utility method that converts a list ofStreamingChunks into a HaystackChatMessage