feat: Add finish_reason field to StreamingChunk

[vblagoje]

Why:

Introduces a dedicated finish_reason field to StreamingChunk class to improve type safety and provide better developer experience when handling streaming finish reasons. This enhancement adds structured access to finish reasons while maintaining full backward compatibility with the existing meta-based approach.

• fixes Add finish_reason field to StreamingChunk #9471

What:

• New Field: Added finish_reason: Optional[FinishReason] field to StreamingChunk class with strict typing
• Type Safety: Created FinishReason Literal type with standard values ("stop", "length", "tool_calls", "content_filter", "tool_call_results")
• Dual Support: Updated generators to populate both new field and legacy meta field for maximum compatibility
• Enhanced Utilities: Modified utility functions to use the new structured field for better type safety
• Comprehensive Testing: Added thorough test coverage for both access patterns and all finish reason values

How can it be used:

Both approaches currently work, but we recommend using the dedicated StreamingChunk field:

# Recommended approach with better type safety and future compatibility
if chunk.finish_reason == "stop":
    add_spacing()

# Legacy approach (works now, but not guaranteed in future versions)
if chunk.meta.get("finish_reason") == "stop":
    add_spacing()

How did you test it:

• Dual Compatibility: Tested both field and meta access patterns work correctly in current version
• Type Safety: Verified strict typing works with all standard finish reason values
• Generator Coverage: Updated OpenAI and HuggingFace generators with appropriate type handling
• Utility Functions: Confirmed utility functions use new field for improved type safety

Notes for the reviewer:

Dual Support Strategy: This implementation provides current support for both access patterns:

• Structured Access: chunk.finish_reason provides type safety, better IDE support, and future compatibility
• Legacy Access: chunk.meta["finish_reason"] works for existing code but will likely be removed in future versions
• No Breaking Changes: All existing code continues to work without modification in current version

[coveralls]

Pull Request Test Coverage Report for Build 15871955598

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 59 unchanged lines in 9 files lost coverage.
• Overall coverage increased (+0.04%) to 90.219%

---

Files with Coverage Reduction	New Missed Lines	%
components/rankers/sentence_transformers_similarity.py	1	98.75%
components/generators/hugging_face_api.py	2	96.43%
components/routers/init.py	2	46.15%
dataclasses/init.py	2	23.08%
dataclasses/streaming_chunk.py	4	91.53%
components/generators/chat/openai.py	6	96.39%
components/generators/chat/hugging_face_api.py	10	94.79%
components/tools/tool_invoker.py	14	83.94%
utils/base_serialization.py	18	89.09%

Totals
Change from base Build 15819151860:	0.04%
Covered Lines:	11677
Relevant Lines:	12943

---

💛 - Coveralls

[vblagoje]

@sjrl please review and make sure that PR and migration plan make sense to you 🙏

[julian-risch]

The issue corresponding to this PR is already in the 2.15.0 milestone. That's the reason why I removed the PR from the milestone again. It's still planned to be part of the release.

[sjrl]

Hey @vblagoje thanks for your work on this! I'll do a more in-depth review today but first I had a few comments and questions.

• I don't think we need to have a migration. As in I think it's fine if we always leave the raw openai finish reason in the metadata. I don't think there is much value in removing this information from there and if users want to continue using that value then they can. So I'd ask to drop all the warnings and plan on leaving the original finish reason in the metadata.
• Could you also update the other Generators and ChatGenerators in haystack main to set finish_reason? So this would be the HuggingFace type Generators and ChatGenerators.

[sjrl]

@vblagoje could we also drop the migration and deprecation info in the PR description?

[vblagoje]

Thank you @sjrl - much appreciate taking care of this one through the finish line!