fix(sglang): treat streamed usage metadata as optional

[Cirilus]

Overview:

Fixes a SGLang token-stream failure where a final chunk with finish_reason can omit completion usage fields in meta_info, causing the decode worker to raise KeyError: 'prompt_tokens'.

Details:

SGLang streaming responses do not guarantee usage metadata on every streamed chunk. SGLang's own streaming path treats these fields as optional and accumulates them via meta.get(...): https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/openai/serving_responses.py#L2033-L2036.

_process_token_stream previously accessed meta_info["prompt_tokens"], meta_info["completion_tokens"], and meta_info["cached_tokens"] unconditionally when finish_reason was present. This can raise KeyError on final chunks that omit usage metadata.

This PR changes the handler to read those fields optionally and only attach completion_usage when both prompt_tokens and completion_tokens are available. It also adds regression coverage for a final token-stream chunk with finish_reason but without usage metadata.

Where should the reviewer start?

• components/src/dynamo/sglang/request_handlers/llm/decode_handler.py
• components/src/dynamo/sglang/tests/test_sglang_decode_handler.py

Related Issues

🔗 This PR is linked to an issue:

• Closes #XXXX

---

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of streamed completion results when token-usage details are missing, reducing the chance of crashes during generation.
  • The app now more reliably returns finish information even when some usage metadata isn’t available.
• Tests
  • Added coverage for streamed responses that omit completion-usage fields, ensuring the expected finish output is still produced.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi Cirilus! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[coderabbitai]

Walkthrough

_process_token_stream in DecodeWorkerHandler now uses .get() lookups for prompt_tokens, completion_tokens, and cached_tokens from meta_info, emitting completion_usage only when both token counts are present. A new async test covers the case where usage fields are absent.

Defensive completion_usage handling

Layer / File(s)	Summary
Optional completion_usage emission and test components/src/dynamo/sglang/request_handlers/llm/decode_handler.py, components/src/dynamo/sglang/tests/test_sglang_decode_handler.py	_process_token_stream switches from hard indexing to .get() for token-count fields and conditionally builds completion_usage; new async test asserts that a stream item without usage fields still yields a chunk with correct finish_reason and empty token_ids.

🎯 1 (Trivial) | ⏱️ ~5 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is mostly complete, but the required Related Issues section still uses the placeholder Closes #XXXX instead of a real issue reference.	Replace Closes #XXXX with the actual issue number, or mark the PR as not linked to an issue using the required checkbox.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main fix: streamed usage metadata is now treated as optional.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[rmccorm4]

/ok to test 8f1f652

[rmccorm4]

Hi @Cirilus, thanks for the contribution! Please address the open comments 🙏

[datadog-official]

⚠️ Warnings

🚦 1 Pipeline job failed

Docs link check | lychee     

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b76cc97 | Docs | Give us feedback!}

[Cirilus]

Hi @Cirilus, thanks for the contribution! Please address the open comments 🙏

Hi @rmccorm4, I replied to the review thread as well. Could you please take another look?

[devin-ai-integration]

Devin Review found 0 new potential issues.

[rmccorm4]

/ok to test b76cc97