fix: emit native Bedrock invoke stream chunks#144
Conversation
|
cc @jpr5 — bringing this to your attention since you authored the original Bedrock streaming and Bedrock ResponseOverrides paths (PRs #133, #135) and Summary:
Happy to iterate on anything you want adjusted (event field naming, stream sequencing, fixture matching semantics, etc.). |
commit: |
jpr5
force-pushed
the
fix/bedrock-invoke-stream-tool-use
branch
from
5fd6183 to
edf6fb1
Compare
InvokeModelWithResponseStream wraps Anthropic native stream events in Bedrock chunk frames, while ConverseStream keeps the Converse event schema.
The "no thinking block when reasoning is absent" test for invoke-with-response-stream used Converse-style event filters (eventType "contentBlockDelta") instead of Anthropic-native filters (payload.type "content_block_delta"), making it pass vacuously.
Adds integration tests for the isContentWithToolCallsResponse path through both invoke-with-response-stream (Anthropic-native format) and converse-stream (Converse camelCase format). This path previously had zero integration test coverage.
jpr5
force-pushed
the
fix/bedrock-invoke-stream-tool-use
branch
from
edf6fb1 to
7ff27b0
Compare
|
@sf-jin-ku Thanks, appreciate you! 🙏 Had to rewrite commits to pass commitlint and prettier, plus fix a minor item or two, but retained your authorship. Welcome to the club! |
## Summary Remediates all 15 debt items from the PR #144 code review backlog plus 8 pre-existing issues surfaced during CR. Includes the PR #144 native stream format changes that landed after v1.16.1. ### Fixed - **Bedrock invoke native stream format** — `invoke-with-response-stream` now emits Anthropic-native snake_case payloads (`content_block_delta`, `input_json_delta`) wrapped in Bedrock EventStream `chunk` frames, instead of Converse-style camelCase events. Converse-stream retains camelCase format. (PR #144, sf-jin-ku) - **Bedrock invoke false-green test** — Reasoning negative test used wrong event filters, masking a real bug (PR #144) - **Bedrock invoke/stream hardening** — `completionReq.stream = true` in streaming handler; deterministic `tool_use_${index}` fallback IDs; `textContent ?? null` to preserve empty strings; unsupported content block and unexpected role warnings; webSearches warning on tool-call-only responses - **Converse stream shape alignment** — Wrap `contentBlockStop` and `messageStop` payloads to match real AWS Converse API; remove duplicate top-level `contentBlockIndex`; add trailing `metadata` events (usage + latencyMs) to all stream builders - **Converse request conversion** — Filter empty-string text blocks; unwrap `inputSchema` from `{ json: {...} }` Converse API wrapper; `completionReq.stream = true` in streaming handler; content-loss warnings; error type `??` fix ### Changed - Shared test helpers (`createMockReq`/`createMockRes`/`createDefaults`) in `helpers/mock-res.ts` - Per-test server lifecycle in reasoning-all-providers - Content+toolCalls streaming integration coverage for both invoke and converse paths (PR #144) ## Test plan - [x] All 2645 tests pass (`pnpm test`) - [x] Format check clean (`pnpm run format:check`) - [x] Lint clean (`pnpm run lint`) - [x] CI green (22/22 checks on prior push; monitoring current push)
2 participants
Summary
invoke-with-response-streamto emit Anthropic-native Claude invoke stream payloads wrapped in Bedrock EventStreamchunkframes.converse-streamon Converse-style camelCase payloads by moving those builders into the Converse implementation.input_json_delta.partial_json, so realboto3Bedrock Runtime consumers can parse tool calls correctly.Why
invoke_model_with_response_streamandconverse_streamare separate Bedrock Runtime APIs with different stream payload shapes. The invoke stream endpoint was returning Converse-style events such ascontentBlockDelta/delta.toolUse.input, but Anthropic native invoke streams return events such ascontent_block_delta/delta.type = input_json_delta/delta.partial_json.This breaks clients that use
boto3.client("bedrock-runtime").invoke_model_with_response_stream(...)and parse Anthropic Claude invoke payloads.Test plan
corepack pnpm -C /Users/jinku/sendbird/tmp/aimock test src/__tests__/bedrock-stream.test.ts src/__tests__/bedrock.test.ts src/__tests__/reasoning-all-providers.test.ts— 176 passedcorepack pnpm -C /Users/jinku/sendbird/tmp/aimock test— 2573 passed, 36 skipped