Skip to content

Add defensive gzip decompression for backend responses#63

Open
clintonb wants to merge 2 commits into
stainless-api:mainfrom
voriteam:gzip-guard-backend-responses
Open

Add defensive gzip decompression for backend responses#63
clintonb wants to merge 2 commits into
stainless-api:mainfrom
voriteam:gzip-guard-backend-responses

Conversation

@clintonb

Copy link
Copy Markdown

Streamable backend responses occasionally reach mcp-go's JSON decoder still gzip-compressed, failing with invalid character '\x1f' looking for beginning of value. One known trigger is an explicit Accept-Encoding on the outgoing request disabling Go's transparent decompression (fixed at the source in #62), but compressed bytes reaching the decoder is worth guarding against regardless — a double-encoding edge or any future header regression produces the same hard-to-diagnose failure.

A gzipGuardTransport RoundTripper now wraps the streamable backend HTTP client. If a response still carries Content-Encoding: gzip, or an application/json body starts with the gzip magic bytes, the body is wrapped in a gzip reader — looping to handle double-encoded bodies, capped at depth 5. The sniff uses a non-consuming bufio.Reader.Peek, so plain JSON, empty bodies, and false alarms pass through byte-identical, and text/event-stream responses are never touched. When the guard fires it logs a WARN with the backend host, request method, response status, the outgoing Accept-Encoding, content headers, and which trigger fired — in production this telemetry is what identified the root cause fixed in #62.

Unit tests cover plain passthrough, header-path and sniffed decompression, double-gzip, SSE passthrough, empty bodies, close propagation to the underlying body, and a gzip header with a non-gzip body.

🤖 Generated with Claude Code

Some hosted MCP backends intermittently return tool-call responses that
are still gzip-compressed when they reach mcp-go's JSON decoder, which
fails with "invalid character '\x1f' looking for beginning of value". A
response-side RoundTripper on streamable backends now sniffs
Content-Encoding and the gzip magic bytes on application/json bodies,
decompresses (looping for double-encoded bodies), and logs a WARN with
the request method, response status, Accept-Encoding, and trigger so
occurrences reveal whether the backend double-encodes or Go skipped
transparent decompression. SSE responses pass through untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a gzipGuardTransport (an http.RoundTripper) to transparently handle and decompress gzip-encoded response bodies (including nested/double-encoded bodies) before they reach the JSON decoder in the streamable HTTP client. It also includes comprehensive unit tests covering various response scenarios. The review feedback suggests optimizing memory allocation and reducing GC pressure by using bufio.NewReaderSize with a minimal buffer size instead of the default 4KB bufio.NewReader when peeking for gzip magic bytes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/client/client.go Outdated
return resp, nil
}

br := bufio.NewReader(resp.Body)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

By default, bufio.NewReader allocates a 4KB buffer. Since this transport wraps every streamable HTTP client response, allocating 4KB per response can lead to significant memory overhead and GC pressure under high throughput. Since we only need to peek at the first 2 bytes to check for the gzip magic header, we can use bufio.NewReaderSize with a small buffer size (e.g., 16 bytes, which is the minimum buffer size in Go's bufio package) to dramatically reduce memory allocations.

Suggested change
br := bufio.NewReader(resp.Body)
br := bufio.NewReaderSize(resp.Body, 16)

Comment thread internal/client/client.go Outdated
break
}
body.closers = append(body.closers, gz)
br = bufio.NewReader(gz)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similarly, we can use bufio.NewReaderSize with a small buffer size here to avoid allocating a 4KB buffer for each nested decompression level.

Suggested change
br = bufio.NewReader(gz)
br = bufio.NewReaderSize(gz, 16)

gzipGuardTransport wraps every streamable response, so the default 4KB
bufio.Reader buffer was allocated per response. Only two bytes are
peeked, so use bufio.NewReaderSize with 16 (bufio's minimum); large body
reads bypass the buffer, leaving throughput unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@clintonb

Copy link
Copy Markdown
Author

Addressed the buffer-size feedback in 1766bd9: switched both bufio.NewReader calls to bufio.NewReaderSize(..., 16) via a named gzipPeekBufferSize constant, since this transport wraps every streamable response and only two bytes are peeked. 16 is bufio's minimum buffer size; large body reads bypass the buffer, so throughput is unaffected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant