Add defensive gzip decompression for backend responses#63
Conversation
Some hosted MCP backends intermittently return tool-call responses that are still gzip-compressed when they reach mcp-go's JSON decoder, which fails with "invalid character '\x1f' looking for beginning of value". A response-side RoundTripper on streamable backends now sniffs Content-Encoding and the gzip magic bytes on application/json bodies, decompresses (looping for double-encoded bodies), and logs a WARN with the request method, response status, Accept-Encoding, and trigger so occurrences reveal whether the backend double-encodes or Go skipped transparent decompression. SSE responses pass through untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a gzipGuardTransport (an http.RoundTripper) to transparently handle and decompress gzip-encoded response bodies (including nested/double-encoded bodies) before they reach the JSON decoder in the streamable HTTP client. It also includes comprehensive unit tests covering various response scenarios. The review feedback suggests optimizing memory allocation and reducing GC pressure by using bufio.NewReaderSize with a minimal buffer size instead of the default 4KB bufio.NewReader when peeking for gzip magic bytes.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| return resp, nil | ||
| } | ||
|
|
||
| br := bufio.NewReader(resp.Body) |
There was a problem hiding this comment.
By default, bufio.NewReader allocates a 4KB buffer. Since this transport wraps every streamable HTTP client response, allocating 4KB per response can lead to significant memory overhead and GC pressure under high throughput. Since we only need to peek at the first 2 bytes to check for the gzip magic header, we can use bufio.NewReaderSize with a small buffer size (e.g., 16 bytes, which is the minimum buffer size in Go's bufio package) to dramatically reduce memory allocations.
| br := bufio.NewReader(resp.Body) | |
| br := bufio.NewReaderSize(resp.Body, 16) |
| break | ||
| } | ||
| body.closers = append(body.closers, gz) | ||
| br = bufio.NewReader(gz) |
gzipGuardTransport wraps every streamable response, so the default 4KB bufio.Reader buffer was allocated per response. Only two bytes are peeked, so use bufio.NewReaderSize with 16 (bufio's minimum); large body reads bypass the buffer, leaving throughput unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the buffer-size feedback in 1766bd9: switched both |
Streamable backend responses occasionally reach mcp-go's JSON decoder still gzip-compressed, failing with
invalid character '\x1f' looking for beginning of value. One known trigger is an explicitAccept-Encodingon the outgoing request disabling Go's transparent decompression (fixed at the source in #62), but compressed bytes reaching the decoder is worth guarding against regardless — a double-encoding edge or any future header regression produces the same hard-to-diagnose failure.A
gzipGuardTransportRoundTripper now wraps the streamable backend HTTP client. If a response still carriesContent-Encoding: gzip, or anapplication/jsonbody starts with the gzip magic bytes, the body is wrapped in a gzip reader — looping to handle double-encoded bodies, capped at depth 5. The sniff uses a non-consumingbufio.Reader.Peek, so plain JSON, empty bodies, and false alarms pass through byte-identical, andtext/event-streamresponses are never touched. When the guard fires it logs a WARN with the backend host, request method, response status, the outgoingAccept-Encoding, content headers, and which trigger fired — in production this telemetry is what identified the root cause fixed in #62.Unit tests cover plain passthrough, header-path and sniffed decompression, double-gzip, SSE passthrough, empty bodies, close propagation to the underlying body, and a gzip header with a non-gzip body.
🤖 Generated with Claude Code