bug: ext_proc body parsing fails on chunked Transfer-Encoding requests

[yossiovadia]

Summary

The ext_proc handler fails to parse request bodies when clients send Transfer-Encoding: chunked. Two distinct failure modes observed, both traced to FULL_DUPLEX_STREAMED request body mode interacting with chunked TE:

1. Framing contamination — chunked TE hex/CRLF bytes leak into body → "invalid character 'a' looking for beginning of value"
2. Cross-stage truncation — when two ext_proc filters are chained, the second filter receives a truncated body → "unexpected end of JSON input"

Both are 100% reproducible with chunked TE. Neither occurs with Content-Length.

Root Cause

Why FULL_DUPLEX_STREAMED is used

The framework removed buffered mode in PR #38 ("remove buffered mode, use streaming only") and PR #169 added response chunk processing for streaming metering. FULL_DUPLEX_STREAMED is needed for responses (SSE streaming from providers), but requests don't need streaming — the framework accumulates the full request body in memory before parsing (server.go:162-168).

The EnvoyFilter sets FULL_DUPLEX_STREAMED for both directions because the framework is streaming-only. However, Envoy's ext_proc processing_mode allows setting request and response body modes independently.

What goes wrong

When request_body_mode: FULL_DUPLEX_STREAMED and the client sends Transfer-Encoding: chunked:

Bug 1 (single filter): Envoy passes raw chunked framing bytes into ext_proc body chunks. The accumulated body contains a0f\r\n{"model":"gpt-5.5"...}\r\n0\r\n\r\n instead of clean JSON. json.Unmarshal at request.go:67 fails.

Bug 2 (chained filters): Cross-stage log tracing with a single x-request-id shows:

• payload-pre-processing: receives ~170 body chunks → reassembles correctly → parses model: gpt-5.5 ✅
• payload-processing: receives one truncated chunk with EoS:true → unexpected end of JSON input ❌

The first filter gets the complete body; the second filter gets a truncated copy. The context canceled on the first filter is an effect (stage 2 fails → Envoy tears down the chain), not the cause.

Reproduction

# Generate a test body (~43KB)
python3 -c "
import json
msgs = [{'role':'user','content':'analyze this: ' + 'x'*500}]
for i in range(20):
    msgs.append({'role':'assistant','content':[{'type':'output_text','text':'result ' + 'y'*2000}]})
    msgs.append({'role':'user','content':'check function ' + str(i)})
json.dump({'model':'gpt-5.5','input':msgs}, open('body.json','w'))
"

# FAILS — 100% failure rate (chunked TE)
curl -T body.json -H "Transfer-Encoding: chunked" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  https://<gateway>/v1/responses

# WORKS — 100% success rate (Content-Length)
curl -d @body.json \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  https://<gateway>/v1/responses

Production frequency: ~7 occurrences per 48h with Claude Code / Codex clients, auto-retried.

Suggested Fix

Primary fix (EnvoyFilter / operator): Change request_body_mode from FULL_DUPLEX_STREAMED to BUFFERED in the EnvoyFilter. Envoy then de-chunks and buffers the complete request body before sending to ext_proc, fixing both symptoms. Response body mode stays FULL_DUPLEX_STREAMED for SSE streaming — the two modes are independent per the ext_proc spec.

The framework already accumulates the full request body in memory before parsing (server.go:162-168), so BUFFERED matches the actual behavior. The pre-processing filter (read-only model extraction) especially has no need for request streaming.

The EnvoyFilter is generated by the MaaS operator (maas.opendatahub.io/v1alpha1 Config/default), so the one-line change belongs in the operator's kustomize manifests (deployment/base/payload-processing/manager/envoy-filter.yaml in opendatahub-io/models-as-a-service).

Defensive fix (framework, PR #210): tryDecodeChunked() in HandleRequestBody detects and strips chunked TE framing before json.Unmarshal. This fixes bug 1 (framing contamination) for any deployment using FULL_DUPLEX_STREAMED, but cannot fix bug 2 (truncated body — the bytes are genuinely missing). Validated on cluster: 15/15 chunked TE requests pass with the framework fix.

Environment

• llm-d-inference-payload-processor v0.1.0-rc.3
• Envoy ext_proc with request_body_mode: FULL_DUPLEX_STREAMED
• Two chained ext_proc filters: payload-pre-processing + payload-processing
• Observed with OpenAI Responses API (/v1/responses) and Anthropic Messages API (/v1/messages)
• Clients: Codex CLI, Claude Code

[yossiovadia]

cc @nirrozenbaum — PR #210 has a fix with tests and cluster validation. Happy to adjust the approach if you prefer a different solution.

[yossiovadia]

Update: root cause fully traced — two symptoms, one bug

Cross-stage log tracing on the cluster (single x-request-id across both ext_proc stages) revealed the full picture:

Setup: The gateway chains two ext_proc filters (payload-pre-processing → payload-processing), both configured with request_body_mode: FULL_DUPLEX_STREAMED.

Trace for a failing request:

• payload-pre-processing: receives ~170 body chunks (EoS:false...→EoS:true) → reassembles correctly → parsed field from body: model=gpt-5.5 ✅
• payload-processing: receives one truncated chunk with EoS:true → unexpected end of JSON input ❌
• The context canceled on pre-processing is an effect (stage 2 fails → Envoy tears down the chain), not the cause.

One bug, two symptoms:

1. Framing contamination → "invalid character 'a'" — chunked TE hex/CRLF bytes leak into body. Fixed by PR fix: handle chunked transfer-encoding in ext_proc request body #210's tryDecodeChunked().
2. Truncated body → "unexpected end of JSON input" — second ext_proc stage receives incomplete body. PR fix: handle chunked transfer-encoding in ext_proc request body #210 cannot fix this (the bytes are genuinely missing, not contaminated with framing).

Both only reproduce with Transfer-Encoding: chunked, NOT with Content-Length. That's why the 2.5MB Content-Length stress test showed 50/50 success.

Architectural fix: Change request_body_mode from FULL_DUPLEX_STREAMED to BUFFERED in the EnvoyFilter. This makes Envoy de-chunk and buffer the complete body before sending to ext_proc, fixing both symptoms. The pre-processing stage (read-only model extraction) has no need for streaming. The main processing stage already accumulates the full body in memory (server.go:133-164) before parsing, so BUFFERED mode matches the actual behavior without losing functionality.

The EnvoyFilter is owner-referenced to maas.opendatahub.io/v1alpha1 Config/default, so the fix belongs in the operator that renders it, not as a direct patch.

PR #210 remains valuable as a defensive fallback in the framework for deployments that use FULL_DUPLEX_STREAMED, but the root fix is in the Envoy configuration layer.

[yossiovadia]

Update: BUFFERED mode doesn't work — framework removed support in PR #38

We tested switching request_body_mode to BUFFERED on the cluster. Result: 504 Gateway Timeout on all requests. The framework only handles FULL_DUPLEX_STREAMED body delivery since PR #38 removed buffered mode. With BUFFERED, Envoy sends the body differently (single RequestBody message) and the framework doesn't process it.

Reverted immediately — cluster is back to normal.

Where we are

• PR fix: handle chunked transfer-encoding in ext_proc request body #210 (this repo) fixes bug 1 (chunked TE framing contamination) — validated, works.
• Bug 2 (cross-stage body truncation) remains unfixed. The BUFFERED config fix we proposed needs the framework to support it first.

Two paths forward

Path A — Re-add buffered request body support to the framework:
The framework already accumulates the full request body before parsing (server.go:162-168). Re-adding BUFFERED mode for requests only (not responses) would let the EnvoyFilter switch request body delivery to BUFFERED while keeping FULL_DUPLEX_STREAMED for response streaming. This is the cleanest architectural split — requests don't need streaming, responses do.

Path B — Fix the streaming handler to be resilient to cross-stage truncation:
When two ext_proc filters are chained with FULL_DUPLEX_STREAMED, the second filter intermittently receives a truncated body. The framework could detect this (e.g., validate that the accumulated body is complete JSON before parsing, and if not, return a retryable error instead of BadRequest) or implement a retry/wait mechanism for the missing chunks.

@nirrozenbaum — you removed buffered mode in PR #38 for simplification. Given that FULL_DUPLEX_STREAMED has issues with chunked TE and chained filters, what's your preferred approach? Happy to implement whichever path you think is right.