Codec: token-native binary transport for completions streaming

[wdunn001]

Summary

Adds a binary streaming wire format (MessagePack and length-prefixed Protocol Buffers) that ships raw uint32 token IDs end-to-end instead of detokenizing every chunk to UTF-8 and wrapping it in JSON Server-Sent Events. Opt-in via the request-body field stream_format. Default behaviour is byte-identical to current sglang.

Why

Real cross-stack measurements (run id 2026-05-15T20-00-00Z, 2,000-token completion on Qwen2.5-0.5B-Instruct; full matrix at https://github.com/wdunn001/Codec/blob/main/packages/bench/results/2026-05-15T20-00-00Z/MATRIX.md):

Engine	JSON-SSE	Best Codec	Reduction
sglang	485.2 KB	291 B (msgpack + dict-zstd)	1,707×
vllm	517.8 KB	3.9 KB (msgpack + gzip)	137×
llama.cpp	528.8 KB	140 B (msgpack + dict-zstd, fp16)	3,868×

The savings come from:

1. Not detokenizing at the serving server (no UTF-8 / JSON envelope per chunk).
2. Letting the client opt into HTTP-level compression (gzip, br, zstd with pre-trained dictionaries) on the binary stream.
3. Skipping the re-tokenize round-trip in agent-to-agent and tool-dispatch hops — the consumer reads uint32 IDs directly and feeds them to the next model.

What this PR adds (additive only)

13 files, +2,690 / -2 lines. All under python/sglang/srt/entrypoints/.

New modules:

• codec_frame.py — MessagePack + Protocol Buffers encoders for CodecFrame {ids[], done, finish_reason?, tool_calls?}. Hand-rolled protobuf (no codegen step). Accepts Union[List[int], numpy.ndarray[uint32], array.array('I'), bytes] on the ids parameter so a future upstream change to surface output_ids as a buffer can be wired in without further encoder churn.
• codec_compression.py — Accept-Encoding negotiation for zstd (with pre-trained dictionary), br, gzip, identity. Emits Codec-Zstd-Dict: sha256:<hex> on every zstd response so clients can verify they have the matching dict loaded before decompressing.
• codec_agent.py — ToolWatcher: a uint32-compare state machine that detects delimited regions (tool calls, reasoning blocks, multimodal spans) in the raw token stream without detokenizing. ~100× faster than detokenize+regex in the steady state. Per-request opt-in via the tool_watcher request field.
• codec_dispatcher.py — bolt-on tool dispatch (default off, CODEC_BOLT_ON_DISPATCH=1). Reads tool manifests from CODEC_TOOL_MANIFEST_URLS at boot, hash-validates each manifest's tokenizerHash against the active model's tokenizer, POSTs CodecToolCall (msgpack-framed) to registered tools when ToolWatcher fires, reinjects response_ids into the generation stream — full <tool_call>...</tool_call> loop without ever decoding the model's stream to text.
• codec_version.py — protocol-version negotiation (Codec-Client-Version, Codec-Min-Version headers + 426 Upgrade Required + VERSION_INCOMPATIBLE frame).

5 modified existing files:

• openai/serving_completions.py + openai/serving_chat.py — dispatch to the binary generator when stream_format != "json"; preserve the existing JSON-SSE path byte-for-byte when unset or "json".
• openai/protocol.py — stream_format and tool_watcher fields on the request types.
• engine.py + http_server.py — /codec/schema endpoint registration (returns the protobuf schema text for client code generation).

Plus 3 new test files: test_codec_agent.py, test_codec_compression.py, test_codec_version.py.

Trust posture

• Default behaviour byte-identical to current sglang. A client that does NOT set stream_format gets JSON-SSE exactly as today.
• No new mandatory dependencies. The wire emit uses msgspec (already in sglang's reqs); compression uses brotli + zstandard (missing → graceful fallthrough to identity per the Accept-Encoding negotiation rules).
• Engine boot unchanged unless opt-in. CODEC_BOLT_ON_DISPATCH=1 loads manifests lazily; without it nothing in the codec path runs at boot.

Test plan

• pytest python/sglang/srt/entrypoints/test_codec_*.py green (3 new test files in this PR).
• stream_format=msgpack round-trip via curl against a Qwen2.5-0.5B-Instruct engine; response Content-Type application/x-msgpack, decode reproduces token IDs end-to-end.
• Existing stream_format unset / stream_format=json requests produce byte-identical JSON-SSE to current main (responses regression-tested in the bench harness referenced above).
• Accept-Encoding: zstd, br, gzip negotiates correctly per RFC 7231 §5.3.4 preference order; Codec-Zstd-Dict header present on zstd responses.

Cross-stack reference + spec

• Wire format spec: https://github.com/wdunn001/Codec/blob/main/spec/versions/v0.5.md
• .well-known/codec/ discovery surface: https://github.com/wdunn001/Codec/blob/main/spec/WELL_KNOWN_DISCOVERY.md
• 6 client-language reference implementations consume this wire format byte-equally: https://github.com/wdunn001/Codec/tree/main/packages
• Cross-stack bench (the table above + per-cell breakdown): https://github.com/wdunn001/Codec/blob/main/packages/bench/RESULTS.md

Companion PR against vllm-project/vllm filed in parallel. The wdunn001/sglang fork has been carrying this code in production-grade wdunn001/codec-sglang Docker images for two releases (v0.4 and v0.5); cross-client byte-equality is 24/24 cells unanimous as of v0.4.1.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #26120396284
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.

[gemini-code-assist]

Code Review

This pull request introduces the 'Codec' binary transport protocol to SGLang, enabling efficient token-ID streaming via MessagePack and Protobuf to eliminate text detokenization overhead. Key features include server-side tool-call detection, negotiated transport compression (zstd with dictionaries, Brotli, Gzip), and a version negotiation system for graceful downgrades. Feedback focuses on critical performance and concurrency issues, specifically the synchronous fetching of tool manifests and blocking HTTP calls within the asynchronous request loop. Additionally, improvements were suggested for the efficiency of binary unpacking and the robustness of varint decoding to prevent potential overflows or infinite loops.