[Bug] U+FFFD corruption reported when Kimi/Opus writes Korean files — wire paths verified clean, need the failing capture

[lidge-jun]

Report

User report (maintainer): "kimi나 opus가 한국어 파일을 쓸때 fffd 오류가 발생" — U+FFFD (�) appears when Kimi or Opus writes files containing Korean text. This issue tracks the investigation. Related-but-different prior: #417 (Korean realtime voice transcript U+FFFD, upstream — not an ocx relay bug).

What was verified CLEAN (2026-08-03, direct reproduction against the current tree)

Every realistic Korean path through the proxy was exercised with 한국어 파일 내용 — ①②③ 🎆 and checked for U+FFFD — all clean:

Path	Shape	Result
Baseline concat/decoder	string + byte splits	clean
buildResponseJSON bridge	tool_call_delta, custom_tool_call splits at 7 positions	clean
Anthropic adapter parseStream	input_json_delta fragments, string cuts AND arbitrary byte cuts	clean
OpenAI-chat adapter parseStream	function.arguments fragments, string cuts AND byte cuts	clean
E2E via real startServer	chat completions → openai-chat upstream (kimi)	upstream receives intact Korean
E2E via real startServer	chat completions → anthropic upstream (opus)	intact
E2E via real startServer	Responses + custom_tool_call/function_call_output → openai-chat	intact
E2E via real startServer	Responses + custom_tool_call/function_call_output → anthropic	intact
Bun.TOML.parse	Korean basic strings	clean

The apply_patch argument accumulation, SSE byte-level decoding (all decoders use { stream: true }), the responses→chat history conversion, and both providers' request serializers do not corrupt Korean text for these shapes.

Remaining hypotheses (not yet ruled out)

1. A feature-specific path not covered above: local/remote compaction with Korean history, reasoning summary handling, a provider-specific transform (antigravity/xAI/kiro transports), or the boundedInjectedInstruction UTF-16 slice in src/adapters/kiro.ts:384 (kiro-only, can split a surrogate pair at the budget boundary).
2. The corruption is introduced upstream or client-side (the Tracking (upstream): Korean realtime voice transcript renders U+FFFD — not an ocx relay bug #417 precedent), not in ocx.
3. The failing shape differs from the tested ones (Korean filename, Korean in a system/AGENTS slot, a specific model + client combination).

How to capture the failing frame

The relay has a U+FFFD frame detector built in (src/server/live.ts). To pin the exact surface:

OCX_LIVE_FRAME_LOG=/tmp/ocx-live.jsonl ocx start
# reproduce the failure with kimi/opus, then:
grep '"fffd":true' /tmp/ocx-live.jsonl | head -5

The log records direction, frame kind, and byte length per frame (payloads are never written — only the U+FFFD flag), which identifies whether corruption enters on the upstream→client leg, the client→upstream leg, or is already present in the client's own bytes.

Needed next

• The concrete scenario: client (Codex app/CLI vs Claude Code), model + provider preset, and where the FFFD appears (file bytes on disk vs model-visible content vs GUI).
• The OCX_LIVE_FRAME_LOG capture from a failing run.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Follow-up from the coverage audit: three adjacent lone-surrogate U+FFFD producers were found and fixed (all require an astral character at an exact boundary, so they do not explain a strictly Hangul-only reproduction — the capture above is still needed for attribution):

1. src/responses/compaction.ts — the v1 compaction retained tail could begin on a lone low surrogate when the 80k budget cut landed mid-pair (repro: an 80,001-code-unit message beginning with 🎆).
2. src/adapters/kiro-thinking.ts — the reasoning carry/send boundary could emit a reasoning_raw_delta ending on a lone high surrogate.
3. src/adapters/kiro-tools.ts — tool-description truncation at the 1,024-char cap could end on a lone high surrogate.

Plus the earlier src/adapters/kiro.ts:384 instruction-budget slice (same class). All four are fixed with boundary-pinned regressions on the unified bug-fix branch (see PR #892).