diff --git a/docs/test-specs/thinking-sanitize-efficacy.md b/docs/test-specs/thinking-sanitize-efficacy.md new file mode 100644 index 00000000..1ffd9a18 --- /dev/null +++ b/docs/test-specs/thinking-sanitize-efficacy.md @@ -0,0 +1,61 @@ +# Test spec: thinking-sanitize efficacy (real-API replay) + +**For:** Proxy Test Agent. **Owner of the feature under test:** Proxy Builder (#162). **Resolves:** #162 Open Question 1. +**Status:** spec — **requires Chris's go-ahead before execution** (it makes real Anthropic API calls). +**References:** anthropics/claude-code#63147 (root cause + reproducible triggers), cache-fix #162 (the mitigation: drop prior-turn omitted thinking on the request path), #157, #158. + +## Goal + +Empirically answer: **does the #162 transform actually clear the `400 ... thinking blocks in the latest assistant message cannot be modified`?** The 400 names the *latest* assistant message, but #162 drops *prior-turn* thinking and conservatively leaves the latest message alone — so its efficacy is unproven. Produce a results table that tells PB the exact "which turns to drop" rule before #162's implementation locks. + +## Why a real-API experiment (not the fake-upstream smoke) + +The 400 is **real-Anthropic-API validation behavior**. The docker smoke's fake upstream returns only canned shapes — it can't reproduce the 400, and hardcoding it would just encode our assumption circularly. And the failing request requires **genuine server-signed thinking blocks** — signatures are server-issued and cannot be forged or synthesized. So the experiment must hit the real API. + +## Method + +### Phase 0 — obtain genuine signed thinking blocks +Make a real API call (Opus 4.7 and, separately, 4.8; extended/interleaved thinking on; include at least one tool so the turn interleaves `thinking` + `tool_use`). Capture the full assistant response, preserving the real `signature` on each thinking block. Use a **minimal throwaway conversation**, never a production working session. + +### Phase 1 — reproduce the 400 (establish ground truth) +Construct follow-up requests that replay that assistant turn with thinking **text emptied to `""`** (the omitted shape CC persists) + the **real signature retained**. Build the trigger variants: +- **(a) latest turn completed** — thinking + tool_use + tool_result all resolved, then a new user message (resume/`--continue` shape). +- **(b) latest turn mid-continuation** — thinking + tool_use, then a tool_result that continues that same turn (the interleaved-thinking-with-tools continuation). +- **(c) ordering variant** — thinking block positioned *after* a tool_use in the same assistant message, if reproducible (the MicHuang hypothesis). + +Send each to the **real `api.anthropic.com`**. Record, per variant: 400 vs 200, and the exact `messages.N.content.M` pointer. This is the ground-truth map of which shapes actually fail. + +### Phase 2 — test the transform (the actual question) +For each variant that 400'd, run two transform scopes and re-send: +- **Scope A (#162 as written):** drop prior-turn omitted thinking; never touch the latest assistant message. +- **Scope B (candidate fix):** also drop omitted thinking from the latest *completed* turn — but never from a latest turn that is an active tool-continuation. +Record, per (variant × scope): does the 400 clear (200) or not? + +### Phase 3 — regression + safety +- **Healthy passthrough:** a normal request (no omitted-thinking issue) yields the same outcome (200, equivalent response) with the transform on vs off. +- **Determinism:** same input → byte-identical transformed body. +- **Env-lever — statically CONFIRMED via 2.1.148 binary analysis** (runtime-verify optional): `CLAUDE_CODE_DISABLE_THINKING=1` (`Y$ = q.type!="disabled" && !mH(env.CLAUDE_CODE_DISABLE_THINKING)`) and `MAX_THINKING_TOKENS=0` (`if(env.MAX_THINKING_TOKENS) return parseInt(...)>0`) disable thinking entirely; `DISABLE_INTERLEAVED_THINKING=1` only drops the interleaved beta (thinking still emits); `CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1` only gates adaptive effort on 4.6 models. **User-side answer for variant (b): `CLAUDE_CODE_DISABLE_THINKING=1` or `MAX_THINKING_TOKENS=0`.** Optionally runtime-confirm via 0 `type:thinking` blocks. + +## Harness + +Reuse the docker rig, but point `CACHE_FIX_PROXY_UPSTREAM` at the **real `api.anthropic.com`** (the Dockerfile default) instead of the fake. A/B by toggling the transform: `CACHE_FIX_THINKING_SANITIZE=on` vs `off` (note: #162 ships opt-in/default-off, so the on-arm must set it explicitly). A standalone replay script that posts the captured bodies is an acceptable alternative if simpler. + +## Non-functional / safety constraints + +- **Cost:** small N, minimal token bodies. Throwaway context only — never a live working session. +- **Sensitive data:** the captured bodies contain real thinking content + signatures. Treat as sensitive: do **not** commit raw thinking text or signatures; record only variant description, 400/200, the `content.M` pointer, and transform on/off outcome. Honor the repo's public-info-hygiene rules (no secrets/IPs/tokens in any committed artifact). +- **Auth:** use a test key or a throwaway subscription context; never embed the key in committed files or logs. + + +## Refinements from #63147 overnight evidence (2026-05-29) + +- **Latest-turn *removal* may be accepted (beemusicco):** the API rejects only thinking blocks that are *present but altered* in the latest assistant message; it does NOT require thinking on any turn. So **removing** (not altering) the latest turn's omitted thinking may clear the 400 — Scope B must explicitly test *removing latest-turn* thinking, not just prior-turn. If it works, #162's coverage expands past the resume-replay subset. +- **Preserve tool_use/tool_result pairing when removing blocks:** an orphaned `tool_use_id` (tool_use with no matching tool_result, or vice-versa) 400s on replay. The transform must keep pairs balanced after any removal. +- **Wedge-detection signal:** do NOT key off empty-text+signature (normal on every block). The unambiguous live-wedge signal is the latest transcript entry being the `400 … cannot be modified` error text. + +## Deliverable + +A results table mapping **{trigger variant (a/b/c)} × {transform scope (A/B/off)} → {400 cleared?}**, posted to #162. It resolves Open Question 1 and tells PB the exact "which turns to drop" rule. Expected high-value outcomes: +- If Scope A clears (a) but not (b): #162-as-written covers resume/replay; (b) stays the env-disable case (whichever lever the determination step confirms). +- If only Scope B clears (a): #162 must also drop the latest *completed* turn's omitted thinking (update the directive's "which turns" rule). +- If neither scope clears the continuation case (b): confirms the proxy cannot fix it — elevate the confirmed thinking-disable env lever as the sole answer there.