feat(proxy): opt-in same-target 429 wait-and-retry before key failover (#487)

devlog/_plan/260802_429_same_target_retry/000_research.md

@@ -0,0 +1,39 @@
+# 000 — same-target 429 retry: why the client cannot do it
+
+## 요약
+
+Codex turns die instantly when an upstream provider (e.g. BLSC with a single API key)
+returns HTTP 429. The proxy forwards the error with `Retry-After` (#514), but Codex
+never acts on it.
+
+## Evidence
+
+- **Upstream [openai/codex#30471](https://github.com/openai/codex/issues/30471) (open):**
+  `codex-rs/codex-api/src/provider.rs` has a retry policy with an explicit `retry_429` flag,
+  and the endpoint configs set it to `false`; HTTP 429 is only mapped to `UsageLimitReached`
+  when the body matches the Codex usage-limit shape, otherwise it falls through to a generic
+  transport/API error and surfaces as the misleading "exceeded retry limit" message. The
+  suggested fix is to reword the error, deliberately preserving `retry_429=false` — there is
+  no user-facing knob to enable client-side 429 retry.
+- **opencodex issue #487 (auto-closed as `not_planned` for missing detail):** the author
+  requested exactly this feature and noted "The Codex client itself does not retry 429
+  (upstream: openai/codex#30471) — it only retries 5xx — so a proxy-side retry would fill a
+  real gap."
+- **opencodex PR #514 (merged):** attaches a client `Retry-After` (upstream header → message
+  hint → default 2) across Responses, chat completions, Claude messages, and passthrough
+  paths. Its own test plan lists the Codex recovery check as *unverified*. Claude Code honors
+  `Retry-After` and absorbs 429s (#507); Codex does not.
+
+## Current proxy behavior (v2.8.0 / dev)
+
+- `src/server/responses/core.ts` recovery loop: the only 429 retry is multi-key failover
+  (`hasKeyPoolFailover` requires ≥2 keys in `apiKeyPool`). Single-key pools no-op, and the
+  429 falls through to `rate_limit_error` with `Retry-After`.
+- `src/server/chat-completions.ts` and the routed Claude path reuse `handleResponses`, so one
+  insertion point covers all three inbound surfaces.
+
+## Conclusion
+
+Client-side 429 retry does not exist and is not planned. The fix must live in the proxy:
+an opt-in wait-and-retry that replays the identical pre-stream request on the same key before
+any failover.

devlog/_plan/260802_429_same_target_retry/010_design.md

@@ -0,0 +1,114 @@
+# 010 — design: `retryOn429` same-target wait-and-retry
+
+의존: `000_research.md`
+
+## Goal
+
+Provider-level opt-in knob: on HTTP 429, wait (upstream `Retry-After` or a fixed interval)
+and replay the identical request on the same key, up to `attempts` extra times, before the
+existing multi-key failover runs. Default off → zero behavior change for existing setups.
+
+## Surface
+
+```jsonc
+// ~/.opencodex/config.json → providers.<name>
+"retryOn429": {
+  "enabled": true,            // object presence also enables; false disables
+  "attempts": 3,              // extra replays after the first 429 (1..20)
+  "intervalMs": 5000,         // fixed wait when no usable Retry-After
+  "maxIntervalMs": 60000,     // cap for any single wait
+  "respectRetryAfter": true   // prefer the upstream Retry-After when parseable
+}
+```
+
+## Implementation
+
+- `src/types.ts`: `RateLimitRetryPolicy` interface + `OcxProviderConfig.retryOn429`.
+- `src/config.ts`: zod schema entry (zod's default strip inside the object — an unknown key is
+  dropped, never a config-rejecting error; the outer provider schema stays passthrough).
+  Load-time degradation: one hand-edited invalid optional field (e.g. `attempts: 0`) is dropped
+  with a warning instead of tripping the whole schema and hiding all providers behind a default
+  config; misnamed keys (e.g. `attempt`) are warned about too; the management write boundary
+  still rejects invalid policies.
+- `src/providers/key-failover.ts`: `rateLimitRetryPolicyFor` (normalize/default) and
+  `rateLimitRetryDelayMs` (Retry-After seconds/HTTP-date → capped, else `intervalMs`),
+  reusing the existing `parseRetryAfterMs` cooldown parser; the fixed fallback is capped at
+  `maxIntervalMs` too, so a single wait never exceeds the cap. Fail closed: only `authMode:
+  "key"` (or the documented omitted default for custom API-key providers) may use replays —
+  OAuth/forward are never replayed on the same token, local runtimes have no remote key to
+  preserve, and unknown values are rejected. `providerConfigSeed` preserves the registry auth
+  kind (including `"local"`) so the gate survives the seed round-trip.
+- `src/usage/log.ts`: new `AttemptRecoveryKind` member `"rate-limit-429"`.
+- `src/server/responses/core.ts`: in the pre-stream recovery loop, BEFORE the multi-key
+  failover `while`, wait then `rebuildAndRefetch("rate-limit-429")`. Abort during the wait
+  cancels the client request (the unread 429 body is released first). The retry budget lives
+  OUTSIDE the recovery loop, so a 413/401 replay that comes back 429 cannot re-arm a fresh
+  budget — bounded to `attempts` per request. After attempts are exhausted the existing
+  failover and error mapping run unchanged. The same wait-and-replay applies to the other
+  key-auth surfaces that bypass that loop:
+  - Responses passthrough wire (`openai-responses` key-auth gateways, e.g. the built-in
+    DeepSeek preset) — pre-relay, before the forward-pool logic;
+  - image/video bridge and web-search sidecar loops (`src/images/loop.ts`,
+    `src/web-search/loop.ts`) — before their `on429` key rotation;
+  - Anthropic terminal-guard continuations — before key/account failover.
+  Every surface releases (and awaits the cancellation of) the unread 429 body BEFORE the
+  backoff, records the `rate-limit-429` recovery kind on replay sends, and (bridges) clears the
+  old response-header deadline before the wait and starts a fresh one afterward, re-checking
+  client cancellation before telemetry and replay.
+  Covers Responses, chat completions, and routed Claude messages (they all enter
+  `handleResponses`).
+
+## Safety
+
+- Pre-stream only: a 429 arrives before any bytes are relayed, so replaying the string-body
+  request is lossless (same invariant as the transient-5xx layer in `lib/upstream-retry.ts`).
+- Ordering: same-key retries run before failover, so "primary-first" users keep their key on
+  rate-limit blips; failover still works after retries exhaust.
+- Retry-wait bound: the SLEEP component is at most `attempts × maxIntervalMs` (default
+  3 × 60 s = 180 s) when honoring upstream `Retry-After`; `attempts × intervalMs`
+  (default 15 s) when `respectRetryAfter=false` or no header is present. Total request
+  latency is higher: every attempt also consumes its own connect/response time (bounded by
+  `connectTimeoutMs`), so the documented bound covers deliberate waits only.
+- Identical replay: rebuilds are deterministic for the same parsed request (same serialized
+  body and auth headers); the passthrough/continuation/e2e tests assert byte-identical bodies
+  and identical auth headers across replays, not just send counts.
+- Abort during the wait: the sleep is abort-aware — when the server observes the client
+  disconnect (Bun propagates this asynchronously, observed 1–10 s), the wait is interrupted,
+  the unread 429 body is released, and the request is cancelled with 499 before any replay.
+  Because the propagation is async, a replay can still precede the cancel if the interval
+  elapses first; that is bounded by the same `attempts` budget. Terminal continuations sleep
+  on the upstream signal, so a body-cancel (SSE already streaming) aborts the wait too.
+- Concurrency: each request honors its own policy independently — no process-wide cooldown is
+  shared between concurrent requests (unlike the Kiro 429 pattern). Upstream volume per
+  request: same-key replays add at most `attempts` sends, then multi-key failover adds up to
+  `poolKeys − 1` more (or Anthropic account rotations), so the combined bound is
+  `attempts + poolKeys` sends — a storm multiplies by that factor per request, not by
+  `attempts + 1`.
+- Header deadlines: the image/video and web-search bridge loops restart their response-header
+  deadline after each deliberate wait, so backoffs never consume the connect budget and a
+  rate-limit wait is never misattributed as a 504 header timeout. The old deadline is cleared
+  BEFORE the sleep and client cancellation is re-checked after it, so 499 always wins over a
+  stale-deadline edge.
+- Expired `Retry-After`: a valid HTTP-date already in the past retries immediately (same as
+  numeric `Retry-After: 0`) instead of falling back to the fixed interval.
+- Recovery observability: every retry surface records the `rate-limit-429` recovery kind
+  (normal loop, passthrough wire, image/video bridge, web-search sidecar, terminal
+  continuations), so usage logs explain the extra sends.
+- Final 429 still carries `Retry-After` for clients that honor it (Claude Code).
+
+## Tests
+
+- `tests/rate-limit-retry.test.ts` — policy normalization (incl. OAuth/forward gating) +
+  delay computation (seconds, HTTP-date, `0`, malformed, cap; deterministic) + abort during
+  the wait: a directly-invoked `handleResponses` with a controlled abort signal returns 499,
+  cancels the unread 429 body, and performs no further upstream sends (deterministic — no
+  real-socket disconnect timing involved).
+- `tests/usage-log.test.ts` — the `rate-limit-429` recovery kind survives persisted usage logs.
+- `tests/server-rate-limit-retry-e2e.test.ts` — single-key replay to success, immediate
+  passthrough without the knob, exhausted attempts surface 429, and retry-before-failover
+  ordering with a 2-key pool, plus key-auth `openai-responses` passthrough replaying 429 on
+  the same key.
+- `tests/terminal-guard-server.test.ts` — an Anthropic terminal-guard continuation that 429s
+  is replayed on the same key before the error surfaces (3 upstream sends).
+- `tests/images/loop.test.ts` + `tests/web-search.test.ts` — the bridge loops replay 429 on
+  the same key before `on429` rotation runs (same-key sends counted, rotations zero).

docs-site/src/content/docs/guides/providers.md

@@ -39,6 +39,12 @@ labels local presets separately; those normally omit both `authMode` and `apiKey
 | `forward` | Relays **your incoming Codex auth headers** verbatim to the provider — no key stored. This is the ChatGPT-login passthrough. | OpenAI (`openai-responses` adapter). |
 | `oauth` | Resolves a stored OAuth access token (auto-refreshed before expiry) and uses it as the bearer key. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor, GitHub Copilot. |
 
+The [`retryOn429`](/reference/configuration/) same-key 429 replay applies only to API-key
+providers (`authMode: "key"`). OAuth, forward, and local presets are excluded — their
+credentials must never be replayed on the same token, and local runtimes have no remote key to
+preserve. It is opt-in: when the option is absent the feature is off; object presence enables
+it unless `enabled: false`.
+
 ## 1. ChatGPT login (forward / passthrough)
 
 The `openai` provider needs **no API key**. Direct forwards credentials from your existing

docs-site/src/content/docs/ja/guides/providers.md

@@ -35,6 +35,11 @@ max input 922,000 で `*-pro` virtual ID は公開状態を維持し、wire で
 | `forward` | **受け取った Codex 認証ヘッダーを**プロバイダーにそのまま中継します — キーを保存しません。ChatGPT ログインのパススルーです。 | OpenAI(`openai-responses` アダプター)。 |
 | `oauth` | 保存された OAuth アクセストークンを読み込み bearer キーとして使い、期限切れ前に自動更新します。 | xAI、Anthropic、Kimi、Kiro、Google Antigravity、Cursor。 |
 
+[`retryOn429`](/ja/reference/configuration/)（同一キーでの 429 リトライ）は API キー プロバイダー
+（`authMode: "key"`）のみに適用されます。OAuth・forward・ローカル プリセットは除外されます —
+同じトークンを再送すべきではなく、ローカルランタイムには保存すべきリモートキーがありません。
+オプトインです: オプションが無ければ無効、オブジェクトがあれば `enabled: false` でない限り有効です。
+
 ## 1. ChatGPT ログイン(forward / パススルー)
 
 デフォルトプロバイダーは**API キー不要**です。既存の `codex login` の認証情報を OpenAI Responses バックエンドに

docs-site/src/content/docs/ja/reference/adapters.md

@@ -38,6 +38,11 @@ interface ProviderAdapter {
 **対象:** OpenAI **Responses API**。**`passthrough: true`** — 元のリクエスト本文をそのまま渡し、レスポンスを **変換せずに** ストリーミングします。
 **認証:** `forward`（呼び出し元ヘッダー中継）または `key`。
 
+`key` 認証では、[`retryOn429`](/ja/reference/configuration/) もここに適用されます: プリストリームの
+429 は、翻訳された `openai-chat` / Anthropic リクエスト経路と同様に、他の処理やフェイルオーバーに
+先立って、同じキーで同一リクエストを待機して再送します。カスタム `runTurn` トランスポートは
+HTTP リトライ ループの対象外です。
+
 - `forward` URL → `{baseUrl}/responses`。`key` provider はデフォルトで従来の `{baseUrl}/v1/responses` 構築を使います。
 - `key` provider は検証済みの相対 `responsesPath` を設定できます。adapter は `baseUrl` 末尾の `/` を 1 つ除き、`{trimmedBaseUrl}{responsesPath}` に送信します。Ark Agent Plan では `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` と `responsesPath: "/responses"` を使います。
 - `forward` モードでは安全なヘッダー許可リスト（`FORWARD_HEADERS`）だけを中継します。authorization、ChatGPT account id、OpenAI beta/originator/session ヘッダーが対象です。この ChatGPT ログイン経路は [サイドカー](/ja/guides/sidecars/) にも使われます。

docs-site/src/content/docs/ja/reference/configuration/providers.md

@@ -74,6 +74,7 @@ description: プロバイダー エントリ、認証、エンドポイント、
 | `noPenaltyModels?` | `string[]` |存在/周波数ペナルティを拒否するモデル。 |
 | `parallelToolCalls?` | `boolean` |並列ツール呼び出しを切り替えます。 OpenAI Chat はデフォルトでオンになっています。非チャット アダプターは明示的な `true` でのみアドバタイズします。 |
 | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` |正確なプレースホルダー ID および欠落している端末 ID に対するダウンストリーム SSE 修復はデフォルトで無効になっています。関数呼び出し ID は決して書き換えられません。 |
+| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key プロバイダーのみ(`authMode: "key"`)。オプトインの同一ターゲット 429 リトライ: `retryOn429` が無ければ無効で、オブジェクトがあれば `enabled: false` でない限り有効になります。429 時に待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します — メインのテキストターン回復ループ、Responses passthrough、画像/動画ブリッジ、web-search サイドカー、ターミナル継続要求をすべてカバーします。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。デフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は `maxIntervalMs` で上限、その上限は 600000)、`respectRetryAfter: true`。 |
 | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` が `auto` または `none` のみを受け入れるモデル。強制的な選択は格下げされます。 |
 | `preserveReasoningContentModels?` | `string[]` |チャット履歴に以前のアシスタント `reasoning_content` が必要なモデル。 |
 | `thinkingToggleModels?` | `string[]` |エフォート ラダーではなく `thinking.enabled` を使用してモデルをチャットします。 |

docs-site/src/content/docs/ko/guides/providers.md

@@ -35,6 +35,11 @@ shipped v1 config는 marker 2의 단일 옵션 행으로 자동 이관됩니다.
 | `forward` | **수신된 Codex 인증 헤더를** 프로바이더에 그대로 중계합니다 — 키를 저장하지 않습니다. ChatGPT 로그인 패스스루입니다. | OpenAI (`openai-responses` 어댑터). |
 | `oauth` | 저장된 OAuth 액세스 토큰을 불러와 bearer 키로 사용하며, 만료 전에 자동 갱신합니다. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor. |
 
+[`retryOn429`](/ko/reference/configuration/)（동일 키 429 재시도）는 API 키 프로바이더
+（`authMode: "key"`）에만 적용됩니다. OAuth·forward·로컬 프리셋은 제외됩니다 — 같은 토큰을
+재전송해서는 안 되며, 로컬 런타임에는 보존할 원격 키가 없습니다. 옵트인입니다: 옵션이 없으면
+꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다.
+
 ## 1. ChatGPT 로그인 (forward / 패스스루)
 
 기본 프로바이더는 **API 키가 필요 없습니다**. 기존 `codex login`의 자격 증명을 OpenAI Responses 백엔드로

docs-site/src/content/docs/ko/reference/adapters.md

@@ -45,6 +45,11 @@ interface ProviderAdapter {
 **변환하지 않은 채** 스트리밍합니다.
 **인증:** `forward`(호출자 헤더 중계) 또는 `key`.
 
+`key` 인증에서는 [`retryOn429`](/ko/reference/configuration/)도 여기에 적용됩니다: 사전 스트림
+429는 번역된 `openai-chat`/Anthropic 요청 경로와 동일하게 다른 처리나 페일오버보다 먼저
+같은 키로 동일 요청을 대기 후 재전송합니다. 커스텀 `runTurn` 전송은 HTTP 재시도 루프에
+포함되지 않습니다.
+
 - `forward` URL → `{baseUrl}/responses`. `key` provider는 기본적으로 기존 `{baseUrl}/v1/responses` 구성을 사용합니다.
 - `key` provider는 검증된 상대 `responsesPath`를 설정할 수 있습니다. adapter는 `baseUrl` 끝의 `/` 하나를 제거하고 `{trimmedBaseUrl}{responsesPath}`로 전송합니다. Ark Agent Plan은 `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"`와 `responsesPath: "/responses"`를 사용합니다.
 - `forward` 모드에서는 안전한 헤더 허용 목록(`FORWARD_HEADERS`)만 중계합니다. authorization,