You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix: detect token invalidation in 401 responses and stop cascade rotation (#496)
* fix: detect token invalidation in 401 responses and stop cascade rotation
Upstream responses containing explicit token-invalidation messages
("invalidated oauth token", "authentication token has been invalidated",
etc.) are now detected separately from generic 401 errors. When detected:
- A long cooldown (default 5 min, CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS)
is applied to the affected account.
- The 401 is returned directly to the client instead of rotating to the
next account, preventing the cascade where each successive account token
is invalidated in turn by OpenAI's anti-abuse detection.
- Session affinity for that session key is cleared.
Generic 401 responses (expired tokens, wrong credentials) continue to
rotate as before.
Fixes#495
* test: address CR review — cooldown duration assertions, edge cases, phrase provenance
- Add comment above TOKEN_INVALIDATION_PHRASES explaining observed source
providers (OpenAI/Microsoft), how to update the list, and reference to
issue #495 for context.
- Invalidation test: assert coolingDownUntil is ~5min (not 30s generic),
confirming the long cooldown path is reached.
- Generic 401 test: assert coolingDownUntil is ~30s (not 5min), confirming
the short fallback path is unchanged.
- Add edge case: empty 401 body does not trigger invalidation detection,
rotation proceeds normally.
- Add edge case: HTML 401 body containing invalidation phrase is detected
and stops cascade rotation.
* feat: add minRotationIntervalMs to throttle cross-request account switching
Adds a global per-proxy rotation throttle that biases account selection
toward the last successfully-served account within a configurable time
window (default 60 seconds, env: CODEX_AUTH_MIN_ROTATION_INTERVAL_MS).
When the last served account is within the window and still available,
it receives a large score boost (1000) in the hybrid selection algorithm,
overriding the freshness weight that previously caused the proxy to
eagerly rotate to idle accounts on every request.
This reduces how often different OAuth tokens are presented from the same
IP in quick succession -- the primary fingerprint that triggers OpenAI's
anti-abuse detection and causes cascade token invalidation (issue #495).
The boost is applied only to available accounts, so rate-limited or
cooling-down accounts are still skipped and rotation proceeds naturally.
Setting minRotationIntervalMs to 0 disables the throttle entirely.
* fix: detect token invalidation in refresh failures and stop cascade
When the OAuth token refresh endpoint itself returns an explicit
invalidation message (e.g. Microsoft/Outlook SSO tokens being revoked
server-side on first use through the proxy), apply the long cooldown and
return 401 directly to the client without rotating to the next account.
This covers the second cascade vector reported in issue #495: Account 4
(Outlook) was getting invalidated immediately on the first request
because ensureFreshAccessToken was calling queuedRefresh on a token
with a missing or short expiry, and the refresh itself triggered
Microsoft's session revocation. Rotating to the next account after that
would then present another fresh token, cascading the invalidation.
The fix adds tokenInvalidationCooldownMs to ensureFreshAccessToken,
checks isTokenInvalidationError against the refresh failure message, and
when matched: applies the long cooldown, clears session affinity, and
returns the auth error to the client instead of continuing the loop.
* fix: sliding sticky window, monotonic cooldown, and session affinity tests
- Sliding sticky window: update lastGlobalSwitchAt on every successful
serve, not only when the account index changes. Previously a request
at t=61s would rotate even if the same account served at t=55s (only
6s ago) because the anchor stayed at t=0. Now the window slides so
the 60s interval is measured from the last actual serve.
- Monotonic auth-failure cooldown: replace direct markAccountCoolingDown
calls in auth-failure paths with applyMonotonicAuthCooldown helper that
only extends the cooldown if the proposed deadline exceeds the existing
one. Two concurrent requests can no longer race so that a generic 401
truncates a longer invalidation cooldown already written by a parallel
request.
- Regression test: three requests across the window boundary using
vi.useFakeTimers to verify the sliding behavior.
- Session affinity test: refresh-invalidation test sends session_id and
verifies subsequent request with same session routes to healthy account.
* docs: document token-invalidation anti-abuse mitigations and Microsoft SSO warning
- troubleshooting.md: add rows for progressive OAuth token invalidation
and Microsoft/Outlook SSO immediate invalidation, with recovery steps
and CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS guidance.
- configuration.md: add CODEX_AUTH_MIN_ROTATION_INTERVAL_MS and
CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS to the env var table.
Expand the Runtime Rotation Proxy section with an "Anti-abuse
protection" note explaining the two mitigations added in issue #495
(token-invalidation detection + rotation-rate throttle), and a
Microsoft/Outlook SSO note for accounts that get invalidated on first
proxy request.
|`CODEX_AUTH_MIN_ROTATION_INTERVAL_MS=<ms>`| Minimum time between global account switches (default `60000`). The proxy biases selection toward the last-served account within this window to reduce the rate at which different OAuth tokens appear from the same IP. Set to `0` to disable. |
77
+
|`CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS=<ms>`| Cooldown applied to an account when the upstream or token-refresh endpoint explicitly revokes its OAuth token (default `300000`, 5 minutes). Raise this if accounts continue to be re-invalidated after re-login. |
76
78
77
79
---
78
80
@@ -110,6 +112,13 @@ Keep these enabled for most environments:
110
112
111
113
The proxy preserves request bodies and streaming responses, replaces outbound auth headers with the selected managed account, and rotates to another account before response bytes are streamed when it sees rate limits, server errors, network failures, or refresh failures. It removes hop-by-hop headers, private account metadata headers, and stale decoded `content-encoding` from client responses. If every account is unavailable, the proxy returns a structured pool-exhaustion error that points to `codex-multi-auth rotation status`.
112
114
115
+
**Anti-abuse protection.** Rapidly switching OAuth tokens from the same IP can trigger OpenAI's anti-abuse detection and cause accounts to be invalidated in sequence. The proxy includes two mitigations:
116
+
117
+
-**Token-invalidation detection**: when the upstream or the token-refresh endpoint returns an explicit OAuth revocation message, the proxy returns the error directly to the client instead of rotating to the next account. The affected account receives a 5-minute cooldown (`tokenInvalidationCooldownMs`, default `300000`) instead of the generic 30-second auth-failure cooldown. Configure via `CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS`.
118
+
-**Rotation-rate throttle**: the proxy biases account selection toward the last-served account for a configurable window (default 60 seconds, `minRotationIntervalMs`). Accounts that are rate-limited or cooling down are still rotated around. Configure via `CODEX_AUTH_MIN_ROTATION_INTERVAL_MS` or set to `0` to disable.
119
+
120
+
Microsoft/Outlook SSO accounts may be more sensitive to proxy-mediated token use. If an Outlook-linked account is invalidated on every first request through the proxy but works normally on ChatGPT web, the root cause is likely IP or device binding on the Microsoft side. Raising `CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS` and re-logging in the affected account typically resolves the cascade. If the problem persists, consider excluding the Microsoft account from the rotation pool via `codex-multi-auth switch`.
121
+
113
122
For `codex app` launches that go through the wrapper, the wrapper automatically starts a small internal helper so rotation can keep working if the desktop app launcher detaches. The helper stores only local runtime status, uses the same per-session proxy client key as the CLI path, and exits after an idle timeout.
114
123
115
124
`codex-multi-auth rotation enable` also binds the packaged desktop app to a persistent localhost router. This backs up the real Codex `config.toml`, writes the `codex-multi-auth-runtime-proxy` provider into the real Codex home, starts the router immediately, and installs a user login startup entry: a Startup `.cmd` on Windows or a LaunchAgent on macOS. The persistent provider is marked as not requiring OpenAI auth and uses a local app-bind client token, so the desktop runtime does not display the selected multi-auth account while codex-multi-auth status and quota views still read the router's last-account telemetry. `codex-multi-auth rotation disable` and `codex-multi-auth rotation unbind-app` stop that router, remove the startup entry, and restore the backed-up Codex config. The official app files are not patched.
Copy file name to clipboardExpand all lines: docs/troubleshooting.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -76,6 +76,8 @@ The package does not publish a global `codex` binary. `codex-multi-auth ...` is
76
76
|`codex-multi-auth rotation status` says disabled | Stored setting or env override is off | Run `codex-multi-auth rotation enable`, remove `CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0`, or set `CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=1` for one process |
77
77
| Forwarded Codex session does not show the local provider | Command is help/non-requesting, rotation is disabled, or the official CLI was not launched through the wrapper | Check `where codex-multi-auth-codex`, then run `codex-multi-auth rotation status`|
78
78
| Pool exhausted error from the proxy | Every managed account is unavailable for that model/family | Run `codex-multi-auth rotation status`, then `codex-multi-auth forecast --live`|
79
+
| Accounts progressively lose OAuth tokens while the proxy is active | Rapid account rotation triggers OpenAI's anti-abuse detection, which invalidates tokens in sequence | The proxy detects explicit token-invalidation responses and stops rotating; re-login any invalidated accounts and ensure `minRotationIntervalMs` is at least `60000` (default) |
80
+
| Microsoft/Outlook SSO account gets invalidated on every first request through the proxy | Microsoft OAuth tokens may be invalidated when the proxy presents them from a different IP or device context than where they were issued | The proxy now detects invalidation at both the upstream request and the token-refresh stage; if the problem persists, set `CODEX_AUTH_TOKEN_INVALIDATION_COOLDOWN_MS=600000` (10 min) and re-login, or keep the Microsoft account disabled from the rotation pool via `codex-multi-auth rotation status`|
79
81
| Packaged app still uses normal Codex routing | App bind was not installed or was removed | Run `codex-multi-auth rotation bind-app`, then reopen the app |
80
82
| Codex Desktop history disappears after app bind | Current Codex Desktop builds can filter local threads by the active provider, and app bind switches the real config to `codex-multi-auth-runtime-proxy`| The data is normally still under `~/.codex`; run `codex-multi-auth rotation unbind-app` or `codex-multi-auth rotation disable` to restore the original provider/config before browsing old history |
81
83
| Model speed controls are not visible with rotation | Speed/reasoning controls remain owned by Codex config or CLI flags; the app bind only routes Responses traffic | Set `model_reasoning_effort` in `~/.codex/config.toml` or pass `-c model_reasoning_effort=<level>` for wrapper-launched CLI sessions |
0 commit comments