fix stale update jobs and combo quota fallback

Author: Ingwannu

docs-site/src/content/docs/guides/web-dashboard.md

@@ -117,7 +117,7 @@ The GUI is a thin client over the proxy's JSON management API. Useful endpoints
 | `POST /api/startup-action` | Install the background service or Codex launcher shim through fixed, allowlisted actions. |
 | `GET` / `POST /api/windows-tray` | Read or change the Windows tray installation and visible-process state. POST accepts `install`, `start`, `stop`, or `uninstall`. |
 | `POST /api/sync` | Rebuild the shared model catalog and stale the Codex model cache. |
-| `GET /api/update/check` · `POST /api/update/run` · `GET /api/update/status` | Check, run, and monitor self-update jobs. |
+| `GET /api/update/check` · `POST /api/update/run` · `GET /api/update/status` | Check, run, and monitor self-update jobs. Worker PIDs are persisted so a crashed job recovers automatically; legacy no-PID jobs recover after ten minutes. |
 | `GET` / `PUT /api/sidecar-settings` | Read or set search/vision sidecar model settings. |
 | `GET` / `PUT /api/injection-model` | Read or set the shared sub-agent model/effort selection and the independent guidance/native-default switches. |
 | `GET` / `PUT /api/v2` | Read or set the surface mode, Codex feature flag, and v2 thread limit. |

docs-site/src/content/docs/reference/cli.md

@@ -530,3 +530,8 @@ Two dispatch targets are intentionally omitted from normal help: `__refresh-vers
 refreshes the update-notification cache in a detached process, and
 `__gui-update-worker <job-id> [latest|preview] [restart]` runs a dashboard update job. They are
 implementation details, not stable user-facing commands.
+
+The dashboard persists the detached update worker PID and automatically recovers an active job if
+that worker is no longer alive. Active records written by older releases without a PID are treated
+as stale after ten minutes. A live worker remains protected from concurrent updates even if the
+update takes longer than that window.

src/server/responses/core.ts

@@ -72,6 +72,7 @@ import {
   type CodexAuthContext,
 } from "../../codex/auth-context";
 import {
+  computeQuotaCooldown,
   formatCodexProviderForLog,
   previewCodexAccountForRequest,
   recordCodexUpstreamOutcome,
@@ -251,6 +252,7 @@ interface CodexPoolAccountRetryArgs {
   options: {
     abortSignal?: AbortSignal;
     onCodexAuthContextResolved?: (ctx: CodexAuthContext) => void;
+    deferCodexResetDerivedCooldown?: boolean;
   };
   firstAuthCtx: Extract<CodexAuthContext, { kind: "pool" | "main-pool" }>;
   firstResponse: Response;
@@ -276,6 +278,31 @@ type CodexPoolAccountRetryResult =
     authCtx: Extract<CodexAuthContext, { kind: "pool" | "main-pool" }>;
   };
 
+function codexQuotaOutcomeMeta(response: Response): {
+  retryAfter: string | null;
+  resetAt: string[];
+} {
+  return {
+    retryAfter: response.headers.get("retry-after"),
+    resetAt: [
+      response.headers.get("x-codex-primary-reset-at"),
+      response.headers.get("x-codex-secondary-reset-at"),
+      response.headers.get("x-codex-tertiary-reset-at"),
+    ].filter((value): value is string => !!value),
+  };
+}
+
+/**
+ * A reset timestamp describes a quota window, not an explicit instruction to
+ * stop using the whole account. A combo may therefore try a later model in the
+ * same request, while Retry-After and headerless quota failures remain blocking.
+ */
+function shouldDeferCodexResetDerivedCooldown(response: Response, enabled?: boolean): boolean {
+  return enabled === true
+    && (response.status === 429 || response.status === 402)
+    && computeQuotaCooldown(codexQuotaOutcomeMeta(response)).source === "reset-derived";
+}
+
 /**
  * One bounded alternate-account retry for Codex pool auth. Used for allow-listed
  * model-400 and for pre-stream 429/402 quota failures (#584).
@@ -306,23 +333,20 @@ async function retryCodexPoolOnAlternateAccount(
     return { kind: "no-alternate" };
   }
 
-  const retryAfterRaw = firstResponse.headers.get("retry-after");
+  const quotaMeta = codexQuotaOutcomeMeta(firstResponse);
   if (outcomeStatus === 429 || outcomeStatus === 402) {
     const { applyAccountQuotaFromUpstreamHeaders } = await import("../../codex/auth-api");
     applyAccountQuotaFromUpstreamHeaders(firstAuthCtx.accountId, firstResponse.headers);
   }
-  recordCodexUpstreamOutcome(config, firstAuthCtx.accountId, outcomeStatus, {
-    retryAfter: retryAfterRaw,
-    resetAt: [
-      firstResponse.headers.get("x-codex-primary-reset-at"),
-      firstResponse.headers.get("x-codex-secondary-reset-at"),
-      firstResponse.headers.get("x-codex-tertiary-reset-at"),
-    ].filter(Boolean),
-    threadId: req.headers.get("x-codex-parent-thread-id"),
-    probeLeaseId: codexProbeLeaseId(firstAuthCtx),
-    // Retry already advanced the RR ring via excludeAccountId — reuse for promotion.
-    ...(retryAuthCtx.accountId ? { promoteAccountId: retryAuthCtx.accountId } : {}),
-  });
+  if (!shouldDeferCodexResetDerivedCooldown(firstResponse, options.deferCodexResetDerivedCooldown)) {
+    recordCodexUpstreamOutcome(config, firstAuthCtx.accountId, outcomeStatus, {
+      ...quotaMeta,
+      threadId: req.headers.get("x-codex-parent-thread-id"),
+      probeLeaseId: codexProbeLeaseId(firstAuthCtx),
+      // Retry already advanced the RR ring via excludeAccountId — reuse for promotion.
+      ...(retryAuthCtx.accountId ? { promoteAccountId: retryAuthCtx.accountId } : {}),
+    });
+  }
 
   const retryHeaders = headersForCodexAuthContext(req.headers, retryAuthCtx);
   const retryProvider = applyCodexAuthContextToProvider(
@@ -469,6 +493,8 @@ export interface HandleResponsesOptions {
   promptCacheKeyIsSharedCohort?: boolean;
   /** Internal recursion guard; callers outside this module must not set it. */
   comboAttempt?: boolean;
+  /** Internal combo handoff: allow a later same-provider model after a reset-derived 429/402. */
+  deferCodexResetDerivedCooldown?: boolean;
   /** 030-owned handoff when a child consumed the original failure under bounds. */
   onConsumedComboFailure?: (failure: ConsumedComboFailure) => void;
 }
@@ -902,9 +928,17 @@ export async function handleComboResponses(
     const callbackGate = createChildPassthroughCallbackGate(options);
     let response: Response;
     try {
+      const currentTargetProvider = pick.target.provider;
+      const deferCodexResetDerivedCooldown = combo.strategy === "failover"
+        && combo.targets.slice(pick.targetIndex + 1).some(target =>
+          target.provider === currentTargetProvider
+          && payloadEligible(target)
+          && !isComboTargetInCooldown(comboId, target),
+        );
       response = await handleResponses(childRequest, config, childLog, {
         ...options,
         comboAttempt: true,
+        deferCodexResetDerivedCooldown,
         // Attempt-relative TTFT is recorded HERE (not via childLog.firstOutputMs — a later
         // Object.assign(logCtx, childLog) would overwrite the request-relative value).
         onFirstOutput: () => {
@@ -1488,7 +1522,7 @@ export async function handleResponses(
    if (usesCodexForwardPoolAuth(authCtx, route.provider)) {
       // primary was the 5h window; it now carries weekly data for GPT plans.
       // Prefer primary when present, fall back to secondary for compatibility.
-      const retryAfterRaw = upstreamResponse.headers.get("retry-after");
+      const quotaMeta = codexQuotaOutcomeMeta(upstreamResponse);
       const { applyAccountQuotaFromUpstreamHeaders } = await import("../../codex/auth-api");
       applyAccountQuotaFromUpstreamHeaders(authCtx.accountId, upstreamResponse.headers);
       if (terminalBodyWillRecord) {
@@ -1512,16 +1546,14 @@ export async function handleResponses(
           }
           options.onNativePassthroughTerminal?.(status);
         });
-      } else {
+      } else if (!shouldDeferCodexResetDerivedCooldown(
+        upstreamResponse,
+        options.deferCodexResetDerivedCooldown,
+      )) {
         recordCodexUpstreamOutcome(config, authCtx.accountId, upstreamResponse.status, {
-        retryAfter: retryAfterRaw,
-         resetAt: [
-           upstreamResponse.headers.get("x-codex-primary-reset-at"),
-           upstreamResponse.headers.get("x-codex-secondary-reset-at"),
-           upstreamResponse.headers.get("x-codex-tertiary-reset-at"),
-         ].filter(Boolean),
-         threadId: req.headers.get("x-codex-parent-thread-id"),
-         probeLeaseId: codexProbeLeaseId(authCtx),
+          ...quotaMeta,
+          threadId: req.headers.get("x-codex-parent-thread-id"),
+          probeLeaseId: codexProbeLeaseId(authCtx),
         });
       }
     }

src/update/job.ts

@@ -3,7 +3,7 @@ import { existsSync, mkdirSync, readFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { atomicWriteFile, getConfigDir, loadConfig, readPid, readRuntimePort } from "../config";
-import { killProxy } from "../lib/process-control";
+import { isProcessAlive, killProxy } from "../lib/process-control";
 import { reclaimListenPort } from "../server/port-reclaim";
 import { isOpencodexHealthz, probeHostname, proxyIdentityAt, type HealthzIdentity } from "../server/proxy-liveness";
 import { isServiceInstalled } from "../service";
@@ -28,6 +28,8 @@ const UPDATE_TIMEOUT_MS = 180_000;
 const RESTART_TIMEOUT_MS = 60_000;
 const RESTART_HEALTH_TIMEOUT_MS = 15_000;
 const RESTART_STABILITY_WINDOW_MS = 15_000;
+/** Legacy active records did not persist a worker PID, so age is their only safe recovery signal. */
+export const UPDATE_JOB_LEGACY_STALE_MS = 10 * 60_000;
 /** How long update restart waits for the captured port to become bindable after stop. */
 export const RESTART_PORT_RECLAIM_MS = 30_000;
 
@@ -77,6 +79,19 @@ export interface UpdateCheckDeps {
   latestVersion: (tag: Channel) => string | null;
 }
 
+interface UpdateWorkerProcess {
+  pid?: number;
+  unref(): void;
+  once(event: "error", listener: (error: Error) => void): unknown;
+}
+
+export interface StartUpdateJobDeps {
+  checkForUpdateFn: (channel: Channel) => UpdateCheckResult;
+  spawnWorkerFn: (jobId: string, channel: Channel, restart: boolean) => UpdateWorkerProcess;
+  isProcessAliveFn: (pid: number) => boolean;
+  nowMs: () => number;
+}
+
 const defaultCheckDeps: UpdateCheckDeps = {
   currentVersion,
   detectInstall,
@@ -225,19 +240,77 @@ function newJobId(): string {
   return `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
 }
 
-export function startUpdateJob(channel: Channel, restart: boolean): UpdateJobState {
+/**
+ * [Decision Log]
+ * - Purpose: recover dashboard updates after a detached worker dies without unlocking concurrent live updates.
+ * - Existing constraints: legacy records have no PID, while a healthy update may legitimately run for minutes.
+ * - Alternatives considered: clear every active record by age, or require operators to delete the file manually.
+ * - Chosen approach: trust PID liveness first and use a conservative age limit only for legacy no-PID records.
+ * - Why: age-only recovery can start two installers, while never recovering leaves the dashboard permanently blocked.
+ * - Impact: live PID records remain locked regardless of age; dead PIDs recover immediately; legacy records recover after ten minutes.
+ */
+export function staleActiveUpdateJobReason(
+  job: Pick<UpdateJobState, "status" | "pid" | "updatedAt">,
+  now = Date.now(),
+  isAlive: (pid: number) => boolean = isProcessAlive,
+): string | null {
+  if (job.status !== "running" && job.status !== "restarting") return null;
+  if (typeof job.pid === "number" && Number.isSafeInteger(job.pid) && job.pid > 0) {
+    return isAlive(job.pid) ? null : `update worker PID ${job.pid} is no longer running`;
+  }
+  const updatedAt = Date.parse(job.updatedAt);
+  if (Number.isFinite(updatedAt) && now - updatedAt >= UPDATE_JOB_LEGACY_STALE_MS) {
+    return "legacy active update record has no worker PID and exceeded the stale window";
+  }
+  return null;
+}
+
+const defaultStartUpdateJobDeps: StartUpdateJobDeps = {
+  checkForUpdateFn: channel => checkForUpdate(channel),
+  spawnWorkerFn: (jobId, channel, restart) => spawn(
+    process.execPath,
+    [process.argv[1], "__gui-update-worker", jobId, channel, restart ? "restart" : "no-restart"],
+    {
+      detached: true,
+      stdio: "ignore",
+      windowsHide: true,
+      env: { ...process.env, OCX_SERVICE: "1" },
+    },
+  ),
+  isProcessAliveFn: isProcessAlive,
+  nowMs: Date.now,
+};
+
+export function startUpdateJob(
+  channel: Channel,
+  restart: boolean,
+  deps: Partial<StartUpdateJobDeps> = {},
+): UpdateJobState {
+  const resolvedDeps = { ...defaultStartUpdateJobDeps, ...deps };
   const running = readUpdateJob();
   if (running?.status === "running" || running?.status === "restarting") {
-    throw new UpdateJobError("An update job is already running", 409, "update_already_running");
+    const staleReason = staleActiveUpdateJobReason(
+      running,
+      resolvedDeps.nowMs(),
+      resolvedDeps.isProcessAliveFn,
+    );
+    if (!staleReason) {
+      throw new UpdateJobError("An update job is already running", 409, "update_already_running");
+    }
+    updateJob(
+      running,
+      { status: "failed", error: `Recovered stale update job: ${staleReason}.`, exitCode: null },
+      `Recovered stale update job: ${staleReason}.`,
+    );
   }
 
-  const check = checkForUpdate(channel);
+  const check = resolvedDeps.checkForUpdateFn(channel);
   if (!check.canUpdate) {
     throw new UpdateJobError(check.reason ?? "No update is available", 409, check.reason ?? "update_unavailable");
   }
 
   const id = newJobId();
-  const now = new Date().toISOString();
+  const now = new Date(resolvedDeps.nowMs()).toISOString();
   const job: UpdateJobState = {
     id,
     status: "running",
@@ -254,14 +327,30 @@ export function startUpdateJob(channel: Channel, restart: boolean): UpdateJobSta
   };
   writeJob(job);
 
-  const child = spawn(process.execPath, [process.argv[1], "__gui-update-worker", id, channel, restart ? "restart" : "no-restart"], {
-    detached: true,
-    stdio: "ignore",
-    windowsHide: true,
-    env: { ...process.env, OCX_SERVICE: "1" },
+  let child: UpdateWorkerProcess;
+  try {
+    child = resolvedDeps.spawnWorkerFn(id, channel, restart);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    updateJob(job, { status: "failed", error: `Could not start update worker: ${message}` }, "Update worker failed to start.");
+    throw new UpdateJobError("Could not start update worker", 500, "update_worker_start_failed");
+  }
+  if (typeof child.pid !== "number" || !Number.isSafeInteger(child.pid) || child.pid <= 0) {
+    updateJob(job, { status: "failed", error: "Could not start update worker: no worker PID was returned." }, "Update worker failed to start.");
+    throw new UpdateJobError("Could not start update worker", 500, "update_worker_start_failed");
+  }
+  const startedJob = updateJob(job, { pid: child.pid }, `Update worker started as PID ${child.pid}.`);
+  child.once("error", error => {
+    const current = readUpdateJob(id);
+    if (!current || current.pid !== child.pid || (current.status !== "running" && current.status !== "restarting")) return;
+    updateJob(
+      current,
+      { status: "failed", error: `Update worker failed to start: ${error.message}` },
+      "Update worker emitted a startup error.",
+    );
   });
   child.unref();
-  return { ...job, pid: child.pid };
+  return startedJob;
 }
 
 function runLoggedCommand(job: UpdateJobState, bin: string, args: string[], timeout: number): { status: number | null; signal: NodeJS.Signals | null } {

structure/04_transports-and-sidecars.md

@@ -387,6 +387,26 @@ retried. Guarded paths: the ChatGPT passthrough and generic adapter fetch in
 fallback. Adapters with their own `fetchResponse` (kiro, cursor, google) keep their own retry
 policies; kiro imports the shared abort/sleep helpers from this module.
 
+## Same-provider combo quota fallback
+
+For a failover combo with multiple models on the same Codex-login OpenAI provider, a pre-stream
+429/402 carrying only `x-codex-*-reset-at` may advance to the later model on the same account. The
+failed physical combo target still enters its normal target cooldown. An explicit `Retry-After`
+remains an account-wide instruction and blocks the later target; a quota response with neither an
+explicit retry delay nor a usable reset timestamp keeps the conservative default account cooldown.
+This exception is request-scoped and is not applied to direct requests, round-robin combos, or a
+combo whose remaining eligible targets use other providers.
+
+```text
+[Decision Log]
+- 목적과 의도: Let an ordered combo recover when one model-specific Codex quota window is exhausted but another model on the same account remains usable.
+- 기존 구현 및 제약 조건: Account health is shared across models, and recording a reset-derived 429 before combo advancement rejected the later model locally.
+- 검토한 주요 대안: Make every quota cooldown model-scoped; ignore all combo 429 cooldowns; or defer only reset-derived cooldown recording for an eligible later same-provider failover target.
+- 선택한 방식: Use the narrow request-scoped deferral while retaining target cooldown and all explicit Retry-After/default account cooldown behavior.
+- 다른 대안 대신 이 방식을 선택한 이유: Reset timestamps identify quota windows rather than a literal account-wide retry instruction, but widening the exception would risk hot retries and provider abuse.
+- 장점, 단점 및 영향: Same-account model fallback works without weakening explicit upstream backoff; the account health map intentionally does not remember that one deferred reset-derived failure, while the combo target map does.
+```
+
 ## Sidecars
 
 Web search and vision sidecars only run when the mode-aware `openai` forward ChatGPT authority