fix: recover stale saved reset redemptions

Author: masterkain

docs-site/src/content/docs/operators/upstreams.mdx

@@ -98,9 +98,9 @@ Account cards show a compact saved-reset count badge only when sanitized usage o
 
 The saved-reset dialog explains the banked reset count in plain operator terms and edits auto-redemption policy. If expiration metadata is available, it shows the next available saved reset expiration and the number of reported expiration dates. The dialog and cockpit both require an explicit confirmation before queueing a manual `Redeem one saved reset` action. Saving the policy never enqueues redemption work.
 
-Manual redemption is enabled only when the upstream is not deleted or disabled, credentials are usable, at least one Pool assignment exists, a saved-reset count has been reported, the reported count is greater than zero, and no redemption is already in progress. Otherwise the action is disabled with the matching blocker, such as missing credentials, no Pool assignment, unreported count, no available resets, or an in-progress redemption.
+Manual redemption is enabled only when the upstream is not deleted or disabled, credentials are usable, at least one Pool assignment exists, a saved-reset count has been reported, the reported count is greater than zero, and no fresh redemption is already in progress. A stale in-progress attempt is not a blocker; confirming manual redemption lets the worker recover the stale attempt state before claiming a new redemption. Otherwise the action is disabled with the matching blocker, such as missing credentials, no Pool assignment, unreported count, no available resets, or a fresh in-progress redemption.
 
-Auto redemption is a per-upstream opt-in policy available from the account-card `Saved resets` dialog and the upstream cockpit. New accounts default to auto redemption off. Operators choose whether automatic redemption waits until weekly quota is blocked or may start earlier when fresh weekly quota evidence shows every eligible candidate account is near the configured `Near-limit threshold`. A saved reset with a known expiration inside the next 24 hours may also redeem early when that same account already has weekly usage to recover and the natural weekly reset is not close. The `Natural reset buffer` prevents spending when the weekly quota will reset naturally soon, and `Keep credits` reserves banked resets from automatic use.
+Auto redemption is a per-upstream opt-in policy available from the account-card `Saved resets` dialog and the upstream cockpit. New accounts default to auto redemption off. Operators choose whether automatic redemption waits until weekly quota is blocked or may start earlier when fresh weekly quota evidence shows every eligible candidate account is near the configured `Near-limit threshold`. The blocked mode is still allowed to rescue a saved reset with a known expiration inside the next 24 hours when that same account already has weekly usage to recover and the natural weekly reset is not close. The `Natural reset buffer` prevents spending when the weekly quota will reset naturally soon, and `Keep credits` reserves banked resets from automatic use.
 
 Saved reset observations and redemption attempts stay metadata-only on `upstream_identities.metadata` under `saved_resets` and `saved_reset_redemption`. The saved-reset observation may include sanitized aggregate expiration fields such as `available_expires_at`, `next_expires_at`, `expires_observed_at`, and `expires_refresh_attempted_at`; it must not include raw provider credit objects, identifiers, titles, descriptions, or redemption request ids. Policy fields live on the upstream identity as `saved_reset_auto_redeem_enabled`, `saved_reset_auto_redeem_trigger_mode`, `saved_reset_auto_redeem_quota_threshold_percent`, `saved_reset_auto_redeem_min_blocked_minutes`, and `saved_reset_auto_redeem_keep_credits`. Do not copy provider payloads, raw credit objects or identifiers, redemption request identifiers, token material, or account secrets into metadata, logs, docs, or tests.
 

docs-site/src/content/docs/reference/routing-strategies.mdx

@@ -183,9 +183,11 @@ Fallback doesn't bypass hard checks. It won't use a paused account, an account w
 
 Saved reset redemption is quota recovery, not a routing strategy. It cannot make a paused, unassigned, unsupported, unhealthy, or session-conflicting upstream eligible.
 
-Auto redemption is default-disabled per upstream account because saved resets are limited provider capacity. When an operator enables it, the gateway may attempt one redemption only after required quota filtering reaches `quota_exhausted` and every candidate is blocked by weekly quota exhaustion. The selected candidate must have a reported saved-reset count greater than its `Keep credits` policy, no redemption already in progress, and a natural reset far enough away to satisfy `Min blocked minutes`.
+Auto redemption is default-disabled per upstream account because saved resets are limited provider capacity. In blocked mode, the gateway may attempt one redemption only after required quota filtering reaches `quota_exhausted` and every candidate is blocked by weekly quota exhaustion. The selected candidate must have a reported saved-reset count greater than its `Keep credits` policy, no fresh or stale redemption already claimed, and a natural reset far enough away to satisfy `Min blocked minutes`.
 
-If auto redemption applies a saved reset, Codex Pooler refreshes quota evidence and filters candidates again before strategy ordering. If no saved reset can be applied, or the redemption result does not change quota evidence, the original quota exhaustion stands. Manual `Redeem saved reset` from the upstream cockpit uses the same metadata-only redemption boundary but is an operator action, not an in-flight routing fallback.
+Early redemption can also run before hard exhaustion when the operator selects near-limit mode and every eligible candidate account has fresh weekly quota evidence at or above the configured threshold. Separately, a known saved reset expiring inside the next 24 hours may be rescued early when that same account already has weekly usage to recover and the natural weekly reset is not close.
+
+If auto redemption applies a saved reset, Codex Pooler refreshes quota evidence and filters candidates again before strategy ordering. If no saved reset can be applied, or the redemption result does not change quota evidence, the original quota decision stands. Manual `Redeem saved reset` from the upstream cockpit uses the same metadata-only redemption boundary but is an operator action, not an in-flight routing fallback.
 
 Saved-reset observations and redemption attempt state are stored on `upstream_identities.metadata`; routing and redemption metadata stores bounded status, result codes, timestamps, HTTP status, and before/after counts only. It does not store raw provider payloads, credit identifiers, redemption request identifiers, tokens, prompts, request bodies, or response bodies.
 

lib/codex_pooler/gateway/routing/saved_reset_auto_redeem.ex

@@ -261,7 +261,8 @@ defmodule CodexPooler.Gateway.Routing.SavedResetAutoRedeem do
     snapshot = SavedResets.snapshot(identity)
 
     policy.enabled? and snapshot.available_count != nil and
-      snapshot.available_count > policy.keep_credits and not snapshot.in_progress?
+      snapshot.available_count > policy.keep_credits and not snapshot.in_progress? and
+      not snapshot.redemption_stale?
   end
 
   defp redeemable_weekly_window?(

lib/codex_pooler/upstreams/saved_resets.ex

@@ -19,6 +19,8 @@ defmodule CodexPooler.Upstreams.SavedResets do
   @default_quota_threshold_percent 95
   @expiration_refresh_ttl_seconds 6 * 60 * 60
   @expiring_soon_seconds 24 * 60 * 60
+  @redemption_projection_receive_timeout_ms 15_000
+  @redemption_projection_stale_grace_ms 60_000
 
   @type count_parse_result :: {:reported, non_neg_integer()} | :unreported
   @type snapshot_projection :: %{
@@ -37,6 +39,7 @@ defmodule CodexPooler.Upstreams.SavedResets do
           required(:expires_refresh_attempted_at) => String.t() | nil,
           required(:expires_reported?) => boolean(),
           required(:in_progress?) => boolean(),
+          required(:redemption_stale?) => boolean(),
           required(:last_redemption) => map() | nil
         }
   @type auto_policy_projection :: %{
@@ -128,11 +131,16 @@ defmodule CodexPooler.Upstreams.SavedResets do
   end
 
   @spec snapshot(UpstreamIdentity.t() | map() | nil) :: snapshot_projection()
-  def snapshot(%UpstreamIdentity{} = identity), do: snapshot(identity.metadata)
+  def snapshot(identity_or_metadata), do: snapshot(identity_or_metadata, now())
 
-  def snapshot(%{} = metadata) do
+  @spec snapshot(UpstreamIdentity.t() | map() | nil, DateTime.t()) :: snapshot_projection()
+  def snapshot(%UpstreamIdentity{} = identity, %DateTime{} = timestamp),
+    do: snapshot(identity.metadata, timestamp)
+
+  def snapshot(%{} = metadata, %DateTime{} = timestamp) do
     snapshot = Map.get(metadata, "saved_resets", metadata)
     redemption = Map.get(metadata, "saved_reset_redemption")
+    redemption_state = redemption_state(redemption, timestamp)
     status = snapshot_status(snapshot)
     available_count = snapshot_available_count(snapshot, status)
     available_expires_at = snapshot_available_expires_at(snapshot)
@@ -157,13 +165,14 @@ defmodule CodexPooler.Upstreams.SavedResets do
       expires_observed_at: expires_observed_at,
       expires_refresh_attempted_at: expires_refresh_attempted_at,
       expires_reported?: next_expires_at != nil,
-      in_progress?: redemption_in_progress?(redemption),
+      in_progress?: redemption_state == :in_progress,
+      redemption_stale?: redemption_state == :stale,
       last_redemption: redemption_or_nil(redemption)
     }
   end
 
-  def snapshot(_identity_or_metadata) do
-    snapshot(%{"saved_resets" => %{}})
+  def snapshot(_identity_or_metadata, %DateTime{} = timestamp) do
+    snapshot(%{"saved_resets" => %{}}, timestamp)
   end
 
   @spec reset_credit_list_refresh_due?(
@@ -514,8 +523,32 @@ defmodule CodexPooler.Upstreams.SavedResets do
   defp label(@unavailable, _count), do: "Saved resets unavailable"
   defp label(_status, _count), do: "Saved resets not reported"
 
-  defp redemption_in_progress?(%{"status" => "redeeming"}), do: true
-  defp redemption_in_progress?(_redemption), do: false
+  @spec redemption_state(map() | term(), DateTime.t()) :: :in_progress | :stale | :complete
+  defp redemption_state(
+         %{"status" => "redeeming", "started_at" => started_at},
+         %DateTime{} = timestamp
+       )
+       when is_binary(started_at) do
+    case DateTime.from_iso8601(started_at) do
+      {:ok, started_at, _offset} ->
+        if fresh_redemption?(started_at, timestamp), do: :in_progress, else: :stale
+
+      _invalid ->
+        :stale
+    end
+  end
+
+  defp redemption_state(%{"status" => "redeeming"}, _timestamp), do: :stale
+  defp redemption_state(_redemption, _timestamp), do: :complete
+
+  @spec fresh_redemption?(DateTime.t(), DateTime.t()) :: boolean()
+  defp fresh_redemption?(%DateTime{} = started_at, %DateTime{} = timestamp) do
+    DateTime.diff(timestamp, started_at, :millisecond) <
+      @redemption_projection_receive_timeout_ms + @redemption_projection_stale_grace_ms
+  end
+
+  @spec now() :: DateTime.t()
+  defp now, do: DateTime.utc_now() |> DateTime.truncate(:microsecond)
 
   defp redemption_or_nil(%{} = redemption), do: redemption
   defp redemption_or_nil(_redemption), do: nil

lib/codex_pooler_web/live/admin/components/pages/upstreams/cockpit/sections.ex

@@ -311,7 +311,7 @@ defmodule CodexPoolerWeb.Admin.UpstreamCockpitComponents.Sections do
             <div class="grid max-w-3xl gap-1">
               <p class="text-sm font-semibold text-base-content">Auto redeem policy</p>
               <p class="text-xs leading-5 text-base-content/60">
-                Automatic redemption can wait until weekly quota is blocked, start earlier near the quota limit when every eligible account is under pressure, or spend a soon-expiring reset when this account already has weekly usage. The reset buffer prevents spending when the weekly reset is close.
+                Automatic redemption can wait until weekly quota is blocked, start earlier near the quota limit when every eligible account is under pressure, or rescue a soon-expiring reset when this account already has weekly usage. The reset buffer prevents spending when the weekly reset is close.
               </p>
             </div>
             <label
@@ -341,7 +341,7 @@ defmodule CodexPoolerWeb.Admin.UpstreamCockpitComponents.Sections do
               label="Auto trigger"
               class="select select-bordered w-full"
               options={[
-                {"Blocked", "blocked"},
+                {"Blocked or expiring", "blocked"},
                 {"Near limit", "threshold"}
               ]}
             />

lib/codex_pooler_web/live/admin/components/pages/upstreams/cockpit/summary.ex

@@ -342,7 +342,7 @@ defmodule CodexPoolerWeb.Admin.UpstreamCockpitComponents.Summary do
        do: "Auto redeem on · near #{threshold}% · keep #{keep_credits}"
 
   defp saved_reset_policy_description(%{enabled?: true, keep_credits: keep_credits}),
-    do: "Auto redeem on · blocked · keep #{keep_credits}"
+    do: "Auto redeem on · blocked/expiring · keep #{keep_credits}"
 
   defp saved_reset_policy_description(_policy), do: "Auto redeem off"
 

lib/codex_pooler_web/live/admin/components/pages/upstreams/page_components.ex

@@ -605,9 +605,6 @@ defmodule CodexPoolerWeb.Admin.UpstreamPageComponents do
                 name="saved_reset_policy[auto_redeem_enabled]"
                 label="Auto redeem saved resets"
               />
-              <p class="text-xs leading-5 text-base-content/65">
-                Let Codex Pooler spend a banked reset automatically when the Pool is at risk and this account has more resets than the reserve below.
-              </p>
             </div>
 
             <div class="grid gap-4 md:grid-cols-[minmax(0,1.1fr)_minmax(9rem,0.9fr)]">
@@ -619,12 +616,12 @@ defmodule CodexPoolerWeb.Admin.UpstreamPageComponents do
                   name="saved_reset_policy[trigger_mode]"
                   label="When automatic redemption can start"
                   options={[
-                    {"Only after work is blocked", "blocked"},
+                    {"After block or near expiry", "blocked"},
                     {"Before work stops near the quota limit", "threshold"}
                   ]}
                 />
                 <p class="text-xs leading-5 text-base-content/65">
-                  Early mode waits until every eligible account in the Pool is also near the configured weekly quota limit. Expiring saved resets may also redeem early when this account already has weekly usage.
+                  Blocked mode waits for weekly quota exhaustion, except a known reset expiring within 24 hours may be rescued early after this account has weekly usage. Near-limit mode waits until every eligible account in the Pool is also near the configured weekly quota limit.
                 </p>
               </div>
 
@@ -640,7 +637,7 @@ defmodule CodexPoolerWeb.Admin.UpstreamPageComponents do
                   step="1"
                 />
                 <p class="text-xs leading-5 text-base-content/65">
-                  Used only by early mode. 95 means redeem when fresh weekly quota evidence shows at least 95% used.
+                  Used only by near-limit mode. 95 means redeem when every eligible account has fresh weekly quota evidence at or above 95% used.
                 </p>
               </div>
             </div>
@@ -655,7 +652,7 @@ defmodule CodexPoolerWeb.Admin.UpstreamPageComponents do
                 min="0"
               />
               <p class="text-xs leading-5 text-base-content/65">
-                Do not spend a saved reset when the weekly quota will reset naturally within this many minutes. Use 0 only when operators accept aggressive fail-open routing.
+                Do not spend a saved reset when the weekly quota will reset naturally within this many minutes. Set 0 to allow automatic redemption even when the natural reset is close.
               </p>
             </div>
 
@@ -669,7 +666,7 @@ defmodule CodexPoolerWeb.Admin.UpstreamPageComponents do
                 min="0"
               />
               <p class="text-xs leading-5 text-base-content/65">
-                Automatic redemption stops when the available reset count is at or below this reserve. Manual redemption below remains an explicit operator action.
+                Automatic redemption stops when the available reset count is at or below this reserve.
               </p>
             </div>
           </.form>

test/codex_pooler/gateway/routing/route_filtering_test.exs

@@ -96,6 +96,43 @@ defmodule CodexPooler.Gateway.Routing.RouteFilteringTest do
       assert [] = FakeUpstream.requests(upstream)
     end
 
+    test "auto redemption ignores stale in-progress redemption until manual recovery" do
+      {:ok, upstream} =
+        FakeUpstream.start_link(
+          {:path_json,
+           %{
+             "/api/codex/rate-limit-reset-credits/consume" => {200, %{"code" => "reset"}},
+             "/api/codex/usage" => {200, usage_payload(0)}
+           }}
+        )
+
+      started_at = DateTime.utc_now() |> DateTime.add(-5, :minute) |> DateTime.to_iso8601()
+      %{pool: pool, api_key: api_key} = active_api_key_fixture()
+
+      %{identity: identity, assignment: assignment} =
+        active_upstream_assignment_fixture(pool, %{
+          metadata:
+            upstream
+            |> saved_reset_metadata(1)
+            |> Map.put("saved_reset_redemption", %{
+              "status" => "redeeming",
+              "attempt_id" => Ecto.UUID.generate(),
+              "generation" => 1,
+              "trigger_kind" => "gateway_auto",
+              "started_at" => started_at,
+              "finished_at" => nil,
+              "result" => nil
+            })
+        })
+
+      identity = enable_saved_reset_auto_redeem!(identity)
+      upsert_weekly_exhausted_quota!(identity)
+      filter_input = filter_input(pool, api_key, assignment, identity, "auto-stale-redemption")
+
+      assert {:error, %{code: "quota_exhausted"}} = RouteFiltering.filter_candidates(filter_input)
+      assert [] = FakeUpstream.requests(upstream)
+    end
+
     test "auto redeems saved reset and refilters when weekly account quota is exhausted" do
       {:ok, upstream} =
         FakeUpstream.start_link(

test/codex_pooler/upstreams/saved_reset_redemption_test.exs

@@ -132,6 +132,47 @@ defmodule CodexPooler.Upstreams.SavedResetRedemptionTest do
       assert {:error, :redemption_in_progress} = SavedResetRedemption.redeem(assignment)
       assert [] = FakeUpstream.requests(fake)
     end
+
+    test "stale admin in-progress redemption is recovered by manual attempt" do
+      {:ok, fake} =
+        FakeUpstream.start_link(
+          {:path_json,
+           %{
+             "/api/codex/rate-limit-reset-credits/consume" => {200, %{"code" => "reset"}},
+             "/api/codex/usage" => {200, usage_payload(0)}
+           }}
+        )
+
+      stale_started_at =
+        DateTime.utc_now()
+        |> DateTime.add(-5, :minute)
+        |> DateTime.truncate(:microsecond)
+
+      %{identity: identity, assignment: assignment} =
+        assignment_with_fake(fake, "/api/codex/usage", "codex_api",
+          redemption: %{
+            "status" => "redeeming",
+            "attempt_id" => Ecto.UUID.generate(),
+            "generation" => 1,
+            "trigger_kind" => "admin_manual",
+            "started_at" => DateTime.to_iso8601(stale_started_at),
+            "finished_at" => nil,
+            "result" => nil
+          }
+        )
+
+      assert {:ok, %{status: :succeeded, applied?: true, code: "reset"}} =
+               SavedResetRedemption.redeem(assignment)
+
+      assert [consume_request, usage_request] = FakeUpstream.requests(fake)
+      assert consume_request.path == "/api/codex/rate-limit-reset-credits/consume"
+      assert usage_request.path == "/api/codex/usage"
+
+      persisted = Repo.reload!(identity)
+      assert get_in(persisted.metadata, ["saved_reset_redemption", "status"]) == "succeeded"
+      assert get_in(persisted.metadata, ["saved_reset_redemption", "generation"]) == 3
+      assert get_in(persisted.metadata, ["saved_reset_redemption", "result", "code"]) == "reset"
+    end
   end
 
   defp assignment_with_fake(fake, usage_path, path_style, opts \\ []) do