[Refactor] Change Retry Logic in Email Sending (#1191)

### Context
Some of our users' emails were getting stuck in sending. The long delays
in processing the retries caused a vercel function timeout.

### Summary of Changes
We refactor the low level email sending functions to remove the retry
logic there. We kick it up to the email queue step. Additionally, we
flag emails to be retried when they encounter issues but leave it for a
future iteration to actually perform the retry. We perform an
exponential backoff with a random component to decide when they have to
be retried. We also make some small adjustments to the queuing function
to not queue skipped emails.

When an email fails to send during the sending function, we check to see
if it is a retryable error or not. Some errors are transient and trying
again may succeed while others indicate deeper issues. If it is
retryable, and the max number of retry attempts hasn't been reached, we
set `nextSendRetryAt` to a time determined by an exponential backoff
calculation function. When the queuing function looks for emails to
queue, it doesn't just pick up the `SCHEDULED`. emails whose
`scheduledAt` time <= `NOW()`, but also those emails whose
`nextSendRetryAt` time <= `NOW()`. What this means in practice is that
one iteration of the `email-queue-step` will mark emails as retryable
while another iteration will perform the retry. This should be cleaner
and prevent long delays in the `email-queue-step` process due to
retries. This also makes it easier to scale up the number of retries if
need be.

Author: nams1570

AGENTS.md

@@ -101,6 +101,7 @@ To see all development ports, refer to the index.html of `apps/dev-launchpad/pub
 - Fail early, fail loud. Fail fast with an error instead of silently continuing.
 - Do NOT use `as`/`any`/type casts or anything else like that to bypass the type system unless you specifically asked the user about it. Most of the time a place where you would use type casts is not one where you actually need them. Avoid wherever possible.
 - When writing database migration files, assume that we have >1,000,000 rows in every table (unless otherwise specified). This means you may have to use CONDITIONALLY_REPEAT_MIGRATION_SENTINEL to avoid running the migration and things like concurrent index builds; see the existing migrations for examples.
+- Each migration file runs in its own transaction with a relatively short timeout. Split long-running operations into separate migration files to avoid timeouts. For example, when adding CHECK constraints, use `NOT VALID` in one migration, then `VALIDATE CONSTRAINT` in a separate migration file.
 - **When building frontend code, always carefully deal with loading and error states.** Be very explicit with these; some components make this easy, eg. the button onClick already takes an async callback for loading state, but make sure this is done everywhere, and make sure errors are NEVER just silently swallowed.
 - Unless very clearly equivalent from types, prefer explicit null/undefinedness checks over boolean checks, eg. `foo == null` instead of `!foo`.
 

apps/backend/prisma/migrations/20260210000000_deferred_email_retry/migration.sql

@@ -0,0 +1,27 @@
+-- Add deferred retry fields for email sending
+-- These fields allow the email queue to schedule retries for later iterations
+-- instead of blocking the current iteration with inline retries.
+
+ALTER TABLE "EmailOutbox"
+  ADD COLUMN "sendRetries" INTEGER NOT NULL DEFAULT 0,
+  ADD COLUMN "nextSendRetryAt" TIMESTAMP(3),
+  ADD COLUMN "sendAttemptErrors" JSONB;
+
+-- Constraint: nextSendRetryAt can only be set after at least one failed attempt
+-- (if sendRetries is 0, no attempt has failed, so there's nothing to retry)
+-- Use NOT VALID to avoid holding ACCESS EXCLUSIVE lock during full-table validation.
+-- Validation happens in a separate migration to avoid transaction timeout.
+ALTER TABLE "EmailOutbox"
+  ADD CONSTRAINT "EmailOutbox_nextSendRetryAt_requires_failure"
+  CHECK ("nextSendRetryAt" IS NULL OR "sendRetries" > 0) NOT VALID;
+
+-- Constraint: sendAttemptErrors can only be set after at least one failed attempt
+ALTER TABLE "EmailOutbox"
+  ADD CONSTRAINT "EmailOutbox_sendAttemptErrors_requires_failure"
+  CHECK ("sendAttemptErrors" IS NULL OR "sendRetries" > 0) NOT VALID;
+
+-- Constraint: nextSendRetryAt must be null when email has finished sending
+-- (if finishedSendingAt is set, there's nothing more to retry)
+ALTER TABLE "EmailOutbox"
+  ADD CONSTRAINT "EmailOutbox_no_retry_after_finished"
+  CHECK ("finishedSendingAt" IS NULL OR "nextSendRetryAt" IS NULL) NOT VALID;

apps/backend/prisma/migrations/20260210000001_deferred_email_retry_validate/migration.sql

@@ -0,0 +1,7 @@
+-- Validate the deferred retry constraints added in the previous migration.
+-- This runs in a separate transaction to avoid timeout, and only takes
+-- SHARE UPDATE EXCLUSIVE lock (allows concurrent reads/writes).
+
+ALTER TABLE "EmailOutbox" VALIDATE CONSTRAINT "EmailOutbox_nextSendRetryAt_requires_failure";
+ALTER TABLE "EmailOutbox" VALIDATE CONSTRAINT "EmailOutbox_sendAttemptErrors_requires_failure";
+ALTER TABLE "EmailOutbox" VALIDATE CONSTRAINT "EmailOutbox_no_retry_after_finished";

apps/backend/prisma/migrations/20260213004424_email_outbox_is_queued_index/migration.sql

@@ -0,0 +1,6 @@
+-- SPLIT_STATEMENT_SENTINEL
+-- SINGLE_STATEMENT_SENTINEL
+-- RUN_OUTSIDE_TRANSACTION_SENTINEL
+-- Index on isQueued for efficient queueReadyEmails() queries.
+-- Most emails have isQueued=TRUE (already processed), so filtering for FALSE is highly selective.
+CREATE INDEX CONCURRENTLY IF NOT EXISTS "EmailOutbox_isQueued_idx" ON /* SCHEMA_NAME_SENTINEL */."EmailOutbox" ("isQueued");

apps/backend/prisma/schema.prisma

@@ -99,8 +99,8 @@ model ExternalDbSyncMetadata {
 
   singleton BooleanTrue @unique @default(TRUE)
 
-  sequencerEnabled  Boolean @default(true)
-  pollerEnabled     Boolean @default(true)
+  sequencerEnabled Boolean @default(true)
+  pollerEnabled    Boolean @default(true)
 
   createdAt DateTime @default(now())
   updatedAt DateTime @updatedAt
@@ -833,7 +833,7 @@ model EmailOutbox {
   // The scheduled time of when the email should be added to the queue. Can be edited, but only if the email has not yet started sending. Doing so should set isQueued to false.
   scheduledAt DateTime
 
-  // The scheduled time of the email if it is in the future.
+  // Whether the email has been queued for sending. Once queued, this stays true unless the email is retried or rescheduled.
   isQueued Boolean @default(false)
 
   // A generated column that is equal to scheduledAt if isQueued is false, otherwise null. See the note above on EmailOutboxStatus.status for more details on dbgenerated values.
@@ -844,6 +844,14 @@ model EmailOutbox {
   // if startedSendingAt is not set, then finishedSendingAt is also not set
   finishedSendingAt DateTime?
 
+  // Deferred retry fields for email sending
+  // Number of send retries attempted (starts at 0, incremented on each failure). Reset when email content is edited.
+  sendRetries       Int       @default(0)
+  // When to retry sending (null = not waiting for retry). Set when a retryable error occurs. Reset when email content is edited. Must be null if sendRetries is 0 (enforced by EmailOutbox_nextSendRetryAt_requires_failure). Must be null when finishedSendingAt is set (enforced by EmailOutbox_no_retry_after_finished).
+  nextSendRetryAt   DateTime?
+  // JSON array of errors from each failed send attempt. Each entry has: { attemptNumber, timestamp, externalMessage, externalDetails, internalMessage, internalDetails }. Reset when email content is edited. Must be null if sendRetries is 0 (enforced by EmailOutbox_sendAttemptErrors_requires_failure).
+  sendAttemptErrors Json?
+
   // A generated column that is equal to finishedSendingAt if canHaveDeliveryInfo is false, otherwise deliveredAt.
   sentAt DateTime? @default(dbgenerated("\nCASE\n    WHEN (\"canHaveDeliveryInfo\" IS TRUE) THEN \"deliveredAt\"\n    WHEN (\"canHaveDeliveryInfo\" IS FALSE) THEN \"finishedSendingAt\"\n    ELSE NULL::timestamp without time zone\nEND"))
 
@@ -878,6 +886,7 @@ model EmailOutbox {
   @@index([tenancyId, finishedSendingAt(sort: Desc), scheduledAtIfNotYetQueued(sort: Desc), priority, id], map: "EmailOutbox_ordering_idx")
   @@index([tenancyId, simpleStatus], map: "EmailOutbox_simple_status_tenancy_idx")
   @@index([tenancyId, status], map: "EmailOutbox_status_tenancy_idx")
+  @@index([isQueued], map: "EmailOutbox_isQueued_idx")
 }
 
 model EmailOutboxProcessingMetadata {
@@ -1088,7 +1097,7 @@ model OutgoingRequest {
 
   qstashOptions       Json
   startedFulfillingAt DateTime?
-  deduplicationKey String?
+  deduplicationKey    String?
 
   // Partial unique index on deduplicationKey WHERE startedFulfillingAt IS NULL
   // is created in a custom migration (not expressible in Prisma schema)

apps/backend/src/app/api/latest/emails/outbox/crud.tsx

@@ -7,6 +7,7 @@ import { KnownErrors } from "@stackframe/stack-shared";
 import { emailOutboxCrud, EmailOutboxCrud } from "@stackframe/stack-shared/dist/interface/crud/email-outbox";
 import { yupObject, yupString } from "@stackframe/stack-shared/dist/schema-fields";
 import { StackAssertionError, StatusError, throwErr } from "@stackframe/stack-shared/dist/utils/errors";
+import { Json } from "@stackframe/stack-shared/dist/utils/json";
 import { createLazyProxy } from "@stackframe/stack-shared/dist/utils/proxies";
 
 /**
@@ -57,6 +58,25 @@ function prismaModelToCrud(prismaModel: EmailOutbox): EmailOutboxCrud["Server"][
     to = { type: "custom-emails", emails: recipient?.emails ?? [] };
   }
 
+  // Convert sendAttemptErrors from DB format (camelCase) to API format (snake_case)
+  const sendAttemptErrors = prismaModel.sendAttemptErrors
+    ? (prismaModel.sendAttemptErrors as Array<{
+        attemptNumber: number,
+        timestamp: string,
+        externalMessage: string,
+        externalDetails: Record<string, Json>,
+        internalMessage: string,
+        internalDetails: Record<string, Json>,
+      }>).map(e => ({
+        attempt_number: e.attemptNumber,
+        timestamp: e.timestamp,
+        external_message: e.externalMessage,
+        external_details: e.externalDetails,
+        internal_message: e.internalMessage,
+        internal_details: e.internalDetails,
+      }))
+    : null;
+
   // Base fields present on all emails
   const base = {
     id: prismaModel.id,
@@ -68,6 +88,9 @@ function prismaModelToCrud(prismaModel: EmailOutbox): EmailOutboxCrud["Server"][
     variables: (prismaModel.extraRenderVariables ?? {}) as Record<string, any>,
     skip_deliverability_check: prismaModel.shouldSkipDeliverabilityCheck,
     scheduled_at_millis: prismaModel.scheduledAt.getTime(),
+    send_retries: prismaModel.sendRetries,
+    next_send_retry_at_millis: prismaModel.nextSendRetryAt?.getTime() ?? null,
+    send_attempt_errors: sendAttemptErrors,
     // Default flags (overridden in specific statuses)
     is_paused: false,
     has_rendered: false,
@@ -358,6 +381,7 @@ export const emailOutboxCrudHandlers = createLazyProxy(() => createCrudHandlers(
       // Cancel action - mark as skipped
       set("isPaused", Prisma.sql`false`);
       set("isQueued", Prisma.sql`false`);
+      setNull("nextSendRetryAt"); // Clear any pending retry so it won't be picked up
       set("skippedReason", Prisma.sql`'MANUALLY_CANCELLED'::"EmailOutboxSkippedReason"`);
       set("skippedDetails", Prisma.sql`'{}'::jsonb`);
     } else {
@@ -395,13 +419,16 @@ export const emailOutboxCrudHandlers = createLazyProxy(() => createCrudHandlers(
       // If content changed, reset rendering and sending state
       if (needsRerenderReset) {
         set("isQueued", Prisma.sql`false`);
+        // Reset retry fields (sendRetries to 0, others to null)
+        set("sendRetries", Prisma.sql`0`);
         setNull(
           "renderedByWorkerId", "startedRenderingAt", "finishedRenderingAt",
           "renderErrorExternalMessage", "renderErrorExternalDetails",
           "renderErrorInternalMessage", "renderErrorInternalDetails",
           "renderedHtml", "renderedText", "renderedSubject",
           "renderedIsTransactional", "renderedNotificationCategoryId",
           "startedSendingAt", "finishedSendingAt",
+          "nextSendRetryAt", "sendAttemptErrors",
           "sendServerErrorExternalMessage", "sendServerErrorExternalDetails",
           "sendServerErrorInternalMessage", "sendServerErrorInternalDetails",
           "skippedReason", "skippedDetails", "canHaveDeliveryInfo",
@@ -494,6 +521,9 @@ function parseEmailOutboxFromJson(j: Record<string, unknown>): EmailOutbox {
     scheduledAtIfNotYetQueued: dateOrNull("scheduledAtIfNotYetQueued"),
     startedSendingAt: dateOrNull("startedSendingAt"),
     finishedSendingAt: dateOrNull("finishedSendingAt"),
+    sendRetries: j.sendRetries as number,
+    nextSendRetryAt: dateOrNull("nextSendRetryAt"),
+    sendAttemptErrors: j.sendAttemptErrors as Prisma.JsonValue,
     sentAt: dateOrNull("sentAt"),
     sendServerErrorExternalMessage: j.sendServerErrorExternalMessage as string | null,
     sendServerErrorExternalDetails: j.sendServerErrorExternalDetails as Prisma.JsonValue,