fix(permissions,console): retry a transient /me/permissions failure instead of stranding the app on its loading state (#3050) (#3052)

"Not now" is a real answer from this endpoint. On a multi-tenant host it is
served by the environment kernel that owns the session, and a COLD one answers
`503` + `Retry-After` while it warms (objectstack#4159 / cloud#927 —
`KernelWarmingError` states ~15s). The provider treated that like any other
failure: it set `error`. And the render branch falls back to `loadingFallback`
for the error state whenever no `errorFallback` is given — exactly how the
console wires it. So a tenant opening the console during a cold start sat on a
spinner indefinitely, with a `retry` no path could reach.

Both halves are needed; neither alone is enough.

1. The fetch re-attempts a TRANSIENT failure — 408, 425, 429, 502, 503, 504, or
   a thrown fetch (offline / DNS / aborted), which never got an answer at all. A
   server-stated `Retry-After` wins over the exponential backoff (both wire
   forms parsed, clamped to 30s so a hostile value cannot park the UI).
   `loading` stays true across the waits, so the fail-closed state holds and
   consumers never see a permissive flash mid-recovery. 401/403/404/500 are real
   answers about the caller and fail on the first attempt, exactly as before.
   New props, both defaulted so no call site changes: `maxRetries` (3; `0`
   restores the old behaviour) and `retryBaseDelayMs` (500).

2. The console passes an `errorFallback`. Retrying narrows the window but cannot
   close it — a build slower than the retry budget still lands in the error
   state. It now renders `<LoadingScreen error onRetry />`, the affordance that
   component has carried all along and nobody wired.

Also fixes a latent race the retries made much wider: the in-flight fetch is
cancelled when the effect tears down, so a slow answer for a previous `endpoint`
or `fetcher` can no longer overwrite a fast answer for the current one.

The retry primitives live in an internal `./retry` module (not exported from the
package root) — plain functions with plain tests, and a component file that also
exports helpers breaks fast refresh.

Author: os-zhuang

.changeset/me-permissions-retry-transient.md

@@ -0,0 +1,47 @@
+---
+"@object-ui/permissions": minor
+"@object-ui/console": patch
+---
+
+fix(permissions,console): `MePermissionsProvider` retries a transient `/me/permissions` failure instead of stranding the app on its loading state
+
+"Not now" is a real answer from this endpoint. On a multi-tenant host it is served
+by the environment kernel that owns the session, and a COLD one answers `503` +
+`Retry-After` while it warms (objectstack#4159 / cloud#927). The provider treated
+that like any other failure: it set `error` — and a consumer that passes no
+`errorFallback` renders `loadingFallback` for the error state too. The console
+does exactly that (`loadingFallback={<LoadingScreen />}`, no `errorFallback`), so
+the app sat on its spinner indefinitely, with a `retry` nobody could reach.
+
+The fetch now re-attempts a **transient** failure — `408`, `425`, `429`, `502`,
+`503`, `504`, or a thrown fetch (offline / DNS / aborted), which never got an
+answer at all. A server-stated `Retry-After` wins over the exponential backoff
+(both wire forms are read, and clamped to 30s so a hostile value cannot park the
+UI); otherwise the delay doubles from `retryBaseDelayMs`. `loading` stays true
+across the waits, so the fail-closed loading state holds and consumers never see
+a permissive flash mid-recovery.
+
+Unchanged for a real answer about the caller: `401`, `403`, `404` and `500` fail
+on the first attempt exactly as before. `500` is deliberately not retried — a
+genuine server fault neither benefits from hammering nor should be hidden behind
+a spinner.
+
+**New props**, both optional and defaulted so no call site needs to change:
+
+- `maxRetries` (default `3`) — `0` restores the previous single-attempt
+  behaviour.
+- `retryBaseDelayMs` (default `500`) — base for the exponential backoff.
+
+Also fixes a latent race the retries made much wider: the in-flight fetch is now
+cancelled when the effect tears down, so a slow answer for a previous `endpoint`
+or `fetcher` can no longer overwrite a fast answer for the current one. The retry
+primitives (`parseRetryAfterMs`, `backoffMs`, `isTransientFailure`,
+`TRANSIENT_STATUS`, `PermissionsFetchError`) live in a new internal `./retry`
+module — not exported from the package root.
+
+**The console now passes an `errorFallback`.** Retrying narrows the window but
+cannot close it — a kernel build slower than the retry budget still lands in the
+error state, and rendering `loadingFallback` there is what produced the eternal
+spinner. It now renders `<LoadingScreen error={...} onRetry={retry} />`, using the
+error + retry affordance that component has carried all along, so a user is never
+left with a spinner and no way forward.

apps/console/src/AppContent.tsx

@@ -132,7 +132,18 @@ export function AppContent() {
   // rendering fell open (restricted fields rendered editable).
   const authFetch = useMemo(() => createAuthenticatedFetch(), []);
   return (
-    <MePermissionsProvider endpoint={endpoint} fetcher={authFetch} loadingFallback={<LoadingScreen />}>
+    <MePermissionsProvider
+      endpoint={endpoint}
+      fetcher={authFetch}
+      loadingFallback={<LoadingScreen />}
+      // Without this, the provider's error state renders `loadingFallback` too —
+      // an eternal spinner with a `retry` nobody can reach. The provider already
+      // re-attempts a transient failure (a cold environment kernel answers 503 +
+      // `Retry-After` while it warms, objectstack#4159); this is what the user
+      // sees once those are exhausted, and `LoadingScreen` has carried the
+      // error + retry affordance all along.
+      errorFallback={(err, retry) => <LoadingScreen error={err.message} onRetry={retry} />}
+    >
       <LocalizationFetchProvider endpoint={localizationEndpoint}>
         <UploadProvider adapter={uploadAdapter}>
           <DefaultAppContent extraRoutes={systemRoutes} extraRoutesNoApp={systemRoutes} />

packages/permissions/src/MePermissionsProvider.tsx

@@ -7,6 +7,13 @@
  */
 
 import React, { useEffect, useMemo, useState, useCallback } from 'react';
+import {
+  PermissionsFetchError,
+  backoffMs,
+  isTransientFailure,
+  parseRetryAfterMs,
+  sleep,
+} from './retry';
 import type {
   PermissionAction,
   PermissionCheckResult,
@@ -65,6 +72,27 @@ export interface MePermissionsProviderProps {
   loadingFallback?: React.ReactNode;
   /** Rendered when load fails */
   errorFallback?: (err: Error, retry: () => void) => React.ReactNode;
+  /**
+   * How many times to re-attempt a TRANSIENT failure before giving up — a
+   * response that says "not now" (`TRANSIENT_STATUS` in `./retry`) or a network
+   * error. `0` disables retrying.
+   *
+   * This exists because "not now" is a real answer from this endpoint. On a
+   * multi-tenant host it is served by the environment kernel that owns the
+   * session, and a cold one answers `503` + `Retry-After` while it warms
+   * (objectstack#4159). Without a retry the provider keeps `loadingFallback`
+   * on screen forever, because a consumer that passes no `errorFallback` — the
+   * console does exactly that — renders the loading node for the error state
+   * too, and nothing ever calls `retry`.
+   *
+   * @default 3
+   */
+  maxRetries?: number;
+  /**
+   * Base for the exponential backoff between retries, in ms. A `Retry-After`
+   * header on the response wins over it. @default 500
+   */
+  retryBaseDelayMs?: number;
   /** Children */
   children: React.ReactNode;
 }
@@ -88,34 +116,75 @@ export function MePermissionsProvider({
   initialPermissions,
   loadingFallback = null,
   errorFallback,
+  maxRetries = 3,
+  retryBaseDelayMs = 500,
   children,
 }: MePermissionsProviderProps) {
   const [data, setData] = useState<MePermissionsResponse | null>(initialPermissions ?? null);
   const [error, setError] = useState<Error | null>(null);
   const [loading, setLoading] = useState(!initialPermissions);
 
-  const fetchPermissions = useCallback(async () => {
-    setLoading(true);
-    setError(null);
-    try {
+  /**
+   * Run the fetch, re-attempting a transient failure.
+   *
+   * `token` cancels an in-flight attempt. Retries put real time (a backoff, or
+   * a server-stated `Retry-After`) between the request and the `setState`, and
+   * during that window the effect may be torn down — by an unmount, or by
+   * `endpoint`/`fetcher` changing. Without the token a superseded attempt would
+   * still land, so a slow answer for the OLD endpoint could overwrite a fast
+   * answer for the new one.
+   */
+  const fetchPermissions = useCallback(
+    async (token?: { cancelled: boolean }) => {
+      const live = () => !token?.cancelled;
+      setLoading(true);
+      setError(null);
       const doFetch = fetcher ?? fetch;
-      const res = await doFetch(endpoint, {
-        credentials: 'include',
-        headers: { Accept: 'application/json' },
-      });
-      if (!res.ok) throw new Error(`Permissions endpoint returned ${res.status}`);
-      const json = (await res.json()) as MePermissionsResponse;
-      setData(json);
-    } catch (e) {
-      setError(e instanceof Error ? e : new Error(String(e)));
-    } finally {
-      setLoading(false);
-    }
-  }, [endpoint, fetcher]);
+      const attempts = Math.max(0, maxRetries) + 1;
+      for (let attempt = 0; attempt < attempts; attempt++) {
+        try {
+          const res = await doFetch(endpoint, {
+            credentials: 'include',
+            headers: { Accept: 'application/json' },
+          });
+          if (!res.ok) {
+            throw new PermissionsFetchError(
+              res.status,
+              parseRetryAfterMs(res.headers?.get?.('Retry-After'), Date.now()),
+            );
+          }
+          const json = (await res.json()) as MePermissionsResponse;
+          if (!live()) return;
+          setData(json);
+          setLoading(false);
+          return;
+        } catch (e) {
+          const err = e instanceof Error ? e : new Error(String(e));
+          if (!isTransientFailure(err) || attempt === attempts - 1) {
+            if (!live()) return;
+            setError(err);
+            setLoading(false);
+            return;
+          }
+          // `loading` stays true across the wait, so the fail-closed loading
+          // state holds and the recovery is invisible to consumers.
+          const stated = err instanceof PermissionsFetchError ? err.retryAfterMs : undefined;
+          await sleep(backoffMs(attempt, retryBaseDelayMs, stated));
+          if (!live()) return;
+        }
+      }
+    },
+    [endpoint, fetcher, maxRetries, retryBaseDelayMs],
+  );
+
+  /** The `retry` handed to `errorFallback` — uncancelled, it is user-initiated. */
+  const retry = useCallback(() => { void fetchPermissions(); }, [fetchPermissions]);
 
   useEffect(() => {
     if (initialPermissions) return;
-    void fetchPermissions();
+    const token = { cancelled: false };
+    void fetchPermissions(token);
+    return () => { token.cancelled = true; };
   }, [fetchPermissions, initialPermissions]);
 
   const checkField = useCallback(
@@ -232,7 +301,7 @@ export function MePermissionsProvider({
 
   if (loading && !data) return <>{loadingFallback}</>;
   if (error && !data) {
-    if (errorFallback) return <>{errorFallback(error, fetchPermissions)}</>;
+    if (errorFallback) return <>{errorFallback(error, retry)}</>;
     return <>{loadingFallback}</>;
   }
 

packages/permissions/src/__tests__/MePermissionsProvider.retry.test.tsx

@@ -0,0 +1,177 @@
+/**
+ * ObjectUI
+ * Copyright (c) 2024-present ObjectStack Inc.
+ *
+ * "Not now" is a real answer from `/me/permissions`, and it used to strand the
+ * console.
+ *
+ * On a multi-tenant host the endpoint is served by the environment kernel that
+ * owns the session, and a COLD one answers `503` + `Retry-After` while it warms
+ * (objectstack#4159 / cloud#927). The provider treated that like any other
+ * failure: it set `error`, and a consumer that passes no `errorFallback` — the
+ * console passes only `loadingFallback={<LoadingScreen />}` — renders the
+ * loading node for the error state too. So the app sat on its spinner forever,
+ * with a `retry` nobody could reach.
+ *
+ * These pin the retry: transient answers are re-attempted (server-stated delay
+ * first), real answers about the caller are not, and the fail-closed loading
+ * state holds throughout rather than flashing permissive.
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { render, screen, waitFor } from '@testing-library/react';
+import React from 'react';
+import { MePermissionsProvider } from '../MePermissionsProvider';
+import { parseRetryAfterMs } from '../retry';
+import { usePermissions } from '../usePermissions';
+
+function Probe() {
+  const { isLoaded, check } = usePermissions();
+  return (
+    <div>
+      <span data-testid="loaded">{String(isLoaded)}</span>
+      <span data-testid="edit">{String(check('account', 'update').allowed)}</span>
+    </div>
+  );
+}
+
+const PAYLOAD = {
+  authenticated: true,
+  userId: 'u1',
+  tenantId: 't1',
+  roles: [],
+  permissionSets: ['member_default'],
+  objects: { account: { allowRead: true, allowEdit: false } },
+  fields: {},
+};
+
+const ok = () => ({ ok: true, status: 200, headers: new Headers(), json: async () => PAYLOAD });
+const fail = (status: number, headers: Record<string, string> = {}) => ({
+  ok: false,
+  status,
+  headers: new Headers(headers),
+  json: async () => ({}),
+});
+
+/** Render with retries fast enough that the tests need no timer control. */
+const renderProvider = (props: Record<string, unknown> = {}) =>
+  render(
+    <MePermissionsProvider retryBaseDelayMs={1} {...props}>
+      <Probe />
+    </MePermissionsProvider>,
+  );
+
+describe('MePermissionsProvider retries a transient failure', () => {
+  beforeEach(() => { vi.spyOn(global, 'fetch' as any); });
+  afterEach(() => { vi.restoreAllMocks(); });
+
+  it('recovers from a warming 503 and serves the real answer', async () => {
+    (global.fetch as any)
+      .mockResolvedValueOnce(fail(503, { 'Retry-After': '0' }))
+      .mockResolvedValueOnce(ok());
+
+    renderProvider();
+
+    await waitFor(() => expect(screen.getByTestId('loaded').textContent).toBe('true'));
+    // The permission answer really arrived — not the anonymous fail-open.
+    expect(screen.getByTestId('edit').textContent).toBe('false');
+    expect((global.fetch as any)).toHaveBeenCalledTimes(2);
+  });
+
+  it('prefers the server-stated Retry-After over its own backoff', async () => {
+    // Timer-free proof: the backoff base is 100s, so if the header were ignored
+    // this test could not finish. `Retry-After: 0` says "now".
+    (global.fetch as any)
+      .mockResolvedValueOnce(fail(503, { 'Retry-After': '0' }))
+      .mockResolvedValueOnce(ok());
+
+    renderProvider({ retryBaseDelayMs: 100_000 });
+
+    await waitFor(() => expect(screen.getByTestId('loaded').textContent).toBe('true'));
+  });
+
+  it('retries a thrown fetch (offline / DNS / aborted) too', async () => {
+    (global.fetch as any)
+      .mockRejectedValueOnce(new TypeError('Failed to fetch'))
+      .mockResolvedValueOnce(ok());
+
+    renderProvider();
+
+    await waitFor(() => expect(screen.getByTestId('loaded').textContent).toBe('true'));
+  });
+
+  it('holds the fail-closed loading state while retrying — never a permissive flash', async () => {
+    (global.fetch as any)
+      .mockResolvedValueOnce(fail(503))
+      .mockResolvedValueOnce(fail(503))
+      .mockResolvedValueOnce(ok());
+
+    renderProvider({ loadingFallback: <span data-testid="spinner" /> });
+
+    // Children (and therefore any permissive default) are not rendered yet.
+    expect(screen.getByTestId('spinner')).toBeTruthy();
+    expect(screen.queryByTestId('loaded')).toBeNull();
+    await waitFor(() => expect(screen.getByTestId('loaded').textContent).toBe('true'));
+  });
+
+  it('gives up after maxRetries and surfaces the error', async () => {
+    (global.fetch as any).mockResolvedValue(fail(503));
+
+    const errorFallback = vi.fn((err: Error) => <span data-testid="err">{err.message}</span>);
+    renderProvider({ maxRetries: 2, errorFallback });
+
+    await waitFor(() => expect(screen.getByTestId('err')).toBeTruthy());
+    expect(screen.getByTestId('err').textContent).toContain('503');
+    expect((global.fetch as any)).toHaveBeenCalledTimes(3); // 1 attempt + 2 retries
+  });
+});
+
+describe('MePermissionsProvider does NOT retry a real answer about the caller', () => {
+  beforeEach(() => { vi.spyOn(global, 'fetch' as any); });
+  afterEach(() => { vi.restoreAllMocks(); });
+
+  it.each([401, 403, 404, 500])('gives up immediately on %i', async (status) => {
+    (global.fetch as any).mockResolvedValue(fail(status));
+
+    const errorFallback = vi.fn((err: Error) => <span data-testid="err">{err.message}</span>);
+    renderProvider({ errorFallback });
+
+    await waitFor(() => expect(screen.getByTestId('err')).toBeTruthy());
+    expect((global.fetch as any)).toHaveBeenCalledTimes(1);
+  });
+
+  it('maxRetries: 0 disables retrying entirely', async () => {
+    (global.fetch as any).mockResolvedValue(fail(503));
+
+    const errorFallback = vi.fn((err: Error) => <span data-testid="err">{err.message}</span>);
+    renderProvider({ maxRetries: 0, errorFallback });
+
+    await waitFor(() => expect(screen.getByTestId('err')).toBeTruthy());
+    expect((global.fetch as any)).toHaveBeenCalledTimes(1);
+  });
+});
+
+describe('parseRetryAfterMs', () => {
+  const NOW = Date.parse('2026-07-30T12:00:00Z');
+
+  it('reads delta-seconds', () => {
+    expect(parseRetryAfterMs('4', NOW)).toBe(4000);
+    expect(parseRetryAfterMs('0', NOW)).toBe(0);
+    expect(parseRetryAfterMs('  7  ', NOW)).toBe(7000);
+  });
+
+  it('reads an HTTP-date, relative to now', () => {
+    expect(parseRetryAfterMs('Thu, 30 Jul 2026 12:00:03 GMT', NOW)).toBe(3000);
+  });
+
+  it('clamps a far-future value so it cannot park the UI', () => {
+    expect(parseRetryAfterMs('999999', NOW)).toBe(30_000);
+    expect(parseRetryAfterMs('Fri, 31 Jul 2026 12:00:00 GMT', NOW)).toBe(30_000);
+  });
+
+  it('ignores absent, empty, unparseable and past values', () => {
+    for (const v of [null, undefined, '', '   ', 'soon', 'Thu, 30 Jul 2026 11:59:00 GMT']) {
+      expect(parseRetryAfterMs(v, NOW), String(v)).toBeUndefined();
+    }
+  });
+});