fix(fcm): gate native-fallback probe on rolling FCM health

The native-fallback probe previously returned true whenever FCM was
configured AND devices were registered, which suppressed web-push for
the namespace. The HAPI Bot correctly pointed out the gap: if the FCM
pipeline silently breaks (expired service-account key, sustained 5xx,
OAuth token-fetch failure, network blackhole) the operator gets nothing
on either channel until they manually intervene.

Approach (deliberate, not the bot's exact suggested fix):

- FcmService now keeps a small rolling window (last 8 outcomes) of send
  attempts and exposes `isHealthy()`. The threshold is 5+/8 failures =
  unhealthy; the buffer starts empty so a freshly-booted hub is
  optimistic ("innocent until proven guilty") and does not double-fire
  on event #1.
- Token-fetch failure (`getFcmAccessToken` throws) now records exactly
  one health-failure (not one per device), short-circuits the send
  loop, and returns a result so `sendToNamespace` no longer leaks the
  exception.
- `invalid` token responses are explicitly excluded from the health
  buffer because they are per-device facts (rotated/uninstalled token),
  not pipeline failures - FCM was reachable, it just rejected one
  stale token.
- `buildNativeFallbackProbe` now optionally accepts the FcmService and
  short-circuits to "let web-push fire" when health is bad, before it
  even queries the device registry. The single-arg call shape is still
  supported for back-compat.

Why not the bot's exact suggestion ("invert: call FCM first, fall back
on result.sent === 0"):
- Couples PushNotificationChannel to FcmService and FcmSendPayload,
  reversing the clean parallel-channel architecture established earlier
  in this PR.
- Treats every transient single-event failure as fallback-worthy, which
  re-opens the duplicate-notification race that the suppression logic
  was added to close (FCM HTTP timeout that delivers later + the web
  push we sent in the meantime = two pings).
- A rolling health window only flips on sustained breakage, which is
  the actual operational scenario the bot is worried about.

The wrist-first design intent ("FCM fires unconditionally, web-push is
suppressed for the same namespace") documented in
docs/api/native-companion-contract.md is preserved on the happy path.
The probe only re-enables web-push when there is concrete evidence the
native pipeline is not delivering.

Tests:
- New FcmService.isHealthy suite covers empty-buffer, threshold flip,
  recovery as failures age out of the window, invalid-token exclusion,
  and network-error path.
- nativeFallbackProbe gains coverage for the unhealthy-but-registered,
  healthy-and-registered, and absent-fcmService (back-compat) cases.
- All 292 hub tests still pass; typecheck clean.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: heavygee

hub/src/fcm/fcmService.test.ts

@@ -199,3 +199,102 @@ describe('FcmService.sendToNamespace', () => {
         expect(globalThis.fetch).not.toHaveBeenCalled()
     })
 })
+
+describe('FcmService.isHealthy (rolling outcome window)', () => {
+    let originalFetch: typeof globalThis.fetch
+    beforeEach(() => {
+        originalFetch = globalThis.fetch
+    })
+    afterEach(() => {
+        globalThis.fetch = originalFetch
+    })
+
+    it('starts healthy with an empty outcome buffer (innocent until proven guilty)', () => {
+        const store = makeStore([])
+        const svc = new FcmService('proj-id', { client_email: 'x', private_key: 'y' }, store as never)
+        expect(svc.isHealthy()).toBe(true)
+    })
+
+    it('flips to unhealthy after 5 consecutive transient failures', async () => {
+        const store = makeStore([
+            { namespace: 'default', token: 't1', platform: 'phone', deviceId: 'p1' }
+        ])
+        globalThis.fetch = mock(async () =>
+            new Response('Service Unavailable', { status: 503 })
+        ) as unknown as typeof fetch
+
+        const svc = new FcmService('proj-id', { client_email: 'x', private_key: 'y' }, store as never)
+
+        // 4 failures: still healthy (threshold is 5)
+        for (let i = 0; i < 4; i += 1) {
+            await svc.sendToNamespace('default', makePayload())
+        }
+        expect(svc.isHealthy()).toBe(true)
+
+        // 5th failure crosses the threshold
+        await svc.sendToNamespace('default', makePayload())
+        expect(svc.isHealthy()).toBe(false)
+    })
+
+    it('recovers to healthy as recent successes age out the failure tail', async () => {
+        const store = makeStore([
+            { namespace: 'default', token: 't1', platform: 'phone', deviceId: 'p1' }
+        ])
+        let callCount = 0
+        globalThis.fetch = mock(async () => {
+            callCount += 1
+            // First 5 calls fail (503), rest succeed
+            if (callCount <= 5) {
+                return new Response('Service Unavailable', { status: 503 })
+            }
+            return new Response('{"name":"ok"}', { status: 200 })
+        }) as unknown as typeof fetch
+
+        const svc = new FcmService('proj-id', { client_email: 'x', private_key: 'y' }, store as never)
+
+        for (let i = 0; i < 5; i += 1) {
+            await svc.sendToNamespace('default', makePayload())
+        }
+        expect(svc.isHealthy()).toBe(false)
+
+        // 4 successes after 5 failures: window is [F,F,F,F,F,S,S,S,S] -> trim
+        // to last 8: [F,F,F,F,S,S,S,S] -> 4 failures, threshold 5 -> healthy.
+        for (let i = 0; i < 4; i += 1) {
+            await svc.sendToNamespace('default', makePayload())
+        }
+        expect(svc.isHealthy()).toBe(true)
+    })
+
+    it('does NOT count invalid-token responses against health (per-device fact, not pipeline failure)', async () => {
+        const store = makeStore([
+            { namespace: 'default', token: 'rotated', platform: 'phone', deviceId: 'p1' }
+        ])
+        globalThis.fetch = mock(async () =>
+            new Response('{"error":{"status":"UNREGISTERED"}}', { status: 404 })
+        ) as unknown as typeof fetch
+
+        const svc = new FcmService('proj-id', { client_email: 'x', private_key: 'y' }, store as never)
+
+        // 10 invalid-token responses - the pipeline is fine, just the
+        // device is dead. Health must not flip.
+        for (let i = 0; i < 10; i += 1) {
+            await svc.sendToNamespace('default', makePayload())
+        }
+        expect(svc.isHealthy()).toBe(true)
+    })
+
+    it('counts fetch-throw (network error) as a health failure', async () => {
+        const store = makeStore([
+            { namespace: 'default', token: 't1', platform: 'phone', deviceId: 'p1' }
+        ])
+        globalThis.fetch = mock(async () => {
+            throw new Error('ECONNREFUSED')
+        }) as unknown as typeof fetch
+
+        const svc = new FcmService('proj-id', { client_email: 'x', private_key: 'y' }, store as never)
+        for (let i = 0; i < 5; i += 1) {
+            await svc.sendToNamespace('default', makePayload())
+        }
+        expect(svc.isHealthy()).toBe(false)
+    })
+})

hub/src/fcm/fcmService.ts

@@ -55,25 +55,77 @@ type FcmSendResult = {
 type FcmTokenSendResult = 'sent' | 'invalid' | 'failed'
 
 export class FcmService {
+    /**
+     * Rolling window of the last N send outcomes. Drives `isHealthy()`,
+     * which the native-fallback probe consults to decide whether suppressing
+     * web-push for this namespace is still safe. We deliberately do NOT
+     * count `invalid` here - an invalid token is a per-device fact, not an
+     * FCM-pipeline-broken signal (FCM was reachable, it just rejected one
+     * stale token). Only `sent` and `failed` populate the buffer.
+     */
+    private recentOutcomes: Array<'sent' | 'failed'> = []
+    private static readonly HEALTH_WINDOW = 8
+    private static readonly HEALTH_FAILURE_THRESHOLD = 5
+
     constructor(
         private readonly projectId: string,
         private readonly serviceAccount: ServiceAccount,
         private readonly store: Store
     ) {}
 
+    /**
+     * Health gate for the native-fallback probe. Returns false when the
+     * recent-outcome window is dominated by failures (broken Firebase
+     * credentials, sustained 5xx, network blackhole). When unhealthy, the
+     * probe lets web-push fire as a last-resort surface for this namespace.
+     *
+     * Empty buffer is treated as healthy (innocent until proven guilty) so
+     * a freshly-started hub does not silently double-fire on event #1.
+     */
+    isHealthy(): boolean {
+        if (this.recentOutcomes.length === 0) return true
+        const failures = this.recentOutcomes.filter((o) => o === 'failed').length
+        return failures < FcmService.HEALTH_FAILURE_THRESHOLD
+    }
+
+    private recordOutcome(outcome: 'sent' | 'failed'): void {
+        this.recentOutcomes.push(outcome)
+        if (this.recentOutcomes.length > FcmService.HEALTH_WINDOW) {
+            this.recentOutcomes.shift()
+        }
+    }
+
     async sendToNamespace(namespace: string, payload: FcmSendPayload): Promise<FcmSendResult> {
         const devices = this.store.fcm.getDevicesByNamespace(namespace)
         if (devices.length === 0) {
             return { sent: 0, failed: 0, invalidTokens: [] }
         }
 
-        const accessToken = await getFcmAccessToken(this.serviceAccount)
+        let accessToken: string
+        try {
+            accessToken = await getFcmAccessToken(this.serviceAccount)
+        } catch (e) {
+            // Token-fetch failure (expired service account key, OAuth
+            // outage, network) - count one health-failure (not one per
+            // device, that would over-weight the buffer) and return.
+            console.error('[FcmService] Token fetch failed:', e instanceof Error ? e.message : e)
+            this.recordOutcome('failed')
+            return { sent: 0, failed: devices.length, invalidTokens: [] }
+        }
+
         const invalidTokens: string[] = []
         let sent = 0
         let failed = 0
 
         await Promise.all(devices.map(async (device) => {
             const result = await this.sendToToken(accessToken, device.token, payload, device.platform)
+            // `invalid` is a per-device fact, not a pipeline signal -
+            // exclude it from the health buffer (see field doc above).
+            if (result === 'sent') {
+                this.recordOutcome('sent')
+            } else if (result === 'failed') {
+                this.recordOutcome('failed')
+            }
             if (result === 'sent') {
                 sent += 1
                 return

hub/src/fcm/nativeFallbackProbe.test.ts

@@ -55,4 +55,36 @@ describe('buildNativeFallbackProbe', () => {
         expect(probe('empty')).toBe(false)
         expect(probe('untouched')).toBe(false)
     })
+
+    it('returns false when fcmService reports unhealthy, even with registered devices', () => {
+        const store = makeStore({ default: 3 })
+        const fcmConfig = { projectId: 'p', serviceAccount: { client_email: 'x', private_key: 'y' } }
+        const fcmService = { isHealthy: mock(() => false) }
+        const probe = buildNativeFallbackProbe(store as unknown as Store, fcmConfig, fcmService)
+
+        expect(probe('default')).toBe(false)
+        expect(fcmService.isHealthy).toHaveBeenCalled()
+        // We short-circuit before hitting the device store - the namespace
+        // has registered devices but a broken FCM pipeline means web-push
+        // is the only surface still able to reach the operator.
+        expect(store.fcm.getDevicesByNamespace).not.toHaveBeenCalled()
+    })
+
+    it('returns true when fcmService reports healthy and devices exist', () => {
+        const store = makeStore({ default: 1 })
+        const fcmConfig = { projectId: 'p', serviceAccount: { client_email: 'x', private_key: 'y' } }
+        const fcmService = { isHealthy: mock(() => true) }
+        const probe = buildNativeFallbackProbe(store as unknown as Store, fcmConfig, fcmService)
+
+        expect(probe('default')).toBe(true)
+        expect(fcmService.isHealthy).toHaveBeenCalled()
+    })
+
+    it('treats absent fcmService as healthy (back-compat with single-arg call site)', () => {
+        const store = makeStore({ default: 2 })
+        const fcmConfig = { projectId: 'p', serviceAccount: { client_email: 'x', private_key: 'y' } }
+        const probe = buildNativeFallbackProbe(store as unknown as Store, fcmConfig)
+
+        expect(probe('default')).toBe(true)
+    })
 })

hub/src/fcm/nativeFallbackProbe.ts

@@ -1,25 +1,37 @@
 import type { Store } from '../store'
+import type { FcmService } from './fcmService'
 
 /**
  * Build a per-namespace probe used by the web-push channel to decide whether
  * to defer to the native FCM channel. The probe MUST only return true when
- * BOTH conditions hold:
+ * ALL of these hold:
  *
  *   1. FCM is actually configured on this hub start (fcmConfig is truthy),
  *      so a registered FCM channel exists to deliver the notification.
- *   2. At least one device row is registered for the namespace.
+ *   2. The FCM service is currently healthy (recent sends are not all
+ *      failing). When credentials expire or the FCM pipeline blackholes,
+ *      we let web-push run again so the operator does not get silently
+ *      cut off from all notifications.
+ *   3. At least one device row is registered for the namespace.
  *
- * Failing either condition means web-push must fire normally - otherwise
- * notifications go to /dev/null when, for example, an operator removes
- * `FCM_SERVICE_ACCOUNT_PATH` from the env without clearing stored device
- * registrations from the database.
+ * Failing any condition means web-push must fire normally. The default
+ * "happy path" remains: native is the canonical wrist-first surface and
+ * web-push is suppressed to avoid duplicate OS notifications - this probe
+ * only re-enables web-push when we have evidence the native pipeline is
+ * not actually delivering.
  */
 export function buildNativeFallbackProbe(
     store: Store,
-    fcmConfig: unknown
+    fcmConfig: unknown,
+    fcmService?: Pick<FcmService, 'isHealthy'>
 ): (namespace: string) => boolean {
     if (!fcmConfig) {
         return () => false
     }
-    return (namespace: string) => store.fcm.getDevicesByNamespace(namespace).length > 0
+    return (namespace: string) => {
+        if (fcmService && !fcmService.isHealthy()) {
+            return false
+        }
+        return store.fcm.getDevicesByNamespace(namespace).length > 0
+    }
 }

hub/src/startHub.ts

@@ -201,13 +201,28 @@ export async function startHub(options: StartHubOptions = {}): Promise<HubInstan
 
     const fcmConfig = resolveFcmConfig()
 
+    // Build the optional FCM service early so the native-fallback probe
+    // can consult its health gate. When FCM is configured, `fcmService` is
+    // shared between the FcmNotificationChannel and the probe so a broken
+    // pipeline (expired credentials, sustained 5xx) lets web-push run as
+    // a last-resort surface for the namespace instead of silently muting
+    // both channels.
+    const fcmService = fcmConfig
+        ? new FcmService(fcmConfig.projectId, fcmConfig.serviceAccount, store)
+        : null
+
     // Only suppress web-push when a native FCM channel is actually live
-    // AND a device is registered for the namespace. The fcmConfig gate is
-    // critical: without it, stale device rows from a prior FCM-enabled
-    // boot would silently drop web-push on a hub started without the
-    // service-account env var (notifications would vanish entirely).
-    // See `buildNativeFallbackProbe` for the contract.
-    const nativeFallbackProbe = buildNativeFallbackProbe(store, fcmConfig)
+    // AND a device is registered for the namespace AND the FCM pipeline is
+    // currently healthy. The fcmConfig gate is critical: without it, stale
+    // device rows from a prior FCM-enabled boot would silently drop
+    // web-push on a hub started without the service-account env var
+    // (notifications would vanish entirely). See `buildNativeFallbackProbe`
+    // for the full contract.
+    const nativeFallbackProbe = buildNativeFallbackProbe(
+        store,
+        fcmConfig,
+        fcmService ?? undefined
+    )
 
     const notificationChannels: NotificationChannel[] = [
         new PushNotificationChannel(
@@ -219,8 +234,7 @@ export async function startHub(options: StartHubOptions = {}): Promise<HubInstan
         )
     ]
 
-    if (fcmConfig) {
-        const fcmService = new FcmService(fcmConfig.projectId, fcmConfig.serviceAccount, store)
+    if (fcmConfig && fcmService) {
         notificationChannels.push(new FcmNotificationChannel(fcmService, sseManager, visibilityTracker, store))
         console.log('[Fcm] Native companion push enabled (project:', fcmConfig.projectId + ')')
     }