[superlog] Downgrade transient Redis errors in insights worker to WARN

[superlog-app]

Summary

During a Redis server upgrade the insights worker emitted two worker.error ERROR logs (READONLY Writes are temporarily rejected due to server upgrade and ERR caller gone), creating a false-positive incident. The BullMQ worker is configured with maxRetriesPerRequest: null so ioredis reconnects automatically; all jobs continued to succeed after both errors.

The root cause is that worker.on("error") unconditionally logs every Worker-level connection error at ERROR severity, even for well-known transient Redis signals that require no operator action.

This patch adds a TRANSIENT_REDIS_ERROR_PATTERNS list (READONLY, ERR caller gone, ECONNRESET, Connection is closed, Socket closed unexpectedly) and routes matched errors to WARN level. Unexpected Worker errors continue to be logged at ERROR. An alternative approach would be to suppress transient errors entirely (no log at all), but WARN preserves visibility for debugging without triggering incidents.

Incident on Superlog

---

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

---

Summary by cubic

Downgraded transient Redis errors in the insights worker from ERROR to WARN to avoid false-positive incidents during upgrades/failovers. Unexpected errors still log at ERROR; job processing is unchanged.

• Bug Fixes
  • Added detection for transient Redis errors (READONLY, ERR caller gone, ECONNRESET, Connection is closed, Socket closed unexpectedly) and treat them as WARN, with unit tests.
  • Updated worker.on("error") to route WARN vs ERROR in emitInsightsEvent. bullmq uses maxRetriesPerRequest: null, so ioredis reconnects and jobs keep succeeding.

^{Written for commit 5993422. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dashboard	Ready	Preview, Comment	Jun 30, 2026 3:47pm
databuddy-status	Ready	Preview, Comment	Jun 30, 2026 3:47pm
documentation	Ready	Preview, Comment	Jun 30, 2026 3:47pm

[unkey-deploy]

The latest updates on your projects. Learn more about Unkey Deploy

Name	Status	Preview	Inspect	Updated (UTC)
api (preview)	Ready	Visit Preview	Inspect	Jun 16, 2026 9:25am

[greptile-apps]

Greptile Summary

This PR addresses a false-positive incident caused by transient Redis errors during a server upgrade being logged at ERROR severity in the insights worker. It introduces a TRANSIENT_REDIS_ERROR_PATTERNS list and an isTransientRedisError helper that routes matched errors to WARN while keeping unrecognized errors at ERROR.

• Adds five regex patterns covering known BullMQ/ioredis transient signals (READONLY, ERR caller gone, ECONNRESET, Connection is closed, Socket closed unexpectedly) and branches the worker.on("error") handler on whether the error matches, preserving full observability at WARN level.
• The emitInsightsEvent signature already accepts "warn" as a valid LogLevel, so no type changes are needed and the integration is correct.

Confidence Score: 4/5

Safe to merge — the change is narrow, type-correct, and the fallback path (unknown errors stay at ERROR) is sound.

The new isTransientRedisError helper and patterns are unexported with no unit tests, so the exact regex boundary between WARN and ERROR cannot be regression-tested. Everything else — the type compatibility with LogLevel, the correct scoping to the worker.on("error") event (not job failures), and the safe default for non-matching errors — is solid.

apps/insights/src/worker.ts — specifically the untested isTransientRedisError function and pattern list.

Important Files Changed

Filename	Overview
apps/insights/src/worker.ts	Adds TRANSIENT_REDIS_ERROR_PATTERNS and isTransientRedisError to selectively downgrade known transient Redis errors from ERROR to WARN in the worker error handler; logic is correct and type-safe, but the helper is unexported with no unit tests.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[BullMQ Worker emits error event] --> B{isTransientRedisError?}
    B -- "matches READONLY / ERR caller gone\nECONNRESET / Connection is closed\nSocket closed unexpectedly" --> C[level = 'warn']
    B -- "no match" --> D[level = 'error']
    C --> E[emitInsightsEvent 'warn' worker.error]
    D --> F[emitInsightsEvent 'error' worker.error]
    E --> G[log.warn → Axiom + Superlog drain]
    F --> H[log.error → Axiom + Superlog drain\n⚠️ triggers incident alert]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[BullMQ Worker emits error event] --> B{isTransientRedisError?}
    B -- "matches READONLY / ERR caller gone\nECONNRESET / Connection is closed\nSocket closed unexpectedly" --> C[level = 'warn']
    B -- "no match" --> D[level = 'error']
    C --> E[emitInsightsEvent 'warn' worker.error]
    D --> F[emitInsightsEvent 'error' worker.error]
    E --> G[log.warn → Axiom + Superlog drain]
    F --> H[log.error → Axiom + Superlog drain\n⚠️ triggers incident alert]

Loading

_{Reviews (1): Last reviewed commit: "[superlog] Downgrade transient Redis err..." | Re-trigger Greptile}

[greptile-apps]

isTransientRedisError is unexported and untested

The patterns in TRANSIENT_REDIS_ERROR_PATTERNS are the sole gate between WARN and ERROR alerting. There is no worker.test.ts, and the function is unexported, so there is currently no way to verify that the regexes correctly match the intended Redis messages (e.g. READONLY Writes are temporarily rejected…) and do not accidentally match unrelated errors. If a pattern is wrong or a Redis error message format changes, operators would silently receive WARN for a real incident with no indication anything is wrong. Exporting isTransientRedisError (or the pattern list) and adding a few unit tests would make the boundary explicit and regression-proof.

[izadoesdev]

Reviewed with a subagent pass. This looks useful and safe in principle: it only downgrades known transient Redis worker error events to WARN while keeping unexpected worker errors at ERROR, and its original checks are green. I am not merging it because this PR targets main, while repo policy and this merge pass require staging. Please retarget/recreate this against staging and rerun checks; it should be a good candidate after that. Minor non-blocking follow-up: a small test around transient-vs-unexpected Redis worker errors would keep the matcher from drifting.

[izadoesdev]

Approved after refreshing onto staging and adding focused coverage for the transient Redis error classifier.

Local verification:

• cd apps/insights && bun test src/worker-errors.test.ts
• cd apps/insights && bun run check-types
• bun run lint
• bun run test

The change keeps expected Redis failover/connection-close worker errors at WARN while preserving ERROR for unknown worker failures.