[superlog] Downgrade worker.job_stalled log from ERROR to WARN

[superlog-app]

Summary

The insights worker's stalled BullMQ event handler logged at ERROR severity, creating false-positive incidents in Superlog every time the insights worker restarted or was redeployed while an AI insight generation job was in flight.

BullMQ's stalled event is part of its built-in resilience mechanism: when a worker loses its heartbeat (e.g., during a Rolling deploy on Railway), active jobs have their locks expire and are automatically moved back to waiting for retry. This is expected behavior — not an operational failure. Terminal failures (after all retry attempts are exhausted) are already captured at ERROR by the existing worker.on('failed', ...) handler.

This patch:

1. Changes worker.job_stalled from ERROR → WARN so normal recovery during deploys no longer pages.
2. Adds job_name inference from the job ID prefix (e.g. insights-website-* → insights-generate-website) so stalled-event logs are easier to diagnose without querying the job object.

An alternative approach would be to suppress the stalled event entirely (just not log it), but keeping it at WARN preserves visibility for cases where stalls happen outside of deploys (e.g. OOM kills or runaway jobs).

Incident on Superlog

---

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

---

Summary by cubic

Downgraded worker.job_stalled logs from error to warn in the insights worker to stop false Superlog incidents during normal bullmq recovery. Added job_name inference from job ID prefixes (dispatch, maintenance, rollup, website) to make stalled logs easier to diagnose; terminal failures remain logged as errors via the existing failed handler.

^{Written for commit 3368aa2. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
databuddy-status	Ready	Preview, Comment	Jun 27, 2026 6:39am

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
dashboard	Skipped		Jun 27, 2026 6:39am
documentation	Skipped		Jun 27, 2026 6:39am

[unkey-deploy]

The latest updates on your projects. Learn more about Unkey Deploy

Name	Status	Preview	Inspect	Updated (UTC)
api (preview)	Ready	Visit Preview	Inspect	Jun 27, 2026 6:39am

[greptile-apps]

Greptile Summary

This PR downgrades the worker.job_stalled log event from ERROR to WARN in the insights worker and adds a inferJobNameFromId helper so stalled-event logs include a human-readable job name without needing to fetch the full job object.

• Log level change (apps/insights/src/worker.ts): emitInsightsEvent(\"error\", ...) → emitInsightsEvent(\"warn\", ...) for the stalled event; this correctly reflects that BullMQ stalls are a normal part of the lock-expiry / retry cycle and are not terminal failures (which remain at ERROR via the failed handler).
• inferJobNameFromId helper: Covers all four currently known job name constants (INSIGHTS_GENERATE_WEBSITE_JOB_NAME, INSIGHTS_ROLLUP_JOB_NAME, INSIGHTS_DISPATCH_JOB_NAME, INSIGHTS_MAINTENANCE_JOB_NAME) and returns \"unknown\" for anything unrecognised — a safe, best-effort fallback.

Confidence Score: 5/5

Safe to merge — the change is isolated to one event handler and reduces alert noise without dropping observability.

The only runtime behaviour that changes is the log level emitted on a stalled job, and the new inferJobNameFromId helper covers every job name constant currently imported by the file. Terminal failures remain at ERROR through the existing failed handler. There is no data-path or control-flow change.

No files require special attention. apps/insights/src/worker.ts is the only changed file and the diff is straightforward.

Important Files Changed

Filename	Overview
apps/insights/src/worker.ts	Downgrades worker.job_stalled from ERROR to WARN and adds inferJobNameFromId helper; all four known job types are covered, fallback returns "unknown"

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant BullMQ
    participant Worker as Insights Worker
    participant Logger as emitInsightsEvent

    BullMQ->>Worker: job active (lock acquired)
    Note over Worker: Worker restarts / deploy rolling
    BullMQ->>Worker: stalled event (jobId)
    Worker->>Worker: inferJobNameFromId(jobId)
    Worker->>Logger: "warn("worker.job_stalled", {job_id, job_name})"
    BullMQ->>BullMQ: move job back to "waiting"
    BullMQ->>Worker: job retried (normal retry path)
    alt all retries exhausted
        BullMQ->>Worker: failed event (job, error)
        Worker->>Logger: "error("worker.job_failed", {...})"
    else job succeeds
        BullMQ->>Worker: completed event (job)
        Worker->>Logger: "info("worker.job_completed", {...})"
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant BullMQ
    participant Worker as Insights Worker
    participant Logger as emitInsightsEvent

    BullMQ->>Worker: job active (lock acquired)
    Note over Worker: Worker restarts / deploy rolling
    BullMQ->>Worker: stalled event (jobId)
    Worker->>Worker: inferJobNameFromId(jobId)
    Worker->>Logger: "warn("worker.job_stalled", {job_id, job_name})"
    BullMQ->>BullMQ: move job back to "waiting"
    BullMQ->>Worker: job retried (normal retry path)
    alt all retries exhausted
        BullMQ->>Worker: failed event (job, error)
        Worker->>Logger: "error("worker.job_failed", {...})"
    else job succeeds
        BullMQ->>Worker: completed event (job)
        Worker->>Logger: "info("worker.job_completed", {...})"
    end

Loading

_{Reviews (1): Last reviewed commit: "[superlog] Downgrade worker.job_stalled ..." | Re-trigger Greptile}

[izadoesdev]

Parking this as superseded by #479 for the stalled uptime job false-positive incident. #479 is the focused one-file staging PR for the uptime worker stalled event; this PR is currently conflicting and broader. Please keep this one unmerged unless #479 is abandoned and this branch is refreshed/revalidated.

[izadoesdev]

This still has unique value for the insights worker stalled-job path, but the branch is dirty and includes stale unrelated files already present on staging. Please rebase on latest staging and reduce the diff to the insights worker change; keeping this open for a clean follow-up review.

[izadoesdev]

Closed as superseded by #515, which landed on staging as 8fdb1b1.\n\nWhy not merge this original Superlog branch: it was not a clean staging branch and did not preserve all of the related worker-error behavior we wanted from the earlier Redis warning change. The clean replacement keeps the useful behavior only: stalled insights jobs now log WARN, include deterministic job_name context, and unknown job IDs fall back to "unknown" without Redis lookups. Verification on #515: targeted insights tests, scoped Ultracite, root lint, root check-types, root test, and GitHub CI/health checks passed.