Skip to content

Commit 436d2a5

Browse files
fix(nuq-postgres): reap orphaned NULL times_out_at backlog rows (firecrawl#3413)
nuq_queue_scrape_backlog_reaper previously only deleted rows where times_out_at < now(), leaving any row with times_out_at = NULL in place forever. Combined with nuq_group_crawl_finished's NOT EXISTS check on queue_scrape_backlog, a single orphaned NULL row on a group blocks its crawl from ever flipping to 'completed'. Add a fallback clause: also delete NULL-timeout rows whose created_at is older than 25h (the crawl Redis state TTL + 1h buffer). This unwedges existing stuck groups and protects against any producer that forgets to set times_out_at in the future. Paired with the queue-jobs.ts fix that stops producing NULL times_out_at for crawl-linked backlog rows. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev>
1 parent b31b574 commit 436d2a5

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

apps/nuq-postgres/nuq.sql

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,8 +120,13 @@ SELECT cron.schedule('nuq_queue_scrape_lock_reaper', '15 seconds', $$
120120
$$);
121121

122122
SELECT cron.schedule('nuq_queue_scrape_backlog_reaper', '* * * * *', $$
123+
-- Also sweep orphaned rows with NULL times_out_at older than the crawl
124+
-- Redis TTL window, so a missing timeout on a backlog row can't wedge
125+
-- nuq_group_crawl_finished forever via its NOT EXISTS check on backlog.
123126
DELETE FROM nuq.queue_scrape_backlog
124-
WHERE nuq.queue_scrape_backlog.times_out_at < now();
127+
WHERE nuq.queue_scrape_backlog.times_out_at < now()
128+
OR (nuq.queue_scrape_backlog.times_out_at IS NULL
129+
AND nuq.queue_scrape_backlog.created_at < now() - interval '25 hours');
125130
$$);
126131

127132
SELECT cron.schedule('nuq_queue_scrape_reindex', '0 9 * * *', $$

0 commit comments

Comments
 (0)