Commit 436d2a5
fix(nuq-postgres): reap orphaned NULL times_out_at backlog rows (firecrawl#3413)
nuq_queue_scrape_backlog_reaper previously only deleted rows where
times_out_at < now(), leaving any row with times_out_at = NULL in place
forever. Combined with nuq_group_crawl_finished's NOT EXISTS check on
queue_scrape_backlog, a single orphaned NULL row on a group blocks its
crawl from ever flipping to 'completed'.
Add a fallback clause: also delete NULL-timeout rows whose created_at
is older than 25h (the crawl Redis state TTL + 1h buffer). This unwedges
existing stuck groups and protects against any producer that forgets to
set times_out_at in the future.
Paired with the queue-jobs.ts fix that stops producing NULL times_out_at
for crawl-linked backlog rows.
Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
Co-authored-by: micahstairs <micah@sideguide.dev>1 parent b31b574 commit 436d2a5
1 file changed
Lines changed: 6 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
123 | 126 | | |
124 | | - | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
125 | 130 | | |
126 | 131 | | |
127 | 132 | | |
| |||
0 commit comments