[superlog] Add queryTimeoutMs to cacheable() to prevent indefinite DB hangs#489
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
The latest updates on your projects. Learn more about Unkey Deploy
|
|
Found 7 test failures on Blacksmith runners: Failures
|
Greptile SummaryThis PR adds an optional
Confidence Score: 4/5The core fix is correct and safe to merge. The The apps/basket/src/hooks/auth.ts — Important Files Changed
|
|
|
||
| const results = await Promise.allSettled([ | ||
| cached(), | ||
| cached(), | ||
| cached(), | ||
| ]); | ||
|
|
||
| for (const result of results) { | ||
| expect(result.status).toBe("rejected"); | ||
| if (result.status === "rejected") { | ||
| expect(result.reason.message).toBe("Query timeout"); | ||
| } | ||
| } | ||
| }, 5000); | ||
|
|
||
| it("does not time out fast functions", async () => { | ||
| const cached = cacheable(async () => "quick", { | ||
| expireInSec: 60, | ||
| prefix: "qtimeout-fast", | ||
| queryTimeoutMs: 1000, | ||
| }); |
There was a problem hiding this comment.
Missing test for background stale-while-revalidate timeout path
The triggerBackgroundRevalidation function now accepts queryTimeoutMs and uses withTimeout around the fn() call (line 223–225 of cacheable.ts), but the test suite has no case that exercises this path. A test with staleWhileRevalidate: true and a slow function would confirm the background fetch is bounded and that the stale value is still served to the caller (i.e. the timeout in background doesn't surface to callers).
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
izadoesdev
left a comment
There was a problem hiding this comment.
Thanks for tackling the basket hang path. I am parking this one for now because the PR currently has PR-owned lint blockers in packages/redis/cacheable.ts: CacheOptions members need to be sorted, and the new queryTimeoutMs != null ternaries trip noNegationElse at the timeout call sites. The timeout approach itself looks useful, and the Redis cacheable tests cover the important miss/inflight/bypass paths, but this needs the lint fixes plus a fresh green or explicitly waived CI run before we merge to staging. One caveat to keep in mind: Promise.race bounds caller wait time, but it does not cancel the underlying DB/query promise.
izadoesdev
left a comment
There was a problem hiding this comment.
Fixed the lint blockers in packages/redis/cacheable.ts and verified locally. Verification: cd packages/redis && bun test cacheable.test.ts, cd packages/redis && bun run check-types, bun run lint, plus pre-push check-types/test all pass. Approving for staging merge. Note: queryTimeoutMs bounds caller wait time but does not cancel the underlying DB promise; DB-level cancellation can be a follow-up.
Summary
The basket
/batchendpoint returned after ~943 seconds (nearly 16 minutes) for a single request. Thecacheable()wrapper inpackages/redishas a 2-second timeout for Redis operations but no timeout for the underlying function call. When the PostgreSQLwebsitestable lookup stalled, the request — and all concurrent requests for the same website ID deduplicated by theinflightRequestsmap — were frozen for the entire duration of the DB stall.The
cacheable()wrapper was extended with an optionalqueryTimeoutMsoption. When set, the underlyingfn()call is wrapped withwithTimeout(..., queryTimeoutMs, "Query timeout")on all execution paths: cache miss, Redis-unavailable bypass, and background stale-while-revalidate.getWebsiteByIdWithOwnerCachedinapps/basket/src/hooks/auth.tsis set toqueryTimeoutMs: 10_000(10 seconds), matching the p99 expectation for a simple PK lookup.An alternative remediation would be to set
statement_timeouton the PostgreSQL connection pool so the database itself enforces the limit — this would also protect all other Drizzle queries that don't go throughcacheable. Both approaches are complementary and can be combined.Incident on Superlog
Was this PR helpful? Leave feedback — goes straight to the Superlog team.
Summary by cubic
Add a per-call timeout to
cacheable()to prevent long DB stalls from freezing requests. Applied a 10s timeout to Basket’s website lookup so requests fail fast instead of hanging.New Features
queryTimeoutMstopackages/rediscacheable(). Wraps the underlying call withwithTimeout(..., "Query timeout")on miss, Redis-bypass, and background revalidate.inflightRequestson timeout; concurrent callers get the same timeout error.queryTimeoutMs: 10_000togetWebsiteByIdWithOwnerCachedinapps/basket/src/hooks/auth.ts.Refactors
Written for commit 140ce2c. Summary will update on new commits.