Fix insights rarity-scores 500s and data-completeness timeout logging#375
Merged
Conversation
…meouts
Production insights logs showed two recurring scheduler failures:
1. `release_rarity` failed every run with a 500 from
`GET /api/internal/insights/rarity-scores`. The API logs traced it to a
Neo4j `TransientError: MemoryPoolOutOfMemoryError` — `fetch_all_rarity_signals`
fired eight full-graph Cypher scans concurrently via `asyncio.gather`, summing
their working sets past `dbms.memory.transaction.total.max` (16 GiB). Two of
the scans also used `size([(r)-[]-() | 1])`, which materialises a list element
per relationship for every Release.
Fix: run the eight queries sequentially (caps peak transaction memory at one
query; fine for a daily background job) and replace the list-comprehension
degree with `COUNT { }`. The endpoint now also catches `TransientError` and
returns 503 instead of a bare 500, so the caller logs a meaningful, retryable
failure.
2. `data_completeness` failed intermittently with `error=""` — an empty,
undiagnosable log line. The cause was an `httpx.ReadTimeout` (whose `str()`
is empty) at the 600s client timeout; the underlying full-table scan varies
from ~230s to >600s.
Fix: a `_describe_error` helper that always includes the exception type
(applied to both the data-completeness and rarity handlers), and a larger
timeout (1200s) giving the cold-cache scan headroom. The rarity client
timeout is likewise raised, since its scans now run sequentially.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
E2E Coverage (webkit)
|
Contributor
E2E Coverage (chromium)
|
Contributor
E2E Coverage (firefox)
|
Contributor
E2E Coverage (webkit - iPhone 15)
|
Contributor
E2E Coverage (webkit - iPad Pro 11)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
discogsography-insightsscheduler logs showed two computations failing on every run:1.
release_rarity→ 500 every run. The API logs traced it to:fetch_all_rarity_signalsfired 8 full-graph Cypher scans concurrently viaasyncio.gather, summing their working sets past the Neo4j transaction-memory pool. Two scans also usedsize([(r)-[]-() | 1]), which materialises a list element per relationship for everyRelease.2.
data_completeness→ intermittent failure logged aserror="". Anhttpx.ReadTimeout(whosestr()is empty) at the 600s client timeout; the underlying full-table scan ranges from ~230s to >600s, so it timed out on 3 of 4 observed days — with no diagnosable log line.What changed
I1 — rarity-scores OOM:
asyncio.gather— caps peak transaction memory at one query (it's a daily background job, so wall-clock is fine).size([(r)-[]-() | 1])withCOUNT { }in the two degree queries — counts without materialising a per-relationship list.neo4j.exceptions.TransientError→ returns 503 (retryable) instead of a bare 500.I2 — data-completeness blind timeout:
_describe_errorhelper that always includes the exception type (soReadTimeoutno longer logs aserror=""); applied to both the data-completeness and rarity error handlers.Verification
tests/api+tests/insightssuites passTransientErrorendpoint behaviour;_describe_errornames the type for empty-message exceptions🤖 Generated with Claude Code