Performance: avoid full-table scan in /recordings/unavailable#23586
Performance: avoid full-table scan in /recordings/unavailable#23586JackDanger wants to merge 1 commit into
Conversation
The query filtered on (end_time >= after AND start_time <= before). With before defaulting to now, start_time <= before matches every row, so SQLite range-scans the entire recordings table for any window. Add a start_time lower bound (segments are short, so a recording overlapping the window starts within an hour of it). Also fix a 500 on cameras=all, where cameras was reassigned to a list and later .split().
6169339 to
bd6aed4
Compare
There was a problem hiding this comment.
Pull request overview
This PR targets the /recordings/unavailable API path to improve query performance on large SQLite recordings tables and to fix a crash when cameras=all is requested.
Changes:
- Avoids a full-table scan by adding a
start_timelower bound to the/recordings/unavailablequery. - Fixes
cameras=allhandling so the endpoint no longer errors when parsing cameras.
Comments suppressed due to low confidence (1)
frigate/api/record.py:295
- For
cameras=all, the query currently does not restrict results toallowed_cameras(it only setscamera_list = allowed_camerasbut doesn't apply it). Now thatcameras=allno longer 500s, this effectively considers recordings from cameras the caller may not have access to, which can leak recording availability.
Always apply the camera filter using camera_list (computed from either the requested cameras or allowed_cameras).
if cameras != "all":
camera_list = cameras.split(",")
clauses.append((Recordings.camera << camera_list))
else:
camera_list = allowed_cameras
| @@ -283,7 +283,11 @@ async def no_recordings( | |||
| ) | |||
| clauses = [ | ||
| (Recordings.start_time >= after - 3600) | ||
| & (Recordings.start_time <= before) | ||
| & (Recordings.end_time >= after) | ||
| ] |
|
Thanks. The additional lower bound seems fine, but the |
Proposed change
no_recordingsfilters onend_time >= after AND start_time <= before. Withbeforedefaulting to now,start_time <= beforematches every row, so SQLite scans the wholerecordingstable regardless of the requested window — the cost grows with total retention, not the window size.Adding a
start_timelower bound makes it an index range-scan. Recording segments are short, so a recording overlapping[after, before]starts within ~an hour ofafter;after - 3600is a safe bound.Also fixes a 500 on
cameras=all:camerasis reassigned to a list and then.split()is called on it further down.I have this running 24/7 at my home. Measurements are on an i9-14900K and an RTX 2060. There are about ~450k
recordingsrows. The query time is as follows:Type of change
Additional information
AI disclosure
AI tool(s) used: Claude (Claude Code)
How AI was used: profiling/diagnosis, the code change, and drafting this description.
Extent of AI involvement: identified the full-table scan and the
cameras=allcrash and made the targeted fixes (start_time lower bound + keep the"all"sentinel).Human oversight: ran the patched query against a live Frigate DB (~450k rows); confirmed it returns the same recordings for the window, that
cameras=allnow returns 200 with valid output (was a 500), and measured query times (table above).Checklist
enlocale.ruff format frigate)