Skip to content

Performance: avoid full-table scan in /recordings/unavailable#23586

Open
JackDanger wants to merge 1 commit into
blakeblackshear:devfrom
JackDanger:perf/recordings-unavailable-scan
Open

Performance: avoid full-table scan in /recordings/unavailable#23586
JackDanger wants to merge 1 commit into
blakeblackshear:devfrom
JackDanger:perf/recordings-unavailable-scan

Conversation

@JackDanger

@JackDanger JackDanger commented Jun 28, 2026

Copy link
Copy Markdown

Proposed change

no_recordings filters on end_time >= after AND start_time <= before. With before defaulting to now, start_time <= before matches every row, so SQLite scans the whole recordings table regardless of the requested window — the cost grows with total retention, not the window size.

Adding a start_time lower bound makes it an index range-scan. Recording segments are short, so a recording overlapping [after, before] starts within ~an hour of after; after - 3600 is a safe bound.

Also fixes a 500 on cameras=all: cameras is reassigned to a list and then .split() is called on it further down.

I have this running 24/7 at my home. Measurements are on an i9-14900K and an RTX 2060. There are about ~450k recordings rows. The query time is as follows:

window before after
24h 330ms 58ms
7d 520ms 386ms
cameras=all HTTP 500 works

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code
  • Documentation Update

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to discussion with maintainers (required for any large or "planned" features):

AI disclosure

  • No AI tools were used in this PR.
  • AI tools were used in this PR. Details below:

AI tool(s) used: Claude (Claude Code)

How AI was used: profiling/diagnosis, the code change, and drafting this description.

Extent of AI involvement: identified the full-table scan and the cameras=all crash and made the targeted fixes (start_time lower bound + keep the "all" sentinel).

Human oversight: ran the patched query against a live Frigate DB (~450k rows); confirmed it returns the same recordings for the window, that cameras=all now returns 200 with valid output (was a 500), and measured query times (table above).

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I can explain every line of code in this PR if asked.
  • UI changes including text have used i18n keys and have been added to the en locale.
  • The code has been formatted using Ruff (ruff format frigate)

@JackDanger JackDanger marked this pull request as ready for review June 28, 2026 19:42
Copilot AI review requested due to automatic review settings June 28, 2026 19:43
The query filtered on (end_time >= after AND start_time <= before). With
before defaulting to now, start_time <= before matches every row, so SQLite
range-scans the entire recordings table for any window. Add a start_time lower
bound (segments are short, so a recording overlapping the window starts within
an hour of it). Also fix a 500 on cameras=all, where cameras was reassigned to a
list and later .split().
@JackDanger JackDanger force-pushed the perf/recordings-unavailable-scan branch from 6169339 to bd6aed4 Compare June 28, 2026 19:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR targets the /recordings/unavailable API path to improve query performance on large SQLite recordings tables and to fix a crash when cameras=all is requested.

Changes:

  • Avoids a full-table scan by adding a start_time lower bound to the /recordings/unavailable query.
  • Fixes cameras=all handling so the endpoint no longer errors when parsing cameras.
Comments suppressed due to low confidence (1)

frigate/api/record.py:295

  • For cameras=all, the query currently does not restrict results to allowed_cameras (it only sets camera_list = allowed_cameras but doesn't apply it). Now that cameras=all no longer 500s, this effectively considers recordings from cameras the caller may not have access to, which can leak recording availability.

Always apply the camera filter using camera_list (computed from either the requested cameras or allowed_cameras).

    if cameras != "all":
        camera_list = cameras.split(",")
        clauses.append((Recordings.camera << camera_list))
    else:
        camera_list = allowed_cameras

Comment thread frigate/api/record.py
Comment on lines 279 to 283
@@ -283,7 +283,11 @@ async def no_recordings(
)
Comment thread frigate/api/record.py
Comment on lines +286 to +290
clauses = [
(Recordings.start_time >= after - 3600)
& (Recordings.start_time <= before)
& (Recordings.end_time >= after)
]
@hawkeye217

Copy link
Copy Markdown
Collaborator

Thanks. The additional lower bound seems fine, but the cameras=all fix is not correct. Setting cameras = "all" makes the query skip the camera filter entirely, so it runs across every camera regardless of the caller's allowed_cameras. The UI always passes a single camera, so there is no way a 500 would be produced unless you are making an authenticated API call directly. This has already been improved in this commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants