Record watchdog: corroborate heartbeat staleness against the cache dir before restarting

[teajaysee]

Proposed change

The per-camera record watchdog restarts the record ffmpeg when the maintainer heartbeat (latest_cache_segment_time / valid-segment time) is older than record_stale_threshold. That heartbeat is published by the recording maintainer, not measured locally — so whenever the maintainer lags behind (the Unable to keep up with recording segments in cache overload, #9661-class), every camera goes "stale" at the same moment and every record process is restarted while recording is actually healthy. The synchronized restart cuts segments short at the restart points, giving the maintainer even more files per pass — a self-reinforcing storm.

We traced this live on a 26-camera production deployment (0.17.1): fleet-wide record-ffmpeg restarts on a steady cadence (52/hour at peak) while the cameras' newest cache segments on disk were only seconds old at the very moment each watchdog declared them >120s stale. (On 0.17 the effect is severely amplified when a user raises ffmpeg.retry_interval, which there is also the watchdog lap period; dev's 1s lap already removes that amplifier, but genuine maintainer lag still produces the same false positive on dev.)

This change corroborates against reality before restarting on staleness: stat the camera's newest CACHE_DIR segment, and if a segment fresher than the staleness threshold exists, the recorder is demonstrably writing — log a warning, adopt the disk mtime as the heartbeat, and skip the restart. The invalid_stale path (an actual invalid-segment report) is deliberately untouched, as are restarts when the cache is genuinely dry.

Validation on the production deployment (backport of exactly this check to 0.17.1, running since 2026-06-11): synchronized mass restarts went 52/hour → 0; heartbeat-stale events still occur under load (~2/hour) and are now logged as maintainer lag and skipped, with recordings confirmed continuous on disk throughout; the storm-driven 1-2s segment shredding stopped entirely.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code
• Documentation Update

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue: [Support]: Unable to keep up with recording segments in cache #9661

AI disclosure

• No AI tools were used in this PR.
• AI tools were used in this PR. Details below:

AI tool(s) used (e.g., Claude, Copilot, ChatGPT, Cursor): Claude (Claude Code)

How AI was used (e.g., code generation, code review, debugging, documentation): Debugging/diagnosis of the production storm (live tracing, log analysis), drafting the patch, and writing this description.

Extent of AI involvement (e.g., generated entire implementation, assisted with specific functions, suggested fixes): The diagnosis and the patch were produced with AI assistance end-to-end, directed and reviewed by the maintainer of the affected deployment.

Human oversight: The equivalent change has been running on a 26-camera production 0.17.1 deployment since 2026-06-11 with the results described above (mass restarts 52/hour → 0, recordings verified continuous on disk during heartbeat-stale events). The deployment owner reviewed the change and the claims in this PR.

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I can explain every line of code in this PR if asked.
• UI changes including text have used i18n keys and have been added to the en locale.
• The code has been formatted using Ruff (ruff format frigate)

[NickM-27]

this is incorrect. The stale check not only validates if a camera is writing segments but if they are actually valid. If a camera creates multiple segments that are corrupt, missing video, or other issues then it should still be marked as stale and restarted.

[hawkeye217]

@NickM-27 is right, this is incorrect. invalid_stale requires the most recent invalid report to be older than the stale window, so a camera continuously writing corrupt segments refreshes latest_invalid_segment_time every few seconds and invalid_stale never fires. The condition that actually catches that camera is valid_stale, which this PR suppresses. Corrupt segments still produce fresh mtimes, and a stat only proves bytes are being written, not valid video, so that camera would never be restarted.

There are other issues with the PR:

• The skip only refreshes latest_cache_segment_time. valid_stale stays stale for the entire lag, so the 1s watchdog lap re-globs the cache dir and re-logs the warning every second, per camera.
• The "storm" exists because the stale-restart path resets nothing, so once stale fires, ffmpeg relaunches every 1s lap until a heartbeat refreshes. That cadence also applies to genuine failures (a hung recorder gets hammered with reconnects every second), and this PR leaves it in place.
• "Unable to keep up" fires at pass durations around 70s+; the false positive needs the heartbeat to age past max(120, 2*segment_time+30), i.e. 2+ minute passes. Repeated "unable to keep up" does not imply mass restarts of ffmpeg, and the overload itself, including segment discarding, is untouched by this change.

See the dev docs on the common causes of the "Unable to keep up" messages here.