[Bug]: macOS window recording can leak stalled recordly-window-list helpers and exhaust system memory

[borkasm]

Summary

On macOS, Recordly can launch many overlapping recordly-window-list helper processes while stopping a native window recording. If SCShareableContent stalls during ScreenCaptureKit teardown, those helpers can remain alive and consume enough memory/CPU to make the entire machine unresponsive.

This report is based on a macOS force-reset stackshot correlated with the v1.3.3 source. Raw diagnostics are intentionally not attached because they contain private system data.

Environment

• Recordly 1.3.3
• macOS 26.5.1
• Apple Silicon (M1 Pro), 16 GB RAM
• Native ScreenCaptureKit recording path

Observed behavior

• Recording stopped and the output was saved.
• During a single 20.3-second interval around recording finalization, 26 recordly-window-list processes entered the same wait state.
• The captured main-thread frame symbolicated to main + 432 in recordly-window-list, consistent with the helper's DispatchGroup.wait().
• Every helper was waiting on a spinlock. The helpers remained alive after Recordly was no longer recording.
• Each helper showed approximately 2.6 GB of resident mappings. This likely includes shared ScreenCaptureKit/window-surface mappings, so the sum is not necessarily unique physical memory.
• The kernel later reported 100% of compressed pages limit (BAD) and 31 swapfiles. The machine became unresponsive and required a forced power-button reset.
• There was no preceding spontaneous kernel panic or sleep/wake failure report.

Source-level cause

The behavior appears to come from several lifecycle and concurrency issues acting together:

1. electron/ipc/cursor/bounds.ts starts a fire-and-forget bounds refresh every 250 ms:

   void refreshSelectedWindowBounds();
   setWindowBoundsCaptureInterval(setInterval(() => {
     void refreshSelectedWindowBounds();
   }, 250));

   There is no in-flight guard, so a slow or stalled refresh does not prevent another refresh from launching.

2. getNativeMacWindowSources() starts a new recordly-window-list process for every cache miss. The cache is only populated after the helper completes, and there is no shared in-flight promise.

3. electron/native/ScreenCaptureKitWindowList.swift calls:

   try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: true)

   and the main thread waits on a DispatchGroup without an internal timeout. The source itself notes that this helper can stall sporadically when run as a standalone process.

4. In src/hooks/useScreenRecorder.ts, the renderer waits for native recording shutdown before disabling recording state:

   const result = await window.electronAPI.stopNativeScreenRecording();
   await window.electronAPI?.setRecordingState(false);

   Consequently, the 250 ms bounds poll continues during SCStream.stopCapture() and asset-writer finalization, when ScreenCaptureKit is already tearing down the active stream.

5. stopWindowBoundsCapture() only clears future interval callbacks. It cannot cancel helpers that are already running.

6. The Node execFile timeout has no explicit SIGKILL escalation or child-process registry, and the app's before-quit cleanup does not terminate window-list helpers.

7. The native recorder separately performs another full SCShareableContent enumeration every 500 ms to validate that the selected window still exists, adding more ScreenCaptureKit contention.

The critical bounds/helper code is unchanged on current main, so the issue does not appear fixed after v1.3.3.

Likely failure sequence

1. A window recording is active and bounds polling runs every 250 ms.
2. The user stops the recording.
3. Native ScreenCaptureKit teardown begins, but bounds polling remains active.
4. One window-list helper stalls in SCShareableContent.
5. Additional interval ticks launch more helpers because there is no single-flight guard.
6. Recording shutdown eventually clears the interval, but already-running helpers remain.
7. The leaked helpers retain substantial ScreenCaptureKit/window resources and eventually exhaust compressed memory and swap.

Suggested fixes

• Call setRecordingState(false) before awaiting stopNativeScreenRecording().
• Replace the 250 ms setInterval with a serialized/recursive refresh that cannot overlap.
• Add a shared in-flight promise to getNativeMacWindowSources() and set the cache timestamp after successful completion.
• Track all window-list child processes and terminate them on recording stop and app quit.
• Escalate timed-out helpers to SIGKILL after a short grace period.
• Add an internal watchdog to the Swift helper so DispatchGroup.wait() cannot block indefinitely.
• Consider a persistent or window-specific bounds API instead of enumerating all shareable content four times per second.
• Add a regression test with a never-resolving helper and assert that at most one helper exists and that it is cleaned up on stop/quit.

Related symptoms may overlap with #700 and #673, but neither issue documents the orphaned helper/process-exhaustion mechanism.