Renderer: dispose live frames before re-alloc on source switch

[lrohrmann]

Summary

Re-opening a source on a Player (i.e. switching streams via Open/OpenAsync
without a full Stop()/renderer reset) leaks native GPU memory. The process's
committed memory grows on every switch and is only released when the device is
destroyed (Stop/dispose). On integrated Intel GPUs (where "VRAM" is system RAM) this
ratchets up until the machine runs out of commit charge.

Root cause

A VideoFrame can leave the VideoCache and never be Dispose()d — e.g. the frame
held by the renderer/screamer at the moment of a source switch, seek, or hand-off.
VideoFrame has no finalizer, so the native objects it owns — the D3D11
Texture/ShaderResourceView/VideoProcessorInputView and the pinned AVFrame —
are released only when the COM wrappers are finalized by the GC.

Because the managed heap stays tiny during playback, Gen2 collections almost never
run, so those finalizers almost never run. Each switch therefore leaves the previous
source's frames (and their surfaces) committed while the new decoder allocates its
own → the concurrent peak is old + new, and committed memory never recovers.

A heap dump of a leaking process shows thousands of VideoFrame and
ID3D11VideoProcessorInputView instances that are unrooted (no GC root) but whose
native resources were never freed, while only a handful of ID3D11Texture2D remain
(HW frames share one texture array) — i.e. the managed wrappers are dead garbage and
the native memory leaked.

Reproduction (current master)

1. Take any sample that drives a Player (e.g. the FlyleafPlayer sample).
2. Repeatedly switch the source on the same player without calling Stop()
   between switches — e.g. alternate player.OpenAsync(A) / player.OpenAsync(B)
   on a ~1s timer, or loop a playlist of a few items.
3. Watch the process's Commit size (Task Manager → add the "Commit size" column,
   or VMMap "Private Committed"). Do not rely on the Working Set — it is trimmed
   by the OS and hides the leak.
4. Committed memory rises by roughly one source's worth of surfaces per switch and
   never drops while the player keeps running.
   • Most visible with large software-decoded frames (e.g. big MJPEG/JPEG stills,
     several megapixels) and on Intel integrated GPUs; hardware-decoded streams leak
     more slowly but still ratchet up.
   • Calling Stop() (full renderer reset) before each Open releases the memory —
     confirming the leak is specific to the in-place re-open path.

Fix

Track every frame the renderer hands out through the single FillPlanes choke point,
and dispose any survivors deterministically:

• in Player.OpenInternal, before the new decoder allocates (so the freed
  surfaces are reused instead of accumulating), and
• on renderer teardown (DisposeLocal).

A Flush() after disposal lets D3D11 actually destroy the released resources (there
is no Present on a paused/just-switched player to trigger deferred destruction).

Changes

• Renderer.Device.cs: per-renderer live-frame registry (TrackFrame/UntrackFrame/
  DisposeLiveFrames) + FlushContext(); DisposeLiveFrames() is also called from
  DisposeLocal.
• Renderer.VP.cs: FillPlanes becomes a small wrapper around the (now private)
  fillPlanes delegate that records each returned frame.
• VideoFrame: gains an Owner back-reference and untracks itself on Dispose().
• Player.Open.cs: dispose cached + escaped frames and flush before decoder.Open(...).

No public API change (the affected members are internal). The per-frame tracking is
a HashSet add/remove guarded by a lock — negligible overhead on the frame path.

Testing

Verified on an Intel integrated GPU (committed-memory growth is most pronounced there)
with a setup of several concurrent Players, repeatedly switching one player's source
between an H.264 RTSP stream (hardware-decoded) and large multi-megapixel MJPEG stills
(software-decoded).

• Committed memory (Task Manager "Commit size" / VMMap "Private Committed", not
  Working Set): before the fix it rose by roughly one source's worth of surfaces on
  every switch and never recovered (~tens of MB per switch with the large SW frames,
  unbounded over ~30 switches). After the fix it stays bounded — it plateaus at roughly
  a single source's peak instead of ratcheting up.
• Heap dump cross-check (WinDbg + SOS): with the fix, !dumpheap -stat -type VideoFrame and -type ID3D11VideoProcessorInputView no longer show thousands of
  accumulating instances after repeated switches (they stay at a small steady count),
  and !gcroot on a sampled survivor confirms old frames are actually disposed rather
  than left as unrooted objects whose native resources were never released.
• Both decode paths: confirmed for the HW (D3D11 VideoProcessor) and SW frame
  paths; the regression was reproducible and is resolved on both.
• No rendering/playback regression: source switching, seeking and normal playback
  behave as before; no black frames or flicker introduced by the deterministic disposal
  • flush.

[SuRGeoNix]

@lrohrmann Thanks for the PR. I will need to look deeper into this, however are you sure that is general or only for intel GPUs that they have also a bug with that? Additionally, the solution it should be to identity where the frame leaks and not to try to track everything and then dispose it independently.

[lrohrmann]

I'm not sure if it is related to Intel GPU, but I was able to reproduce it on different Intel platforms pretty easy by just playing a high resolution stream and switch between different streams. The committed memory is raising unlimited and never returned to OS. Maybe there are better ways to fix this, but that was what has helped at the end. The fastest way to see the issue is configuring some http URLs with pretty big static jpeg images and switch between those sources. I have used 30mb jpeg files and with every switch the ram was growing 100-150mb and never returned. But you need to check committed ram, the active ram is freed but committed not.