perf(macos): add high-FPS ScreenCaptureKit VideoToolbox path

[bdjben]

Description

Adds a macOS ScreenCaptureKit display capture path and a native VTCompressionSession VideoToolbox encoder path for high-FPS streaming. The encode loop is paced from an independent output clock so streams can continue emitting encoded frames at the requested cadence even when macOS capture delivers fresh frames at a lower rate.

This also preserves the client-requested stream dimensions, includes the cursor in ScreenCaptureKit output, and adds capture/encode/send rate logging useful for diagnosing macOS high-FPS behavior.

Local validation was done on an Apple Silicon Mac mini with a Windows Moonlight client and two simultaneous Sunshine-derived app instances: one stream at 120 FPS and one stream at 60 FPS. The successful run showed approximately 120 FPS and 60 FPS output respectively, with native VideoToolbox reporting failed_packets=0 in the tested windows.

Known follow-up areas: the high-FPS pacing strategy is intentionally aggressive and should be reviewed for CPU efficiency, and broader testing is needed across macOS versions, display configurations, codecs, HDR/SDR modes, and client settings.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[sonarqubecloud]

Quality Gate failed

Failed conditions
5 New issues
5 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[ReenigneArcher]

https://docs.lizardbyte.dev/latest/developers/contributing.html

[martona]

To the extent it matters I would love to have an SCKit patch that's accepted because it's the only clean fix to an AVFoundation bug that never re-shows the macOS mouse cursor after it's been hidden for typing assistance.

However, SCKit's scaler is absolutely garbage quality; it's not even close to what AVFoundation or the Lanczos swscaler produce. To maintain quality, you'd have to match the capture resolution to the streaming resolution 1:1, or use an alternate scaler. The first is an unrealistic expectation, and the second basically boils down to Lanczos (good, free with Sunshine, but very cpu-intensive) or a GPU-scaler that you have to write: and it's a non-trivial amount of code.

[andygrundman]

I am working on my own SCK implementation because I haven't been happy with any of the submissions we've received, partly the blame goes to the existing Mac capture code, which is actually fundamentally flawed and a bad example. I found a good reference implementation in OBS and I would like to base ours on that.

It will definitely have full HDR and 4:4:4 support, and hopefully avoid the use of kCVPixelFormatType_32BGRA because I think that is very slow to process. I like the idea of native VT compression code, but it would have to offer some feature that is missing from ffmpeg's VT encoder, or be faster.

I also think it would be a nice-to-have to include support for window capture and app capture modes, via some optional string field you could enter. Capture app "application=Safari" to just stream only browser windows for example.

High FPS has been a struggle, for some reason I get limited around 90fps on M1 Pro, but I think the problem is not SCK but overhead from other stuff.

[ReenigneArcher]

@andygrundman

I also think it would be a nice-to-have to include support for window capture and app capture modes, via some optional string field you could enter. Capture app "application=Safari" to just stream only browser windows for example.

This has been a consistently requested feature across all platforms. Now sure how hard it would be to implement though. Likely definitely need a UI selector to help with this as the naming probably won't always be logical. And then there's the case when multiple instances are running for some apps, or apps that open secondary windows.