fix(windows): correct shader selection for sRGB-encoded FP16 framebuffer (Win11 SDR)

[eastginza]

Description

Fixes white-out / blown-out highlights when streaming an SDR session on Windows 11 where the desktop framebuffer ends up in FP16 storage with gamma-encoded (sRGB / G22) pixel values.

Symptom

This affects any Windows 11 22H2+ host where the active display causes the desktop to be composed in FP16 storage with gamma-encoded (sRGB / G22) values rather than linear scRGB. The same FP16 + G22 state is produced under multiple OS-level triggers:

• Auto Color Management (ACM) enabled on a regular display — the most commonly documented trigger; Win11 22H2+ feature whose default-enabled footprint has been expanding.
• Virtual display EDID advertising HDR-class luminance and/or BT.2020 wide-gamut colorimetry — common in remote-development / VM setups using Indirect Display Drivers (e.g. VDD-based virtual monitors). Even after stripping the HDR Static Metadata block from the EDID (reducing the advertised Max Luminance to SDR-class), Win11 can still promote the framebuffer to FP16 when the EDID continues to advertise BT.2020 wide-gamut colorimetry via the CTA Colorimetry Data Block. This happens even when ACM is disabled.
• Other Win11 build-specific behaviors that put the framebuffer in the same state.

The trigger is OS-level (Win11 composition behavior + Sunshine's existing FP16 shader assumptions); it is independent of GPU vendor, display type, host hardware, or whether the host runs on bare metal or in a VM. The affected user population is expected to grow as both ACM rollout and Sunshine-based remote-development setups (which often rely on virtual displays) become more common.

The Moonlight client shows the streamed desktop with whites and bright tones clipped against the top of the range. The affected code path in display_vram.cpp is the shader-selection logic shared by all five Windows capture formats (NV12, P010, AYUV, Y410, R16_UINT); the patch covers all five. Empirical verification in this report is the NV12 path (HEVC NVENC SDR session).

Root cause

Under the triggers listed above, Windows 11 composites the desktop framebuffer in FP16 storage (DXGI_FORMAT_R16G16B16A16_FLOAT) even when both the application and the display are SDR. The values stored in that FP16 buffer are still sRGB-encoded (DXGI reports DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709), not linear scRGB.

The existing capture path in display_vram.cpp treats every FP16 source as linear scRGB and routes it through the linear shader chain. That chain ends in convert_linear_base.hlsl:

float3 CONVERT_FUNCTION(float3 input) {
    return ApplySRGBCurve(saturate(input));
}

This applies the sRGB encoding curve to values that are already sRGB-encoded, producing a double-encoded output. Highlights are crushed against the top of the range and the streamed image looks blown out on the client.

Log signature of an affected session:

Colorspace         : DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709
Bits Per Color     : 8
Max Luminance      : 270 nits
Desktop format     : DXGI_FORMAT_R16G16B16A16_FLOAT
[Error]: BT.2020 SDR colorspace expects 10-bit color depth, falling back to Rec. 709
Color coding: SDR (Rec. 709)
Color depth: 8-bit

Note that Max Luminance can be SDR-class (here, 270 nits) and the colorspace is G22_NONE_P709 (gamma-encoded sRGB) — both are SDR signals — yet the Desktop format is still the FP16 buffer. That mismatch is the defining fingerprint of this bug class.

Historical context

This change extends the case matrix opened by #1178 (cgutman, 2023), which added the linear-FP16 capture path for Advanced Color SDR displays — the case where Windows delivers SDR content in FP16 storage as linear scRGB. Windows 11 22H2+ has since introduced a distinct FP16 SDR variant under multiple triggers (ACM, HDR-class virtual display EDID, etc.): the framebuffer still uses DXGI_FORMAT_R16G16B16A16_FLOAT storage, but the pixel values are gamma-encoded (G22), not linear. The existing shader-selection logic does not distinguish between these two FP16 cases, so the gamma-encoded FP16 variant is incorrectly routed through the linear shader chain. This PR adds the discriminator (is_source_gamma_encoded_fp16()) that the new variant requires.

Fix

Detect the gamma-encoded FP16 case via DXGI ColorSpace (G22_NONE_P709 / G22_NONE_P2020) and route those sources through the non-linear shader path (whose convert_base.hlsl uses saturate only, with no additional sRGB curve).

• src/platform/windows/display.h — new helper bool is_source_gamma_encoded_fp16() declared alongside is_hdr().
• src/platform/windows/display_base.cpp — implementation that queries IDXGIOutput6::GetDesc1() and returns true for G22_NONE_P709 / G22_NONE_P2020.
• src/platform/windows/display_vram.cpp — every existing "FP16 → linear shader" decision now also checks is_source_gamma_encoded_fp16() and, when true, binds the non-linear shader to the FP16 slot. Covers all five capture formats (NV12 / P010 / R16_UINT / AYUV / Y410), nine call sites total.

Backward compatibility

The new is_source_gamma_encoded_fp16() helper returns false unless the source delivers FP16 storage with G22_NONE_P709 or G22_NONE_P2020 colorspace. All other configurations — HDR streaming (G10 / G2084), Advanced Color SDR (the case fixed by #1178), pre-22H2 Windows installs, and non-FP16 capture paths — retain the existing shader-selection behavior unchanged. The helper also returns false when IDXGIOutput6 is unavailable on the host, so older Windows/GPU configurations fall back safely to the existing path. There is no behavioural change for HDR streaming.

Alternatives considered

Shader-only fix (earlier iteration): an initial attempt simply removed ApplySRGBCurve from convert_linear_base.hlsl. That resolves the gamma-encoded FP16 case but breaks the linear HDR path (G10 / G2084 sources lose their sRGB encoding step). The current C++ approach keeps convert_linear_base.hlsl unchanged and routes only the gamma-encoded sources to the non-linear shader, leaving HDR streaming intact.

External workarounds were exhaustively evaluated by this contributor at the driver / OS / EDID layer and all failed empirically. They are documented here because reviewers seeing similar wash-out reports may otherwise assume one of them is sufficient:

• Disabling Auto Color Management in Windows settings: widely cited as the standard workaround, but does not resolve the symptom when the active display's EDID continues to advertise HDR-class luminance or BT.2020 wide-gamut colorimetry. In this contributor's environment, ACM was disabled and the FP16 + G22 capture state still persisted. So "just turn off ACM" is incomplete advice for any setup that involves a virtual display, and also costs the user the legitimate benefits of ACM elsewhere.
• Stripping the HDR Static Metadata block from the virtual display's EDID (reducing the advertised Max Luminance from HDR-class ~1670 nits to SDR-class 270 nits): tried via a custom EDID binary loaded by the VDD. The Desktop format: DXGI_FORMAT_R16G16B16A16_FLOAT state still persisted — most likely because the EDID continued to advertise BT.2020 wide-gamut colorimetry via the CTA Colorimetry Data Block, which is sufficient on its own to trigger the FP16 promote in Win11 22H2+.
• Switching the VDD to its auto-generated EDID (<CustomEdid>false</CustomEdid> in vdd_settings.xml): same outcome — the auto-generated EDID still produced the FP16 + G22 capture state.
• Building a custom Indirect Display Driver to hand-author a minimal SDR-only EDID: forking Microsoft's IddSampleDriver, signing it with a WDK test certificate, installing it, and promoting it to the primary display path. Empirically confirmed: even with an SDR-only EDID, no CEA extension, AdapterCaps.Flags = 0, and the base IddCx 1.02 API, DXGI_FORMAT_R16G16B16A16_FLOAT capture persisted. This is strong evidence that no driver- or EDID-layer fix can suppress the FP16 promote in this Win11 build family — the decision happens above the IddCx surface. Additionally, this approach requires Windows test signing mode (bcdedit /set testsigning on) and permanently displays a desktop watermark, making it unsuitable as an end-user workaround even if it had worked.
• Forcing 10-bit HEVC encoding on the Moonlight client side: empirically tested in this contributor's environment and did not resolve the symptom. The 10-bit capture path exercises a related shader chain that is affected by the same FP16 + G22 shader mis-routing, so the symptom carries over.

The exhaustive failure of all driver- / EDID- / encoder-layer workarounds is what narrowed the root cause to Sunshine's shader-selection logic. The C++ patch in this PR fixes that root cause once for all FP16 + G22 SDR sources, regardless of which OS-level trigger (ACM, HDR-class luminance EDID, BT.2020 wide-gamut EDID, or future Win11 behaviors) put the framebuffer in that state.

Reproduction

1. Use Windows 11 22H2 or later, configured so the desktop framebuffer is composed in FP16 with G22 colorspace. Any of:
   a. ACM trigger: enable Auto Color Management (Settings → System → Display → Color profile → Automatically manage colour for apps); or
   b. HDR-class luminance trigger: run an Indirect Display Driver whose EDID advertises HDR-class max luminance (commonly seen with VDD-based virtual monitors used for remote development / VM setups), with ACM disabled; or
   c. BT.2020 wide-gamut trigger: run a display (physical or virtual) whose EDID advertises BT.2020 colorimetry via the CTA Colorimetry Data Block, even with HDR Static Metadata stripped and Max Luminance reduced to SDR-class.
2. Confirm in Sunshine's log (config/sunshine.log) that the Desktop format is DXGI_FORMAT_R16G16B16A16_FLOAT and the Colorspace is DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709 (see "Log signature" above).
3. Connect from Moonlight to a Sunshine host capturing an SDR application or the desktop.
4. Compare the streamed desktop against a local view: whites and bright tones look "blown out" on the client.

Empirically verified on: Windows 11 25H2 with ACM disabled + a VDD-based virtual display whose custom EDID had its HDR Static Metadata block stripped (Max Luminance reduced from ~1670 nits to 270 nits, SDR-class) but still advertised BT.2020 wide-gamut colorimetry via the CTA Colorimetry Data Block, NVIDIA RTX 5090, AORUS FO32U2P 4K@240Hz QD-OLED, NV12 HEVC NVENC 8-bit Rec. 709 SDR session. The same display_vram.cpp shader-selection path is exercised by every FP16 + G22 capture regardless of which trigger produced that state, so the ACM-triggered variant, the HDR-class-luminance-triggered variant, and other GPU vendors (AMD/Intel) are expected to exhibit identical behavior. Several long-standing wash-out reports (e.g. #1853, #4377) describe similar symptoms on Windows but each has its own underlying trigger; this PR does not claim to resolve them, but reporters whose setup matches the FP16 + G22 SDR case documented above may find that this patch addresses their specific instance.

Screenshot

Before (Photoshop reconstruction — the patched binary is already deployed on this contributor's host so a live before-capture is not easily obtainable; the reconstruction approximates the on-wire highlight-clipping visual of the bug, consistent with the symptom described in the wash-out reports cited in Reproduction):
After (patched binary, actual capture from the affected host):

Issues Fixed or Closed

(none — several long-standing wash-out reports exist on the tracker (e.g. #1853, #3500, #4377), but on close inspection each has a distinct root cause that is not directly fixed by this patch. They are mentioned here only for context, not for auto-close.)

Roadmap Issues

(none)

Type of Change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

(The two unchecked boxes: the affected path is the DXGI capture / shader-selection chain inside display_vram.cpp, which currently has no unit-test harness for branch coverage. Manual reproduction steps are documented above.)

AI Usage

• Moderate

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud