The native DwmIsCompositionEnabled dllimport function currently uses PreserveSig = false to automatically "convert" the signature.
The actual native Windows API signature in C is this:
HRESULT DwmIsCompositionEnabled(BOOL *enabled);
That is, it returns a HRESULT, not a BOOL, and the actual BOOL is returned as an output parameter (given via a pointer). However, because of PreserveSig = false, this is now specified as:
BOOL DwmIsCompositionEnabled(void);
But mono doesn't support PreserveSig = false for dllimports, and so wine-mono (which also relies on wpf) doesn't either. When this native function is called, the PreserveSig is ignored, so it calls the second version, which is wrong.
For a 64-bit application (such as the one described below), since no parameter was supplied (it's empty / void), it will have whatever value rcx was had when it was invoked, due to the MS x64 ABI. In my particular case, it had the value rcx=0x00000000FFFFFFFF.
This was used as the "BOOL *enabled" pointer, which is an invalid address. wine's DwmIsCompositionEnabled then tried to write a value of FALSE to this address, which causes a crash. (it's not wine's fault here, it was simply supplied an invalid pointer due to wrong signature, even though the function always returns FALSE, it shouldn't be a problem).
A simple fix would be to do the same as was done for the imported native DwmGetColorizationColor API—use the correct signature with PreserveSig = true, and call the API with its correct signature.
Background: This happened while trying to get the launcher working for the game Divinity: Original Sin II Definitive Edition with wine-mono. Note that there are several issues with it right now, and just this patch isn't enough, but to get to this bug, the following patches (PRs) are also needed on mono first:
mono/mono#20831
mono/mono#20832
There's another fix for wine-mono required before this bug can be reproduced, but a workaround is to simply set the environment variable before launching it: MONO_THREADS_SUSPEND=preemptive
After this, it will arrive at this bug, i.e. crash in native DwmIsCompositionEnabled due to the wrong pointer supplied.
The native DwmIsCompositionEnabled dllimport function currently uses PreserveSig = false to automatically "convert" the signature.
The actual native Windows API signature in C is this:
HRESULT DwmIsCompositionEnabled(BOOL *enabled);
That is, it returns a HRESULT, not a BOOL, and the actual BOOL is returned as an output parameter (given via a pointer). However, because of PreserveSig = false, this is now specified as:
BOOL DwmIsCompositionEnabled(void);
But mono doesn't support PreserveSig = false for dllimports, and so wine-mono (which also relies on wpf) doesn't either. When this native function is called, the PreserveSig is ignored, so it calls the second version, which is wrong.
For a 64-bit application (such as the one described below), since no parameter was supplied (it's empty / void), it will have whatever value rcx was had when it was invoked, due to the MS x64 ABI. In my particular case, it had the value rcx=0x00000000FFFFFFFF.
This was used as the "BOOL *enabled" pointer, which is an invalid address. wine's DwmIsCompositionEnabled then tried to write a value of FALSE to this address, which causes a crash. (it's not wine's fault here, it was simply supplied an invalid pointer due to wrong signature, even though the function always returns FALSE, it shouldn't be a problem).
A simple fix would be to do the same as was done for the imported native DwmGetColorizationColor API—use the correct signature with PreserveSig = true, and call the API with its correct signature.
Background: This happened while trying to get the launcher working for the game Divinity: Original Sin II Definitive Edition with wine-mono. Note that there are several issues with it right now, and just this patch isn't enough, but to get to this bug, the following patches (PRs) are also needed on mono first:
mono/mono#20831
mono/mono#20832
There's another fix for wine-mono required before this bug can be reproduced, but a workaround is to simply set the environment variable before launching it: MONO_THREADS_SUSPEND=preemptive
After this, it will arrive at this bug, i.e. crash in native DwmIsCompositionEnabled due to the wrong pointer supplied.