Fix DwmIsCompositionEnabled crash on Mono/wine-mono

[williamzujkowski]

Description

Fixes WPF applications crashing on minimize/restore when running under Wine on Linux.

History

This is a revival of #4155 (2021), which was closed because it depended on two mono PRs:

• Don't assume result is non-NULL if its type is not TYPE_VOID when tracing. mono/mono#20832 — ✅ Merged
• Allow both NULL path and mapName to mono_mmap_open_file. mono/mono#20831 — ⏳ Still open (required for wine-mono, allows NULL path in mono_mmap_open_file)

The original issue #4166 was closed as completed in 2021, but the WPF-side fix was never actually merged. The bug still exists in the current codebase.

Root cause

The P/Invoke for DwmIsCompositionEnabled in NativeMethods.cs uses PreserveSig = false:

[DllImport("dwmapi.dll", EntryPoint = "DwmIsCompositionEnabled", PreserveSig = false)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool _DwmIsCompositionEnabled();

This transforms the native signature from HRESULT DwmIsCompositionEnabled(BOOL *pfEnabled) to bool DwmIsCompositionEnabled(void). On .NET Framework/CoreCLR this works via automatic HRESULT-to-exception conversion, but Mono and wine-mono do not support PreserveSig = false for DllImports. The marshalling is skipped, so dwmapi.dll receives a garbage pointer in rcx (x64) instead of a valid BOOL*, causing an access violation.

Fix

Change to PreserveSig = true with an explicit out parameter, matching the pattern already used by DwmGetColorizationColor on line 2513 of the same file:

[DllImport("dwmapi.dll", EntryPoint = "DwmIsCompositionEnabled", PreserveSig = true)]
private static extern HRESULT _DwmIsCompositionEnabled(
    [Out, MarshalAs(UnmanagedType.Bool)] out bool pfEnabled);

The WindowsBase/Utilities.cs version already uses the correct pattern via PInvoke.DwmIsCompositionEnabled(out BOOL).

Impact

• Windows: No behavior change — HRESULT.S_OK is always returned, enabled reflects actual DWM state
• Wine/Mono: Fixes crash on minimize/restore for all WPF applications
• Affected apps: Any WPF app triggering SystemParameters.IsGlassEnabled or WindowChromeWorker

Note on mono dependency

This WPF-side fix is correct regardless of mono/mono#20831's status — it changes the P/Invoke to use the proper Windows API signature. The mono PR is a separate prerequisite for wine-mono to fully support the PreserveSig attribute, but this change is independently valid since it switches FROM PreserveSig = false (the problematic path) TO PreserveSig = true (the working path).

Fixes #4166
Supersedes #4155