[Bug] MSAL occasionally hangs dotnet.exe when doing restore (Azure Artifact Feeds + CredProvider)

[majaeger]

Library version used

4.64

.NET version

Restoring .net 8 project

Scenario

PublicClient - desktop app

Is this a new or an existing app?

None

Issue description and reproduction steps

In our repo, we frequently see dotnet restore hanging when restoring against Azure Artifact Feeds. I can provide a full dump offline (majaeger@microsoft.com)

It looks like there is a hang/deadlock during shutdown when in plugin mode. Here are the offending thread stacks:
ntdll.dll!ZwWaitForSingleObject() Line 268 at minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm(268) KERNELBASE.dll!00007ffe0d82c9df() msalruntime.dll!neosmart::WaitForEvent(neosmart::neosmart_event_t_ * event, unsigned __int64) Line 572 at D:\a\_work\1\s\deps\pevents\src\pevents.cpp(572) msalruntime.dll!Msai::ThreadPool::CancelBackgroundRequests() Line 482 at D:\a\_work\1\s\source\xplat\threading\ThreadPool.cpp(482) msalruntime.dll!Msai::ThreadPool::Stop() Line 207 at D:\a\_work\1\s\source\xplat\threading\ThreadPool.cpp(207) msalruntime.dll!Msai::RequestDispatcherWithPool::Stop() Line 56 at D:\a\_work\1\s\source\xplat\threading\RequestDispatcherWithPool.cpp(56) msalruntime.dll!Msai::AuthenticatorFactoryInternal::Shutdown() Line 551 at D:\a\_work\1\s\source\xplat\api_impl\AuthenticatorFactoryInternalImpl.cpp(551) [Managed to Native Transition] Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.API.x64.Shutdown() Line 158 at /_/msalruntime/interop/net/Module.cs(158) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Module.RemoveRef(string handleName) Line 60 at /_/msalruntime/interop/net/Module.cs(60) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Handle.Dispose(bool disposing) Line 45 at /_/msalruntime/interop/net/Handle.cs(45) System.Private.CoreLib.dll!System.Runtime.InteropServices.SafeHandle.Dispose() Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Core.ReleaseCallback() Line 175 at /_/msalruntime/interop/net/Core.cs(175) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Core.Dispose(bool disposing) Line 163 at /_/msalruntime/interop/net/Core.cs(163) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Core.Dispose() Line 155 at /_/msalruntime/interop/net/Core.cs(155) Microsoft.Identity.Client.Broker.dll!Microsoft.Identity.Client.Platforms.Features.RuntimeBroker.RuntimeBroker.OnProcessExit(object sender, System.EventArgs e) System.Private.CoreLib.dll!System.AppContext.OnProcessExit() [Native to Managed Transition] kernel32.dll!00007ffe0f01e8d7() ntdll.dll!RtlUserThreadStart(long(*)(void *) StartAddress, void * Argument) Line 1184 at minkernel\ldr\rtlstrt.c(1184)

[Waiting on lock owned by Thread 53144, double-click or press enter to switch to thread] ntdll.dll!ZwWaitForMultipleObjects() Line 964 at minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm(964) KERNELBASE.dll!WaitForMultipleObjectsEx(unsigned long nCount, void * const * lpHandles, int bWaitAll, unsigned long dwMilliseconds, int bAlertable) Line 1551 at minkernel\kernelbase\synch.c(1551) [Managed to Native Transition] System.Private.CoreLib.dll!System.Threading.Monitor.Enter(object obj, ref bool lockTaken) Line 52 at /_/src/System.Private.CoreLib/src/System/Threading/Monitor.cs(52) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Module.AddRef(string handleName) Line 33 at /_/msalruntime/interop/net/Module.cs(33) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.Handle.Handle(bool releaseModule) Line 18 at /_/msalruntime/interop/net/Handle.cs(18) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.MSALRUNTIME_DISCOVER_ACCOUNTS_RESULT_HANDLE.MSALRUNTIME_DISCOVER_ACCOUNTS_RESULT_HANDLE(System.IntPtr hndl) Line 67 at /_/msalruntime/interop/net/API.DiscoverAccountsResult.cs(67) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.API.DiscoverAccountsCallbackCompletion.AnonymousMethod__44_0(System.IntPtr h) Line 90 at /_/msalruntime/interop/net/API.DiscoverAccountsResult.cs(90) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.API.CallbackCompletion<Microsoft.Identity.Client.NativeInterop.MSALRUNTIME_DISCOVER_ACCOUNTS_RESULT_HANDLE>(System.IntPtr hResponse, System.IntPtr callbackData, System.Func<System.IntPtr, Microsoft.Identity.Client.NativeInterop.MSALRUNTIME_DISCOVER_ACCOUNTS_RESULT_HANDLE> convert) Line 116 at /_/msalruntime/interop/net/Platforms/Shared/API.cs(116) Microsoft.Identity.Client.NativeInterop.dll!Microsoft.Identity.Client.NativeInterop.API.DiscoverAccountsCallbackCompletion(System.IntPtr hResponse, System.IntPtr callbackData) Line 90 at /_/msalruntime/interop/net/API.DiscoverAccountsResult.cs(90) [Native to Managed Transition] [Inline Frame] msalruntime.dll!std::_Func_class<void,std::shared_ptr<Msai::SignOutResultInternal> const &>::operator()(const std::shared_ptr<Msai::SignOutResultInternal> &) Line 951 at C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.39.33519\include\functional(951) msalruntime.dll!Msai::SignOutEventSinkImpl::OnComplete(const std::shared_ptr<Msai::SignOutResultInternal> & signOutResult) Line 17 at D:\a\_work\1\s\source\api\msal_xplat\impl\SignOutEventSinkImpl.cpp(17) msalruntime.dll!Msai::DiscoverAccountsRequest::FireCallback(const std::shared_ptr<Msai::DiscoverAccountsResultInternal> & result) Line 128 at D:\a\_work\1\s\source\xplat\requests\DiscoverAccountsRequest.cpp(128) msalruntime.dll!Msai::DiscoverAccountsRequest::Execute() Line 100 at D:\a\_work\1\s\source\xplat\requests\DiscoverAccountsRequest.cpp(100) msalruntime.dll!Msai::ThreadPool::ExecuteQueueItemThreadProc(const std::shared_ptr<Msai::BackgroundRequestQueueItem> & queueItem) Line 406 at D:\a\_work\1\s\source\xplat\threading\ThreadPool.cpp(406) msalruntime.dll!Msai::ThreadWorkLoopImpl::WaitForWork() Line 115 at D:\a\_work\1\s\source\xplat\threading\ThreadWorkLoopImpl.cpp(115) msalruntime.dll!anonymous namespace'::ThreadProc(void * lpParameter) Line 20
at D:\a_work\1\s\source\windows\threading\ThreadManagerImpl.cpp(20)
msalruntime.dll!thread_start<unsigned int (__cdecl*)(void ),1>(void * const parameter) Line 97
at minkernel\crts\ucrt\src\appcrt\startup\thread.cpp(97)
kernel32.dll!BaseThreadInitThunk(unsigned long RunProcessInit, long()(void ) StartAddress, void * Argument) Line 77
at clientcore\base\win32\client\thread.c(77)
ntdll.dll!RtlUserThreadStart(long()(void *) StartAddress, void * Argument) Line 1184
at minkernel\ldr\rtlstrt.c(1184)
`

Relevant code snippets

Command line of hung process:

"C:\Program Files\dotnet\dotnet.exe" "C:\Users\majaeger\.nuget\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.dll" -Plugin

Expected behavior

No response

Identity provider

Microsoft Entra ID (Work and School accounts and Personal Microsoft accounts)

Regression

No response

Solution and workarounds

No response