Profiling timing out unless using debug logging

[Piedone]

Very curious issue, and I'd appreciate some guidance on troubleshooting this.

If I set up my project locally just by adding the package and services.AddServiceProfiler(); (so, no custom configuration, everything on default), I get the following messages in the Application Insights logs:

StopProfiler failed.
StopProfiler triggered.
StartProfiler succeeded.
StartProfiler triggered.

Then, when I launch the profiler with the "Profile now" button from the Azure Portal, it says "Profiling is in progress" for about 18 minutes, then fails with "Profiling timeout. Refresh and try again".

However, if I run the exact same app but with Debug logging enabled, see below, profiling succeeds within about 4 minutes.

{
  "Logging": {
    "LogLevel": {
      "Microsoft.ApplicationInsights.Profiler": "Debug"
    }
}

I get this exception:

Unexpected exception in the last cycle. System.InvalidOperationException: Uploader can't be located. at System.InvalidOperationException:
   at Microsoft.ApplicationInsights.Profiler.Shared.Services.UploaderProxy.UploaderPathProvider.GetUploaderFullPath (Microsoft.ApplicationInsights.Profiler.Shared, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at Microsoft.ApplicationInsights.Profiler.Shared.Services.UploaderProxy.UploaderPathProvider.TryGetUploaderFullPath (Microsoft.ApplicationInsights.Profiler.Shared, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at Microsoft.ApplicationInsights.Profiler.Shared.Services.PostStopProcessor+<ExecuteUploadAsync>d__14.MoveNext (Microsoft.ApplicationInsights.Profiler.Shared, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.ApplicationInsights.Profiler.Shared.Services.PostStopProcessor+<PostStopProcessAsync>d__13.MoveNext (Microsoft.ApplicationInsights.Profiler.Shared, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.ApplicationInsights.Profiler.Core.ServiceProfilerProvider+<StopServiceProfilerAsync>d__27.MoveNext (Microsoft.ApplicationInsights.Profiler.Core, Version=3.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.ApplicationInsights.Profiler.Shared.Orchestrations.OrchestratorEventPipe+<StopProfilingAsync>d__16.MoveNext (Microsoft.ApplicationInsights.Profiler.Shared, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.ServiceProfiler.Orchestration.SchedulingPolicy+<StartPolicyAsync>d__1.MoveNext (Microsoft.ServiceProfiler.Orchestration, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.ServiceProfiler.Orchestration.SchedulingPolicy+<StartPolicyAsync>d__1.MoveNext (Microsoft.ServiceProfiler.Orchestration, Version=1.0.2510.2201, Culture=neutral, PublicKeyToken=null)

How can this be fixed? Thank you!

[xiaomi7732]

Hi @Piedone , thanks for the feedback.

I think you are going toward the right direction for troubleshooting, these 4 events need to success before profiler's actually gonna work:

StopProfiler failed.
StopProfiler triggered.
StartProfiler succeeded.
StartProfiler triggered.

It appears that the stopping process failed for some reason.

Usually, this indicates that the uploader (an out of proc assembly) was somehow not located. Since you have the logging enabled already, looking for logging entries about locating uploader.

According to the code here, you would want to check for logs around Locating uploader by locator. Priority: {priority}., and find out why the uploader doesn't exist on neither of the locations:

• https://github.com/Azure/azuremonitor-opentelemetry-profiler-net/blob/fe01671e742670879c4e98e2c70bf038d7ad8b1a/src/Microsoft.ApplicationInsights.Profiler.Shared/Services/UploaderProxy/UploaderLocatorBase.cs#L26C1-L27C1

It could usually cause by some project setup issue or file system permission issue. Find out why the target assembly Microsoft.ApplicationInsights.Profiler.Uploader.dll doesn't exist.

[Piedone]

Thanks for your reply! I investigated this, and it turned out that (regardless of tracing settings) the TraceUpload.zip file wasn't copied to the publish folder, and thus not deployed to Azure either. Fixing that solved the uploader issue, which is now loaded:

Calling execute out of proc on dotnet with arguments: /tmp/ServiceProfiler/1.0.2510.2201/Uploader/Microsoft.ApplicationInsights.Profiler.Uploader.dll -t "/tmp/SPTraces/e0ddf4a5-3c08-4c96-983b-f8b6253e3355.netperf" -i c1cb0bd4-8340-4e6a-a6f9-6360b199ad52 --sessionId "2025-12-30T02:34:41.0223993Z" -s "" --host https://profiler.monitor.azure.com/ --metadata "/tmp/SPTraces/e0ddf4a5-3c08-4c96-983b-f8b6253e3355.metadata" --uploadMode "OnSuccess" --sampleActivityFilePath "" --pipeName "a17834dd-7d5c-462f-92ca-a6b129ce1ea8" --roleName "dotnesttenantscore-eus" --trigger "OneTimeSchedulingPolicy" --environment "Production" --traceFileFormat "Netperf". Intended priority: BelowNormal
Uploader to be used: /tmp/ServiceProfiler/1.0.2510.2201/Uploader/Microsoft.ApplicationInsights.Profiler.Uploader.dll
Uploader found: /tmp/ServiceProfiler/1.0.2510.2201/Uploader/Microsoft.ApplicationInsights.Profiler.Uploader.dll
Found zipped uploader at /home/site/wwwroot/TraceUpload.zip. Extracting...

However, I unfortunately still get timeouts. I don't see anything relevant in the logs among the traces, and there are no exceptions. Can you please provide some hints on what to look for?

BTW this is for our Application Insights-integrating Orchard Core module. Thus I haven't added the Microsoft.ApplicationInsights.Profiler.AspNetCore package to the root web project, but to a project referenced by it (the module is an MVC area). This was the solution: Lombiq/Orchard-Azure-Application-Insights@3c3c486.

Could you perhaps be so kind and help me out under #242 too?

[Piedone]

Hmm, today profiling worked, no timeout. Weird. Anyway, thanks for the help here!

[xiaomi7732]

Hey @Piedone I am glad to hear that the profiler works for you. See this for details about uploader not exist, it looks like you might have a similar project structure, and there is something to try:
#241 (comment)

[Piedone]

Ah, that's a clever solution, much better than adding TraceUpload.zip as a content file. I changed the implementation to that: Lombiq/Orchard-Azure-Application-Insights@fd7e52f.