`ConfigureFunctionsApplicationInsights` is not idempotent and multiple calls can lead to runtime crashes

[yasmoradi]

Summary

AppServiceEnvironmentVariableMonitor uses a plain Dictionary<string, string?> that is mutated inside ExecuteAsync without any lock or concurrent collection. Under certain conditions (described below), this might cause:

System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access.
   at System.Collections.Generic.Dictionary`2.TryInsert(...)
   at Microsoft.Azure.Functions.Worker.ApplicationInsights.Initializers.AppServiceEnvironmentVariableMonitor+<ExecuteAsync>d__9.MoveNext()

Because BackgroundService defaults to BackgroundServiceExceptionBehavior.StopHost, this unhandled exception would tear down the entire Function App host.

Affected file

src/DotNetWorker.ApplicationInsights/Initializers/AppServiceEnvironmentVariableMonitor.cs

Details

The class is registered as a singleton IHostedService:

services.AddSingleton<AppServiceEnvironmentVariableMonitor>();
services.AddSingleton<IHostedService>(p => p.GetRequiredService<AppServiceEnvironmentVariableMonitor>());

The _monitoredVariableCache field is declared as a non-thread-safe Dictionary:

private readonly Dictionary<string, string?> _monitoredVariableCache = new(StringComparer.OrdinalIgnoreCase);

The dictionary is both read and written inside ExecuteAsync without synchronization:

_monitoredVariableCache.TryGetValue(envVar, out string? cachedVal);
// ...
_monitoredVariableCache[envVar] = currentVal;

It is worth noting that the fields immediately below in the same class — _cancellationTokenSource and _changeToken — are correctly protected against concurrent access using Interlocked.Exchange:

var oldTokenSource = Interlocked.Exchange(ref _cancellationTokenSource, new CancellationTokenSource());
Interlocked.Exchange(ref _changeToken, new CancellationChangeToken(_cancellationTokenSource.Token));

The dictionary, however, has no equivalent protection.

The FunctionEnvironmentReloadRequest or something similar could, directly or indirectly, cause the hosting infrastructure to invoke StartAsync more than once on the same singleton instance, resulting in two concurrent ExecuteAsync loops writing to the same dictionary. We were unable to confirm this from the code alone and would appreciate any insight into whether there is another path that could cause concurrent access.

Impact

• The unhandled exception propagates out of ExecuteAsync, triggering StopHost behavior.
• The host begins tearing down. The logging infrastructure is disposed early in this process, causing subsequent log attempts from application code to throw AggregateException, masking the original root cause.
• In-flight HTTP requests time out or drop, producing server error spikes.

Suggested fix

Replace the plain Dictionary with ConcurrentDictionary<string, string?>:

// Before
private readonly Dictionary<string, string?> _monitoredVariableCache = new(StringComparer.OrdinalIgnoreCase);

// After
private readonly ConcurrentDictionary<string, string?> _monitoredVariableCache = new(StringComparer.OrdinalIgnoreCase);

The write (_monitoredVariableCache[envVar] = currentVal) is valid on ConcurrentDictionary via the indexer, so no other changes to the loop body would be required.

Workaround (until a fix is released)

Callers can prevent the host from being torn down by configuring HostOptions:

builder.Services.Configure<HostOptions>(options =>
{
    options.BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.Ignore;
});

Note: this suppresses the crash but does not fix the race condition. If the exception fires repeatedly, the monitor will silently stop functioning and environment variable changes will no longer propagate to the telemetry initializers.

Package version

Microsoft.Azure.Functions.Worker.ApplicationInsights observed on v2.50.0.

[yasmoradi]

Azure/azure-functions-host#11576

[jviau]

@yasmoradi the ExecuteAsync method is not expected to be thread safe. There shouldn't be any concurrent callers to it. This is a IHostedService, where when started via host.StartAsync, the dotnet generic host will manage calling ExecuteAsync only once. The exception you are seeing means this expectation has been violated and something is calling that concurrently. Considering this is the first report we have seen for a class that has been unchanged for 3 years, I would start with examining your startup to see if you are starting hosted services multiple times in your app startup.

As for the other fields have interlocked access, that is intentional. Those_fields have concurrent access between ExecuteAsync and GetChangeToken. But the dictionary is accessed purely through ExecuteAsync, which again should only ever be single caller.

[yasmoradi]

Thanks! We're calling Microsoft.Azure.Functions.Worker.ApplicationInsights's ConfigureFunctionsApplicationInsights twice and as it's registering IHostedService using lambda services.AddSingleton<IHostedService>(p => p.GetRequiredService<AppServiceEnvironmentVariableMonitor>()); it would end up calling ExecuteAsync twice on the same instance.

We've installed latest version of the Microsoft.Azure.Functions.Worker.ApplicationInsights, but this package installs Microsoft.Azure.Functions.Worker.Core version 2.50.0, not 2.52.0 which has fixed this issue 3273

Here is the chain of events based on the logs and source code:

1. The Crash: If we rely on the default BackgroundServiceExceptionBehavior.StopHost, an unhandled exception in any HostedService kills our worker process.
2. The Exit Code: As called out in the .NET 11 breaking changes release notes, in modern .NET versions (prior to .NET 11), a host termination caused by a background service failure cleanly exits the process with Exit Code 0.
3. The Azure Zombie State: According to Azure Functions Host issue #11576, when a worker process exits with code 0 outside of an intentional shutdown, the Azure Host would restart it, but even though that we're using ~4, which means that fix should be applied already, we STILL have to stop it manually, wait for a few seconds and then start it.

[jviau]

We've installed latest version of the Microsoft.Azure.Functions.Worker.ApplicationInsights, but this package installs Microsoft.Azure.Functions.Worker.Core version 2.50.0, not 2.52.0 which has fixed this issue #3273

Include both package references.

<PackagReference Include="Microsoft.Azure.Functions.Worker" Version="2.52.0" />
<PackagReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="2.50.0" />

We're calling Microsoft.Azure.Functions.Worker.ApplicationInsights's ConfigureFunctionsApplicationInsights twice and as it's registering IHostedService using lambda services.AddSingleton(p => p.GetRequiredService()); it would end up calling ExecuteAsync twice on the same instance.

Are you able to remove this double call?

Looking at the code, I do see how ConfigureFunctionsApplicationInsights is not idempotent. We can track that fix here. I am going to update your PR title to capture that issue.

ROOT ISSUE

ConfigureFunctionsApplicationInsights when called multiple times will register multiple IHostedService copies with the exact same instance of AppServiceEnvironmentVariableMonitor, leading to ExecuteAsync being called twice.

This method needs to be idempotent. Multiple calls should not re-register all of these services.

[yasmoradi]

To me, it's Idempotent

The problem is people mostly install Microsoft.Azure.Functions.Worker.ApplicationInsights, not Microsoft.Azure.Functions.Worker.Core. Because Microsoft.Azure.Functions.Worker.ApplicationInsights automatically brings in Microsoft.Azure.Functions.Worker.Core.

Just like the way people install Microsoft.EntityFrameworkCore.SqlServer and they don't install Microsoft.EntityFrameworkCore itself again in the project.

And to me, it sounds like a very bad idea that you've pushed new version of Microsoft.Azure.Functions.Worker.Core 2.52.0 (Which contains the fix that I've pasted its screenshot), but you've not pushed new version for Microsoft.Azure.Functions.Worker.ApplicationInsights, and as I mentioned people mostly install the Microsoft.Azure.Functions.Worker.ApplicationInsights, they've no reason to install all of its dependencies directly to their project.

So, if you wanna have my 2 cents, please publish a new version for Microsoft.Azure.Functions.Worker.ApplicationInsights ASAP

===

As I mentioned, .NET 11 Prev 3 fixes the 0 Exit Code of the IHostedService crash, and I'm also aware of #11576, but each time this was happening for us, we've had to manually stop, wait for a few seconds and then start the container.

[jviau]

Ah I see there is an idempotence check. Sounds like this issue is resolved then by upgrading your Microsoft.Azure.Functions.Worker package.

The problem is people mostly install Microsoft.Azure.Functions.Worker.ApplicationInsights, not Microsoft.Azure.Functions.Worker.Core. Because
Microsoft.Azure.Functions.Worker.ApplicationInsights automatically brings in Microsoft.Azure.Functions.Worker.Core.

Just like the way people install Microsoft.EntityFrameworkCore.SqlServer and they don't install Microsoft.EntityFrameworkCore itself again in the project.

This is a broad generalization that I don't think actually applies to everyone. Our templates add Microsoft.Azure.Functions.Worker package directly. Ultimately, it is up to the end developer to determine their balance of explicit vs transitive package references. We won't be taking any action here.