Hi,
Since almost a year, we are running a large application in production and getting a specific exception randomly.
The application contains a web api, hangfire job processor and an Azure Function. Both the api and hangfire applications are highly used which can result in many parallel operations (api requests; hangfire job processes).
We are using .NET10 and MediatR.
Exception we get (more stacktraces are attached here stacktraces.txt ):
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at bool System.Linq.Enumerable+SelectManySingleSelectorIterator<TSource, TResult>.MoveNext()
at List<TSource> System.Linq.Enumerable+WhereEnumerableIterator<TSource>.ToList()
at Action<IEmitter> LightInject.ServiceContainer.CreateEmitMethodForEnumerableServiceServiceRequest(Type serviceType, string serviceName)
I asked Claude what could be the issue, and this is the output:
Root cause: a LightInject concurrency race on first resolution of open generics
This is not a bug in your registration code per se — it's the well‑known LightInject thread‑safety race when open‑generic registrations are first resolved under concurrent load:
- You register everything as open generics: IPipelineBehavior<,>, IRequestPreProcessor<> (in HandlingCompositionRoot and the container.Register(typeof(IRequestPreProcessor<>), typeof(IdempotencyPreProcessor<>)) in Program.cs).
- The first time MediatR requests IEnumerable<IPipelineBehavior<GetGenericPagedResult, PagedResult>>, LightInject has no closed registration for that exact constructed type yet. So CreateEmitMethodForEnumerableServiceServiceRequest walks AvailableServices, and lazily adds the closed‑generic registrations into the shared registration collection while it is enumerating it.
- Under production load, two requests for two different (or the same not‑yet‑seen) request types hit this path at the same time. Thread A mutates the registration dictionary while thread B is doing .Where(...).ToList() over it → “Collection was modified; enumeration operation may not execute.”
- Once a closed type has been compiled successfully, subsequent calls are served from the cached delegate and never touch this path again — which is exactly why:
• it only happens in production (real concurrency), not in dev/single‑request tests,
• it’s intermittent and tends to happen right after a cold start / new deploy / scale‑out, and after the daily config reload window when fresh request types first appear.
How to fix it
There are two complementary fixes. The first is the reliable one.
-
Warm up the container at startup (single‑threaded) — recommended
Force LightInject to compile the closed‑generic enumerable resolutions once, on the startup thread, before any concurrent traffic arrives. After warm‑up the mutation path is never taken again, so the race can't occur.
You already enumerate all known request types in RegisterGenericEntityRequestHandler via MappingConstants.GenericEntityTypeMappings. Do a matching pass that resolves the pipeline once per request/response shape, e.g. after app.Build() in Program.cs: (code sample)
and a helper that, for every IRequest type in your loaded handler assemblies, resolves the enumerables MediatR will ask for: (code sample)
Because this runs on a single thread before requests are served, all closed‑generic registrations get added then, and live request handling only ever hits the cached compiled delegates
-
Upgrade LightInject and align with ASP.NET settings
• Bump LightInject 7.0.1 → 7.1.0 (and the LightInject.Microsoft.DependencyInjection adapter to its matching version) and re‑test under load; several emit/registration concurrency fixes have gone in over the 7.x line.
Upgrading alone is not guaranteed to remove the race for open‑generic enumerables — the warm‑up in step 1 is the deterministic fix.
Can this be fixed in the LightInject code itself?
Yes — the root cause lives in LightInject and is fixable there. The crash is in ServiceContainer.CreateEmitMethodForEnumerableServiceServiceRequest, which does a LINQ .Where(...).ToList() over the shared availableServices/registration collection while, on the unknown-service path (CreateEmitMethodForUnknownService), another thread lazily registers the closed-generic variant into that same collection — classic read-while-write on a non-thread-safe collection.
A proper fix in LightInject would be one of:
• Snapshot before enumerating: materialize the registrations to an array/ToList() under the container's registration lock before running the SelectMany/Where, so enumeration runs over an immutable copy.
• Lock the lazy open→closed generic registration path so it can't mutate the dictionary concurrently with enumeration (the emit/registration phase already has locking primitives that aren't fully covering this path).
• Use a concurrency-safe / copy-on-write collection for availableServices so enumeration never throws even mid-mutation.
Can you please fix this issue? If you want more information about our current fix (generated by AI), let me know.
Hi,
Since almost a year, we are running a large application in production and getting a specific exception randomly.
The application contains a web api, hangfire job processor and an Azure Function. Both the api and hangfire applications are highly used which can result in many parallel operations (api requests; hangfire job processes).
We are using .NET10 and MediatR.
Exception we get (more stacktraces are attached here stacktraces.txt ):
I asked Claude what could be the issue, and this is the output:
Root cause: a LightInject concurrency race on first resolution of open generics
This is not a bug in your registration code per se — it's the well‑known LightInject thread‑safety race when open‑generic registrations are first resolved under concurrent load:
• it only happens in production (real concurrency), not in dev/single‑request tests,
• it’s intermittent and tends to happen right after a cold start / new deploy / scale‑out, and after the daily config reload window when fresh request types first appear.
How to fix it
There are two complementary fixes. The first is the reliable one.
Warm up the container at startup (single‑threaded) — recommended
Force LightInject to compile the closed‑generic enumerable resolutions once, on the startup thread, before any concurrent traffic arrives. After warm‑up the mutation path is never taken again, so the race can't occur.
You already enumerate all known request types in RegisterGenericEntityRequestHandler via MappingConstants.GenericEntityTypeMappings. Do a matching pass that resolves the pipeline once per request/response shape, e.g. after app.Build() in Program.cs: (code sample)
and a helper that, for every IRequest type in your loaded handler assemblies, resolves the enumerables MediatR will ask for: (code sample)
Because this runs on a single thread before requests are served, all closed‑generic registrations get added then, and live request handling only ever hits the cached compiled delegates
Upgrade LightInject and align with ASP.NET settings
• Bump LightInject 7.0.1 → 7.1.0 (and the LightInject.Microsoft.DependencyInjection adapter to its matching version) and re‑test under load; several emit/registration concurrency fixes have gone in over the 7.x line.
Upgrading alone is not guaranteed to remove the race for open‑generic enumerables — the warm‑up in step 1 is the deterministic fix.
Can this be fixed in the LightInject code itself?
Yes — the root cause lives in LightInject and is fixable there. The crash is in ServiceContainer.CreateEmitMethodForEnumerableServiceServiceRequest, which does a LINQ .Where(...).ToList() over the shared availableServices/registration collection while, on the unknown-service path (CreateEmitMethodForUnknownService), another thread lazily registers the closed-generic variant into that same collection — classic read-while-write on a non-thread-safe collection.
A proper fix in LightInject would be one of:
• Snapshot before enumerating: materialize the registrations to an array/ToList() under the container's registration lock before running the SelectMany/Where, so enumeration runs over an immutable copy.
• Lock the lazy open→closed generic registration path so it can't mutate the dictionary concurrently with enumeration (the emit/registration phase already has locking primitives that aren't fully covering this path).
• Use a concurrency-safe / copy-on-write collection for availableServices so enumeration never throws even mid-mutation.
Can you please fix this issue? If you want more information about our current fix (generated by AI), let me know.