[BUG]: Fody update causing `Common Language Runtime detected an invalid program`

[mrtristan]

Tracer Version(s)

3.43.0

Operating system and platform

fargate mcr.microsoft.com/dotnet/sdk:10 / mcr.microsoft.com/dotnet/aspnet:10

Instrumentation Mode

automatic

TFM

net10

Bug Report

Very difficult to express steps to reproduce. We have an api receiving a request that fires a MassTransit message out to a receiver that's trying to process some work. Down that call stack we have an explosion that looks very much like something Fody would cause. Confirmed by reverting the pinned tracer version in docker and nuget to the last release before the Fody update (#7764)

System.InvalidProgramException: Common Language Runtime detected an invalid program.
   at OurNamespace.OurMethod(our method params)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   ... steps in our namespace ...
   at OurMassTransitConsumer.Consume(ConsumeContext`1 context) in /app/QueueConsumers/ProcessDeferredOrderCommandConsumer.cs:line 61
   at Datadog.Trace.ClrProfiler.CallTarget.Handlers.Continuations.TaskContinuationGenerator`3.SyncCallbackHandler.ContinuationAction(Task previousTask, TTarget target, CallTargetState state)
   at MassTransit.DependencyInjection.ScopeConsumerFactory`1.Send[TMessage](ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/DependencyInjection/DependencyInjection/ScopeConsumerFactory.cs:line 22
   at MassTransit.DependencyInjection.ScopeConsumerFactory`1.Send[TMessage](ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/DependencyInjection/DependencyInjection/ScopeConsumerFactory.cs:line 22
   at MassTransit.Middleware.ConsumerMessageFilter`2.MassTransit.IFilter<MassTransit.ConsumeContext<TMessage>>.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ConsumerMessageFilter.cs:line 48

Note this would also be followed usually with an unceremonious task crash with not-so-pretty logs. sometimes the stack trace is properly captured like above and other times it looks like this and tags aren't attached and then the process dies.

again, reverting to 3.42.0 works

Reproduction Code

No response

[andrewlock]

Hi @mrtristan, thanks for raising this - the Fody update seems like a bit of a stretch to be honest 🤔 The only weaver we have is one that replaces method bodies with inline IL, and that currently only touches a single ASM method when you're using .NET 6 🤔

Could you please contact support directly so that we can collect the extra details we need to figure this out ASAP? Do mention that we asked you to contact them, so they can raise an expediated escalation with us. If you're able to provide a crash dump or anything else at the same time that would be very helpful, thanks!

[mrtristan]

interesting. assumed fody involvement once rewriting comes in to play but not an area i'm super familiar with.

i want to be able to give you what you need but honestly not sure how to go about that in a fargate environment. once this happens the task is killed. the whole thing is very transient so to speak. any familiarity there?

[andrewlock]

honestly not sure how to go about that in a fargate environment. once this happens the task is killed. the whole thing is very transient so to speak. any familiarity

Not personally 😬 But our our support engineers might well do!

TBh, it's worth opening a support ticket if you're able to anyway, so we can dig into all the additionally things like which products are you using, what config is set, etc!

Thanks!

[Bobrovskiy]

Avoid Problematic Async Extension Patterns
If you control the code, avoid:
public static async Task<T> MyExtAsync<T>(this T obj) { ... }

Why This Happens Datadog’s .NET profiler uses IL rewriting to inject tracing calls into async methods. Async methods are compiled into state machines by the C# compiler. If the profiler modifies the IL in a way that the JIT doesn’t expect (especially in generic async extension methods), the runtime can throw: InvalidProgramException MethodAccessException An invalid asynchronous invocation was detected This is more common in: .NET 6/7/8 with aggressive inlining/JIT optimizations. Azure Functions / Durable Functions where async chaining is heavy. Extension methods returning Task or ValueTask.