[Azure Functions] Fix span parenting in ASP.NET Core integration#7628
Merged
lucaspimentel merged 42 commits intoApr 23, 2026
Conversation
lucaspimentel
changed the title
This comment has been minimized.
This comment has been minimized.
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (7628) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (72ms) : 70, 75
master - mean (73ms) : 69, 77
section Bailout
This PR (7628) - mean (78ms) : 75, 81
master - mean (80ms) : 76, 84
section CallTarget+Inlining+NGEN
This PR (7628) - mean (1,082ms) : 1024, 1139
master - mean (1,081ms) : 1038, 1124
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (114ms) : 110, 118
master - mean (117ms) : 111, 123
section Bailout
This PR (7628) - mean (119ms) : 114, 124
master - mean (118ms) : 112, 124
section CallTarget+Inlining+NGEN
This PR (7628) - mean (777ms) : 753, 801
master - mean (779ms) : 749, 810
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (101ms) : 96, 106
master - mean (102ms) : 98, 106
section Bailout
This PR (7628) - mean (102ms) : 98, 105
master - mean (106ms) : 99, 112
section CallTarget+Inlining+NGEN
This PR (7628) - mean (937ms) : 904, 970
master - mean (942ms) : 908, 976
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (103ms) : 97, 110
master - mean (101ms) : 97, 104
section Bailout
This PR (7628) - mean (102ms) : 99, 106
master - mean (101ms) : 99, 104
section CallTarget+Inlining+NGEN
This PR (7628) - mean (822ms) : 792, 852
master - mean (823ms) : 788, 858
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (207ms) : 194, 221
master - mean (206ms) : 196, 216
section Bailout
This PR (7628) - mean (211ms) : 199, 222
master - mean (210ms) : 201, 219
section CallTarget+Inlining+NGEN
This PR (7628) - mean (1,220ms) : 1158, 1282
master - mean (1,211ms) : 1144, 1277
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (298ms) : 280, 315
master - mean (294ms) : 281, 308
section Bailout
This PR (7628) - mean (298ms) : 279, 317
master - mean (296ms) : 281, 310
section CallTarget+Inlining+NGEN
This PR (7628) - mean (974ms) : 940, 1008
master - mean (962ms) : 927, 997
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (290ms) : 274, 306
master - mean (285ms) : 277, 293
section Bailout
This PR (7628) - mean (292ms) : 273, 311
master - mean (287ms) : 274, 300
section CallTarget+Inlining+NGEN
This PR (7628) - mean (1,166ms) : 1124, 1207
master - mean (1,155ms) : 1116, 1194
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7628) - mean (287ms) : 272, 303
master - mean (285ms) : 275, 294
section Bailout
This PR (7628) - mean (288ms) : 270, 306
master - mean (285ms) : 276, 293
section CallTarget+Inlining+NGEN
This PR (7628) - mean (1,061ms) : 972, 1150
master - mean (1,048ms) : 974, 1121
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
lucaspimentel
force-pushed
the
lpimentel/APMSVLS-58-azfunc-host-parenting
branch
2 times, most recently
from
12a23e2 to
0c87ab6
Compare
BenchmarksBenchmark execution time: 2026-04-23 19:06:53 Comparing candidate commit 7b4da9c in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 27 metrics, 0 unstable metrics, 59 known flaky benchmarks, 28 flaky benchmarks without significant changes.
|
lucaspimentel
force-pushed
the
lpimentel/APMSVLS-58-azfunc-host-parenting
branch
6 times, most recently
from
9dd669c to
27ba999
Compare
lucaspimentel
changed the title
lucaspimentel
force-pushed
the
lpimentel/APMSVLS-58-azfunc-host-parenting
branch
from
27ba999 to
584134a
Compare
lucaspimentel
force-pushed
the
lpimentel/APMSVLS-58-azfunc-host-parenting
branch
8 times, most recently
from
24a8537 to
aae2cfb
Compare
lucaspimentel
force-pushed
the
lpimentel/APMSVLS-58-azfunc-host-parenting
branch
7 times, most recently
from
974a81e to
613d6bb
Compare
Extracted context from gRPC propagation headers must take priority over InternalActiveScope. Enabling the AspNetCoreDiagnosticObserver in isolated workers caused InternalActiveScope to be a gRPC listener span with an unrelated trace ID, breaking host-to-worker context flow. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
When ASP.NET Core integration is active, check if the retrieved scope is already the active scope before creating a new span. If it's active, reuse it and update the root span tags instead of creating a duplicate azure_functions.invoke span. This prevents extra spans that break integration tests expecting specific span counts. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
Replace direct Microsoft.AspNetCore.Http.HttpContext type reference with IHttpContextItems duck type to avoid FileNotFoundException in non-ASP.NET Core Azure Functions workers where the Microsoft.AspNetCore.Http.Abstractions assembly is not available. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
When the ASP.NET Core scope is already active, we update its tags but no longer assign it to `scope`. This prevents OnAsyncMethodEnd from disposing a scope it doesn't own. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
An exception during duck typing / scope retrieval from HttpContext.Items is unexpected and should be logged at Error level. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
…hrows The worker's FunctionExecutionMiddleware catches user-function exceptions internally, so the ASP.NET Core diagnostic observer never sees an unhandled exception and HttpContext.Response.StatusCode stays at 200. Propagate the exception onto the aspnet_core.request span explicitly. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
Set Error: 1, error tags, and http.status_code 500 on the aspnet_core.request span when the isolated worker function throws. 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes incorrect span parenting for isolated Azure Functions using ASP.NET Core integration by bridging the ASP.NET Core request scope into the Functions worker middleware pipeline, so azure_functions.invoke spans correctly parent under the ASP.NET Core request span instead of the host root span.
Changes:
- Bridge
aspnet_core.requestscope viaHttpContext.Itemsand retrieve it fromFunctionContext.Itemsto establish correct parent/child relationships. - Adjust isolated HTTP trigger context extraction to avoid using stale gRPC propagation headers in ASP.NET Core proxying mode.
- Update integration tests and snapshots to reflect the corrected span hierarchy and updated span counts.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tracer/src/Datadog.Trace/PlatformHelpers/AspNetCoreHttpRequestHandler.cs | Stores the active ASP.NET Core request Scope in HttpContext.Items for Azure Functions. |
| tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Azure/Functions/AzureFunctionsCommon.cs | Retrieves ASP.NET Core scope from FunctionContext.Items and updates parenting/propagation behavior for isolated functions. |
| tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Azure/Functions/Isolated/IFunctionContext.cs | Adds duck-typed access to FunctionContext.Items. |
| tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Azure/Functions/Isolated/IHttpContextItems.cs | Adds duck type for HttpContext.Items without a hard ASP.NET Core reference. |
| tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Azure/Functions/Isolated/FunctionExecutionMiddlewareInvokeIntegration.cs | Propagates isolated user exceptions onto the ASP.NET Core request span. |
| tracer/src/Datadog.Trace/ClrProfiler/Instrumentation.cs | Enables ASP.NET Core diagnostic observer for isolated worker + extension v4 (and skips in other Azure Functions scenarios). |
| tracer/src/Datadog.Trace/Tagging/AzureFunctionsTags.cs | Refactors root-span tagging helper to work on ITags and handle AzureFunctionsTags vs generic tags. |
| tracer/test/Datadog.Trace.Tests/ClrProfiler/AutoInstrumentation/Azure/Functions/AzureFunctionsCommonTests.cs | Updates mock duck type to include Items. |
| tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/AzureFunctionsTests.cs | Updates expectations/assertions for new span counts and structure in ASP.NET Core integration mode. |
| tracer/test/snapshots/AzureFunctionsTests.Isolated.V4.AspNetCore*.verified.txt | Updates snapshots to match corrected span hierarchy and additional spans. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Fixes incorrect span parenting in isolated Azure Functions when using ASP.NET Core integration for HTTP triggers. Worker process spans are now correctly parented to the ASP.NET Core request span, instead of being incorrectly parented to the root host span.
Reason for change
When using isolated Azure Functions with ASP.NET Core integration and HTTP proxying enabled, spans created in the worker process were being parented to the wrong span, causing disconnected or incorrectly structured traces. This made it difficult to understand the complete request flow and latency attribution.
Current (incorrect) behavior:
Fixed (correct) behavior:
Implementation details
The root cause was that
AsyncLocalcontext doesn't flow correctly through Azure Functions middleware, causing the worker'sazure_functions.invokespan to have no local parent. The instrumentation would then fall back to extracting trace context from gRPC message headers, which contained stale context (the host's root span context), resulting in incorrect parenting.The Fix: Use
HttpContext.Itemsas an explicit bridge to pass scope between ASP.NET Core and Azure Functions middleware layers:Store scope in HttpContext.Items (
AspNetCoreHttpRequestHandler.cs:159-171)aspnet_core.requestscope, store it inHttpContext.Items[HttpContextActiveScopeKey]Skip stale gRPC header extraction (
AzureFunctionsCommon.cs:243-259)"HttpRequestContext"key inFunctionContext.ItemsRetrieve scope from
HttpContext.Items(AzureFunctionsCommon.cs:287-367)tracer.InternalActiveScopeis null (AsyncLocal didn't flow), callGetAspNetCoreScope()FunctionContext.Items["HttpRequestContext"](set byFunctionsHttpProxyingMiddleware)HttpContext.Items[HttpContextActiveScopeKey]Added Items property to
IFunctionContext(IFunctionContext.cs:21)IDictionary<object, object>? Items { get; }for duck-typed access to FunctionContext.ItemsThis preserves existing behavior for non-proxying scenarios (timer triggers, non-ASP.NET Core HTTP triggers) while fixing the proxying case.
Test coverage
Covered by existing tests in
AzureFunctionsTests.cs.Fixed test snapshots to reflect correct span counts and hierarchy.
Screenshots
Before
https://dd-dotnet.datadoghq.com/apm/trace/69129102000000009bd07f7872769a84
After
https://dd-dotnet.datadoghq.com/apm/trace/69150c5500000000cb655a7e34732dd6
More recent examples with these changes:
Other details
Fixes APMSVLS-58