Commit 14c47d8
[Azure Functions] Fix span parenting in ASP.NET Core integration (#7628)
## Summary of changes
Fixes incorrect span parenting in isolated Azure Functions when using
ASP.NET Core integration for HTTP triggers. Worker process spans are now
correctly parented to the ASP.NET Core request span, instead of being
incorrectly parented to the root host span.
## Reason for change
When using isolated Azure Functions with ASP.NET Core integration and
HTTP proxying enabled, spans created in the worker process were being
parented to the wrong span, causing disconnected or incorrectly
structured traces. This made it difficult to understand the complete
request flow and latency attribution.
**Current (incorrect) behavior:**
```
ROOT: azure_functions.invoke: GET /api/httptest [HOST]
├─ http.request: GET localhost:40521/api/HttpTest [HOST → WORKER]
└─ azure_functions.invoke: Http HttpTest [WORKER] ❌ wrong parent
└─ (worker child spans)
```
**Fixed (correct) behavior:**
```
ROOT: azure_functions.invoke: GET /api/httptest [HOST]
└─ http.request: GET localhost:40521/api/HttpTest [HOST → WORKER]
└─ aspnet_core.request [WORKER]
└─ azure_functions.invoke: Http HttpTest [WORKER] ✅ correct parent
└─ (worker child spans)
```
## Implementation details
The root cause was that `AsyncLocal` context doesn't flow correctly
through Azure Functions middleware, causing the worker's
`azure_functions.invoke` span to have no local parent. The
instrumentation would then fall back to extracting trace context from
gRPC message headers, which contained **stale context** (the host's root
span context), resulting in incorrect parenting.
**The Fix**: Use `HttpContext.Items` as an explicit bridge to pass scope
between ASP.NET Core and Azure Functions middleware layers:
1. **Store scope in HttpContext.Items**
(`AspNetCoreHttpRequestHandler.cs:159-171`)
- After creating the `aspnet_core.request` scope, store it in
`HttpContext.Items[HttpContextActiveScopeKey]`
- Only done when running in Azure Functions isolated worker
2. **Skip stale gRPC header extraction**
(`AzureFunctionsCommon.cs:243-259`)
- Detect ASP.NET Core integration by checking for `"HttpRequestContext"`
key in `FunctionContext.Items`
- Skip extracting trace context from gRPC message headers (which contain
stale host root span context)
- Only extract headers in non-ASP.NET Core mode (timer triggers,
non-proxying HTTP triggers)
3. **Retrieve scope from `HttpContext.Items`**
(`AzureFunctionsCommon.cs:287-367`)
- When `tracer.InternalActiveScope` is null (AsyncLocal didn't flow),
call `GetAspNetCoreScope()`
- Get HttpContext from `FunctionContext.Items["HttpRequestContext"]`
(set by `FunctionsHttpProxyingMiddleware`)
- Get scope from `HttpContext.Items[HttpContextActiveScopeKey]`
- Use retrieved scope as parent if found, otherwise fall back to
extracted context or create root
4. **Added Items property to `IFunctionContext`**
(`IFunctionContext.cs:21`)
- Added `IDictionary<object, object>? Items { get; }` for duck-typed
access to FunctionContext.Items
This preserves existing behavior for non-proxying scenarios (timer
triggers, non-ASP.NET Core HTTP triggers) while fixing the proxying
case.
## Test coverage
Covered by existing tests in `AzureFunctionsTests.cs`.
Fixed test snapshots to reflect correct span counts and hierarchy.
## Screenshots
### Before
<img width="3267" height="293" alt="image"
src="https://github.com/user-attachments/assets/f0942ae5-1c27-4f7c-a1e7-d6b3559ba486"
/>
https://dd-dotnet.datadoghq.com/apm/trace/69129102000000009bd07f7872769a84
### After
<img width="3264" height="246" alt="image"
src="https://github.com/user-attachments/assets/4074aa46-5577-44fc-b0da-bc0ed9b44c05"
/>
https://dd-dotnet.datadoghq.com/apm/trace/69150c5500000000cb655a7e34732dd6
More recent examples with these changes:
-
https://dd-dotnet.datadoghq.com/apm/trace/697be6d40000000000710c25e093bbc4
-
https://dd-dotnet.datadoghq.com/apm/trace/6994c3a200000000008c4ab320c9c4b7
## Other details
Fixes APMSVLS-58
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>1 parent 8840657 commit 14c47d8
11 files changed
Lines changed: 962 additions & 447 deletions
File tree
- tracer
- src/Datadog.Trace
- ClrProfiler
- AutoInstrumentation/Azure/Functions
- Isolated
- PlatformHelpers
- Tagging
- test
- Datadog.Trace.ClrProfiler.IntegrationTests
- Datadog.Trace.Tests/ClrProfiler/AutoInstrumentation/Azure/Functions
- snapshots
Lines changed: 199 additions & 42 deletions
Large diffs are not rendered by default.
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
48 | 55 | | |
49 | 56 | | |
50 | 57 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
Lines changed: 23 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| |||
525 | 525 | | |
526 | 526 | | |
527 | 527 | | |
528 | | - | |
529 | | - | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
530 | 573 | | |
531 | | - | |
| 574 | + | |
532 | 575 | | |
533 | 576 | | |
534 | 577 | | |
| |||
Lines changed: 18 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
9 | 8 | | |
10 | 9 | | |
11 | | - | |
12 | 10 | | |
13 | 11 | | |
14 | 12 | | |
| |||
20 | 18 | | |
21 | 19 | | |
22 | 20 | | |
23 | | - | |
24 | 21 | | |
25 | | - | |
26 | 22 | | |
27 | 23 | | |
| 24 | + | |
28 | 25 | | |
29 | 26 | | |
30 | 27 | | |
| 28 | + | |
31 | 29 | | |
32 | 30 | | |
33 | 31 | | |
34 | 32 | | |
35 | 33 | | |
36 | 34 | | |
37 | 35 | | |
| 36 | + | |
38 | 37 | | |
39 | 38 | | |
40 | 39 | | |
| |||
174 | 173 | | |
175 | 174 | | |
176 | 175 | | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
177 | 191 | | |
178 | 192 | | |
179 | 193 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
44 | | - | |
| 43 | + | |
45 | 44 | | |
46 | 45 | | |
47 | 46 | | |
| |||
51 | 50 | | |
52 | 51 | | |
53 | 52 | | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
58 | 67 | | |
59 | 68 | | |
60 | 69 | | |
Lines changed: 28 additions & 26 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
| 148 | + | |
148 | 149 | | |
149 | 150 | | |
150 | 151 | | |
151 | 152 | | |
152 | 153 | | |
153 | 154 | | |
154 | 155 | | |
155 | | - | |
156 | | - | |
| 156 | + | |
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
| |||
179 | 179 | | |
180 | 180 | | |
181 | 181 | | |
| 182 | + | |
182 | 183 | | |
183 | 184 | | |
184 | 185 | | |
185 | 186 | | |
186 | 187 | | |
187 | 188 | | |
188 | 189 | | |
189 | | - | |
| 190 | + | |
190 | 191 | | |
191 | 192 | | |
192 | 193 | | |
| |||
213 | 214 | | |
214 | 215 | | |
215 | 216 | | |
| 217 | + | |
216 | 218 | | |
217 | 219 | | |
218 | 220 | | |
219 | 221 | | |
220 | 222 | | |
221 | | - | |
222 | 223 | | |
| 224 | + | |
| 225 | + | |
223 | 226 | | |
224 | 227 | | |
225 | 228 | | |
| |||
245 | 248 | | |
246 | 249 | | |
247 | 250 | | |
248 | | - | |
| 251 | + | |
249 | 252 | | |
250 | 253 | | |
251 | 254 | | |
252 | 255 | | |
253 | 256 | | |
254 | | - | |
| 257 | + | |
255 | 258 | | |
256 | | - | |
257 | | - | |
258 | 259 | | |
259 | 260 | | |
260 | 261 | | |
| |||
283 | 284 | | |
284 | 285 | | |
285 | 286 | | |
286 | | - | |
| 287 | + | |
| 288 | + | |
287 | 289 | | |
288 | 290 | | |
| 291 | + | |
289 | 292 | | |
| 293 | + | |
290 | 294 | | |
291 | 295 | | |
292 | 296 | | |
293 | 297 | | |
294 | 298 | | |
295 | 299 | | |
296 | 300 | | |
| 301 | + | |
297 | 302 | | |
298 | 303 | | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | 304 | | |
304 | 305 | | |
305 | 306 | | |
306 | | - | |
| 307 | + | |
307 | 308 | | |
308 | 309 | | |
309 | 310 | | |
| |||
331 | 332 | | |
332 | 333 | | |
333 | 334 | | |
334 | | - | |
| 335 | + | |
| 336 | + | |
335 | 337 | | |
336 | 338 | | |
337 | 339 | | |
338 | 340 | | |
| 341 | + | |
339 | 342 | | |
340 | 343 | | |
341 | 344 | | |
342 | 345 | | |
343 | | - | |
344 | 346 | | |
345 | 347 | | |
| 348 | + | |
| 349 | + | |
346 | 350 | | |
347 | | - | |
348 | 351 | | |
349 | | - | |
350 | 352 | | |
351 | 353 | | |
352 | 354 | | |
| 355 | + | |
353 | 356 | | |
354 | 357 | | |
355 | 358 | | |
| |||
374 | 377 | | |
375 | 378 | | |
376 | 379 | | |
| 380 | + | |
377 | 381 | | |
378 | 382 | | |
379 | | - | |
| 383 | + | |
380 | 384 | | |
381 | 385 | | |
382 | 386 | | |
| |||
385 | 389 | | |
386 | 390 | | |
387 | 391 | | |
388 | | - | |
389 | | - | |
390 | | - | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
391 | 395 | | |
392 | 396 | | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
| 397 | + | |
| 398 | + | |
397 | 399 | | |
398 | 400 | | |
399 | 401 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
| 125 | + | |
| 126 | + | |
125 | 127 | | |
126 | 128 | | |
127 | 129 | | |
| |||
0 commit comments