fix(azure-durable-functions): link activity and entity spans to the HTTP trace#9394
fix(azure-durable-functions): link activity and entity spans to the HTTP trace#9394chemystery09 wants to merge 9 commits into
Conversation
…rable chunks Extract the W3C traceparent/tracestate that the Durable Functions host supplies on the invocation traceContext and continue that trace when starting activity/entity spans, so they join the same trace as the HTTP root instead of starting new roots. The host re-propagates with the sampled flag cleared, so also force the continued span's priority to USER_KEEP to prevent durable chunks from being dropped independently. Co-authored-by: Cursor <cursoragent@cursor.com>
Overall package sizeSelf size: 7.47 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 2d65ca1 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-07-20 18:34:52 Comparing candidate commit 2d65ca1 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2326 metrics, 32 unstable metrics.
|
…recisely Rework the durable trace-continuation keep so it no longer force-keeps every activity/entity invocation: - Only re-apply a keep when the Durable Functions host cleared the W3C sampled flag (traceparent `-00`) but the datadog tracestate decision (`s`) still indicates keep. Genuine upstream drops (no dd tracestate, or `s` <= 0) are now honored so the customer's sampling config is respected instead of retaining 100% of durable chunks. - Continue with the exact propagated priority (tracestate `s`) rather than always USER_KEEP, so the durable chunk matches the rest of the trace and the propagated decision maker is preserved. - Normalize a failed `extract()` (null) to undefined so startSpan still falls back to an in-process parent instead of re-rooting. Co-authored-by: Cursor <cursoragent@cursor.com>
chemystery09
changed the title
…nd integration tests Add plugin unit tests for trace extraction, sampling re-application, and entity metadata so patch coverage meets the codecov threshold. Extend the integration test to assert activity and entity spans share the HTTP trace id. Co-authored-by: Cursor <cursoragent@cursor.com>
…ce integration check Wire the azure-durable-functions job through plugins/test so the new unit spec is exercised in CI. Remove the integration assertion for shared trace ids because the local Functions host fixture does not propagate traceContext to activities. Co-authored-by: Cursor <cursoragent@cursor.com>
…st step SPEC=index limits test:plugins to index.spec.js so the unit job does not re-run integration tests or install azure-functions-core-tools into the plugin sandbox. Co-authored-by: Cursor <cursoragent@cursor.com>
When the loader already runs init/instrument.mjs, the separate ESM instrumentation test spawns the same child back-to-back and can hang on Node 22.0.0 under DD_INJECTION_ENABLED. Co-authored-by: Cursor <cursoragent@cursor.com>
This change belongs in a separate PR; keep azure-durable-functions work scoped. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
ojproductions
left a comment
ojproductions
left a comment
There was a problem hiding this comment.
this looks great! for now, as @jcstorms1 said, lets keep this as a draft, until we get a proper rfc and approval for all the durable functions work.
I do have 2 notes:
- I tried running your code and realized it doesnt work unless the function's host.json config has the following:
"durableTask": {
"tracing": {
"distributedTracingEnabled": true,
"version": "V2"
}
}This is what allows the traceContext to be propagated to the activity/entity spans. I think itd be good to include this in your PR description as I had no clue about this config
Secondly: this might be a personal peeve, but I think its still worth mentioning. Some of these comments are unnecessarily long. it adds extra context and detail that a human reader ends up having to decipher alongside the code. this ends up just making code harder to read. Please consider, shortening some of it 🙏
Trim the inline comment per review feedback while keeping the key behavior: re-apply propagated tracestate priority when the host clears the W3C sampled flag. Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
Azure Durable Functions runs HTTP triggers, activities, and entities as separate invocations. The host forwards W3C trace context between them, but dd-trace wasn't using it — so activity and entity spans often showed up as separate traces (or got dropped).
This PR:
traceparent/tracestatefrom the host'straceContexton activity and entity invocationstraceparent-00) but Datadogtracestatestill says keep — without overriding genuine upstream drop decisionsSample trace:
https://ddserverless.datadoghq.com/u/06064c4b/uYC-ybf-nHW
Note: Orchestration spans are out of scope; this only covers activity and entity invocations. Customers may also need
DD_TRACE_DISABLED_PLUGINS=http2if the http2 plugin interferes with the host–worker gRPC channel.Required
host.jsonconfigurationTrace context propagation only works when distributed tracing is enabled in the Durable Functions host. Add the following under
extensionsinhost.json:{ "extensions": { "durableTask": { "tracing": { "distributedTracingEnabled": true, "version": "V2" } } } }Without this, the host does not forward
traceContext(traceparent/tracestate) to activity and entity invocations, and spans will not be linked to the HTTP root trace.Test plan