Expose activity retry metadata via trigger bindings, OpenTelemetry, and status history#3455
Open
bachuv wants to merge 18 commits into
Open
Expose activity retry metadata via trigger bindings, OpenTelemetry, and status history#3455bachuv wants to merge 18 commits into
bachuv wants to merge 18 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a cross-stack “activity retry visibility” contract by parsing retry-related TaskScheduledEvent.Tags and surfacing them consistently via activity trigger metadata, OpenTelemetry span attributes, and (for client status queries) preserved history tags.
Changes:
- Introduces a shared retry metadata/tag contract (
dt.retry.*,durabletask.*) and parsing/aggregation helpers. - Emits per-attempt retry metadata to out-of-proc activity trigger metadata and (partially) to OpenTelemetry spans; emits per-instance aggregates on orchestration completion/failure spans.
- Preserves
TaskScheduledEvent.Tagswhen folding scheduled events into aggregated history items returned byDurableClient.getStatus(showHistory: true).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/WebJobs.Extensions.DurableTask/RetryMetadataConstants.cs | Defines the wire-contract keys for history tags, trigger metadata, and OTel attributes. |
| src/WebJobs.Extensions.DurableTask/ActivityRetryMetadata.cs | Adds parsing logic for retry metadata from TaskScheduledEvent.Tags. |
| src/WebJobs.Extensions.DurableTask/RetryHistoryAggregator.cs | Adds orchestration-terminal aggregation of retry metrics to OTel spans. |
| src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs | Adds out-of-proc activity span stamping + warning, and orchestration-terminal aggregation stamping. |
| src/WebJobs.Extensions.DurableTask/Listener/TaskOrchestrationShim.cs | Adds in-proc orchestration-terminal aggregation stamping. |
| src/WebJobs.Extensions.DurableTask/Listener/TaskActivityShim.cs | Propagates retry metadata into the activity context (but currently doesn’t stamp OTel attempt attributes). |
| src/WebJobs.Extensions.DurableTask/DurableTaskExtension.cs | Plumbs retry metadata into the in-proc activity shim. |
| src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableActivityContext.cs | Adds RetryMetadata storage on the activity context for binding metadata emission. |
| src/WebJobs.Extensions.DurableTask/Bindings/ActivityTriggerAttributeBindingProvider.cs | Exposes retry metadata as trigger binding data keys when available. |
| src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs | Preserves TaskScheduledEvent.Tags through history folding so clients can still read dt.retry.*. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
nytian
reviewed
Jun 15, 2026
nytian
left a comment
nytian
left a comment
Contributor
There was a problem hiding this comment.
Did an initial review. Thanks for the PR! Also can we add some tests?
|
|
||
| // Emit per-instance retry aggregate on the orchestration span for the | ||
| // "failed for some other reason" path as well — same terminal status. | ||
| RetryHistoryAggregator.EmitToActivity(runtimeState.Events, Activity.Current); |
Contributor
There was a problem hiding this comment.
Just confirm: runtimeState.Events include all history events or just session's events?
| // Tags" or mixed-version scenarios. | ||
| if (scheduledEvent.Tags != null | ||
| && scheduledEvent.Tags.ContainsKey(RetryMetadataConstants.HistoryTagAttempt) | ||
| && retryMetadata == null |
Contributor
There was a problem hiding this comment.
what is interlock/exchange for?
Contributor
Author
There was a problem hiding this comment.
The Interlocked.Exchange ensures the warning fires exactly once per process. Without it, the same warning logging thousands of times would crowd out other diagnostics in the log stream.
nytian
approved these changes
Jun 17, 2026
nytian
left a comment
nytian
left a comment
Contributor
There was a problem hiding this comment.
LGTM. the two comments are non-blocking
sophiatev
reviewed
Jun 30, 2026
sophiatev
reviewed
Jun 30, 2026
sophiatev
reviewed
Jun 30, 2026
| } | ||
|
|
||
| anyTaggedFound = true; | ||
| if (meta.Value.Attempt > 1) |
Contributor
There was a problem hiding this comment.
If I'm understanding correctly, it seems that this counter actually counts the cumulative attempts across all retried Activities. Is this what we want? Do we even need this cumulative counter if we have the per-Activity retry counter (I remember seeing this in a previous file)?
Contributor
Author
There was a problem hiding this comment.
It would help answer which orchestration retried the most / hit their retry ceiling, but I'm open to discussing this further.
| } | ||
| } | ||
|
|
||
| activity.SetTag(RetryMetadataConstants.SpanAttrRetryAttemptCount, retryAttemptCount); |
Contributor
There was a problem hiding this comment.
I am actually wondering if it even makes sense to add this information at the orchestration-level at all.. it just seems like we are repeating the information already included in the spans for (Durable) Activities
Comment on lines
+573
to
+577
| // Parse per-attempt retry metadata from the scheduling event's tags (if any) and attach | ||
| // it to the activity context so the binding layer can forward it to the worker via | ||
| // triggerMetadata. | ||
| ActivityRetryMetadata? retryMetadata = ActivityRetryMetadata.TryParseFromTags(scheduledEvent.Tags); | ||
| inputContext.RetryMetadata = retryMetadata; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What changed?
activity invocation when the activity was scheduled with a
RetryOptionspolicy:
durabletask.attempt(int),durabletask.maxAttempts(int), anddurabletask.isMaxAttempt(bool). Workers read these from theirlanguage-specific binding metadata API (e.g.
context.triggerMetadatainthe Node.js worker). Keys are absent entirely when no retry policy is in
use.
span (
durabletask.attempt,durabletask.max_attempts,durabletask.is_max_attempt) for both in-proc and out-of-proc paths.durabletask.retry_attempt_count,durabletask.retry_max_attempts_reached) are emitted once on theorchestration-completion span on the terminal turn. Aggregator
short-circuits when no retry-tagged events exist.
DurableClient.getStatus(showHistory: true)now preservesTaskScheduledEvent.Tagswhen folding TaskScheduled events intoaggregated TaskCompleted / TaskFailed history items, so client-side
consumers walking the returned history can still see
dt.retry.*tags.Only attaches when non-empty to keep the response shape unchanged for
the common no-tags case.
present but malformed — narrows the mixed-version / backend-stripped-tags
detection signal without spamming the no-retry-policy case.
Why is this change needed?
attempt they're on, how many attempts remain, or whether the current
attempt is the final one. User code typically reimplements this with
ad-hoc counters in instance state or by parsing exception messages,
neither of which is robust.
attempt 3 of 5 looks identical to a fresh invocation, making retry
failures hard to triage in distributed traces.
surface retry history because the host's response post-processor folds
TaskScheduled events away and drops their Tags, hiding the metadata
even when the backend persists it correctly.
Issues / work items
Project checklist
pending_docs.mdrelease_notes.mdWebJobs.Extensions.DurableTaskpackage/src/Worker.Extensions.DurableTask/AssemblyInfo.csEventSourcelogsv2.xbranchWebJobs.Extensions.DurableTaskv3.x and will be retained only in thedevandmainbranchesAI-assisted code disclosure (required)
Was an AI tool used? (select one)
If AI was used:
ActivityRetryMetadata.cs,RetryHistoryAggregator.cs,RetryMetadataConstants.csBindings/ActivityTriggerAttributeBindingProvider.cs,ContextImplementations/DurableActivityContext.cs,ContextImplementations/DurableClient.cs,DurableTaskExtension.cs,Listener/TaskActivityShim.cs,Listener/TaskOrchestrationShim.cs,OutOfProcMiddleware.csAI verification (required if AI was used):
Testing
Automated tests