Skip to content

Commit fba02a4

Browse files
Robert Karpclaude
andcommitted
fix: suppress auto-instrumented AmqpReceiver TaskCanceledException APM errors during shutdown (5.7.5)
Root cause: Elastic APM creates "AzureServiceBus RECEIVE" transactions via DiagnosticSource auto-instrumentation. The Azure SDK ends its Activity before firing ProcessErrorAsync, so Agent.Tracer.CurrentTransaction is null in OnReceiveCancelled() — the TX ID is never tracked and the filter passes the error through. Fix: add culprit-based suppression for TaskCanceledException on the AmqpReceiver path (post-WebSockets fix, only fires on shutdown). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 8cc09ec commit fba02a4

2 files changed

Lines changed: 20 additions & 3 deletions

File tree

docs/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77
## 5.7.5
88
- Fixed
99
- `ApmTransactionManager` now registers the APM error filter **at construction time** (application startup) instead of lazily inside `OnReceiveCancelled()`. The lazy approach lost a race: during pod graceful shutdown the APM agent flushes its internal buffer concurrently with Service Bus processor teardown, so error events could be sent to APM before `ReceiverWrapper.OnExceptionOccured` ran and had a chance to register the filter. Registering at construction time — before any message processing starts — closes this window. A fallback call in `OnReceiveCancelled()` handles the edge case where the APM agent was not yet configured at construction.
10+
- The filter now also suppresses `TaskCanceledException` / `OperationCanceledException` errors whose culprit originates in `AmqpReceiver.ReceiveMessagesAsyncInternal`. These error events are produced by Elastic APM's **auto-instrumented** Azure Service Bus transactions (`"AzureServiceBus RECEIVE from …"`): the Azure SDK ends its underlying `Activity` (and therefore the APM transaction) before calling `ProcessErrorAsync`, so `Agent.Tracer.CurrentTransaction` is already `null` when `OnReceiveCancelled()` runs — the transaction ID is never added to `_cancelledTransactionIds` and the transaction-ID-based filter path cannot suppress them. After switching to WebSockets transport, `TaskCanceledException` from this code path only occurs during pod graceful shutdown.
1011

1112
## 5.7.4
1213
- Fixed

src/Ev.ServiceBus.Apm/ApmTransactionManager.cs

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -120,9 +120,25 @@ private static void RegisterShutdownErrorFilter()
120120

121121
// Returning null from the filter drops the error event before it reaches the APM server.
122122
Agent.AddFilter((IError error) =>
123-
error.TransactionId is not null && _cancelledTransactionIds.ContainsKey(error.TransactionId)
124-
? null
125-
: error);
123+
{
124+
// Case 1: transaction explicitly tracked via OnReceiveCancelled().
125+
if (error.TransactionId is not null && _cancelledTransactionIds.ContainsKey(error.TransactionId))
126+
return null;
127+
128+
// Case 2: Elastic APM auto-instrumented "AzureServiceBus RECEIVE" transactions.
129+
// The Azure SDK ends its Activity (and therefore the APM transaction) before firing
130+
// ProcessErrorAsync, so Agent.Tracer.CurrentTransaction is null by the time
131+
// OnReceiveCancelled() runs — the transaction ID is never added to
132+
// _cancelledTransactionIds. Identify these by culprit pattern instead.
133+
// After switching to WebSockets transport, TaskCanceledException originating in
134+
// AmqpReceiver.ReceiveMessagesAsyncInternal only occurs during pod graceful shutdown.
135+
if (error.Exception?.Type is "System.Threading.Tasks.TaskCanceledException"
136+
or "System.OperationCanceledException" &&
137+
error.Culprit?.Contains("AmqpReceiver", StringComparison.Ordinal) == true)
138+
return null;
139+
140+
return error;
141+
});
126142
}
127143

128144
private static bool IsTraceEnabled()

0 commit comments

Comments
 (0)