Add otel.sdk.component.shutdown self-observability event#3723
Add otel.sdk.component.shutdown self-observability event#3723cijothomas wants to merge 25 commits into
Conversation
|
This PR has been labeled as stale due to lack of activity. It will be automatically closed if there is no further activity over the next 7 days. |
Per open-telemetry#2975, the convention is attribute = '.result' (per-event value), metric = '.outcome' (aggregated count). Renames the attribute and updates the brief to use 'result' for consistency. Addresses review feedback from @thompson-tomo on open-telemetry#3723.
7b0c6d8 to
2330b52
Compare
The '_count' suffix is redundant for an int attribute named 'dropped'. Aligns with @thompson-tomo review feedback on open-telemetry#3723.
Per open-telemetry#2975, the convention is attribute = '.result' (per-event value), metric = '.outcome' (aggregated count). Renames the attribute and updates the brief to use 'result' for consistency. Addresses review feedback from @thompson-tomo on open-telemetry#3723.
The '_count' suffix is redundant for an int attribute named 'dropped'. Aligns with @thompson-tomo review feedback on open-telemetry#3723.
Steady-state queue overflow and retry-exhausted drops are already covered by the existing otel.sdk.processor.* metrics and now by otel.component.dropped. What was previously silent is the case where items were successfully buffered, but could not be drained before shutdown terminated (timeout or failure). Adds otel.component.shutdown.dropped as a new recommended int attribute on the shutdown event, and tightens otel.component.dropped to explicitly exclude shutdown-drain losses (clean boundary between the two counters). The severity rule is extended to WARN on shutdown.dropped > 0 as well.
Two clarifications to prevent implementation drift:
1. Invariant: shutdown.dropped MUST be 0 when shutdown.result is success.
A successful shutdown by definition drained everything; any non-zero
value implies result in {failed, timed_out}.
2. Upper-bound semantics: the count is what the SDK could not CONFIRM
delivered, not what was actually lost. In-flight export requests that
the SDK abandoned at timeout may still succeed on the wire after the
SDK gave up - those items are nevertheless counted because the SDK
cannot confirm. Spelled out as a SHOULD list covering: items still
buffered, items in abandoned in-flight requests, and items rejected
by the final export attempt (e.g. OTLP partial-success rejections).
8cf502d to
ee52ee7
Compare
…ent-shutdown-event
- Applicability: use SHOULD (aligns with other events in the repo) - Emission rules: declarative 'at most one per instance per lifetime' - Drop 'body SHOULD be omitted' (unnecessary) - Drop 'Relationship with error.type' section (design justification, not convention) - Drop 'SHOULD NOT sum durations' (implied by nesting sentence) - Intro: shorter, references metrics complementarity
There was a problem hiding this comment.
Two things worth fixing:
-
[BLOCKER]
TonicLogsClientmaps all errors tofailed—
timed_outcase is missing. Match onOTelSdkError::Timeout(_)
likeBatchLogProcessordoes, otherwise the POC doesn't fully
validate the spec. -
[NIT]
cfg_attroncomponent_namewill need manual update
when a third feature uses it. Allocating it always is simpler.
Rest looks good for a POC.
In case of OTel Rust, timeout is not respected today, so it won't ever produce |
|
Thanks for the thorough review! Thinking about the severity/granularity feedback -- would it be simpler to start with a provider-level event (one per On modeling as a span: I chose an event specifically so it flows through the logging pipeline, not the tracing pipeline. The inevitable shutdown ordering makes it challenging to ensure a shutdown-span can flow -- Thoughts?
|
- Emit from providers (TracerProvider, LoggerProvider, MeterProvider) instead of per-component (processors, exporters). Simpler first step; per-component events can follow as DEBUG-level refinement. - Replace custom otel.component.shutdown.result enum with canonical error.type (absent = success, 'timeout' or 'failed' on failure). - Add logger_provider, tracer_provider, meter_provider to otel.component.type well-known values. - Remove otel.component.shutdown.result from registry (no longer needed). - Severity: INFO when no error.type, WARN when error.type present. Addresses review feedback from @lmolkova.
|
@open-telemetry/semconv-sdk-health-approvers Could you take a look ? This is the first Event we are attempting for SDK's own self-observability and and want to get the direction agreed by all, so that subsequent Event additions would be relatively smooth. I've POCs linked, and plan to update OTel-Demo project to also showcase this. |
|
This PR has been labeled as stale due to lack of activity. It will be automatically closed if there is no further activity over the next 7 days. |
Not stale. Waiting for reviews only. |
4 participants
Resolves #3718.
Adds
otel.sdk.component.shutdown-- a self-observability event emitted once per SDK provider instance when its shutdown attempt completes. Validated via POC implementations in Rust and .NET.Event shape
otel.component.typelogger_provider/tracer_provider/meter_providerotel.component.namelogger_provider/0error.typetimeoutorfailedotel.component.shutdown.durationSeverity: INFO when successful (no
error.type), WARN when failed (error.typepresent).Design decisions
Provider-level, not per-component. One event per provider gives operators a single "did my pipeline shut down cleanly?" signal at INFO level. Per-component detail (processors, exporters) can follow in a future PR if operators need more granularity.
error.typefor failure classification. Uses the canonical semconv pattern: absence = success, presence = failure class. Operators already know how to alert onerror.type IS NOT NULLacross HTTP, DB, messaging -- this event gets that for free.Event, not span. The shutdown event flows through the logging pipeline, not the tracing pipeline. TracerProvider is typically shut down first; emitting a span through a TracerProvider that is itself shutting down creates ordering problems.
Deferred from the original proposal (#3718)
Dropped count -- unknowable on timeout; per-operation metrics (
otel.sdk.processor.*.processed{error.type=queue_full}) provide continuous accounting.Per-component events -- can be added in a follow-up if operators need to know which specific processor/exporter failed.