Skip to content

Add 0082 OTel error-span rendering and metric#230

Merged
chris-colinsky merged 1 commit into
mainfrom
feature/0082-otel-error-span-and-metric
Jul 23, 2026
Merged

Add 0082 OTel error-span rendering and metric#230
chris-colinsky merged 1 commit into
mainfrom
feature/0082-otel-error-span-and-metric

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Phase A Wave 2 of the v0.17.0 cycle. PR B of the 0082 split (structured-output failure diagnostics, spec v0.77.0): the OTel error-span rendering + the §11 token-usage-on-failure metric. PR A (merged, #229) landed the event + error surface; PR C (Langfuse failed-Generation rendering) is the last.

What

  • The per-attempt LlmRetryAttemptEvent now carries the response-side surface on a structured_output_invalid failed attempt (mirroring PR A's terminal event).
  • The bundled OTel observer renders it on the error openarmature.llm.complete span: a structured_output_invalid failure falls through the failed-attempt early-return to render finish_reason, usage, gated output.content, and the response identity, while keeping ERROR status + category. Every other failure category still ends there with no response.
  • The opt-in openarmature.gen_ai.client.token.usage metric records the failure's usage (it already recorded whenever a usage record was present; a structured failure now carries one). No metric logic change.

Also fixed (observability §4.2)

The failed openarmature.llm.complete span carried ERROR status but no exception event. The node and invocation spans already recorded it via record_exception; the LLM span did not, a gap the 0082 error-span fixtures surfaced. It now records the OTel semconv exception event (exception.type / exception.message) on every failed attempt, matching the sibling spans, which also makes the enriched error_message locator load-bearing. See the CHANGELOG Fixed entry.

Conformance / tests

  • Un-defers observability fixtures 124 (OTel error span, payload on/off) and 125 (token metric on the failure); only Langfuse 123 stays deferred.
  • A dedicated failure-capable span-tree runner for 124, plus two OTel unit tests (surface rendered / output redacted under payload-off).
  • conformance.toml keeps 0082 partial (only Langfuse remains).

Reviewed with a multi-agent adversarial pass. It caught a real blocker (the missing §4.2 exception event) that my initial span-tree runner was too weak to detect; the runner is strengthened to enforce status_description / attributes_absent / exception_recorded, and the observer records the event. Two stale-docstring findings folded in. Full suite green.

PR B of proposal 0082 (the OTel rendering + the token metric; the
Langfuse rendering is the final PR of the split).

The per-attempt LlmRetryAttemptEvent now carries the response-side
surface on a structured_output_invalid failed attempt (mirroring PR A's
terminal event), so the bundled OTel observer renders it on the error
openarmature.llm.complete span: a structured_output_invalid failure
falls through the failed-attempt early-return to render finish_reason,
usage, gated output.content, and the response identity while keeping
ERROR status and category. The opt-in token-usage histogram records the
failure's usage (it already recorded whenever usage was present; a
structured failure now carries it).

Also records the OTel exception event on every failed llm.complete span
(exception.type / exception.message). The node and invocation spans
already did this; the LLM span did not, a gap the 0082 error-span
fixtures surfaced. This makes the enriched error_message locator
load-bearing.

Un-defers observability fixtures 124 (OTel error span, payload on/off)
and 125 (token metric); only the Langfuse fixture 123 stays deferred.
Copilot AI review requested due to automatic review settings July 23, 2026 07:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR advances proposal 0082 observability by ensuring a structured_output_invalid failed attempt carries enough response-side context to be rendered on the OTel openarmature.llm.complete error span (while preserving payload redaction behavior), and by un-deferring the corresponding conformance fixtures, including the §11 token-usage-on-failure metric fixture.

Changes:

  • Extend LlmRetryAttemptEvent/OpenAI retry-attempt event construction so structured_output_invalid failures include response-side fields (finish_reason/usage/response id+model, plus payload-gated output content).
  • Update the OTel observer to (a) record an OTel semconv exception event on failed LLM attempts and (b) render the response-side surface onto the error span for structured_output_invalid.
  • Un-defer and strengthen observability conformance fixture coverage for the OTel error-span rendering (124) and token-usage-on-failure metric (125), plus add focused OTel unit tests.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/unit/test_observability_otel.py Adds unit tests asserting error-span surface rendering and payload redaction for structured-output failures.
tests/conformance/test_observability.py Un-defers fixtures 124/125 and adds a dedicated runner + extra assertions for exception recording and payload redaction claims.
src/openarmature/observability/otel/observer.py Records exception events on failed LLM attempts and renders response-side surface for structured_output_invalid error spans.
src/openarmature/llm/providers/openai.py Includes response-side surface on LlmRetryAttemptEvent for StructuredOutputInvalid failures and enriches error_message.
src/openarmature/graph/events.py Updates LlmRetryAttemptEvent documentation to describe the structured-output failure exception case.
docs/concepts/observability.md Documents the structured-output failure exception for response-side surface rendering on error spans.
conformance.toml Updates proposal 0082 note to reflect PR B’s OTel rendering + metric coverage.
CHANGELOG.md Documents the new OTel error-span surface behavior and the added exception event on failed LLM attempts.

Comment thread tests/conformance/test_observability.py
@chris-colinsky
chris-colinsky merged commit 338e4b8 into main Jul 23, 2026
7 checks passed
@chris-colinsky
chris-colinsky deleted the feature/0082-otel-error-span-and-metric branch July 23, 2026 17:33
chris-colinsky added a commit that referenced this pull request Jul 23, 2026
Address CoPilot review on #230 (the guard landed after the squash-merge).
_assert_error_span_extras picked candidates[0] for a span name; with a
single-attempt fixture (124) names are unique, but a future fixture with
multiple same-named spans (e.g. call-level retry) could silently assert
the wrong span. Assert exactly one candidate so that case fails loudly
here instead, rather than duplicating _assert_span_tree_matches's full
name+attribute disambiguation.

Test-only; no behavior change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants