Add 0082 OTel error-span rendering and metric (#230)

PR B of proposal 0082 (the OTel rendering + the token metric; the
Langfuse rendering is the final PR of the split).

The per-attempt LlmRetryAttemptEvent now carries the response-side
surface on a structured_output_invalid failed attempt (mirroring PR A's
terminal event), so the bundled OTel observer renders it on the error
openarmature.llm.complete span: a structured_output_invalid failure
falls through the failed-attempt early-return to render finish_reason,
usage, gated output.content, and the response identity while keeping
ERROR status and category. The opt-in token-usage histogram records the
failure's usage (it already recorded whenever usage was present; a
structured failure now carries it).

Also records the OTel exception event on every failed llm.complete span
(exception.type / exception.message). The node and invocation spans
already did this; the LLM span did not, a gap the 0082 error-span
fixtures surfaced. This makes the enriched error_message locator
load-bearing.

Un-defers observability fixtures 124 (OTel error span, payload on/off)
and 125 (token metric); only the Langfuse fixture 123 stays deferred.

Author: chris-colinsky

CHANGELOG.md

@@ -10,7 +10,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
 
 - **PromptManager service-wide default cache TTL** (proposal 0086, prompt-management §6, spec v0.79.0). `PromptManager` construction accepts an optional `default_cache_ttl_seconds`, applied to any `fetch` or `get` that omits a per-call `cache_ttl_seconds`. Resolution follows a precedence chain: an explicit per-call value (including `0` force-fresh) wins; otherwise the manager default applies; otherwise nothing is forwarded and the backend's own caching governs. An omitted or explicit-`None` per-call value both select the default, so resolution does not depend on argument presence. A negative default is rejected at construction, and the per-call negative rejection is unchanged. `get` delegates to `fetch` and inherits the chain, and the bundled caching backends need no change (they already honor a resolved `cache_ttl_seconds`). Conformance fixture 036 is un-deferred.
 - **PromptGroup arity enforcement** (proposal 0080, prompt-management §10 / §11, spec v0.75.0). Constructing a `PromptGroup` with fewer than two members (an empty or single-member group) now raises a categorized `PromptGroupInvalid` at construction, before any render or LLM call. This replaces the previous bare `ValueError` (which pydantic folded into a `ValidationError` carrying no error category) and adds a `prompt_group_invalid` category to the prompt-management error set. `PromptGroupInvalid` is non-transient and is exported from `openarmature.prompts`. Conformance fixture 035 is un-deferred.
-- **Structured-output failure diagnostics on the LLM failure event** (proposal 0082, graph-engine §6 + llm-provider §7, spec v0.77.0). A `structured_output_invalid` failure is a completion whose final validation gate failed, so the wire response is intact. `LlmFailedEvent` now carries the response-side surface for that one category (`None` for every other): `output_content` (the content that failed validation, payload-gated), `finish_reason` (the truncation-vs-malformed triage signal, `"length"` vs `"stop"`), `usage`, `response_id`, and `response_model`. The `StructuredOutputInvalid` error additionally exposes `finish_reason` and `usage`, so a caller's exception handler can make a truncation-aware retry decision. Observer rendering of the surface (OTel error span, Langfuse failed Generation) and the token-usage-on-failure metric land in follow-on PRs of the same proposal.
+- **Structured-output failure diagnostics on the LLM failure event** (proposal 0082, graph-engine §6 + llm-provider §7, spec v0.77.0). A `structured_output_invalid` failure is a completion whose final validation gate failed, so the wire response is intact. `LlmFailedEvent` now carries the response-side surface for that one category (`None` for every other): `output_content` (the content that failed validation, payload-gated), `finish_reason` (the truncation-vs-malformed triage signal, `"length"` vs `"stop"`), `usage`, `response_id`, and `response_model`. The `StructuredOutputInvalid` error additionally exposes `finish_reason` and `usage`, so a caller's exception handler can make a truncation-aware retry decision. The bundled OTel observer renders this surface on the error `openarmature.llm.complete` span (`ERROR` status plus `finish_reason`, `usage`, and payload-gated `output.content`), and the opt-in `openarmature.gen_ai.client.token.usage` metric now records a structured-output failure's token usage (other failure categories still record none). Rendering the surface on the Langfuse failed Generation lands in a follow-on PR of the same proposal.
+
+### Fixed
+
+- **The OTel `openarmature.llm.complete` span records the exception event on a failed attempt** (observability §4.2). A failed provider-call span carried `ERROR` status but no exception event; the node and invocation spans already recorded it, the LLM span did not. It now records the OTel semconv exception event (`exception.type` / `exception.message`) on every failed attempt, matching the sibling spans. Surfaced while wiring the proposal 0082 error-span fixtures.
 
 ## [0.16.0] — 2026-07-18
 

conformance.toml

@@ -822,7 +822,7 @@ since = "0.16.0"
 [proposals."0082"]
 status = "partial"
 since = "0.17.0"
-note = "The graph-engine §6 event surface + llm-provider §7 error surface (PR A). LlmFailedEvent gains five response-side fields (output_content / finish_reason / usage / response_id / response_model), populated ONLY for structured_output_invalid (None for every other category); output_content is payload-bearing, the other four are not gated. StructuredOutputInvalid gains finish_reason / usage (the §7-mandated pair) plus response_id / response_model (internal, for the event builder), attached at the OpenAI provider's parse/validate call site where the intact wire response is in hand; the failed-event builder reads them off the enriched error. Un-defers observability fixtures 120 (truncation, finish_reason=length) / 121 (schema-mismatch, finish_reason=stop) / 122 (null-on-non-body companion), driven via the typed-event collector with a new calls_llm.response_schema harness lever; and llm-provider fixtures 022 / 023, whose carries block now asserts finish_reason + usage (the carries reader gained subset matching for a mapping-valued field like usage). PARTIAL: the OTel error-span rendering (124), Langfuse failed-Generation rendering (123), and the §11.2 token-usage-on-failure metric (125) are deferred to the OTel+metrics and Langfuse PRs -- they render the surface the event now carries."
+note = "The graph-engine §6 event surface + llm-provider §7 error surface (PR A). LlmFailedEvent gains five response-side fields (output_content / finish_reason / usage / response_id / response_model), populated ONLY for structured_output_invalid (None for every other category); output_content is payload-bearing, the other four are not gated. StructuredOutputInvalid gains finish_reason / usage (the §7-mandated pair) plus response_id / response_model (internal, for the event builder), attached at the OpenAI provider's parse/validate call site where the intact wire response is in hand; the failed-event builder reads them off the enriched error. Un-defers observability fixtures 120 (truncation, finish_reason=length) / 121 (schema-mismatch, finish_reason=stop) / 122 (null-on-non-body companion), driven via the typed-event collector with a new calls_llm.response_schema harness lever; and llm-provider fixtures 022 / 023, whose carries block now asserts finish_reason + usage (the carries reader gained subset matching for a mapping-valued field like usage). PR B adds the OTel rendering + the metric: the per-attempt LlmRetryAttemptEvent now carries the same response-side surface on a structured_output_invalid failed attempt, so the OTel error span renders it (a structured_output_invalid failure falls through the failed-attempt early-return to render finish_reason / usage / gated output.content / response id+model while keeping ERROR status + category) and the §11.2 token-usage histogram records the failure's usage (the metric already recorded whenever usage was present; a structured failure now carries it). Un-defers observability fixtures 124 (OTel error span, payload on/off) and 125 (token metric on the failure). PARTIAL: only the Langfuse failed-Generation rendering (123) is deferred to the final PR of the split -- it renders the surface the event carries onto a failed Generation."
 
 # Spec v0.78.0 (proposal 0083).  Per-prompt token-budget observability
 # (prompt-management §3 / graph-engine §6 / observability §5.5.15).

docs/concepts/observability.md

@@ -715,9 +715,11 @@ inside a single call. Each attempt emits its own
 first try emits one span at `attempt_index` 0; a call that fails twice
 transiently before succeeding emits three spans (indices 0, 1, 2). Each
 failed attempt's span carries `ERROR` status plus
-`openarmature.error.category`; the final attempt's span carries the
-terminal outcome (`OK` on success, `ERROR` on an exhausted or
-non-transient failure).
+`openarmature.error.category`; a `structured_output_invalid` failure
+additionally carries the response-side surface (`finish_reason`,
+`usage`, and payload-gated `output.content`), since its wire response
+was intact. The final attempt's span carries the terminal outcome
+(`OK` on success, `ERROR` on an exhausted or non-transient failure).
 
 `openarmature.llm.attempt_index` is the **call-level** attempt counter,
 [independent of the node-level `attempt_index`](llms.md#call-level-vs-node-level-retry):

src/openarmature/graph/events.py

@@ -713,7 +713,10 @@ class LlmRetryAttemptEvent:
     ``error_category`` discriminates the outcome: ``None`` for a
     successful attempt (the response-side fields are populated), a
     category string for a failed attempt (the response-side fields are
-    ``None`` — no response was received).
+    ``None`` — no response was received — with one exception: a
+    ``structured_output_invalid`` failure carries the response-side surface,
+    since its wire response was intact and failed only downstream
+    validation, mirroring :class:`LlmFailedEvent`).
 
     Field set:
 
@@ -729,9 +732,13 @@ class LlmRetryAttemptEvent:
     - response side (``response_id`` / ``response_model`` / ``usage`` /
       ``finish_reason`` / ``output_content`` / ``output_tool_calls``):
       populated on a successful attempt; ``None`` / empty list on a
-      failed attempt. ``output_tool_calls`` is the source the OTel
-      observer renders the output tool-call attributes
-      from (this is the per-attempt event that drives the LLM span).
+      failed attempt, except a ``structured_output_invalid`` failure,
+      which populates ``output_content`` / ``finish_reason`` / ``usage`` /
+      ``response_id`` / ``response_model`` from its intact wire response
+      (``output_tool_calls`` stays empty — a structured failure carries no
+      tool calls). ``output_tool_calls`` is the source the OTel observer
+      renders the output tool-call attributes from (this is the per-attempt
+      event that drives the LLM span).
     - failure side (``error_category`` / ``error_message`` /
       ``error_type``): populated on a failed attempt; ``None`` on a
       successful one.

src/openarmature/llm/providers/openai.py

@@ -867,11 +867,27 @@ def _build_llm_retry_attempt_event(
             )
         if exc is None:
             raise ValueError("_build_llm_retry_attempt_event requires response or exc")
+        # Proposal 0082: a structured_output_invalid failed attempt carries the
+        # response-side surface (the wire response was intact); the OTel observer
+        # renders it on the error span and the §11 token metric records from it.
+        # None for every other category. error_message carries the failing locator.
+        error_message = str(exc)
+        response_fields: dict[str, Any] = {}
+        if isinstance(exc, StructuredOutputInvalid):
+            response_fields = {
+                "output_content": exc.raw_content,
+                "finish_reason": exc.finish_reason,
+                "usage": exc.usage,
+                "response_id": exc.response_id,
+                "response_model": exc.response_model,
+            }
+            error_message = f"{exc}: {exc.failure_description}"
         return LlmRetryAttemptEvent(
             **base,
             error_category=exc.category,
             error_type=type(exc).__name__,
-            error_message=str(exc),
+            error_message=error_message,
+            **response_fields,
         )
 
     async def _do_complete(

src/openarmature/observability/otel/observer.py

@@ -1402,11 +1402,13 @@ def _record_llm_metrics(self, event: LlmRetryAttemptEvent) -> None:
         attempt.
 
         Duration is recorded for every attempt (including a failed one,
-        carrying ``error.type``); token usage only when the attempt
-        returned a usage record (a failed attempt has none). Sourced from
-        the per-attempt ``LlmRetryAttemptEvent`` — one duration sample per
-        attempt under call-level retry, matching the per-attempt span
-        model. The attempt index is deliberately not a dimension
+        carrying ``error.type``); token usage only when the attempt carries
+        a usage record. Most failed attempts have none, but a
+        ``structured_output_invalid`` failure does (proposal 0082): the wire
+        response was intact, so its token usage records like a completion's.
+        Sourced from the per-attempt ``LlmRetryAttemptEvent`` — one duration
+        sample per attempt under call-level retry, matching the per-attempt
+        span model. The attempt index is deliberately not a dimension
         (cardinality control).
         """
         if self._duration_histogram is None or self._token_histogram is None:
@@ -1426,9 +1428,10 @@ def _record_llm_metrics(self, event: LlmRetryAttemptEvent) -> None:
             if event.error_category is not None:
                 duration_dims["error.type"] = event.error_category
             self._duration_histogram.record(event.latency_ms / 1000.0, duration_dims)
-        # Token usage: only when a usage record is present (a failed
-        # attempt has none). Two observations for an LLM completion —
-        # input + output — each only when its count was reported.
+        # Token usage: only when a usage record is present. Most failed
+        # attempts have none; a structured_output_invalid failure does
+        # (proposal 0082). Two observations for an LLM completion --
+        # input + output -- each only when its count was reported.
         usage = event.usage
         if usage is not None:
             if usage.prompt_tokens is not None:
@@ -1450,8 +1453,11 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None:
         the calling node span, each carrying
         ``openarmature.llm.attempt_index`` 0..N-1. A successful attempt
         (``error_category is None``) carries the full response surface
-        with OK status; a failed attempt carries ERROR status + the
-        error category and no response attributes.
+        with OK status; a failed attempt carries ERROR status + the error
+        category + the §4.2 exception event. Most failed categories add no
+        response attributes, but a ``structured_output_invalid`` failure
+        also carries the response surface (payload-gated output), since its
+        wire response was intact (proposal 0082).
         """
         # Mid-call metadata augmentation can't reach this span: the
         # typed event arrives only after complete() returns, and the
@@ -1542,13 +1548,30 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None:
             start_time=start_time_ns,
         )
         if event.error_category is not None:
-            # Failed attempt: ERROR + the §4 category, no response
-            # attributes (no response was received).
+            # Failed attempt: ERROR + the §4 category + the §4.2 exception
+            # event. The per-attempt event carries the error type/message (not
+            # the exception object the node span records via record_exception),
+            # so add the OTel semconv exception event directly. error_message
+            # carries the failing locator for a structured_output_invalid
+            # failure (proposal 0082).
             span.set_status(Status(StatusCode.ERROR, description=event.error_category))
             span.set_attribute("openarmature.error.category", event.error_category)
-            self._run_enrichers(span, event)
-            span.end(end_time=end_time_ns)
-            return
+            span.add_event(
+                "exception",
+                attributes={
+                    "exception.type": event.error_type or "",
+                    "exception.message": event.error_message or "",
+                },
+            )
+            # Proposal 0082: a structured_output_invalid failure has an intact
+            # wire response, so the error span ALSO carries the response-side
+            # surface (payload-gated output) rendered below, in addition to
+            # ERROR status + category. Every other category received no
+            # response -- end the span here.
+            if event.error_category != "structured_output_invalid":
+                self._run_enrichers(span, event)
+                span.end(end_time=end_time_ns)
+                return
         usage = event.usage
         if event.finish_reason is not None:
             span.set_attribute("openarmature.llm.finish_reason", event.finish_reason)
@@ -1619,7 +1642,10 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None:
                     "openarmature.llm.output.tool_calls",
                     _truncate_for_attribute(serialized_calls, self.payload_max_bytes),
                 )
-        span.set_status(Status(StatusCode.OK))
+        # A structured_output_invalid failure fell through here with ERROR
+        # already set (proposal 0082); only a real success sets OK.
+        if event.error_category is None:
+            span.set_status(Status(StatusCode.OK))
         self._run_enrichers(span, event)
         span.end(end_time=end_time_ns)