Add 0082 Langfuse failed-Generation rendering (#232)

* Add 0082 Langfuse failed-Generation rendering

PR C of proposal 0082, completing the 3-PR split with the Langfuse
failed-Generation rendering. 0082 is now fully implemented.

For a structured_output_invalid failure the bundled Langfuse observer
renders the response-side surface on the ERROR Generation like a
completion: output (payload-gated) + usage on the Generation, and
finish_reason / response_id / response_model in metadata (the
completion-only isinstance guard widened to include the category),
alongside the existing level=ERROR + statusMessage=category. Every
other failure category is unchanged. Fixture 123 is driven in the
dedicated Langfuse harness (response_schema threaded + expected_error
handled). conformance.toml marks 0082 implemented.

Also folds in the whole-proposal adversarial review over all three PRs:

- Cross-observer consistency: the failure event builders project empty
  output_content to None (matching the success path's content-or-None),
  so both observers omit empty content identically. The OpenAI wire
  cannot currently produce empty content on the structured path, so this
  is a defensive builder projection with a builder-direct test.
- The Langfuse conformance harness now asserts observation statusMessage
  (previously a silent no-op, so 123's category assertion did not run;
  also affected fixtures 130/137/138).
- Refreshed a stale _typed_event_metadata docstring, dropped a docstring
  spec ref, and completed the CHANGELOG response id/model listing.

* Tighten 0082 Langfuse failure-path assertions

Address CoPilot review threads on PR #232 (0082 Langfuse
failed-Generation rendering):

- Assert metadata.response_id on the failed-Generation unit test.
  The event set it but only response_model was pinned, so a
  regression dropping response_id would have passed.
- Assert raised_from against the raised NodeException in the linear
  conformance path. Fixture 123 declares it but nothing checked it.
  Category is not asserted on the exception (its category is always
  node_exception); it rides on the Generation statusMessage.
- Restructure renders_response_side to an explicit if/else so the
  failure branch is obvious. No behavior change.

Author: chris-colinsky

CHANGELOG.md

@@ -10,7 +10,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
 
 - **PromptManager service-wide default cache TTL** (proposal 0086, prompt-management §6, spec v0.79.0). `PromptManager` construction accepts an optional `default_cache_ttl_seconds`, applied to any `fetch` or `get` that omits a per-call `cache_ttl_seconds`. Resolution follows a precedence chain: an explicit per-call value (including `0` force-fresh) wins; otherwise the manager default applies; otherwise nothing is forwarded and the backend's own caching governs. An omitted or explicit-`None` per-call value both select the default, so resolution does not depend on argument presence. A negative default is rejected at construction, and the per-call negative rejection is unchanged. `get` delegates to `fetch` and inherits the chain, and the bundled caching backends need no change (they already honor a resolved `cache_ttl_seconds`). Conformance fixture 036 is un-deferred.
 - **PromptGroup arity enforcement** (proposal 0080, prompt-management §10 / §11, spec v0.75.0). Constructing a `PromptGroup` with fewer than two members (an empty or single-member group) now raises a categorized `PromptGroupInvalid` at construction, before any render or LLM call. This replaces the previous bare `ValueError` (which pydantic folded into a `ValidationError` carrying no error category) and adds a `prompt_group_invalid` category to the prompt-management error set. `PromptGroupInvalid` is non-transient and is exported from `openarmature.prompts`. Conformance fixture 035 is un-deferred.
-- **Structured-output failure diagnostics on the LLM failure event** (proposal 0082, graph-engine §6 + llm-provider §7, spec v0.77.0). A `structured_output_invalid` failure is a completion whose final validation gate failed, so the wire response is intact. `LlmFailedEvent` now carries the response-side surface for that one category (`None` for every other): `output_content` (the content that failed validation, payload-gated), `finish_reason` (the truncation-vs-malformed triage signal, `"length"` vs `"stop"`), `usage`, `response_id`, and `response_model`. The `StructuredOutputInvalid` error additionally exposes `finish_reason` and `usage`, so a caller's exception handler can make a truncation-aware retry decision. The bundled OTel observer renders this surface on the error `openarmature.llm.complete` span (`ERROR` status plus `finish_reason`, `usage`, and payload-gated `output.content`), and the opt-in `openarmature.gen_ai.client.token.usage` metric now records a structured-output failure's token usage (other failure categories still record none). Rendering the surface on the Langfuse failed Generation lands in a follow-on PR of the same proposal.
+- **Structured-output failure diagnostics on the LLM failure event** (proposal 0082, graph-engine §6 + llm-provider §7, spec v0.77.0). A `structured_output_invalid` failure is a completion whose final validation gate failed, so the wire response is intact. `LlmFailedEvent` now carries the response-side surface for that one category (`None` for every other): `output_content` (the content that failed validation, payload-gated), `finish_reason` (the truncation-vs-malformed triage signal, `"length"` vs `"stop"`), `usage`, `response_id`, and `response_model`. The `StructuredOutputInvalid` error additionally exposes `finish_reason` and `usage`, so a caller's exception handler can make a truncation-aware retry decision. The bundled OTel observer renders this surface on the error `openarmature.llm.complete` span (`ERROR` status plus `finish_reason`, `usage`, payload-gated `output.content`, and the response id / model), and the opt-in `openarmature.gen_ai.client.token.usage` metric now records a structured-output failure's token usage (other failure categories still record none). The bundled Langfuse observer renders the same surface on the failed Generation: `level = "ERROR"` + `statusMessage` = the category, plus `output` (payload-gated), `usage`, `metadata.finish_reason`, and `metadata.response_id` / `metadata.response_model`.
 
 ### Fixed
 

conformance.toml

@@ -816,13 +816,13 @@ since = "0.16.0"
 # Spec v0.77.0 (proposal 0082).  Structured-output failure diagnostics
 # (graph-engine §6 -- LlmFailedEvent response-side surface for
 # structured_output_invalid + llm-provider §7 finish_reason / usage).
-# Partial since 0.17.0: the event + error surface is implemented (PR A);
-# the OTel / Langfuse / metrics RENDERING (observability fixtures 123-125)
-# lands in later PRs of the split.
+# Implemented since 0.17.0 across the 3-PR split: the event + error surface
+# (PR A), the OTel error-span rendering + §11 token metric (PR B), and the
+# Langfuse failed-Generation rendering (PR C).
 [proposals."0082"]
-status = "partial"
+status = "implemented"
 since = "0.17.0"
-note = "The graph-engine §6 event surface + llm-provider §7 error surface (PR A). LlmFailedEvent gains five response-side fields (output_content / finish_reason / usage / response_id / response_model), populated ONLY for structured_output_invalid (None for every other category); output_content is payload-bearing, the other four are not gated. StructuredOutputInvalid gains finish_reason / usage (the §7-mandated pair) plus response_id / response_model (internal, for the event builder), attached at the OpenAI provider's parse/validate call site where the intact wire response is in hand; the failed-event builder reads them off the enriched error. Un-defers observability fixtures 120 (truncation, finish_reason=length) / 121 (schema-mismatch, finish_reason=stop) / 122 (null-on-non-body companion), driven via the typed-event collector with a new calls_llm.response_schema harness lever; and llm-provider fixtures 022 / 023, whose carries block now asserts finish_reason + usage (the carries reader gained subset matching for a mapping-valued field like usage). PR B adds the OTel rendering + the metric: the per-attempt LlmRetryAttemptEvent now carries the same response-side surface on a structured_output_invalid failed attempt, so the OTel error span renders it (a structured_output_invalid failure falls through the failed-attempt early-return to render finish_reason / usage / gated output.content / response id+model while keeping ERROR status + category) and the §11.2 token-usage histogram records the failure's usage (the metric already recorded whenever usage was present; a structured failure now carries it). Un-defers observability fixtures 124 (OTel error span, payload on/off) and 125 (token metric on the failure). PARTIAL: only the Langfuse failed-Generation rendering (123) is deferred to the final PR of the split -- it renders the surface the event carries onto a failed Generation."
+note = "The graph-engine §6 event surface + llm-provider §7 error surface (PR A). LlmFailedEvent gains five response-side fields (output_content / finish_reason / usage / response_id / response_model), populated ONLY for structured_output_invalid (None for every other category); output_content is payload-bearing, the other four are not gated. StructuredOutputInvalid gains finish_reason / usage (the §7-mandated pair) plus response_id / response_model (internal, for the event builder), attached at the OpenAI provider's parse/validate call site where the intact wire response is in hand; the failed-event builder reads them off the enriched error. Un-defers observability fixtures 120 (truncation, finish_reason=length) / 121 (schema-mismatch, finish_reason=stop) / 122 (null-on-non-body companion), driven via the typed-event collector with a new calls_llm.response_schema harness lever; and llm-provider fixtures 022 / 023, whose carries block now asserts finish_reason + usage (the carries reader gained subset matching for a mapping-valued field like usage). PR B adds the OTel rendering + the metric: the per-attempt LlmRetryAttemptEvent now carries the same response-side surface on a structured_output_invalid failed attempt, so the OTel error span renders it (a structured_output_invalid failure falls through the failed-attempt early-return to render finish_reason / usage / gated output.content / response id+model while keeping ERROR status + category) and the §11.2 token-usage histogram records the failure's usage (the metric already recorded whenever usage was present; a structured failure now carries it). Un-defers observability fixtures 124 (OTel error span, payload on/off) and 125 (token metric on the failure). PR C completes it with the Langfuse failed-Generation rendering: _handle_typed_llm_failed renders output (payload-gated) + usage on the ERROR Generation and _typed_event_metadata renders finish_reason / response id+model for a structured_output_invalid failure (the isinstance(LlmCompletionEvent) guard widened to include it), alongside the existing level=ERROR + statusMessage=category. Un-defers observability fixture 123, driven in the dedicated test_observability_langfuse harness (calls_llm.response_schema threaded + expected_error handled). 0082 now fully implemented."
 
 # Spec v0.78.0 (proposal 0083).  Per-prompt token-budget observability
 # (prompt-management §3 / graph-engine §6 / observability §5.5.15).

src/openarmature/llm/providers/openai.py

@@ -764,7 +764,10 @@ def _build_llm_failed_event(
         response_model: str | None = None
         error_message = str(exc)
         if isinstance(exc, StructuredOutputInvalid):
-            output_content = exc.raw_content
+            # Empty content projects to None (as the success path does with
+            # ``content or None``), so both observers omit it identically
+            # rather than one rendering "" and the other dropping it.
+            output_content = exc.raw_content or None
             finish_reason = exc.finish_reason
             usage = exc.usage
             response_id = exc.response_id
@@ -875,7 +878,9 @@ def _build_llm_retry_attempt_event(
         response_fields: dict[str, Any] = {}
         if isinstance(exc, StructuredOutputInvalid):
             response_fields = {
-                "output_content": exc.raw_content,
+                # Empty content -> None (as the success path projects), so both
+                # observers omit it identically.
+                "output_content": exc.raw_content or None,
                 "finish_reason": exc.finish_reason,
                 "usage": exc.usage,
                 "response_id": exc.response_id,

src/openarmature/observability/langfuse/observer.py

@@ -1757,9 +1757,11 @@ def _handle_typed_llm_completion(self, event: LlmCompletionEvent) -> None:
 
     def _handle_typed_llm_failed(self, event: LlmFailedEvent) -> None:
         """Open + close an ERROR-level Generation observation from the
-        typed LlmFailedEvent (failure path). Same shape as the success
-        path with ERROR level + error_category as the Generation
-        observation's statusMessage."""
+        typed LlmFailedEvent (failure path): ERROR level + error_category
+        as the Generation's statusMessage. A structured_output_invalid
+        failure additionally carries the response-side surface (payload-gated
+        output, usage, metadata.finish_reason) like a completion, since its
+        wire response was intact."""
         from openarmature.observability.correlation import (
             current_correlation_id,
             current_invocation_id,
@@ -1798,18 +1800,32 @@ def _handle_typed_llm_failed(self, event: LlmFailedEvent) -> None:
         metadata["error_message"] = event.error_message
         model_parameters: dict[str, Any] = dict(event.request_params or {})
         input_value: Any = None
+        output_value: Any = None
+        end_kwargs: dict[str, Any] = {}
         if not self.disable_provider_payload:
             if event.input_messages:
                 input_value = self._maybe_truncate_for_input(event.input_messages)
             if event.request_extras:
                 metadata["request_extras"] = self._maybe_truncate_for_extras(dict(event.request_extras))
+        # Proposal 0082: a structured_output_invalid failure has an intact wire
+        # response, so the failed Generation carries the response-side surface
+        # (output payload-gated, usage) like a completion, alongside the ERROR
+        # level + category. metadata.finish_reason is added for this category by
+        # _typed_event_metadata.
+        if event.error_category == "structured_output_invalid":
+            if not self.disable_provider_payload and event.output_content is not None:
+                output_value = self._maybe_truncate_for_output(event.output_content)
+            usage = self._usage_from_typed_event(event)
+            if usage is not None:
+                end_kwargs["usage"] = usage
         target_trace_id = self._trace_id_for(inv_state, event.namespace, event.fan_out_index)
         handle = self.client.generation(
             trace_id=target_trace_id,
             name="openarmature.llm.complete",
             model=event.model,
             model_parameters=model_parameters,
             input=input_value,
+            output=output_value,
             metadata=metadata,
             parent_observation_id=parent_observation_id,
             prompt=self._resolve_prompt_link_from_typed_event(event),
@@ -1820,6 +1836,7 @@ def _handle_typed_llm_failed(self, event: LlmFailedEvent) -> None:
             end_time=end_time,
             level="ERROR",
             status_message=event.error_category,
+            **end_kwargs,
         )
 
     # Spec proposal 0063: dedicated Langfuse Tool observation (asType="tool").
@@ -2169,13 +2186,11 @@ def _typed_event_metadata(
     ) -> dict[str, Any]:
         """Build the Generation observation's metadata dict from a
         typed LLM event. Shared between the success path
-        (LlmCompletionEvent) and the failure path (LlmFailedEvent);
-        response-side metadata (finish_reason / response_model /
-        response_id) lands only on the success variant for now. The
-        failure variant carries those fields too (populated only for a
-        structured_output_invalid failure, proposal 0082), but rendering
-        them on a failed Generation is deferred to the failed-generation
-        rendering PR via the isinstance guard below."""
+        (LlmCompletionEvent) and the failure path (LlmFailedEvent).
+        Response-side metadata (finish_reason / response_model /
+        response_id) renders for a completion and for a
+        structured_output_invalid failure (its wire response was intact);
+        every other failure category renders none, per the guard below."""
         metadata: dict[str, Any] = {}
         if correlation_id is not None:
             metadata["correlation_id"] = correlation_id
@@ -2201,14 +2216,17 @@ def _typed_event_metadata(
             metadata["prompt_group_name"] = active_group.group_name
         if event.caller_invocation_metadata is not None:
             _apply_caller_metadata(metadata, event.caller_invocation_metadata)
-        # Response-side rendering is success-path-only for now. The failure
-        # variant carries these fields too (populated only for a
-        # structured_output_invalid failure, proposal 0082), but this
-        # deliberate isinstance guard defers rendering them on a failed
-        # Generation to the failed-generation rendering PR; the failure-path
-        # metadata stays focused on the request-side + the failure-specific
-        # fields the caller adds separately.
+        # Response-side metadata. A completion always carries it; a
+        # structured_output_invalid failure also carries it (proposal 0082),
+        # since its wire response was intact, so finish_reason (the truncation
+        # signal) and the response identity render on the failed Generation too.
+        # Every other failure category received no response and renders none.
         if isinstance(event, LlmCompletionEvent):
+            renders_response_side = True
+        else:
+            # event narrows to LlmFailedEvent (the only other union member).
+            renders_response_side = event.error_category == "structured_output_invalid"
+        if renders_response_side:
             if event.finish_reason is not None:
                 metadata["finish_reason"] = event.finish_reason
             if event.response_model is not None:
@@ -2217,7 +2235,7 @@ def _typed_event_metadata(
                 metadata["response_id"] = event.response_id
         return metadata
 
-    def _usage_from_typed_event(self, event: LlmCompletionEvent) -> LangfuseUsage | None:
+    def _usage_from_typed_event(self, event: LlmCompletionEvent | LlmFailedEvent) -> LangfuseUsage | None:
         """Map the typed event's Usage onto the Langfuse Usage record.
         Returns None when no usage was reported."""
         # Spec observability §8.4.3 (Langfuse usage mapping).

tests/conformance/test_observability.py

@@ -355,10 +355,11 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None:
     ),
     # Proposal 0082 (structured-output failure diagnostics, spec v0.77.0).
     # The event surface (120-122), the OTel error-span rendering (124) and the
-    # §11 token-usage metric (125) are implemented; only the Langfuse
-    # failed-Generation rendering (123) defers to the final PR of the 0082 split.
+    # §11 token-usage metric (125) run here; the Langfuse failed-Generation
+    # rendering (123) is driven in the dedicated test_observability_langfuse
+    # harness (its langfuse_trace shape lives there, like 022-024).
     "123-langfuse-failed-generation-renders-output-usage-finish-reason": (
-        "structured-output failure Langfuse rendering (proposal 0082) not-yet implemented"
+        "Langfuse failed-Generation rendering; driven in test_observability_langfuse"
     ),
     # Proposal 0083 (per-prompt token-budget observability, spec v0.78.0).
     # The Prompt.token_budget advisory + budget-exceeded span attribute /