diff --git a/CHANGELOG.md b/CHANGELOG.md index d3b105fb..b9c67538 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The - **PromptManager service-wide default cache TTL** (proposal 0086, prompt-management §6, spec v0.79.0). `PromptManager` construction accepts an optional `default_cache_ttl_seconds`, applied to any `fetch` or `get` that omits a per-call `cache_ttl_seconds`. Resolution follows a precedence chain: an explicit per-call value (including `0` force-fresh) wins; otherwise the manager default applies; otherwise nothing is forwarded and the backend's own caching governs. An omitted or explicit-`None` per-call value both select the default, so resolution does not depend on argument presence. A negative default is rejected at construction, and the per-call negative rejection is unchanged. `get` delegates to `fetch` and inherits the chain, and the bundled caching backends need no change (they already honor a resolved `cache_ttl_seconds`). Conformance fixture 036 is un-deferred. - **PromptGroup arity enforcement** (proposal 0080, prompt-management §10 / §11, spec v0.75.0). Constructing a `PromptGroup` with fewer than two members (an empty or single-member group) now raises a categorized `PromptGroupInvalid` at construction, before any render or LLM call. This replaces the previous bare `ValueError` (which pydantic folded into a `ValidationError` carrying no error category) and adds a `prompt_group_invalid` category to the prompt-management error set. `PromptGroupInvalid` is non-transient and is exported from `openarmature.prompts`. Conformance fixture 035 is un-deferred. -- **Structured-output failure diagnostics on the LLM failure event** (proposal 0082, graph-engine §6 + llm-provider §7, spec v0.77.0). A `structured_output_invalid` failure is a completion whose final validation gate failed, so the wire response is intact. `LlmFailedEvent` now carries the response-side surface for that one category (`None` for every other): `output_content` (the content that failed validation, payload-gated), `finish_reason` (the truncation-vs-malformed triage signal, `"length"` vs `"stop"`), `usage`, `response_id`, and `response_model`. The `StructuredOutputInvalid` error additionally exposes `finish_reason` and `usage`, so a caller's exception handler can make a truncation-aware retry decision. Observer rendering of the surface (OTel error span, Langfuse failed Generation) and the token-usage-on-failure metric land in follow-on PRs of the same proposal. +- **Structured-output failure diagnostics on the LLM failure event** (proposal 0082, graph-engine §6 + llm-provider §7, spec v0.77.0). A `structured_output_invalid` failure is a completion whose final validation gate failed, so the wire response is intact. `LlmFailedEvent` now carries the response-side surface for that one category (`None` for every other): `output_content` (the content that failed validation, payload-gated), `finish_reason` (the truncation-vs-malformed triage signal, `"length"` vs `"stop"`), `usage`, `response_id`, and `response_model`. The `StructuredOutputInvalid` error additionally exposes `finish_reason` and `usage`, so a caller's exception handler can make a truncation-aware retry decision. The bundled OTel observer renders this surface on the error `openarmature.llm.complete` span (`ERROR` status plus `finish_reason`, `usage`, and payload-gated `output.content`), and the opt-in `openarmature.gen_ai.client.token.usage` metric now records a structured-output failure's token usage (other failure categories still record none). Rendering the surface on the Langfuse failed Generation lands in a follow-on PR of the same proposal. + +### Fixed + +- **The OTel `openarmature.llm.complete` span records the exception event on a failed attempt** (observability §4.2). A failed provider-call span carried `ERROR` status but no exception event; the node and invocation spans already recorded it, the LLM span did not. It now records the OTel semconv exception event (`exception.type` / `exception.message`) on every failed attempt, matching the sibling spans. Surfaced while wiring the proposal 0082 error-span fixtures. ## [0.16.0] — 2026-07-18 diff --git a/conformance.toml b/conformance.toml index 5c8d0e86..87aa0f53 100644 --- a/conformance.toml +++ b/conformance.toml @@ -822,7 +822,7 @@ since = "0.16.0" [proposals."0082"] status = "partial" since = "0.17.0" -note = "The graph-engine §6 event surface + llm-provider §7 error surface (PR A). LlmFailedEvent gains five response-side fields (output_content / finish_reason / usage / response_id / response_model), populated ONLY for structured_output_invalid (None for every other category); output_content is payload-bearing, the other four are not gated. StructuredOutputInvalid gains finish_reason / usage (the §7-mandated pair) plus response_id / response_model (internal, for the event builder), attached at the OpenAI provider's parse/validate call site where the intact wire response is in hand; the failed-event builder reads them off the enriched error. Un-defers observability fixtures 120 (truncation, finish_reason=length) / 121 (schema-mismatch, finish_reason=stop) / 122 (null-on-non-body companion), driven via the typed-event collector with a new calls_llm.response_schema harness lever; and llm-provider fixtures 022 / 023, whose carries block now asserts finish_reason + usage (the carries reader gained subset matching for a mapping-valued field like usage). PARTIAL: the OTel error-span rendering (124), Langfuse failed-Generation rendering (123), and the §11.2 token-usage-on-failure metric (125) are deferred to the OTel+metrics and Langfuse PRs -- they render the surface the event now carries." +note = "The graph-engine §6 event surface + llm-provider §7 error surface (PR A). LlmFailedEvent gains five response-side fields (output_content / finish_reason / usage / response_id / response_model), populated ONLY for structured_output_invalid (None for every other category); output_content is payload-bearing, the other four are not gated. StructuredOutputInvalid gains finish_reason / usage (the §7-mandated pair) plus response_id / response_model (internal, for the event builder), attached at the OpenAI provider's parse/validate call site where the intact wire response is in hand; the failed-event builder reads them off the enriched error. Un-defers observability fixtures 120 (truncation, finish_reason=length) / 121 (schema-mismatch, finish_reason=stop) / 122 (null-on-non-body companion), driven via the typed-event collector with a new calls_llm.response_schema harness lever; and llm-provider fixtures 022 / 023, whose carries block now asserts finish_reason + usage (the carries reader gained subset matching for a mapping-valued field like usage). PR B adds the OTel rendering + the metric: the per-attempt LlmRetryAttemptEvent now carries the same response-side surface on a structured_output_invalid failed attempt, so the OTel error span renders it (a structured_output_invalid failure falls through the failed-attempt early-return to render finish_reason / usage / gated output.content / response id+model while keeping ERROR status + category) and the §11.2 token-usage histogram records the failure's usage (the metric already recorded whenever usage was present; a structured failure now carries it). Un-defers observability fixtures 124 (OTel error span, payload on/off) and 125 (token metric on the failure). PARTIAL: only the Langfuse failed-Generation rendering (123) is deferred to the final PR of the split -- it renders the surface the event carries onto a failed Generation." # Spec v0.78.0 (proposal 0083). Per-prompt token-budget observability # (prompt-management §3 / graph-engine §6 / observability §5.5.15). diff --git a/docs/concepts/observability.md b/docs/concepts/observability.md index e5b50487..b1510f15 100644 --- a/docs/concepts/observability.md +++ b/docs/concepts/observability.md @@ -715,9 +715,11 @@ inside a single call. Each attempt emits its own first try emits one span at `attempt_index` 0; a call that fails twice transiently before succeeding emits three spans (indices 0, 1, 2). Each failed attempt's span carries `ERROR` status plus -`openarmature.error.category`; the final attempt's span carries the -terminal outcome (`OK` on success, `ERROR` on an exhausted or -non-transient failure). +`openarmature.error.category`; a `structured_output_invalid` failure +additionally carries the response-side surface (`finish_reason`, +`usage`, and payload-gated `output.content`), since its wire response +was intact. The final attempt's span carries the terminal outcome +(`OK` on success, `ERROR` on an exhausted or non-transient failure). `openarmature.llm.attempt_index` is the **call-level** attempt counter, [independent of the node-level `attempt_index`](llms.md#call-level-vs-node-level-retry): diff --git a/src/openarmature/graph/events.py b/src/openarmature/graph/events.py index 5047dfe4..d7aac2f6 100644 --- a/src/openarmature/graph/events.py +++ b/src/openarmature/graph/events.py @@ -713,7 +713,10 @@ class LlmRetryAttemptEvent: ``error_category`` discriminates the outcome: ``None`` for a successful attempt (the response-side fields are populated), a category string for a failed attempt (the response-side fields are - ``None`` — no response was received). + ``None`` — no response was received — with one exception: a + ``structured_output_invalid`` failure carries the response-side surface, + since its wire response was intact and failed only downstream + validation, mirroring :class:`LlmFailedEvent`). Field set: @@ -729,9 +732,13 @@ class LlmRetryAttemptEvent: - response side (``response_id`` / ``response_model`` / ``usage`` / ``finish_reason`` / ``output_content`` / ``output_tool_calls``): populated on a successful attempt; ``None`` / empty list on a - failed attempt. ``output_tool_calls`` is the source the OTel - observer renders the output tool-call attributes - from (this is the per-attempt event that drives the LLM span). + failed attempt, except a ``structured_output_invalid`` failure, + which populates ``output_content`` / ``finish_reason`` / ``usage`` / + ``response_id`` / ``response_model`` from its intact wire response + (``output_tool_calls`` stays empty — a structured failure carries no + tool calls). ``output_tool_calls`` is the source the OTel observer + renders the output tool-call attributes from (this is the per-attempt + event that drives the LLM span). - failure side (``error_category`` / ``error_message`` / ``error_type``): populated on a failed attempt; ``None`` on a successful one. diff --git a/src/openarmature/llm/providers/openai.py b/src/openarmature/llm/providers/openai.py index e23ec2ff..a5ef4159 100644 --- a/src/openarmature/llm/providers/openai.py +++ b/src/openarmature/llm/providers/openai.py @@ -867,11 +867,27 @@ def _build_llm_retry_attempt_event( ) if exc is None: raise ValueError("_build_llm_retry_attempt_event requires response or exc") + # Proposal 0082: a structured_output_invalid failed attempt carries the + # response-side surface (the wire response was intact); the OTel observer + # renders it on the error span and the §11 token metric records from it. + # None for every other category. error_message carries the failing locator. + error_message = str(exc) + response_fields: dict[str, Any] = {} + if isinstance(exc, StructuredOutputInvalid): + response_fields = { + "output_content": exc.raw_content, + "finish_reason": exc.finish_reason, + "usage": exc.usage, + "response_id": exc.response_id, + "response_model": exc.response_model, + } + error_message = f"{exc}: {exc.failure_description}" return LlmRetryAttemptEvent( **base, error_category=exc.category, error_type=type(exc).__name__, - error_message=str(exc), + error_message=error_message, + **response_fields, ) async def _do_complete( diff --git a/src/openarmature/observability/otel/observer.py b/src/openarmature/observability/otel/observer.py index 23e022f7..49edb741 100644 --- a/src/openarmature/observability/otel/observer.py +++ b/src/openarmature/observability/otel/observer.py @@ -1402,11 +1402,13 @@ def _record_llm_metrics(self, event: LlmRetryAttemptEvent) -> None: attempt. Duration is recorded for every attempt (including a failed one, - carrying ``error.type``); token usage only when the attempt - returned a usage record (a failed attempt has none). Sourced from - the per-attempt ``LlmRetryAttemptEvent`` — one duration sample per - attempt under call-level retry, matching the per-attempt span - model. The attempt index is deliberately not a dimension + carrying ``error.type``); token usage only when the attempt carries + a usage record. Most failed attempts have none, but a + ``structured_output_invalid`` failure does (proposal 0082): the wire + response was intact, so its token usage records like a completion's. + Sourced from the per-attempt ``LlmRetryAttemptEvent`` — one duration + sample per attempt under call-level retry, matching the per-attempt + span model. The attempt index is deliberately not a dimension (cardinality control). """ if self._duration_histogram is None or self._token_histogram is None: @@ -1426,9 +1428,10 @@ def _record_llm_metrics(self, event: LlmRetryAttemptEvent) -> None: if event.error_category is not None: duration_dims["error.type"] = event.error_category self._duration_histogram.record(event.latency_ms / 1000.0, duration_dims) - # Token usage: only when a usage record is present (a failed - # attempt has none). Two observations for an LLM completion — - # input + output — each only when its count was reported. + # Token usage: only when a usage record is present. Most failed + # attempts have none; a structured_output_invalid failure does + # (proposal 0082). Two observations for an LLM completion -- + # input + output -- each only when its count was reported. usage = event.usage if usage is not None: if usage.prompt_tokens is not None: @@ -1450,8 +1453,11 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None: the calling node span, each carrying ``openarmature.llm.attempt_index`` 0..N-1. A successful attempt (``error_category is None``) carries the full response surface - with OK status; a failed attempt carries ERROR status + the - error category and no response attributes. + with OK status; a failed attempt carries ERROR status + the error + category + the §4.2 exception event. Most failed categories add no + response attributes, but a ``structured_output_invalid`` failure + also carries the response surface (payload-gated output), since its + wire response was intact (proposal 0082). """ # Mid-call metadata augmentation can't reach this span: the # typed event arrives only after complete() returns, and the @@ -1542,13 +1548,30 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None: start_time=start_time_ns, ) if event.error_category is not None: - # Failed attempt: ERROR + the §4 category, no response - # attributes (no response was received). + # Failed attempt: ERROR + the §4 category + the §4.2 exception + # event. The per-attempt event carries the error type/message (not + # the exception object the node span records via record_exception), + # so add the OTel semconv exception event directly. error_message + # carries the failing locator for a structured_output_invalid + # failure (proposal 0082). span.set_status(Status(StatusCode.ERROR, description=event.error_category)) span.set_attribute("openarmature.error.category", event.error_category) - self._run_enrichers(span, event) - span.end(end_time=end_time_ns) - return + span.add_event( + "exception", + attributes={ + "exception.type": event.error_type or "", + "exception.message": event.error_message or "", + }, + ) + # Proposal 0082: a structured_output_invalid failure has an intact + # wire response, so the error span ALSO carries the response-side + # surface (payload-gated output) rendered below, in addition to + # ERROR status + category. Every other category received no + # response -- end the span here. + if event.error_category != "structured_output_invalid": + self._run_enrichers(span, event) + span.end(end_time=end_time_ns) + return usage = event.usage if event.finish_reason is not None: span.set_attribute("openarmature.llm.finish_reason", event.finish_reason) @@ -1619,7 +1642,10 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None: "openarmature.llm.output.tool_calls", _truncate_for_attribute(serialized_calls, self.payload_max_bytes), ) - span.set_status(Status(StatusCode.OK)) + # A structured_output_invalid failure fell through here with ERROR + # already set (proposal 0082); only a real success sets OK. + if event.error_category is None: + span.set_status(Status(StatusCode.OK)) self._run_enrichers(span, event) span.end(end_time=end_time_ns) diff --git a/tests/conformance/test_observability.py b/tests/conformance/test_observability.py index 7a6ff6ef..2bb118f5 100644 --- a/tests/conformance/test_observability.py +++ b/tests/conformance/test_observability.py @@ -217,6 +217,12 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: "088-llm-metrics-token-and-duration", "090-metrics-error-type-on-duration", "091-metrics-disabled-no-measurements", + # proposal 0082: the OTel error span (124) renders the response-side + # surface on a structured_output_invalid failure, and the §11 token + # metric (125) records the failure's usage. The Langfuse rendering + # (123) stays deferred to the final PR of the split. + "124-otel-error-span-renders-output-usage-finish-reason", + "125-metrics-token-usage-on-structured-output-failure", # v0.69.0 — proposal 0063 (tool-execution observability). A # calls_tool node enters the with_tool_call scope; the typed # ToolCallEvent / ToolCallFailedEvent drive the OTel tool span + @@ -348,17 +354,12 @@ def _reset_otel_global_tracer_provider(restore_to: object) -> None: "case not yet wired -- harness gap, not unimplemented" ), # Proposal 0082 (structured-output failure diagnostics, spec v0.77.0). - # The event-level response-side surface (120-122) is implemented; the - # OTel / Langfuse / metrics RENDERING fixtures defer to later PRs of the - # 0082 split (they render the surface the event now carries). - **{ - fixture_id: "structured-output failure diagnostics rendering (proposal 0082) not-yet implemented" - for fixture_id in ( - "123-langfuse-failed-generation-renders-output-usage-finish-reason", - "124-otel-error-span-renders-output-usage-finish-reason", - "125-metrics-token-usage-on-structured-output-failure", - ) - }, + # The event surface (120-122), the OTel error-span rendering (124) and the + # §11 token-usage metric (125) are implemented; only the Langfuse + # failed-Generation rendering (123) defers to the final PR of the 0082 split. + "123-langfuse-failed-generation-renders-output-usage-finish-reason": ( + "structured-output failure Langfuse rendering (proposal 0082) not-yet implemented" + ), # Proposal 0083 (per-prompt token-budget observability, spec v0.78.0). # The Prompt.token_budget advisory + budget-exceeded span attribute / # WARNING / metrics are unimplemented until a later v0.16.0 PR. @@ -696,8 +697,11 @@ async def test_observability_fixture(fixture_path: Path) -> None: "088-llm-metrics-token-and-duration", "090-metrics-error-type-on-duration", "091-metrics-disabled-no-measurements", + "125-metrics-token-usage-on-structured-output-failure", }: await _run_metrics_fixture(spec) + elif fixture_id == "124-otel-error-span-renders-output-usage-finish-reason": + await _run_structured_output_error_span_fixture(spec) elif fixture_id in { "092-tool-call-event-dispatch", "093-tool-call-failed-event-dispatch", @@ -3888,6 +3892,91 @@ async def _run_metrics_case(case: Mapping[str, Any]) -> None: _assert_metric_points(points, expected_metrics) +def _assert_error_span_extras(spans: Sequence[Any], expected_tree: list[dict[str, Any]]) -> None: + """Assert the span-tree keys ``_assert_span_tree_matches`` ignores: + ``status_description``, ``attributes_absent``, and ``exception_recorded`` + (an event named ``exception`` on the span). Fixture 124's central + payload-redaction and §4.2 exception-event claims live in these keys.""" + by_name: dict[str, list[Any]] = {} + for s in spans: + by_name.setdefault(s.name, []).append(s) + + def _walk(entries: list[dict[str, Any]]) -> None: + for entry in entries: + name = cast("str", entry["name"]) + candidates = by_name.get(name, []) + assert candidates, f"expected a span named {name!r}; got {sorted(by_name)}" + span = candidates[0] + desc = entry.get("status_description") + if desc is not None: + assert span.status.description == desc, ( + f"span {name!r} status_description: expected {desc!r}, got {span.status.description!r}" + ) + attrs = dict(span.attributes or {}) + for absent in cast("list[str]", entry.get("attributes_absent") or []): + assert absent not in attrs, f"span {name!r} MUST NOT carry {absent!r}; got {sorted(attrs)}" + if entry.get("exception_recorded"): + assert any(getattr(e, "name", None) == "exception" for e in span.events), ( + f"span {name!r} MUST record an exception event; got {[e.name for e in span.events]}" + ) + _walk(cast("list[dict[str, Any]]", entry.get("children") or [])) + + _walk(expected_tree) + + +async def _run_structured_output_error_span_fixture(spec: Mapping[str, Any]) -> None: + """Proposal 0082 OTel error-span fixture (124): a structured_output_invalid + failure renders the response-side surface on the ERROR + openarmature.llm.complete span, payload-gated by disable_provider_payload.""" + for case in cast("list[dict[str, Any]]", spec["cases"]): + try: + await _run_structured_output_error_span_case(case) + except AssertionError as e: + raise AssertionError(f"case {case.get('name')!r}: {e}") from e + + +async def _run_structured_output_error_span_case(case: Mapping[str, Any]) -> None: + from opentelemetry.sdk.trace.export import SimpleSpanProcessor + from opentelemetry.sdk.trace.export.in_memory_span_exporter import ( + InMemorySpanExporter, + ) + + from openarmature.graph import NodeException + + # _build_simple_llm_graph threads calls_llm.response_schema (proposal 0082), + # so the mocked structured-output failure raises structured_output_invalid. + graph, state_cls, provider = _build_simple_llm_graph(case, populate_caller_metadata=False) + exporter = InMemorySpanExporter() + # disable_provider_payload defaults to True per observability §5.5.4; case 1 + # sets it false to keep output.content, case 2 relies on the default to redact. + observer = OTelObserver( + span_processor=SimpleSpanProcessor(exporter), + disable_provider_payload=bool(case.get("disable_provider_payload", True)), + ) + graph.attach_observer(observer) + state = _make_state_instance(case, state_cls) + try: + with pytest.raises(NodeException): + await graph.invoke(state) + await graph.drain() + finally: + await provider.aclose() + observer.shutdown() + + spans = exporter.get_finished_spans() + expected_tree = cast("list[dict[str, Any]]", case["expected"]["span_tree"]) + inv_root = next( + (s for s in spans if s.name == "openarmature.invocation" and cast("Any", s.parent) is None), + None, + ) + assert inv_root is not None, f"invocation root span missing; got {[s.name for s in spans]}" + _assert_span_tree_matches(spans, [inv_root], expected_tree) + # _assert_span_tree_matches ignores status_description / attributes_absent / + # exception_recorded; assert those (124's payload redaction + §4.2 exception + # event) explicitly so the fixture's central claims are actually verified. + _assert_error_span_extras(spans, expected_tree) + + def _collect_metric_points(reader: Any) -> list[tuple[str, float, int, dict[str, Any]]]: """Flatten an InMemoryMetricReader's data into ``(instrument_name, point_sum, point_count, point_attributes)`` diff --git a/tests/unit/test_observability_otel.py b/tests/unit/test_observability_otel.py index 370bafdd..64f4a5ed 100644 --- a/tests/unit/test_observability_otel.py +++ b/tests/unit/test_observability_otel.py @@ -670,6 +670,82 @@ async def test_otel_observer_ignores_terminal_llm_events() -> None: assert llm_spans == [] +async def test_structured_output_failure_span_renders_response_surface() -> None: + # Proposal 0082: a structured_output_invalid failed attempt renders the + # response-side surface on the ERROR openarmature.llm.complete span + # (finish_reason, usage, payload-gated output.content) in addition to + # ERROR status + category -- unlike other failure categories, which carry + # no response. + from opentelemetry.trace import StatusCode + + from openarmature.llm.response import Usage + from openarmature.observability.correlation import _reset_invocation_id, _set_invocation_id + from tests._helpers.typed_event import make_retry_attempt_event + + exporter = InMemorySpanExporter() + observer = OTelObserver(span_processor=SimpleSpanProcessor(exporter), disable_provider_payload=False) + token = _set_invocation_id("inv-soi") + try: + await observer( + make_retry_attempt_event( + error_category="structured_output_invalid", + error_type="StructuredOutputInvalid", + finish_reason="length", + output_content='{"name":"Alice","age":', + usage=Usage(prompt_tokens=20, completion_tokens=16, total_tokens=36), + response_id="cc-xyz", + response_model="gpt-test-v2", + ) + ) + finally: + _reset_invocation_id(token) + observer.shutdown() + + llm_spans = [s for s in exporter.get_finished_spans() if s.name == "openarmature.llm.complete"] + assert len(llm_spans) == 1 + span = llm_spans[0] + assert span.status.status_code == StatusCode.ERROR + attrs = dict(span.attributes or {}) + assert attrs["openarmature.error.category"] == "structured_output_invalid" + assert attrs["openarmature.llm.finish_reason"] == "length" + assert attrs["openarmature.llm.output.content"] == '{"name":"Alice","age":' + assert attrs["openarmature.llm.usage.completion_tokens"] == 16 + assert attrs["gen_ai.response.id"] == "cc-xyz" + + +async def test_structured_output_failure_span_redacts_output_when_payload_disabled() -> None: + # Payload-gated: with disable_provider_payload=True, output.content is + # redacted while finish_reason / usage / ERROR status stay (proposal 0082). + from opentelemetry.trace import StatusCode + + from openarmature.llm.response import Usage + from openarmature.observability.correlation import _reset_invocation_id, _set_invocation_id + from tests._helpers.typed_event import make_retry_attempt_event + + exporter = InMemorySpanExporter() + observer = OTelObserver(span_processor=SimpleSpanProcessor(exporter), disable_provider_payload=True) + token = _set_invocation_id("inv-soi2") + try: + await observer( + make_retry_attempt_event( + error_category="structured_output_invalid", + finish_reason="length", + output_content='{"name":"Alice","age":', + usage=Usage(prompt_tokens=20, completion_tokens=16, total_tokens=36), + ) + ) + finally: + _reset_invocation_id(token) + observer.shutdown() + + span = next(s for s in exporter.get_finished_spans() if s.name == "openarmature.llm.complete") + attrs = dict(span.attributes or {}) + assert span.status.status_code == StatusCode.ERROR + assert "openarmature.llm.output.content" not in attrs + assert attrs["openarmature.llm.finish_reason"] == "length" + assert attrs["openarmature.llm.usage.completion_tokens"] == 16 + + async def _drive_llm_span_with_cached_tokens( *, cached_tokens: int | None,