Add per-attempt LLM spans under call-level retry (0050) (#170)

* Add LlmRetryAttemptEvent for per-attempt LLM spans (0050)

* Emit per-attempt LlmRetryAttemptEvent from complete() (0050)

* Render per-attempt LLM spans from LlmRetryAttemptEvent (0050)

* Activate obs-057 (single-attempt llm.attempt_index) (0050)

* Add N-span call-level-retry integration test (0050)

* Activate call-level-retry per-attempt span fixtures (0050)

* Document per-attempt LLM spans and flip 0050

Flip conformance.toml [proposals."0050"] partial -> implemented
(since 0.15.0): the call-level-retry per-attempt span surface now
ships.

Document the openarmature.llm.attempt_index attribute and the
per-attempt span behavior in the observability concepts page, plus
notes that span enrichers receive LlmRetryAttemptEvent on the LLM
span and that the bundled provider dispatches that internal event
alongside the unchanged terminal events.

Add the 0.15.0 changelog section covering this work and backfilling
the 0061 detached-trace invocation span (which landed without an
entry), plus the v0.60.0 -> v0.61.0 spec-pin bullet.

* Deduplicate per-attempt LLM event builder

_build_llm_retry_attempt_event constructed a full LlmRetryAttemptEvent
twice, repeating ~18 shared identity, scoping, and request-side fields
across the success and failure branches. Hoist them into one base dict
and splat it, leaving each branch to add only its outcome fields. No
behavior change.

* Test OTel observer ignores terminal LLM events

The OTel observer now renders the LLM span solely from the per-attempt
LlmRetryAttemptEvent; terminal LlmCompletionEvent / LlmFailedEvent are
ignored. Add a regression test feeding both terminal events and
asserting zero openarmature.llm.complete spans, guarding against
reintroducing the terminal-event span path.

Also fix a stale docstring in _drive_llm_span_with_cached_tokens that
still referenced "typed LlmCompletionEvent".

* Re-export LlmRetryAttemptEvent; isinstance filter

PR #170 CoPilot review:

- Re-export LlmRetryAttemptEvent from the openarmature.graph package
  (import block + __all__), matching the sibling LlmCompletionEvent /
  LlmFailedEvent so the documented observer import path works.
- Replace the brittle type(event).__name__ name match with an
  isinstance check in the conformance _TypedEventCollector; the
  filter_event_type string comparison stays as-is.

Author: chris-colinsky

CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to `openarmature-python` are documented in this file.
 
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The package follows [Semantic Versioning](https://semver.org/); pre-1.0 minor bumps may carry behavioral changes per [spec governance](https://github.com/LunarCommand/openarmature-spec/blob/main/GOVERNANCE.md).
 
+## [0.15.0] — 2026-06-18
+
+### Added
+
+- **Detached-trace invocation span** (proposal 0061, observability §4.4, spec v0.61.0). The OTel observer now synthesizes an `openarmature.invocation` span at the root of each detached trace (a detached subgraph and each detached fan-out instance), carrying the parent's shared `invocation_id` (detached mode is observer-side trace rendering, not a new run) and the detached unit's own `entry_node`; the detached subgraph / instance span nests under it. A raising detached subgraph surfaces ERROR plus the error category and an OTel exception event on both the parent dispatch span and the detached invocation span. This is observer-side only, with no graph-engine change; the Langfuse observer is unchanged (its Trace entity already plays the invocation-level-container role). Conformance fixtures 008 (rewritten) and 058 (newly wired) run in `test_observability`.
+- **Per-attempt LLM spans under call-level retry** (proposal 0050, observability §5.5 / llm-provider §7.1). Completes proposal 0050, which shipped `partial` in v0.14.0 (failure-isolation middleware and the `complete(retry=...)` loop landed then; the per-attempt span surface was deferred). Under call-level retry the OTel observer now emits one `openarmature.llm.complete` span per attempt, each carrying `openarmature.llm.attempt_index` (0-based, 0..N-1, and 0 for a no-retry call). An intermediate failed attempt's span carries ERROR status plus its error category and the request-side attributes; the final attempt's span carries the terminal outcome and, on success, the full response surface. A python-internal `LlmRetryAttemptEvent`, dispatched once per attempt, is the sole source of the OTel span; the terminal `LlmCompletionEvent` / `LlmFailedEvent` stay one per call (payload, latency, Langfuse Generation) and no longer drive the OTel span. Langfuse renders one terminal Generation per call, with the per-attempt detail on the OTel span surface only (a spec-side §8 clarification to pin this is tracked, non-blocking). `conformance.toml` flips proposal 0050 to `implemented`; the call-level fixtures 056-058 are driven through the provider plus OTel observer and the single-attempt observability fixture 057 is wired.
+
+### Changed
+
+- **Pinned spec advances v0.60.0 → v0.61.0** (proposal 0061, the detached-trace invocation span above). A single step this cycle; `conformance.toml` records proposal 0061 as `implemented`. Proposal 0050 needed no pin bump of its own (it was already within the pin from its v0.42.0 acceptance); its v0.14.0 `partial` entry flips to `implemented` with the per-attempt span surface above.
+
 ## [0.14.0] — 2026-06-17
 
 ### Added

conformance.toml

@@ -372,30 +372,34 @@ status = "implemented"
 since = "0.13.0"
 
 # Spec v0.42.0 (proposal 0050).  Retry & degradation primitives —
-# failure-isolation middleware (§6.3) + call-level retry (§7).  Both
-# primitives implemented across the v0.14.0 cycle:
-# FailureIsolationMiddleware (distinct FailureIsolatedEvent +
-# CaughtException) and the call-level ``retry`` parameter on
-# ``Provider.complete()`` — an in-call loop over transient §7 errors
-# reusing the §6.1 RetryConfig record.  ``partial`` because §7.1's
-# per-attempt span surface — N ``openarmature.llm.complete`` spans +
-# the ``openarmature.llm.attempt_index`` attribute — is DEFERRED: the
-# python LLM span is rendered from the typed event, which is
-# terminal-only per the graph-engine §6 mutual-exclusion contract, so
-# per-attempt spans require a dedicated within-call sub-event
-# (LlmRetryAttemptEvent) scoped to a future cycle.  Call-level retry
-# ships terminal-only: exactly one LlmCompletionEvent / LlmFailedEvent
-# per ``complete()`` call.  Failure-isolation conformance fixtures
-# (058-063) are all wired + passing: the FailureIsolatedEvent's
-# attempt_index reports the final / exhausting attempt per §6.3's
-# lineage-correlation rule (spec ruled this in the attempt-index coord
-# thread; RetryMiddleware now records the final attempt in a
-# terminal-attempt scope the outer isolation reads, rather than the
-# post-reset baseline).  ``partial`` is now solely about the
-# call-level-retry per-attempt span surface above.
+# failure-isolation middleware (§6.3) + call-level retry (§7),
+# including §7.1's per-attempt span surface.  Implemented across the
+# v0.14.0 + v0.15.0 cycles: FailureIsolationMiddleware (distinct
+# FailureIsolatedEvent + CaughtException) and the call-level ``retry``
+# parameter on ``Provider.complete()`` — an in-call loop over transient
+# §7 errors reusing the §6.1 RetryConfig record.  §7.1's per-attempt
+# span surface now ships: a call-level ``retry`` emits N
+# ``openarmature.llm.complete`` spans — one per attempt — each carrying
+# ``openarmature.llm.attempt_index`` (0-based, call-level, independent
+# of the node-level attempt_index).  A python-internal
+# LlmRetryAttemptEvent dispatched once per attempt is the SOLE source of
+# the OTel LLM span (including single no-retry calls, at index 0); the
+# terminal LlmCompletionEvent / LlmFailedEvent stay one-per-call
+# (payload, latency, Langfuse Generation, fixture-072 mutual exclusion)
+# and no longer drive the OTel span.  Langfuse renders one terminal
+# Generation per call.  llm-provider fixtures 056-058 (per-attempt
+# spans) are validated in tests/unit/test_observability_otel.py through
+# the provider + OTel observer; observability fixture 057
+# (single-attempt attempt_index) is wired in test_observability.
+# Failure-isolation fixtures (058-063) are all wired + passing: the
+# FailureIsolatedEvent's attempt_index reports the final / exhausting
+# attempt per §6.3's lineage-correlation rule (spec ruled this in the
+# attempt-index coord thread; RetryMiddleware now records the final
+# attempt in a terminal-attempt scope the outer isolation reads, rather
+# than the post-reset baseline).
 [proposals."0050"]
-status = "partial"
-since = "0.14.0"
+status = "implemented"
+since = "0.15.0"
 
 # Spec v0.43.0 (proposal 0051).  Langfuse trace.input/trace.output
 # implementation-surface caveat.  Purely textual: documents that the

docs/concepts/observability.md

@@ -659,14 +659,17 @@ as nested spans.
 
 When an `OpenAIProvider` (or any [custom Provider](../model-providers/authoring.md)
 that wires the dispatch hook) is used inside a graph with `OTelObserver`
-attached, each `provider.complete()` call emits a dedicated span named
-`openarmature.llm.complete`, parented under the calling node's span.
-The span carries two attribute families.
+attached, each `provider.complete()` attempt emits a dedicated span
+named `openarmature.llm.complete`, parented under the calling node's
+span. A call without retry emits one span; a call-level `retry=` that
+retries emits [one span per attempt](#per-attempt-spans-under-call-level-retry).
+Each span carries two attribute families.
 
 **`openarmature.llm.*` (always on).** The framework's canonical
 namespace: model identifier, finish reason, token counts, prompt
-identity from `with_active_prompt(...)`, error category on failure.
-Set unconditionally whenever the LLM span itself emits.
+identity from `with_active_prompt(...)`, error category on failure, and
+`openarmature.llm.attempt_index` (the 0-based call-level attempt
+counter). Set unconditionally whenever the LLM span itself emits.
 
 **`gen_ai.*` (OpenTelemetry GenAI semantic conventions, default on).**
 Cross-vendor attribute names every LLM-aware backend reads
@@ -702,6 +705,28 @@ when an external auto-instrumentation library (OpenInference,
 `opentelemetry-instrumentation-openai`) is already the canonical
 source on your stack.
 
+#### Per-attempt spans under call-level retry
+
+[Call-level retry](llms.md#retrying-transient-failures)
+(`provider.complete(retry=...)`) retries transient provider errors
+inside a single call. Each attempt emits its own
+`openarmature.llm.complete` span tagged with
+`openarmature.llm.attempt_index` (0-based). A call that succeeds on the
+first try emits one span at `attempt_index` 0; a call that fails twice
+transiently before succeeding emits three spans (indices 0, 1, 2). Each
+failed attempt's span carries `ERROR` status plus
+`openarmature.error.category`; the final attempt's span carries the
+terminal outcome (`OK` on success, `ERROR` on an exhausted or
+non-transient failure).
+
+`openarmature.llm.attempt_index` is the **call-level** attempt counter,
+[independent of the node-level `attempt_index`](llms.md#call-level-vs-node-level-retry):
+the former counts attempts inside one `complete()` call, the latter
+counts node re-executions driven by retry *middleware*. A node retried
+once by middleware, each execution calling a provider that itself
+retries once, produces node `attempt_index` 0/1 and, within each,
+call-level `attempt_index` 0/1.
+
 ### LLM payload attributes
 
 By default, LLM spans do **not** carry the messages sent or the
@@ -834,6 +859,14 @@ correctly; doing it from a `SpanProcessor.on_end` callback does
 not, because the framework has already called `span.end()` and the
 OTel SDK silently drops `set_attribute` on ended spans.
 
+For the `openarmature.llm.complete` span the close event is an
+`LlmRetryAttemptEvent` (one per attempt) rather than a `NodeEvent`;
+that is the per-attempt event the observer renders the LLM span from.
+An enricher scoped to that span (`span.name ==
+"openarmature.llm.complete"`) can read the attempt's outcome straight
+off it: `event.llm_attempt_index`, `event.error_category`,
+`event.usage`, `event.finish_reason`, and so on.
+
 Exceptions raised by an enricher are caught and warned, never
 propagated.
 
@@ -880,6 +913,16 @@ via `current_dispatch()`. See
 [Authoring providers](../model-providers/authoring.md) for the full
 pattern.
 
+Under [call-level retry](#per-attempt-spans-under-call-level-retry) the
+bundled `OpenAIProvider` additionally dispatches a python-internal
+`LlmRetryAttemptEvent` once per attempt; that is the event the OTel
+observer renders each per-attempt span from (including the lone attempt
+of a no-retry call, at index 0). The terminal `LlmCompletionEvent` /
+`LlmFailedEvent` above are unchanged: still one per call, still the
+stable surface for per-call consumption (token accounting, failure
+tracking). An observer that only cares about per-call outcomes can
+ignore `LlmRetryAttemptEvent`.
+
 #### Legacy sentinel-namespace pattern (compatibility surface)
 
 `openarmature.observability.LLM_NAMESPACE` and

src/openarmature/graph/__init__.py

@@ -44,6 +44,7 @@
     InvocationStartedEvent,
     LlmCompletionEvent,
     LlmFailedEvent,
+    LlmRetryAttemptEvent,
     MetadataAugmentationEvent,
     NodeEvent,
 )
@@ -101,6 +102,7 @@
     "InvocationStartedEvent",
     "LlmCompletionEvent",
     "LlmFailedEvent",
+    "LlmRetryAttemptEvent",
     "MappingReferencesUndeclaredField",
     "MetadataAugmentationEvent",
     "Middleware",

src/openarmature/graph/events.py

@@ -656,6 +656,72 @@ class LlmFailedEvent:
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
 
 
+# Python-internal per-attempt LLM event. NOT a spec-normative event type
+# (unlike LlmCompletionEvent / LlmFailedEvent): it is the observer-side
+# vehicle for the observability §5.5 per-attempt span surface under
+# llm-provider §7.1 call-level retry. One is dispatched per in-call
+# attempt (including the single attempt of a no-retry call); the OTel
+# observer renders one openarmature.llm.complete span from each, while
+# the terminal LlmCompletionEvent / LlmFailedEvent stay one-per-call
+# (payload/latency, Langfuse mapping, the fixture-072 mutual exclusion).
+@dataclass(frozen=True)
+class LlmRetryAttemptEvent:
+    """One LLM-call attempt delivered to observers for per-attempt span
+    rendering.
+
+    Carries the full request-side surface plus that attempt's outcome.
+    ``error_category`` discriminates the outcome: ``None`` for a
+    successful attempt (the response-side fields are populated), a
+    category string for a failed attempt (the response-side fields are
+    ``None`` — no response was received).
+
+    Field set:
+
+    - ``llm_attempt_index``: the call-level retry-attempt index, ``0``
+      for the first attempt and ``0..N-1`` across the N attempts of a
+      call-level retry. Distinct from ``attempt_index`` (the node-level
+      retry index used for calling-span resolution); the two are
+      independent.
+    - identity / scoping (``invocation_id`` ... ``call_id``) and the
+      request side (``input_messages`` / ``request_params`` /
+      ``request_extras`` / ``active_prompt`` / ``active_prompt_group``)
+      mirror :class:`LlmCompletionEvent`, carried on every attempt.
+    - response side (``response_id`` / ``response_model`` / ``usage`` /
+      ``finish_reason`` / ``output_content``): populated on a successful
+      attempt; ``None`` on a failed attempt.
+    - failure side (``error_category`` / ``error_message`` /
+      ``error_type``): populated on a failed attempt; ``None`` on a
+      successful one.
+    """
+
+    invocation_id: str
+    correlation_id: str | None
+    node_name: str
+    namespace: tuple[str, ...]
+    attempt_index: int
+    fan_out_index: int | None
+    branch_name: str | None
+    provider: str
+    model: str
+    call_id: str
+    llm_attempt_index: int
+    latency_ms: float | None
+    input_messages: list[dict[str, Any]]
+    request_params: Mapping[str, Any]
+    request_extras: Mapping[str, Any]
+    active_prompt: Any
+    active_prompt_group: Any
+    response_id: str | None = None
+    response_model: str | None = None
+    usage: "Usage | None" = None
+    finish_reason: str | None = None
+    output_content: str | None = None
+    error_category: str | None = None
+    error_message: str | None = None
+    error_type: str | None = None
+    caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+
+
 # Spec: pipeline-utilities §6.3 cause chain (proposal 0068). A ``carrier``
 # link is a graph-engine §4 ``node_exception`` wrapper the engine applies at a
 # non-node placement (§9.7 instance / §11.7 branch / §9.6 / §11.6 parent-node
@@ -758,6 +824,7 @@ class FailureIsolatedEvent:
     "InvocationStartedEvent",
     "LlmCompletionEvent",
     "LlmFailedEvent",
+    "LlmRetryAttemptEvent",
     "MetadataAugmentationEvent",
     "NodeEvent",
     "ParallelBranchesEventConfig",

src/openarmature/graph/observer.py

@@ -40,6 +40,7 @@
     InvocationStartedEvent,
     LlmCompletionEvent,
     LlmFailedEvent,
+    LlmRetryAttemptEvent,
     MetadataAugmentationEvent,
     NodeEvent,
 )
@@ -55,6 +56,9 @@
 # typed LLM provider call event, dispatched on every successful LLM
 # completion), LlmFailedEvent (proposal 0058 typed LLM failure event,
 # dispatched alongside the §7 exception when provider.complete raises),
+# LlmRetryAttemptEvent (proposal 0050 per-attempt LLM span event,
+# python-internal, dispatched once per in-call attempt under call-level
+# retry to drive the per-attempt OTel span surface),
 # and FailureIsolatedEvent (proposal 0050 §6.3 framework-emitted event,
 # dispatched by FailureIsolationMiddleware when it catches an exception
 # escaping the inner chain and substitutes a degraded partial update).
@@ -65,6 +69,7 @@
     | InvocationCompletedEvent
     | LlmCompletionEvent
     | LlmFailedEvent
+    | LlmRetryAttemptEvent
     | FailureIsolatedEvent
 )