Add 0084 event lineage chains and OTel exact-match (#240)

* Add 0084 event lineage chains and OTel exact-match

Proposal 0084 (nested-fan-out span lineage), first of the Wave 3 PRs.

Add fan_out_index_chain / branch_name_chain (the enclosing fan-out /
branch lineage, outermost to innermost) to the provider and tool
observer events and the framework failure-isolation event, mirroring
the fields already on NodeEvent. They are populated from the enclosing-
lineage ContextVars at each event's build site.

Key the OTel node-span registry by those chains in addition to the
innermost scalars, so an inner node running under two concurrent outer
fan-out instances no longer collides on the span key, and the provider
span parents under its own lineage-disambiguated calling-node span
rather than a coincidentally-indexed sibling in the other outer
instance. The fan-out node-span finder used for dispatch-span parenting
is likewise made chain-aware.

Un-defer observability fixture 132 (nested keying + nested-LLM exact
match) with a runner that forces the concurrent interleaving the fix
relies on. The orphan-fallback path (a provider call with no open
calling-node span) and the Langfuse resolver are unchanged; they follow
in a later PR. Also fix a pre-existing test-isolation leak where three
tests set invocation metadata outside an invocation without restoring
the ContextVar.

* Comment the rendezvous timeout fallback (CodeQL empty-except)

Author: chris-colinsky

conformance.toml

@@ -834,10 +834,11 @@ note = "Advisory, observability-only per-prompt token budget. PR A (completion p
 
 # Spec v0.81.0 (proposal 0084).  Nested-fan-out span lineage chain
 # (graph-engine §6 fan_out_index_chain / branch_name_chain +
-# observability §4 / §5.5 lineage-resolved parent).  Not-yet; graph-
-# engine fixture 039 and observability fixtures 132-134 defer with it.
+# observability §4 / §5.5 lineage-resolved parent).
 [proposals."0084"]
-status = "not-yet"
+status = "partial"
+since = "0.17.0"
+note = "The event-surface chains + the nested-LLM exact-match shipped. graph-engine §6: fan_out_index_chain / branch_name_chain (mirroring NodeEvent) now ride the provider/tool events (LlmCompletionEvent / LlmFailedEvent / LlmRetryAttemptEvent / EmbeddingEvent / EmbeddingFailedEvent / RerankEvent / RerankFailedEvent / ToolCallEvent / ToolCallFailedEvent) and the framework FailureIsolatedEvent, populated from the enclosing-lineage ContextVars at call time. observability §4.3 / §5.5: the OTel node-span key carries the scalars AND the enclosing chains, so an inner node under two concurrent outer fan-out instances no longer collides (no dropped / mis-closed spans), and the LLM-parent exact-match (§5.5 Lineage-resolved parent, fallback 1) resolves under the lineage-disambiguated calling-node span rather than the innermost scalar. observability fixture 132 (nested keying + nested-LLM exact-match, OTel) is un-deferred and driven. partial because the orphan fallback (a wrapper / middleware-issued provider call with no open calling-node span, §5.5) is not yet generalized to nearest-open-ancestor-at-any-depth and the Langfuse resolver is unchanged: observability fixtures 133 (OTel orphan) + 134 (Langfuse) and the graph-engine event-chain fixture 039 tightening defer with the orphan-call conformance-adapter primitive (a later PR)."
 
 # Spec v0.80.0 (proposal 0085).  Nested-fan-out checkpoint resume
 # lineage (pipeline-utilities §10.11 enclosing_fan_out_lineage).

src/openarmature/graph/events.py

@@ -602,6 +602,16 @@ class LlmCompletionEvent:
     # Defaulted (default_factory) so existing kwargs-constructors that
     # predate this field keep working; the provider always populates it.
     output_tool_calls: list["ToolCall"] = field(default_factory=list["ToolCall"])
+    # Proposal 0084 (graph-engine §6, spec v0.81.0): per-depth enclosing
+    # fan-out / branch lineage chains, parallel to ``namespace`` -- position
+    # ``i`` is the fan_out_index (or branch_name) at the dispatch boundary
+    # leading to namespace depth ``i+1``, or ``None`` at a non-fan-out /
+    # non-branch boundary. Mirror of NodeEvent's chains; observers resolve the
+    # lineage-disambiguated span parent from these rather than the innermost
+    # scalar (which coincides across concurrent enclosing instances). Empty for
+    # a top-level (non-nested) call; the scalars above carry the innermost value.
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: realizes proposal 0058's second spec-normatively-typed event
@@ -709,6 +719,10 @@ class LlmFailedEvent:
     # (which carries the response-side usage) the budget eval fires as it does
     # on a completion; None-carrying every other category leaves it inert.
     token_budget: Any = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Python-internal per-attempt LLM event. NOT a spec-normative event type
@@ -799,6 +813,10 @@ class LlmRetryAttemptEvent:
     # tool-call span attributes from this field (the per-attempt event is
     # the LLM-span source).
     output_tool_calls: list["ToolCall"] = field(default_factory=list["ToolCall"])
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: realizes graph-engine §6 + observability §5.5.9 -- the typed
@@ -877,6 +895,10 @@ class EmbeddingEvent:
     active_prompt_group: Any
     call_id: str
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: the failure sibling of EmbeddingEvent (proposal 0059). Dispatched
@@ -926,6 +948,10 @@ class EmbeddingFailedEvent:
     error_message: str
     error_type: str | None = None
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: realizes graph-engine §6 -- the typed RerankEvent / RerankFailedEvent
@@ -1010,6 +1036,10 @@ class RerankEvent:
     active_prompt_group: Any
     call_id: str
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: the failure sibling of RerankEvent (proposal 0060). Dispatched
@@ -1062,6 +1092,10 @@ class RerankFailedEvent:
     error_message: str
     error_type: str | None = None
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: realizes pipeline-utilities §6.3 failure-isolation middleware
@@ -1107,6 +1141,15 @@ class FailureIsolatedEvent:
     pre_state: Any
     post_state: Mapping[str, Any]
     caught_exception: CaughtException
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage chains
+    # parallel to ``namespace`` (see LlmCompletionEvent). Carried for surface
+    # consistency with the provider events. The marker's calling-node span is
+    # already closed by delivery time (its completed event precedes this one on
+    # the serial queue), so it parents under the invocation span, not via the
+    # lineage exact-match -- the chains only disambiguate in the rare case that
+    # calling-node span is still open.
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: realizes graph-engine §6 tool-execution observer events
@@ -1169,6 +1212,10 @@ class ToolCallEvent:
     result: Any
     latency_ms: float | None
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 # Spec: the failure variant (proposal 0063). Mirrors ToolCallEvent's
@@ -1217,6 +1264,10 @@ class ToolCallFailedEvent:
     error_type: str | None
     error_message: str
     caller_invocation_metadata: Mapping[str, AttributeValue] | None = None
+    # Proposal 0084 (spec v0.81.0): enclosing fan-out / branch lineage
+    # chains parallel to ``namespace`` (see LlmCompletionEvent).
+    fan_out_index_chain: tuple[int | None, ...] = ()
+    branch_name_chain: tuple[str | None, ...] = ()
 
 
 __all__ = [

src/openarmature/graph/middleware/failure_isolation.py

@@ -45,8 +45,10 @@
     _set_terminal_attempt_index,
     current_attempt_index,
     current_branch_name,
+    current_branch_name_chain,
     current_dispatch,
     current_fan_out_index,
+    current_fan_out_index_chain,
     current_namespace_prefix,
 )
 
@@ -233,6 +235,9 @@ def _emit_event(self, state: Any, classification: CaughtException, degraded: Map
                 attempt_index=attempt_index,
                 fan_out_index=current_fan_out_index(),
                 branch_name=current_branch_name(),
+                # Proposal 0084: enclosing fan-out / branch lineage chains.
+                fan_out_index_chain=current_fan_out_index_chain(),
+                branch_name_chain=current_branch_name_chain(),
                 pre_state=state,
                 post_state=degraded,
                 caught_exception=classification,

src/openarmature/llm/providers/openai.py

@@ -72,9 +72,11 @@
 from openarmature.observability.correlation import (
     current_attempt_index,
     current_branch_name,
+    current_branch_name_chain,
     current_correlation_id,
     current_dispatch,
     current_fan_out_index,
+    current_fan_out_index_chain,
     current_invocation_id,
     current_namespace_prefix,
 )
@@ -702,6 +704,9 @@ def _build_llm_completion_event(
             attempt_index=current_attempt_index(),
             fan_out_index=current_fan_out_index(),
             branch_name=current_branch_name(),
+            # Proposal 0084: enclosing fan-out / branch lineage chains.
+            fan_out_index_chain=current_fan_out_index_chain(),
+            branch_name_chain=current_branch_name_chain(),
             provider=self._genai_system,
             model=self.model,
             response_id=response.response_id,
@@ -792,6 +797,9 @@ def _build_llm_failed_event(
             attempt_index=current_attempt_index(),
             fan_out_index=current_fan_out_index(),
             branch_name=current_branch_name(),
+            # Proposal 0084: enclosing fan-out / branch lineage chains.
+            fan_out_index_chain=current_fan_out_index_chain(),
+            branch_name_chain=current_branch_name_chain(),
             provider=self._genai_system,
             model=self.model,
             latency_ms=latency_ms,
@@ -857,6 +865,9 @@ def _build_llm_retry_attempt_event(
             "attempt_index": current_attempt_index(),
             "fan_out_index": current_fan_out_index(),
             "branch_name": current_branch_name(),
+            # Proposal 0084: enclosing fan-out / branch lineage chains.
+            "fan_out_index_chain": current_fan_out_index_chain(),
+            "branch_name_chain": current_branch_name_chain(),
             "provider": self._genai_system,
             "model": self.model,
             "call_id": call_id,

src/openarmature/observability/otel/observer.py

@@ -137,7 +137,28 @@
 # across sibling parallel-branches branches (pipeline-utilities §11);
 # without it the two inner ``ask`` nodes of two branches with the
 # same namespace + fan_out_index would collide on the same key.
-_StackKey = tuple[tuple[str, ...], int, int | None, str | None]
+# Proposal 0084 (spec v0.81.0): a node span is keyed by the innermost scalars
+# AND the full enclosing fan-out / branch lineage chains. The chains are the
+# addition: an inner node under two concurrent outer fan-out instances shares
+# the same innermost scalar across the outer instances, so a scalar-only key
+# would collide (last-writer-wins drops / mis-closes the second) and the
+# LLM-parent exact-match would resolve to a sibling's span. The scalars are
+# RETAINED (not replaced by the chains) because ``_key_for`` is also used to
+# LOOK UP a callable-parallel-branch event in ``open_spans`` on the publish
+# path (``_publish_active_span``): a callable branch carries its branch_name on
+# the event but never extends branch_name_chain (no subgraph descent), so on
+# the chains alone its key would EQUAL its own parallel-branches NODE's key --
+# the lookup would return that NODE instead of missing, and the branch's
+# per-branch dispatch span would never be synthesized (verified: blanking the
+# scalars drops the dispatch span). The scalar branch_name (key[3]) keeps the
+# two distinct. Past that lookup the scalars are redundant with the chains for
+# storage uniqueness (each equals its chain's innermost non-None entry); the
+# merged spec keys the driving span chains-only, so the retained scalar is a
+# harmless superset kept for the callable-branch lookup. The common
+# single-level case keys as before plus the (empty / length-1) chains.
+_StackKey = tuple[
+    tuple[str, ...], int, int | None, str | None, tuple[int | None, ...], tuple[str | None, ...]
+]
 
 # Lineage-aware dispatch keys (proposal 0045): the fan-out / pb NODE namespace
 # prefix plus the fan-out instance index / branch name chain slices along the
@@ -1304,7 +1325,7 @@ def _collect_augmentation_targets(
         #   identified structurally by their presence in the
         #   parent_node_name caches)
         for key, open_span in inv_state.open_spans.items():
-            ns, _ai, _fi, _bn = key
+            ns = key[0]
             if ns != aug_ns and not is_strict_prefix(ns, aug_ns):
                 continue
             # Shared-parent check: a fan-out NODE or parallel-branches NODE
@@ -1587,6 +1608,8 @@ def _handle_typed_llm_retry_attempt(self, event: LlmRetryAttemptEvent) -> None:
             calling_attempt_index=event.attempt_index,
             calling_fan_out_index=event.fan_out_index,
             calling_branch_name=event.branch_name,
+            calling_fan_out_index_chain=event.fan_out_index_chain,
+            calling_branch_name_chain=event.branch_name_chain,
         )
         attrs: dict[str, Any] = {
             "openarmature.llm.model": event.model,
@@ -1808,6 +1831,8 @@ def _handle_tool_call(self, event: ToolCallEvent | ToolCallFailedEvent) -> None:
             calling_attempt_index=event.attempt_index,
             calling_fan_out_index=event.fan_out_index,
             calling_branch_name=event.branch_name,
+            calling_fan_out_index_chain=event.fan_out_index_chain,
+            calling_branch_name_chain=event.branch_name_chain,
         )
         attrs: dict[str, Any] = {"openarmature.tool.name": event.tool_name}
         if event.tool_call_id is not None:
@@ -1926,6 +1951,8 @@ def _handle_embedding(self, event: EmbeddingEvent | EmbeddingFailedEvent) -> Non
             calling_attempt_index=event.attempt_index,
             calling_fan_out_index=event.fan_out_index,
             calling_branch_name=event.branch_name,
+            calling_fan_out_index_chain=event.fan_out_index_chain,
+            calling_branch_name_chain=event.branch_name_chain,
         )
         attrs: dict[str, Any] = {}
         cid = current_correlation_id()
@@ -2036,6 +2063,8 @@ def _handle_rerank(self, event: RerankEvent | RerankFailedEvent) -> None:
             calling_attempt_index=event.attempt_index,
             calling_fan_out_index=event.fan_out_index,
             calling_branch_name=event.branch_name,
+            calling_fan_out_index_chain=event.fan_out_index_chain,
+            calling_branch_name_chain=event.branch_name_chain,
         )
         attrs: dict[str, Any] = {}
         cid = current_correlation_id()
@@ -2147,6 +2176,8 @@ def _handle_failure_isolated(self, event: FailureIsolatedEvent) -> None:
             calling_attempt_index=event.attempt_index,
             calling_fan_out_index=event.fan_out_index,
             calling_branch_name=event.branch_name,
+            calling_fan_out_index_chain=event.fan_out_index_chain,
+            calling_branch_name_chain=event.branch_name_chain,
         )
         attrs: dict[str, Any] = {
             "openarmature.failure_isolation.event_name": event.event_name,
@@ -2182,16 +2213,25 @@ def _resolve_llm_parent(
         calling_attempt_index: int,
         calling_fan_out_index: int | None,
         calling_branch_name: str | None,
+        calling_fan_out_index_chain: tuple[int | None, ...],
+        calling_branch_name_chain: tuple[str | None, ...],
     ) -> object:
         """Look up the calling node's span using the calling-node
         identity, falling back through the per-instance fan-out dispatch
         span, subgraph dispatch / detached root, and the invocation span."""
-        # 1. Direct match on the calling node's ``_StackKey``.
+        # 1. Direct match on the calling node's ``_StackKey`` -- the
+        #    lineage-disambiguated calling-node span (proposal 0084 §5.5
+        #    "Lineage-resolved parent"). Keyed by the full enclosing fan-out /
+        #    branch chain so a nested provider call parents under ITS OWN
+        #    calling node's span, not a sibling in another concurrent outer
+        #    instance that shares the innermost scalar fan_out_index.
         calling_key: _StackKey = (
             calling_namespace_prefix,
             calling_attempt_index,
             calling_fan_out_index,
             calling_branch_name,
+            calling_fan_out_index_chain,
+            calling_branch_name_chain,
         )
         calling = inv_state.open_spans.get(calling_key)
         if calling is not None:
@@ -2264,7 +2304,17 @@ def _open_invocation_span(
         self._invocation_span[invocation_id] = _OpenSpan(span=span)
 
     def _key_for(self, event: NodeEvent) -> _StackKey:
-        return (event.namespace, event.attempt_index, event.fan_out_index, event.branch_name)
+        # Proposal 0084: scalars (retained) + enclosing lineage chains (added),
+        # so concurrent nested fan-out node spans don't collide while a callable
+        # parallel-branch still stays distinct from its parallel-branches node.
+        return (
+            event.namespace,
+            event.attempt_index,
+            event.fan_out_index,
+            event.branch_name,
+            event.fan_out_index_chain,
+            event.branch_name_chain,
+        )
 
     def _resolve_parent_context(
         self,
@@ -2955,23 +3005,21 @@ def _find_fan_out_node_span(
         self, inv_state: _InvState, prefix: tuple[str, ...], event: NodeEvent
     ) -> _OpenSpan | None:
         """Find the currently-open fan-out / pb NODE span at ``prefix`` on the
-        given event's ENCLOSING lineage. When the NODE is itself nested inside an
-        outer fan-out instance / branch, several instances of the same NODE
-        namespace are open at once under concurrency, so a namespace-only scan
-        would bind the wrong one. The NODE's own event carries the instance /
-        branch it sits in as its fan_out_index / branch_name (key[2] / key[3]);
-        that equals the synthesizing event's chain entry at the level above this
-        NODE."""
-        n = len(prefix)
-        enclosing_fi = (
-            event.fan_out_index_chain[n - 2] if n >= 2 and n - 2 < len(event.fan_out_index_chain) else None
-        )
-        enclosing_bn = (
-            event.branch_name_chain[n - 2] if n >= 2 and n - 2 < len(event.branch_name_chain) else None
-        )
+        given event's ENCLOSING lineage. When the NODE is itself nested inside
+        one or more outer fan-out instances / branches, several instances of the
+        same NODE namespace are open at once under concurrency, so a
+        namespace-only scan would bind the wrong one. Disambiguate by the full
+        enclosing chain (proposal 0084): the NODE span sits on the synthesizing
+        event's call-stack ancestor path iff its stored lineage chain is a
+        prefix of the event's -- the same lineage-boundary rule the augmentation
+        scoping uses (``_span_chain_on_path``). Matching the innermost scalar
+        alone is ambiguous at >=3 levels, where an intermediate instance index
+        repeats across concurrent outer instances (both would match the scalar
+        but only one is on the event's path)."""
         for key, open_span in inv_state.open_spans.items():
-            ns, _attempt, fan_idx, bn = key
-            if ns == prefix and fan_idx == enclosing_fi and bn == enclosing_bn:
+            if key[0] == prefix and _span_chain_on_path(
+                open_span, event.fan_out_index_chain, event.branch_name_chain
+            ):
                 return open_span
         return None