Add adapter crash-injection and mock cause (0070) (#163)

* Add adapter crash-injection and mock cause (0070)

Implement proposal 0070's two conformance-adapter capabilities: a
`crash_injection` directive (after_fan_out_instance / after_node)
that simulates a checkpoint-boundary crash with no asserted first-run
outcome, and a recursive mock `cause` that chains a flaky failure to
an originating error. Wire conformance fixtures 067 (crash-injection
fan-out resume) and 068 (failure-mock cause chain).

A general per-instance execution recorder lets plain-node fan-outs
report executed / skipped on resume, consulted as a fallback so the
existing flaky_per_index fixtures are unchanged. The after_node
boundary has a unit test (no fixture exercises it).

Advance the pinned spec to v0.58.0; test-vocabulary only, no library
behavior change.

* Harden crash-injection directive handling (0070)

Address PR review on the crash_injection plumbing:

- crash_injection now defines the crash boundary exclusively; when set,
  the legacy fan_out.abort_after_instance directive is ignored, so an
  instance-boundary and a node-boundary abort cannot both be active.
- A non-dict crash_injection is coerced to None in the first-run
  handler, matching _find_crash_injection's mapping check, so a
  malformed directive no longer triggers the swallow path.
- after_fan_out_instance now honors its documented node field: the
  parser threads the node name and _maybe_abort filters fan_out_progress
  by it, so a multi-fan-out graph aborts after the named node.
  Single-fan-out fixtures and the legacy path are unchanged.

Author: chris-colinsky

CHANGELOG.md

@@ -18,7 +18,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
 - **Observer privacy flag `disable_llm_payload` renamed to `disable_provider_payload`** (proposal 0059, observability §5.5.4, spec v0.54.0). The observer-level flag on both bundled observers (`OTelObserver` and `LangfuseObserver`) is renamed, and its scope broadens from LLM-completion payload to any provider-call payload (LLM completion today; embedding and rerank when those land). This is a breaking change to both observer constructors: config passing `disable_llm_payload=True` (or `False`) updates to `disable_provider_payload=...` with no other change. The default stays `True` (payload suppressed), and the gating behavior for `LlmCompletionEvent` / `LlmFailedEvent` rendering is unchanged at every existing site. The rename is the only part of proposal 0059 adopted this cycle: the retrieval-provider capability itself (the `EmbeddingProvider` protocol, the `EmbeddingEvent` / `EmbeddingFailedEvent` typed variants, and the embedding span / observation mapping) is not yet implemented and rides as `not-yet` in `conformance.toml`. The §5.5.4 rename touches existing LLM-payload gating, so it lands with the pin.
 - **Fan-out failure-isolation degrade contribution implemented** (proposal 0066, pipeline-utilities §9.3 / §9.8 / §11.7, spec v0.56.0). When `FailureIsolationMiddleware` degrades a fan-out instance, that instance is a success whose contribution is its `degraded_update`, read in subgraph-field-name space and never merged onto the failed instance's pre-failure state. This also fixes a latent bug: an instance `degraded_update`'s `extra_outputs` values were previously looked up by the parent field name and silently dropped (`collect_field` was unaffected). A static `degraded_update` that omits the node's `collect_field` is now a compile-time error (`FanOutDegradedUpdateMissingCollectField`); a callable `degraded_update` that omits it yields a graceful null slot rather than raising, preserving one collection slot per item. The parallel-branches counterpart (a branch `degraded_update` omitting a projected `outputs` field skips that field) was already correct as of the parallel-branches fix above and is now pinned by fixture 065. Success-path and resume behavior for correctly-configured fan-outs is unchanged.
 - **Failure-isolation events carry the full structured cause chain** (proposal 0068, pipeline-utilities §6.3, spec v0.57.0). `FailureIsolatedEvent.caught_exception` gains a `chain`: an ordered list of `CauseLink` records (each carrying `category`, `message`, and a `carrier` flag), from the caught exception (outermost) to the originating raise (innermost), with graph-engine `node_exception` carrier wrappers flagged `carrier=True`. The existing `category` and `message` are retained and redefined as a derivation over the chain: the category of the outermost non-carrier link whose category is a non-empty string (else `category` is `null` and `message` is the outermost non-carrier link's message). This supersedes proposal 0065's single "originating cause" representation, which was ambiguous once the post-carrier chain held more than one non-carrier link; the derivation reproduces 0065's single-carrier values, so fixture 064 is unchanged. A new `CauseLink` type is exported from `openarmature.graph`. The bundled OTel and Langfuse observers continue to render the derived `category`; surfacing the full chain is left to custom observers. The change is additive to the event shape, and catch/degrade behavior is unchanged. Conformance fixture 066 (three cases: an instance-site carrier chain, a node-level single non-carrier link, and an uncategorized null-category cause) passes.
-- **Pinned spec advances v0.53.0 → v0.57.0 across the v0.14.0 cycle**, in four steps: v0.54.0 (proposal 0059, the observer-flag rename above), v0.55.1 (proposal 0065 above; the v0.55.1 patch also carries an observability §11 span-links text reconciliation that narrows an *Out of scope* bullet, with no python-observable change), v0.56.0 (proposal 0066, the fan-out degrade contribution above), and v0.57.0 (proposal 0068, the failure-isolation cause chain above). `conformance.toml` records 0065, 0066, and 0068 as `implemented` and 0059 as `not-yet` (only its cross-spec flag rename was adopted).
+- **Pinned spec advances v0.53.0 → v0.58.0 across the v0.14.0 cycle**, in five steps: v0.54.0 (proposal 0059, the observer-flag rename above), v0.55.1 (proposal 0065 above; the v0.55.1 patch also carries an observability §11 span-links text reconciliation that narrows an *Out of scope* bullet, with no python-observable change), v0.56.0 (proposal 0066, the fan-out degrade contribution above), v0.57.0 (proposal 0068, the failure-isolation cause chain above), and v0.58.0 (proposal 0070, conformance-adapter crash-injection and cause-chaining test vocabulary: a `crash_injection` directive and a recursive mock `cause`, with conformance fixtures 067 and 068, no library behavior change). `conformance.toml` records 0065, 0066, 0068, and 0070 as `implemented` and 0059 as `not-yet` (only its cross-spec flag rename was adopted).
 
 ### Fixed
 

conformance.toml

@@ -32,7 +32,7 @@
 
 [manifest]
 implementation = "openarmature-python"
-spec_pin = "v0.57.0"
+spec_pin = "v0.58.0"
 
 # Status values:
 #   implemented   — shipped behavior matches the proposal's contract
@@ -648,3 +648,18 @@ since = "0.14.0"
 [proposals."0068"]
 status = "implemented"
 since = "0.14.0"
+
+# Spec v0.58.0 (proposal 0070).  Conformance-adapter crash/resume vocabulary,
+# crash-injection, and cause-chaining (conformance-adapter §5.1 / §5.6 / §5.8).
+# Two new adapter capabilities: ``crash_injection`` (``after_fan_out_instance``
+# + ``after_node``) simulates a crash at a checkpoint boundary independent of
+# an instance failure, and a recursive mock ``cause`` chains a failure mock's
+# raised error to an originating cause.  The crash/resume + saved-record +
+# resume-outcome directives the proposal formalizes were already implemented.
+# Fixture 067 (crash-injection fan-out resume) drives after_fan_out_instance
+# end-to-end; after_node has a unit test (no fixture exercises it).  Fixture
+# 068 (failure-mock cause chain) pins 0068's outermost-wins derivation via the
+# mock ``cause``.
+[proposals."0070"]
+status = "implemented"
+since = "0.14.0"

openarmature-spec

@@ -63,7 +63,7 @@ Specification = "https://github.com/LunarCommand/openarmature-spec"
 openarmature = "openarmature.cli:main"
 
 [tool.openarmature]
-spec_version = "0.57.0"
+spec_version = "0.58.0"
 
 [dependency-groups]
 dev = [

pyproject.toml

@@ -1,6 +1,6 @@
 # OpenArmature — Agent documentation
 
-*This is the agent guide bundled with the openarmature Python package, version 0.13.0 (spec v0.57.0). For the full docs site see [openarmature.ai](https://openarmature.ai). For the canonical spec text see [openarmature.org/capabilities](https://openarmature.org/capabilities/). For project-specific conventions for the code you're editing, see the host project's `AGENTS.md` or `CLAUDE.md`.*
+*This is the agent guide bundled with the openarmature Python package, version 0.13.0 (spec v0.58.0). For the full docs site see [openarmature.ai](https://openarmature.ai). For the canonical spec text see [openarmature.org/capabilities](https://openarmature.org/capabilities/). For project-specific conventions for the code you're editing, see the host project's `AGENTS.md` or `CLAUDE.md`.*
 
 ## TL;DR
 
@@ -10,7 +10,7 @@ OpenArmature is a workflow framework for LLM pipelines and tool-calling agents:
 
 ## Capability contracts
 
-_Sourced from openarmature-spec v0.57.0. Each entry below reproduces §1 (Purpose) and §2 (Concepts) of the capability's `spec.md` verbatim — including additions from accepted proposals that this Python implementation may not yet ship. For per-proposal implementation status (implemented / partial / textual-only / not-yet), see the `conformance.toml` manifest at the repo root. For the full spec text (execution model, error semantics, determinism, observer hooks, etc.) see the linked docs site._
+_Sourced from openarmature-spec v0.58.0. Each entry below reproduces §1 (Purpose) and §2 (Concepts) of the capability's `spec.md` verbatim — including additions from accepted proposals that this Python implementation may not yet ship. For per-proposal implementation status (implemented / partial / textual-only / not-yet), see the `conformance.toml` manifest at the repo root. For the full spec text (execution model, error semantics, determinism, observer hooks, etc.) see the linked docs site._
 
 ### Capability: `graph-engine`
 

src/openarmature/AGENTS.md

@@ -25,7 +25,7 @@
 """
 
 __version__ = "0.13.0"
-__spec_version__ = "0.57.0"
+__spec_version__ = "0.58.0"
 # Proposal 0052 (spec observability §5.1 / §8.4.1): canonical
 # package-registry name for this implementation. Surfaces on every
 # OTel invocation span as ``openarmature.implementation.name`` and on

src/openarmature/__init__.py

@@ -210,6 +210,31 @@ def __init__(self, message: str, category: str) -> None:
         self.category = category
 
 
+# Conformance-adapter §5.1 ``cause`` (proposal 0070): a failure mock's raised
+# error MAY chain to an originating cause, recursively, so a consumer walking
+# the cause chain (pipeline-utilities §6.3 failure isolation) observes each
+# link's category / message.
+def _build_mock_cause(cause_spec: Mapping[str, Any] | None) -> Exception | None:
+    """Build the chained originating exception from a failure mock's ``cause``
+    directive. ``cause: {category, message, cause: {...}}`` nests recursively;
+    each link becomes a ``_CategorizedException`` (or a bare ``Exception`` when
+    its category is null) linked via ``__cause__``. Returns ``None`` when no
+    cause is configured."""
+    if cause_spec is None:
+        return None
+    inner = _build_mock_cause(cause_spec.get("cause"))
+    message = str(cause_spec.get("message", ""))
+    category = cause_spec.get("category")
+    exc: Exception = (
+        _CategorizedException(message=message, category=category)
+        if isinstance(category, str) and category
+        else Exception(message)
+    )
+    if inner is not None:
+        exc.__cause__ = inner
+    return exc
+
+
 def _make_pure_update_fn(
     node_name: str,
     update: Mapping[str, Any],
@@ -512,10 +537,16 @@ async def fn(_state: Any) -> Mapping[str, Any]:
             entry = sequence[idx]
             if entry is None:
                 return copy.deepcopy(success_update)
-            raise _CategorizedException(
+            # An entry MAY carry a recursive ``cause`` (proposal 0070 §5.1)
+            # that chains the raised error to an originating cause.
+            cause_exc = _build_mock_cause(entry.get("cause"))
+            exc = _CategorizedException(
                 message=entry.get("message", "flaky"),
                 category=entry.get("category", "provider_unavailable"),
             )
+            if cause_exc is not None:
+                raise exc from cause_exc
+            raise exc
         return copy.deepcopy(success_update)
 
     return fn
@@ -538,6 +569,32 @@ async def fn_with_sleep(state: Any) -> Mapping[str, Any]:
     return fn_with_sleep
 
 
+def _wrap_with_execution_recorder(
+    fn: Callable[[Any], Awaitable[Mapping[str, Any]]],
+    node_name: str,
+    recorders: dict[str, dict[int, list[int]]],
+) -> Callable[[Any], Awaitable[Mapping[str, Any]]]:
+    """Wrap a node body so that, when it runs inside a fan-out instance, it
+    records the executing instance's ``current_fan_out_index()`` into
+    ``recorders`` (keyed by node name then index). Lets the checkpoint resume
+    driver tell which fan-out instances executed vs. rolled forward for a
+    plain-node fan-out (e.g. the crash_injection fixture 067), where no
+    ``flaky_per_index`` body records execution. Records at body entry, so an
+    instance whose body ran counts as executed even if it then fails."""
+    from openarmature.observability.correlation import (  # noqa: PLC0415
+        current_attempt_index,
+        current_fan_out_index,
+    )
+
+    async def fn_recording(state: Any) -> Mapping[str, Any]:
+        idx = current_fan_out_index()
+        if idx is not None:
+            recorders.setdefault(node_name, {}).setdefault(idx, []).append(current_attempt_index())
+        return await fn(state)
+
+    return fn_recording
+
+
 @dataclass(frozen=True)
 class _TracingFanOutNode(FanOutNode[State, State]):
     """Conformance helper: a FanOutNode that appends its name to a shared
@@ -658,6 +715,7 @@ def build_graph(
     fan_out_instance_middleware: Mapping[str, Sequence[Any]] | None = None,
     parallel_branches_branch_middleware: Mapping[str, Mapping[str, Sequence[Any]]] | None = None,
     flaky_per_index_attempt_recorders: dict[str, dict[int, list[int]]] | None = None,
+    instance_execution_recorders: dict[str, dict[int, list[int]]] | None = None,
 ) -> BuiltGraph:
     """Translate a graph-shaped fixture block into a `BuiltGraph`.
 
@@ -767,6 +825,13 @@ def build_graph(
         if sleep_ms is not None:
             body = _wrap_with_sleep(body, int(sleep_ms))
 
+        # Record per-instance execution for plain-node fan-outs so the
+        # checkpoint resume driver can tell executed from rolled-forward
+        # instances. flaky_per_index records its own per-instance attempts,
+        # so it is skipped here; this covers the rest.
+        if instance_execution_recorders is not None and "flaky_per_index" not in node_spec:
+            body = _wrap_with_execution_recorder(body, node_name, instance_execution_recorders)
+
         builder.add_node(node_name, body, middleware=per_node_mw)
 
     for edge_spec in spec.get("edges", []):