observability: phase 6.1 PR-C.2 — proposal 0013 + v0.10.0#26
Merged
chris-colinsky merged 6 commits intoMay 10, 2026
Conversation
Implement spec graph-engine + observability v0.10.0 (proposal 0013): non-detached fan-out instances now synthesize per-instance dispatch spans nested between the fan-out node span and the inner-node spans (mirroring the detached path's layout, but landing in the parent trace rather than a fresh trace_id). The fan-out node span carries the four §5.4 attributes (item_count / concurrency / error_policy / parent_node_name) sourced from the new ``NodeEvent.fan_out_config`` field. Mechanism: typed ``FanOutEventConfig`` dataclass on the canonical event payload (events.py). Engine populates eagerly at fan-out entry in ``_step_fan_out_node`` — resolves item_count (from ``count`` mode or ``len(items_field)``) + concurrency once, threads through every dispatch site (started, all four error-path completed, deferred success-case completed). Pre-resolved values flow into ``FanOutNode.run_with_context`` via new ``pre_resolved_count`` / ``pre_resolved_concurrency`` kwargs so callable resolvers fire at most once per fan-out attempt scope (retry middleware re-uses the outer-scope resolution). The four contract corrections from PR #34's CoPilot review: 1. ``concurrency: int | None`` on the canonical payload (positive int or None for unbounded per pipeline-utilities §9.2). The ``None → 0`` translation lives only at the OTel attribute layer in ``_node_attrs`` per observability §5.4's bare-int sentinel convention. 2. All four ``FanOutEventConfig`` fields structurally required (frozen dataclass; partial resolution would raise on construction). The implementation-defined corner from spec-thread 03 (config resolution itself failing) doesn't surface in current fixtures; natural error propagation covers it. 3. Retried fan-out attempts carry ``fan_out_config``: the resolved config is constructed in the outer scope of ``_step_fan_out_node``; retry middleware re-enters ``innermost`` and re-uses the same outer-scope reference. No per-attempt re-resolution; partition is by node type, not event category. 4. ``parent_node_name`` is on per-instance spans only; the observer caches it from the fan-out node's ``started`` event in ``_InvState.fan_out_parent_node_name`` keyed by namespace prefix, applies at per-instance dispatch span synthesis, and clears at the fan-out node's ``completed`` event. Observer changes (``observer.py``): - ``_InvState`` gains ``fan_out_instance_spans`` (per-instance dispatch span store, keyed by ``prefix + (str(fan_out_index),)``) and ``fan_out_parent_node_name`` (cache). - ``_handle_started`` populates the cache when fan-out node events land. - ``_handle_completed`` closes per-instance dispatch spans on the fan-out node's own completion (children-before-parents ordering) and clears the cache entry. - ``_sync_subgraph_spans`` routes non-detached fan-out instance namespaces through the new ``_open_fan_out_instance_dispatch_span`` helper instead of opening a shared subgraph span at the prefix. - ``_resolve_parent_context`` finds the per-instance dispatch span before falling through, so inner-node events parent under their own per-instance dispatch span (not the shared fan-out node span). - ``_node_attrs`` populates the three §5.4 fan-out node span attributes from ``event.fan_out_config`` when present. - ``_drain_inv_state`` extended to drain ``fan_out_instance_spans`` in child→parent order. Pin sites bumped to v0.10.0 (three-place sync per CLAUDE.md): ``openarmature-spec`` submodule pointer to ``ff86945``, ``pyproject.toml`` ``tool.openarmature.spec_version``, ``src/openarmature/__init__.py`` ``__spec_version__``, ``tests/test_smoke.py`` drift-guard literal. ``OTelObserver.spec_version`` updates automatically via PR-A's ``_read_spec_version``. Tests: - ``tests/conformance/test_observability.py`` removes ``006-otel-fan-out-instance-attribution`` from ``_DEFERRED_FIXTURES`` and adds ``_run_fixture_006`` driver. Verifies the fixture's expected tree: 1 fan-out NODE span with item_count/concurrency/error_policy attributes; 3 per-instance dispatch spans with fan_out_index 0..2 and parent_node_name="process"; 3 compute spans (one per instance) parented under their own per-instance dispatch span. - ``tests/conformance/adapter.py``: ``_TracingFanOutNode.run_with_context`` accepts and forwards the new pre-resolved kwargs (pyright override-compatibility). - Existing fan-out tests (resolution-once-per-entry, concurrency-callable-once-per-entry, instance-middleware retry, fan-in ordering, etc.) stay green — pre-resolved- values plumbing didn't introduce double-resolution. 399 tests pass (was 392; net +7 from new fan-out path verification + fixture 006 going from skip to pass + harness work). 3 skipped (was 4; only fixture 010 remains, which is PR-C.3's scope). Pyright clean. PR-C.3 (observer ``prepare_sync`` + fixture 010) sits independently behind its own architectural piece; lands in either order. Phase 6.1 closes when both merge.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Implements spec proposal 0013 / v0.10.0 fan-out observability semantics: fan-out node events now carry resolved fan-out configuration, and the OTel observer synthesizes non-detached per-instance dispatch spans so inner-node spans parent under their instance dispatch span.
Changes:
- Add
FanOutEventConfigto the canonicalNodeEventpayload and populate it for fan-out node started/completed events (including retries). - Thread pre-resolved fan-out count/concurrency into
FanOutNode.run_with_contextto avoid double-resolving callables. - Update the OTel observer to create and parent non-detached per-instance dispatch spans; enable conformance fixture 006 and bump spec version pins to 0.10.0.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_smoke.py | Updates drift-guard spec version assertion to 0.10.0. |
| tests/conformance/test_observability.py | Enables and implements fixture 006 verification for per-instance fan-out attribution. |
| tests/conformance/adapter.py | Extends tracing adapter to accept/forward pre-resolved fan-out config kwargs. |
| src/openarmature/observability/otel/observer.py | Adds per-instance dispatch span synthesis for non-detached fan-outs and §5.4 fan-out attributes. |
| src/openarmature/graph/fan_out.py | Accepts pre-resolved count/concurrency to prevent double resolution. |
| src/openarmature/graph/events.py | Introduces FanOutEventConfig and adds it to NodeEvent. |
| src/openarmature/graph/compiled.py | Eagerly resolves fan-out config and threads it through dispatch + node execution. |
| src/openarmature/init.py | Bumps __spec_version__ to 0.10.0. |
| pyproject.toml | Bumps tool.openarmature.spec_version to 0.10.0. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Move fan_out runtime imports from compiled.py module top into function scope to break the textual import cycle CodeQL's py/cyclic-import rule flagged. fan_out has a TYPE_CHECKING back-reference to compiled (no runtime issue), but the static analyzer doesn't see the gate. Lazy-imports inside _invoke (FanOutNode for the isinstance check) and _step_fan_out_node (_resolve_concurrency / _resolve_count). Type-only FanOutNode reference moves to TYPE_CHECKING with `from __future__ import annotations` so signatures resolve without the runtime import. - Fix _drain_inv_state ordering: per-instance dispatch spans in fan_out_instance_spans are children of the fan-out NODE span (which lives in open_spans at depth 1). The previous shape closed all of open_spans before draining fan_out_instance_spans, ending the parent before its children during shutdown/abandon. Now drains open_spans in two phases (deep >= 2 / shallow = 1) with the per-instance drain between, so children-before-parents holds. - Remove dead `by_id` map from _run_fixture_006_case — scaffolding from an earlier draft; current assertions don't use it.
Both compiled.py and fan_out.py reference each other's types only via TYPE_CHECKING blocks. CodeQL's analyzer flags the textual cycle regardless of the gate; no runtime cycle exists. Removing either side breaks pyright's type resolution for generics across the boundary. Pyright's reportImportCycles already covers genuine runtime cycles at type-check time, so dropping this CodeQL rule loses no signal.
PR-C.2 hoisted item_count / concurrency resolution out of ``run_with_context`` to step entry so the eager ``FanOutEventConfig`` could ride on every fan-out node event. That moved resolver failures (callable raises, ``getattr`` on malformed state, items_field non-list) outside ``innermost``'s ``except Exception → NodeException`` block, dropping the started/completed event pair and the NodeException wrap that the inner path used to provide. Re-establish the contract by wrapping the resolution block: any failure now dispatches a started+completed pair with ``fan_out_config=None`` (we never built one) and surfaces as NodeException. Also drops the silent 0-coercion when ``items_field`` resolves to a non-list — raises with the same TypeError text ``_build_instance_states`` produces, so fan_out_config never reports a misleading ``item_count=0``.
The ``_: ProjectionStrategy[S, Inner] = BoomProjection()`` line was a defensive belt-and-suspenders structural-conformance check, but the call-site ``add_subgraph_node(projection=BoomProjection())`` exercises the same Protocol check via its parameter annotation — pyright catches a mismatch there without needing the explicit assignment. Drop the line and update the comment to point at the call site.
- Annotate ``step=-1`` in the synthetic LLM-event observer test to point readers at ``OpenAIProvider._llm_event`` (openai.py:643) where the same sentinel is minted for production LLM-provider span events that aren't tied to graph step sequencing. - ``_ParentState`` in the fan-out gating test now uses ``Field(default_factory=list[int])`` instead of the bare ``[]`` default, matching the parametrized factory shape used in ``test_checkpoint.py``'s ``_ParentState`` and the surrounding pyright-strict expectation that field types are fully known.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-C.2 of Phase 6.1: implement spec graph-engine + observability
v0.10.0 (proposal 0013). Non-detached fan-out instances now
synthesize per-instance dispatch spans nested between the
fan-out node span and the inner-node spans (mirroring the
detached path's layout, but landing in the parent trace rather
than a fresh
trace_id). The fan-out node span carries thefour §5.4 attributes (
item_count/concurrency/error_policy/parent_node_name) sourced from the newNodeEvent.fan_out_configfield.Mechanism
New
FanOutEventConfigtyped dataclass on the canonical eventpayload (
graph/events.py). Engine populates eagerly atfan-out entry in
_step_fan_out_node— resolvesitem_count(fromcountmode orlen(items_field)) +concurrencyonce, threads throughevery dispatch site (started, all four error-path completed,
deferred success-case completed). Pre-resolved values flow into
FanOutNode.run_with_contextvia newpre_resolved_count/
pre_resolved_concurrencykwargs so callable resolvers fireat most once per fan-out attempt scope.
Observer (
observability/otel/observer.py):_InvStategainsfan_out_instance_spans(per-instancedispatch span store) +
fan_out_parent_node_name(cache)._open_fan_out_instance_dispatch_spanhelpersynthesizes per-instance dispatch spans as children of the
fan-out node span, attaches
parent_node_namefrom cache._sync_subgraph_spansroutes non-detached fan-out instancenamespaces through the new helper.
_resolve_parent_contextfinds the per-instance dispatchspan before falling through, so inner-node events parent
under their own per-instance dispatch span.
_node_attrspopulates the three §5.4 fan-out node spanattributes from
event.fan_out_config._drain_inv_stateextended for child→parent closeordering.
Four contract corrections
All four corrections from PR #34's CoPilot review on the spec
proposal (per coord thread
phase-6-1-pr-c2-fan-out-per-instance/03)honored cleanly:
concurrency: int | Noneon the canonical payload (notint with 0=unbounded). TheNone → 0translation livesonly at the OTel attribute layer.
FanOutEventConfigfields structurallyrequired (frozen dataclass; partial resolution would raise
on construction).
fan_out_config—resolved config is constructed once at fan-out entry and
re-used by retry middleware re-entries.
parent_node_namecached and applied per-instance —sourced from the fan-out node's started event, applied at
per-instance dispatch span synthesis, cleared at the fan-out's
completed event.
Submodule + version pins
Three-place sync to v0.10.0 per CLAUDE.md:
openarmature-specsubmodule pointer →ff86945(v0.10.0).tool.openarmature.spec_versioninpyproject.toml.__spec_version__insrc/openarmature/__init__.py.tests/test_smoke.py.OTelObserver.spec_versionupdates automatically via PR-A's_read_spec_version— every invocation span now reportsopenarmature.graph.spec_version = "0.10.0".Tests
(
observability/006-otel-fan-out-instance-attribution)removed from
_DEFERRED_FIXTURESand driven via_run_fixture_006intest_observability.py. Verifiesthe fixture's expected tree: 1 fan-out NODE span with
item_count=3/concurrency=2/error_policy="collect";3 per-instance dispatch spans with
fan_out_index 0..2andparent_node_name="process"; 3computespans (one perinstance) parented under their own per-instance dispatch
span.
_TracingFanOutNode.run_with_contextaccepts and forwardsthe new pre-resolved kwargs.
test_count_callable_resolved_exactly_once_at_entry/test_concurrency_callable_resolved_exactly_once_at_entrypass (validates the pre-resolved-values plumbing didn't
introduce double-resolution).
Test plan
uv run pytest -q— 399 passed (was 392; net +7 fromnew fan-out path verification + fixture 006 going from
skip to pass + harness work), 3 skipped (was 4; only
fixture 010 remains, which is PR-C.3's scope).
uv run pyright— 0 errors.uv run ruff check . && uv run ruff format— clean.shape.
Phase 6.1 status after this PR
prepare_sync+ fixture 010) — pending; no spec gatePR-C.3 is independent and can land in either order. Phase 6.1
closes when both merge.