Skip to content

Commit 3805e4f

Browse files
observability: spec v0.7 OTel mapping (proposal 0007) — phase 6.0 (#19)
* observability: spec v0.7 OTel mapping (proposal 0007) — phase 6.0 Land the observability capability per spec proposal 0007. Adds `openarmature.observability` with the §3 correlation_id ContextVar + queue-mediated dispatch primitives (core, no OTel deps), and an extras-gated `openarmature.observability.otel` subpackage with `OTelObserver`, detached trace mode, LLM provider span hook, and the log bridge. Engine integration: `correlation_id` ContextVar set/reset around the outermost `invoke()`; `_active_observers` and `_active_dispatch` ContextVars set/reset around every chain invocation in `_step_function_node` / `_step_subgraph_node` / `_step_fan_out_node`. The LLM provider span hook in `OpenAIProvider.complete` calls `current_dispatch()` to enqueue NodeEvent-shaped records on the same delivery worker the engine uses, preserving spec §6 serial ordering. OTelObserver implementation: - Private TracerProvider per §6 — never registers globally. - Subscribes to `started` / `completed` / `checkpoint_saved` phases. - §4.5 subgraph dispatch span synthesis (the engine wrapper is transparent per fixture 013, but observability mandates a span). - §4.4 detached trace mode for subgraphs (one Link, two traces) and fan-outs (one trace per instance, one Link per). - §5 attribute population including the cross-cutting `openarmature.correlation_id` on every span. - §5.5 LLM provider span with `disable_llm_spans` opt-out. - §10.8 checkpoint_saved → zero-duration `openarmature.checkpoint.save` span. - `spec_version` reads from `openarmature.__spec_version__` so the three-place version sync covers the OTel surface automatically. Charter alignment: OTel deps are optional via `pip install openarmature[otel]`; correlation primitives stay in core. Plan to lift `observability.otel` into a sibling `openarmature-otel` package at the v1.0 launch alongside `openarmature-eval` per charter §3.2. `opentelemetry-sdk>=1.27,<2` upper-bound guards against the SDK 2.x LoggingHandler removal until the migration to opentelemetry-instrumentation-logging lands. Test coverage: 4 of 11 conformance fixtures driven end-to-end — 001 (basic trace), 005 (LLM provider span + isolation under external auto-instrumentation), 008 (detached subgraph + detached fan-out), 009 (correlation_id cross-cutting). The Phase 5 deferred fixture 031 span/log assertions activate. The remaining 7 fixtures defer to Phase 6.1 with per-fixture wiring notes (tracked in openarmature-coord). 15 unit tests cover the engine path independently of conformance harness wiring. * observability: PR #19 review followups + CI extras fix Three changes: - **CI** — ``uv sync --frozen`` didn't install the ``[otel]`` extras, so pyright type-checked ``tests/unit/test_observability_otel.py`` against missing ``opentelemetry.*`` symbols and produced 373 false-positive errors. Switched to ``--all-extras`` so every optional-dependency group is present in CI. - **OTelObserver.spec_version factory** — ``default_factory=lambda: _read_spec_version()`` simplified to ``default_factory=_read_spec_version``; the lambda was a redundant wrapper. - **test_correlation.py** — replaced ``lambda s: _pre(s)`` with the bare ``_pre``; the type-ignore came off too. The other two unnecessary-lambda findings (``detached_subgraphs`` / ``detached_fan_outs`` factories) are kept with an explanatory comment — pyright strict mode infers ``default_factory=frozenset`` as ``frozenset[Unknown]`` and flags it; the lambda preserves the ``frozenset[str]`` annotation. * observability: PR #19 round-2 review followups Five Copilot findings addressed: - **`_close_subgraph_span` token leak** — was ending the synthetic subgraph span without detaching the OTel context token, leaving the current-span context pinned to a closed span. Added the detach with the same cross-context guard as ``_close_detached_root``. - **Detached fan-out instance parent lookup** — added an explicit ``namespace[:1] + (str(fan_out_index),)`` check at the top of ``_resolve_parent_context`` so inner-instance spans don't depend on the attach-then-resolve ordering of the surrounding ``_sync_subgraph_spans`` call. - **`isdigit()` heuristic** — replaced the string-shape guess for identifying fan-out instance roots with an explicit ``_fan_out_instance_root_prefixes: set[tuple[str, ...]]`` so a pure-digit node name (e.g. ``"123"``) doesn't get misclassified. - **Invocation-span memory leak in long-lived observers** — closing prior-correlation_id invocation spans when a new correlation_id's first event arrives. Plus a public ``OTelObserver.close_invocation(correlation_id)`` method for callers who want explicit control without driving a follow-on invocation. The very-last invocation still closes on ``shutdown()``. - **`current_dispatch` re-export** — added to ``openarmature.observability.__all__`` to match the public-API intent in ``correlation.py``'s ``__all__`` and the PR description. Plus one harness fix: - **test_observability.py collection-time skip** — added ``pytest.importorskip("opentelemetry.trace")`` before the OTel imports so the file skips cleanly when the ``otel`` extras aren't installed (matches ``tests/unit/test_observability_otel.py``). The two re-flagged ``frozenset`` lambdas are kept with the inline comment explaining why pyright strict mode needs them — same position as the prior PR review pass. * observability: PR #19 round-3 review followups Seven Copilot/code-quality findings addressed: - **`_close_subgraph_span` empty-except comment** — added inline rationale inside the `except ValueError` block (the explanation was already above the `try`; rule wants it on the empty branch). - **Concurrency limitation documented** — class docstring now spells out that ``OTelObserver`` instances are NOT safe across concurrent invocations (overlapping namespaces collide; close-prior- correlation_id closes other in-flight invocations). Recommended pattern: one observer per ``CompiledGraph`` for sequential workloads, fresh per-invocation observers for parallel services. Phase 6.1 tracks the proper correlation_id-scoped state fix. - **Module docstring drift** — rewrote to reflect the ``(namespace, attempt_index, fan_out_index)`` leaf-span key (not the prior ``(trace_id, ...)`` shape) plus the dedicated dicts for subgraph dispatch, detached roots, and invocation spans. - **Real ``invocation_id`` on the invocation span** — added ``current_invocation_id()`` ContextVar reader to ``correlation.py`` mirroring ``current_correlation_id``; engine sets it in ``invoke()`` before worker creation. ``OTelObserver`` reads it when opening the invocation span and omits the attribute when None (replaces the prior ``"<unset>"`` sentinel). - **`_make_llm_event` phase typing** — tightened ``phase: str`` to ``Literal["started", "completed"]``; dropped the corresponding ``cast``. - **Test-isolation for global TracerProvider** — saved + restored the prior global provider in both ``test_observer_uses_private_provider_not_global`` and the fixture 005 driver (the only sites that call ``set_tracer_provider``). - **`test_correlation_id_and_invocation_id_are_structurally_distinct`** — rewrote to drive a real invocation with a checkpointer and assert ``record.correlation_id != record.invocation_id`` deterministically; also cross-checks via the new ``current_invocation_id()`` reader inside the node body. The prior assertion was on stdlib ``uuid.uuid4()``, not framework behavior. The Phase 6.1 conformance-fillin tracker (in openarmature-coord) is updated to add the concurrency-safe state scoping work alongside the deferred fixtures. * observability: replace frozenset lambdas with named factory Two github-code-quality findings on ``detached_subgraphs`` / ``detached_fan_outs`` defaults addressed via the bot's named-factory suggestion. Previous rounds kept the lambdas with an explanatory comment because ``default_factory=frozenset`` produced ``frozenset[Unknown]`` under pyright strict mode; the named factory ``_empty_str_frozenset() -> frozenset[str]`` carries the explicit return annotation, satisfying both constraints (no lambda; pyright preserves the typed empty frozenset). Removed the explanatory comment along with the lambdas. * codeql: suppress py/ineffectual-statement (false positives) Adds ``.github/codeql/codeql-config.yml`` with a query-filter exclude for ``py/ineffectual-statement``. The rule has produced zero true positives across the codebase; every finding has been the same false-positive pattern: - PEP 544 ``Protocol`` method bodies (``...``) on the eight Protocol classes the framework defines (``Provider``, ``Checkpointer``, ``Middleware``, ``Observer``, etc.). Canonical Python idiom — never executed. - ``await worker`` lines flagged as no-effect when ``await`` on a task is the entire mechanism that waits for the task to complete (CodeQL bug on async patterns). The genuine "unused statement" cases are caught at type-check time by pyright's ``reportUnusedExpression`` — suppressing the CodeQL rule doesn't lose any signal. To make GitHub's default-setup code scanning honor this config: Settings → Code security → Code scanning → CodeQL default setup → "Custom configuration" → point at ``.github/codeql/codeql-config.yml``. (Or switch to advanced setup; the file works either way once selected.) * codeql: switch to advanced setup with config file Default setup's UI doesn't expose a config-file field, so the ``.github/codeql/codeql-config.yml`` we added to suppress ``py/ineffectual-statement`` had no effect. Switching to advanced setup via an explicit workflow file fixes that — the workflow references the config, and the rule exclusion lands. Workflow matches default setup's coverage: - Runs on ``main`` push, PRs to ``main``, plus a weekly scheduled scan. - Two language jobs (``actions``, ``python``). - Includes the ``security-and-quality`` query suite explicitly so code-quality findings still surface; only the per-rule exclusion in the config file is silenced. - Uses ``security-events: write`` permission per least-privilege. The user disables default setup in Settings → Code security → CodeQL analysis when this PR merges; the workflow takes over. * observability: PR #19 round-5 review followups Two Copilot findings addressed: - **Log-bridge filter on the root logger** — ``install_log_bridge`` was attaching ``_CorrelationIdFilter`` to the OTel ``LoggingHandler`` only. Pre-existing stdout / file / third-party handlers registered ahead of it on the root logger would see log records WITHOUT ``openarmature.correlation_id``, violating spec §7's "every log record emitted during an invocation MUST carry the attribute." Moved the filter to ``root.addFilter(...)`` so it runs at the LOGGER level, applying to every record regardless of which handler processes it. Idempotency is now tracked separately for handler vs filter installation. - **Typed LLM event payload** — ``_make_llm_event`` previously stuffed a plain dict into ``NodeEvent.pre_state`` with ``cast("Any", ...)``, violating the documented ``pre_state: State`` contract. Any observer calling ``event.pre_state.model_dump()`` on an LLM event would crash. Wrapped the payload in a dedicated ``_LlmEventState(State)`` Pydantic subclass with the LLM fields (model, finish_reason, prompt/completion/total tokens, optional error_type / message / category). The OTel observer reads attributes directly via ``isinstance(event.pre_state, _LlmEventState)`` and attribute access. Schema is honored. * codeql: also suppress py/unused-import false positives Same false-positive pattern as the prior ``py/ineffectual-statement`` exclusion — three cases CodeQL flags but isn't actually right about: - Forward-reference casts (``cast("FinishReason", x)``, ``cast("Checkpointer", capturing)``). The string argument isn't parsed by CodeQL but pyright resolves it; removing the import yields ``reportUndefinedVariable``. - Subscripted base classes (``class T(FanOutNode[State, State]):``) — the base class IS used. - Re-exports for public API. Pyright's ``reportUnusedImport`` covers the genuine cases at type-check time; the CodeQL rule duplicates with a worse FP rate. * test_middleware: comment fix + nonlocal counter Two Copilot suggestions on test_middleware.py: - Comment said "more than twice" but the docstring + spec §2 say "more than once" and the test pins at N=5 calls. Aligning the comment. - Replaced the ``inner_calls = [0]`` mutable-list closure trick with ``nonlocal inner_calls`` + a plain int. Cleaner; same behavior. * test_llm_provider: docstring clarifications Two Copilot suggestions on test_llm_provider.py — both pure documentation: - ``_make_provider`` — expanded to explain that the no-op ``MockTransport(lambda _req: httpx.Response(204))`` exists only to satisfy provider construction; the tests using this helper pass their own ``httpx.Response`` objects directly into ``_classify_http_error`` without going through the wire. - ``test_complete_does_not_mutate_messages_or_tools`` — said the input objects are "byte-identical" but the assertion is ``==`` (equality), not ``is`` (identity). Updated wording to "remain equal to their pre-call snapshots." * RELEASING.md: tighten version-string consistency Three Copilot doc-quality findings: - Commit message + PR title in §1 used the hyphenated ``0.5.0-rc1`` form, but ``pyproject.toml`` and ``__init__.py`` use the canonical PEP 440 ``0.5.0rc1`` form (no hyphen). Updated both to ``0.5.0rc1`` so the literal strings copy-pasted from the runbook match what the version files contain. - Added a short paragraph at the top of §2 (Tag the RC) calling out the two forms explicitly: version strings (pyproject / ``__init__.py``) use ``0.5.0rc1``; git tags use ``v0.5.0-rc1``. PEP 440 normalizes them as equivalent (line 139 covers that), but mixing them in commit messages and PR titles muddies the audit trail.
1 parent 689c336 commit 3805e4f

18 files changed

Lines changed: 2993 additions & 35 deletions

File tree

.github/codeql/codeql-config.yml

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: "openarmature-python CodeQL config"
2+
3+
# CodeQL's Python ``code-quality`` suite includes
4+
# ``py/ineffectual-statement``, which produces a fixed false-positive
5+
# stream against this codebase:
6+
#
7+
# - PEP 544 ``Protocol`` method bodies use ``...`` (the canonical
8+
# Python idiom — never executed because Protocols are structural,
9+
# not inheritance-based). Replacing with ``raise NotImplementedError``
10+
# would imply ABC semantics that the protocol classes
11+
# intentionally don't have. Cited sites:
12+
# - ``llm/provider.py`` (Provider.ready / complete)
13+
# - ``checkpoint/protocol.py`` (Checkpointer.save / load / list /
14+
# delete)
15+
# - ``graph/middleware/_core.py`` (Middleware.__call__ + chain
16+
# callable)
17+
# - ``graph/observer.py`` (Observer.__call__)
18+
# - ``await worker`` after a queue sentinel in
19+
# ``tests/unit/test_observer.py`` is flagged as no-effect, but
20+
# ``await`` on a Task waits for completion — the entire reason
21+
# the line exists. CodeQL bug for async patterns.
22+
#
23+
# Each finding has been individually triaged across PR review rounds;
24+
# the rule has produced zero true positives. Suppressing the rule
25+
# repo-wide stops the regenerating noise without leaving real
26+
# unused-statement bugs un-caught (pyright's ``reportUnusedExpression``
27+
# already covers the genuine cases at type-check time).
28+
query-filters:
29+
- exclude:
30+
id: py/ineffectual-statement
31+
# ``py/unused-import`` produces false positives on three patterns
32+
# this codebase relies on:
33+
#
34+
# - Forward-reference casts: ``cast("FinishReason", x)`` /
35+
# ``cast("Checkpointer", capturing)``. CodeQL doesn't look
36+
# inside the string argument; pyright's strict mode does, AND
37+
# raises ``reportUndefinedVariable`` if the name isn't in scope
38+
# (verified empirically — removing ``FinishReason`` from
39+
# ``openai.py``'s import yields the pyright error).
40+
# - Subscripted base classes: ``class _TracingFanOutNode(
41+
# FanOutNode[State, State]):``. The base class IS used; the
42+
# generic subscription happens at class-definition time.
43+
# - Re-exports for the public API surface (some submodule
44+
# ``__init__`` files import names solely to re-expose).
45+
#
46+
# Pyright's ``reportUnusedImport`` (default in strict mode) catches
47+
# the genuine unused-import cases without these false positives,
48+
# so dropping the CodeQL rule doesn't lose signal.
49+
- exclude:
50+
id: py/unused-import

.github/workflows/ci.yml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,14 @@ jobs:
3636
# version developers run locally.
3737

3838
- name: Sync deps
39-
run: uv sync --frozen
39+
# ``--all-extras`` installs every optional-dependency group
40+
# (currently ``[otel]``) so pyright type-checks the
41+
# OTel-using modules with real types and tests covering
42+
# extras-gated code paths run end-to-end. Without this,
43+
# ``opentelemetry.*`` symbols come back Unknown and pyright
44+
# produces hundreds of false-positive errors on
45+
# ``tests/unit/test_observability_otel.py``.
46+
run: uv sync --frozen --all-extras
4047

4148
- name: Lint (ruff check)
4249
run: uv run ruff check .

.github/workflows/codeql.yml

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# CodeQL advanced setup — replaces the prior default setup so we can
2+
# reference ``.github/codeql/codeql-config.yml`` (default setup's UI
3+
# doesn't expose a config-file field). Config excludes
4+
# ``py/ineffectual-statement`` which produced a regenerating stream
5+
# of false positives against PEP 544 Protocol method bodies; see the
6+
# config file for the full rationale.
7+
name: CodeQL
8+
9+
on:
10+
push:
11+
branches: [main]
12+
pull_request:
13+
branches: [main]
14+
schedule:
15+
# Weekly scan to surface new issues that didn't change in any
16+
# PR. Matches default setup's cadence.
17+
- cron: "25 16 * * 5"
18+
19+
# Least-privilege per the CI workflow next door. CodeQL needs:
20+
# - ``actions: read`` to fetch action metadata.
21+
# - ``contents: read`` to check out code.
22+
# - ``security-events: write`` to upload SARIF results to the Security
23+
# tab.
24+
permissions:
25+
actions: read
26+
contents: read
27+
security-events: write
28+
29+
concurrency:
30+
group: ${{ github.workflow }}-${{ github.ref }}
31+
cancel-in-progress: true
32+
33+
jobs:
34+
analyze:
35+
name: Analyze (${{ matrix.language }})
36+
runs-on: ubuntu-latest
37+
strategy:
38+
fail-fast: false
39+
matrix:
40+
language: [actions, python]
41+
steps:
42+
- name: Checkout
43+
uses: actions/checkout@v6
44+
45+
- name: Initialize CodeQL
46+
uses: github/codeql-action/init@v3
47+
with:
48+
languages: ${{ matrix.language }}
49+
# Reference the in-tree config so the rule-suppressions land.
50+
config-file: ./.github/codeql/codeql-config.yml
51+
# Match the prior default setup's query coverage by including
52+
# the security-and-quality suite explicitly. Code-quality
53+
# findings still surface; only the per-rule
54+
# ``py/ineffectual-statement`` exclusion in the config file
55+
# is suppressed.
56+
queries: security-and-quality
57+
58+
- name: Perform CodeQL Analysis
59+
uses: github/codeql-action/analyze@v3
60+
with:
61+
category: "/language:${{ matrix.language }}"

docs/RELEASING.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,9 @@ vim src/openarmature/__init__.py # __version__ = "0.5.0rc1"
4343
vim tests/test_smoke.py # match
4444
# uv.lock auto-updates via pre-commit when you stage pyproject.toml
4545
git add pyproject.toml src/openarmature/__init__.py tests/test_smoke.py uv.lock
46-
git commit -m "release: prep 0.5.0-rc1"
46+
git commit -m "release: prep 0.5.0rc1"
4747
git push origin <branch>
48-
gh pr create --title "release: prep 0.5.0-rc1" ...
48+
gh pr create --title "release: prep 0.5.0rc1" ...
4949
```
5050

5151
If this is the same release cycle as a previous RC, also bump the spec pins
@@ -56,6 +56,14 @@ Merge the PR.
5656

5757
### 2. Tag the RC
5858

59+
Two version forms appear in this guide and they're not interchangeable
60+
in writing even though PEP 440 normalizes them: `pyproject.toml` /
61+
`__init__.py` use `0.5.0rc1` (no hyphen — the canonical PEP 440
62+
form); git tags + the release-workflow's tag-shape regex use
63+
`v0.5.0-rc1` (with hyphen). Mixing them in commit messages or PR
64+
titles muddies the audit trail; stick to the no-hyphen form for
65+
version-string fields and the hyphenated form for git tags.
66+
5967
```bash
6068
git checkout main && git pull --ff-only
6169
git tag v0.5.0-rc1

pyproject.toml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,21 @@ dependencies = [
1616
"httpx>=0.27",
1717
]
1818

19+
[project.optional-dependencies]
20+
# Spec observability §6 OTel mapping. Optional per charter §3.1
21+
# principle 5 — backend mappings are pluggable rather than core
22+
# dependencies. Plan to lift into a sibling ``openarmature-otel``
23+
# package at v1.0 launch alongside ``openarmature-eval``.
24+
otel = [
25+
# Upper bound guards against the SDK 2.x release that removes
26+
# ``opentelemetry.sdk._logs.LoggingHandler`` (currently emits a
27+
# DeprecationWarning). Migration to
28+
# ``opentelemetry-instrumentation-logging`` lands in Phase 6.1
29+
# before bumping the upper bound.
30+
"opentelemetry-api>=1.27,<2",
31+
"opentelemetry-sdk>=1.27,<2",
32+
]
33+
1934
[project.urls]
2035
Repository = "https://github.com/LunarCommand/openarmature-python"
2136
Specification = "https://github.com/LunarCommand/openarmature-spec"

src/openarmature/graph/compiled.py

Lines changed: 81 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,16 @@
4141
CheckpointRecord,
4242
NodePosition,
4343
)
44+
from openarmature.observability.correlation import (
45+
_reset_active_dispatch,
46+
_reset_active_observers,
47+
_reset_correlation_id,
48+
_reset_invocation_id,
49+
_set_active_dispatch,
50+
_set_active_observers,
51+
_set_correlation_id,
52+
_set_invocation_id,
53+
)
4454

4555
from .edges import END, ConditionalEdge, EndSentinel, StaticEdge
4656
from .errors import (
@@ -369,6 +379,18 @@ async def invoke(
369379
pending_resume_states=pending_resume_states,
370380
resume_invocation=resume_invocation,
371381
)
382+
# Spec observability §3.1: the correlation_id MUST be readable
383+
# from anywhere within the invocation's async call tree via the
384+
# language's idiomatic context primitive. Set the ContextVar
385+
# BEFORE creating the delivery worker so the worker's captured
386+
# context sees the correlation_id (asyncio.create_task snapshots
387+
# the current Context at creation time). Reset on return so
388+
# subsequent invocations get a fresh slate. Nested ``invoke()``
389+
# calls (subgraph-as-node uses ``_invoke`` directly, not the
390+
# public ``invoke``, so they don't re-set; see §3.1's
391+
# "per-invocation is OUTERMOST invoke" wording).
392+
correlation_token = _set_correlation_id(resolved_correlation_id)
393+
invocation_token = _set_invocation_id(invocation_id)
372394
worker = asyncio.create_task(deliver_loop(queue))
373395
self._active_workers.add(worker)
374396
# Auto-prune: when the worker completes (after the sentinel is
@@ -378,6 +400,8 @@ async def invoke(
378400
try:
379401
return await self._invoke(starting_state, context)
380402
finally:
403+
_reset_invocation_id(invocation_token)
404+
_reset_correlation_id(correlation_token)
381405
# Sentinel terminates the worker after it processes events
382406
# already on the queue (including any error event we just
383407
# dispatched on the failure path). Drain semantics live on
@@ -585,14 +609,27 @@ async def innermost(s: Any) -> Mapping[str, Any]:
585609
innermost,
586610
)
587611

612+
# Spec observability §3 / Phase 6 LLM-span hook: capability
613+
# backends emitting from inside a node body (the
614+
# llm-provider span instrumentation in OpenAIProvider) need
615+
# to find the observers active for THIS invocation. Set the
616+
# ContextVar around the chain invocation; reset in
617+
# ``try/finally`` so an exception escaping the chain still
618+
# restores the prior value.
619+
observers_token = _set_active_observers(context.full_observers())
620+
dispatch_token = _set_active_dispatch(lambda event: _dispatch(context, event))
588621
try:
589-
final_partial = await chain(state)
590-
except RuntimeGraphError:
591-
raise
592-
except Exception as e:
593-
# A raw exception (node-raised or middleware-raised) escaped
594-
# the chain unrecovered. Wrap as NodeException per §4.
595-
raise NodeException(node_name=current, cause=e, recoverable_state=state) from e
622+
try:
623+
final_partial = await chain(state)
624+
except RuntimeGraphError:
625+
raise
626+
except Exception as e:
627+
# A raw exception (node-raised or middleware-raised) escaped
628+
# the chain unrecovered. Wrap as NodeException per §4.
629+
raise NodeException(node_name=current, cause=e, recoverable_state=state) from e
630+
finally:
631+
_reset_active_dispatch(dispatch_token)
632+
_reset_active_observers(observers_token)
596633
# Engine's canonical merge uses the ORIGINAL state per §2: "the
597634
# transformed state is passed to ``next``, NOT to the engine's
598635
# merge step." If middleware transformed state mid-chain, the
@@ -649,18 +686,29 @@ async def innermost(s: Any) -> Mapping[str, Any]:
649686
list(self.middleware) + list(node.middleware),
650687
innermost,
651688
)
689+
# Same active-observers scope as _step_function_node — parent
690+
# middleware running before the descent should see the parent's
691+
# observer set; the inner _invoke (called via ``node.run``)
692+
# descends into its own context and sets a new scope from
693+
# there.
694+
observers_token = _set_active_observers(context.full_observers())
695+
dispatch_token = _set_active_dispatch(lambda event: _dispatch(context, event))
652696

653697
try:
654-
final_partial = await chain(state)
655-
except RuntimeGraphError:
656-
raise
657-
except Exception as e:
658-
# Same wrap as _step_function_node: a raw exception escaping
659-
# the parent's middleware chain (e.g., a middleware bug or a
660-
# projection error) becomes NodeException tagged with the
661-
# SubgraphNode's wrapper name so §4 recoverable_state is
662-
# preserved.
663-
raise NodeException(node_name=current, cause=e, recoverable_state=state) from e
698+
try:
699+
final_partial = await chain(state)
700+
except RuntimeGraphError:
701+
raise
702+
except Exception as e:
703+
# Same wrap as _step_function_node: a raw exception escaping
704+
# the parent's middleware chain (e.g., a middleware bug or a
705+
# projection error) becomes NodeException tagged with the
706+
# SubgraphNode's wrapper name so §4 recoverable_state is
707+
# preserved.
708+
raise NodeException(node_name=current, cause=e, recoverable_state=state) from e
709+
finally:
710+
_reset_active_dispatch(dispatch_token)
711+
_reset_active_observers(observers_token)
664712
return _merge_partial(state, final_partial, self.reducers, current)
665713

666714
async def _step_fan_out_node(
@@ -740,12 +788,23 @@ async def innermost(s: Any) -> Mapping[str, Any]:
740788
innermost,
741789
)
742790

791+
# Same observability §3 / LLM-span hook contract as
792+
# _step_function_node: set the active observer set in scope
793+
# around the chain invocation so capability backends emitting
794+
# from inside the fan-out's parent dispatch (or any code
795+
# running on its call stack) can find the observers.
796+
observers_token = _set_active_observers(context.full_observers())
797+
dispatch_token = _set_active_dispatch(lambda event: _dispatch(context, event))
743798
try:
744-
final_partial = await chain(state)
745-
except RuntimeGraphError:
746-
raise
747-
except Exception as e:
748-
raise NodeException(node_name=current, cause=e, recoverable_state=state) from e
799+
try:
800+
final_partial = await chain(state)
801+
except RuntimeGraphError:
802+
raise
803+
except Exception as e:
804+
raise NodeException(node_name=current, cause=e, recoverable_state=state) from e
805+
finally:
806+
_reset_active_dispatch(dispatch_token)
807+
_reset_active_observers(observers_token)
749808
merged_outer = _merge_partial(state, final_partial, self.reducers, current)
750809
# Spec §10.3 + §10.7: the fan-out's own completion DOES save —
751810
# one record once the fan-out as a whole has finished and

0 commit comments

Comments
 (0)