Extend production-observability with accumulator pattern

Two pre-release polish items for the v0.12.0 cycle.

CHANGELOG: the Unreleased Changed entry for the spec-pin advance
originally said proposal 0052 "lands in a follow-on PR of this
cycle" and 0054 "lands in a follow-on PR". Both have since landed
(PRs 131 + 132). Rewrites the bullet to factually describe the
final state: three proposals (0048, 0052, 0054) ship as fully
implemented, two (0051, 0053) ship as textual-only.

production-observability example: adds an LlmUsageAccumulator
class plus a terminal persist node that demonstrates the queryable
observer + drain_events_for pattern end-to-end. The accumulator
subscribes to the LLM-namespace event stream, accumulates per-
invocation token totals via current_invocation_id() bucket keys,
and exposes convention-only get_bucket / drop methods. The persist
node calls drain_events_for to synchronize on the deliver loop
before reading the bucket so the rollup reflects every LLM call
in the invocation, drops the bucket per the explicit-cleanup
discipline, and prints a cost summary. The graph grows from
respond -> END to respond -> persist -> END. Module-level
singletons (_accumulator + _compiled_graph) keep the persist
node closure-free and follow the existing _provider_instance
precedent.

Walkthrough doc updates the H1, overview, what-it-teaches list,
captured-output sample, and reading-the-output walkthrough to
cover the new pattern.

Author: chris-colinsky

CHANGELOG.md

@@ -8,7 +8,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
 
 ### Changed
 
-- **Pinned spec advanced from v0.38.0 to v0.46.0.** Submodule + `[tool.openarmature].spec_version` + `conformance.toml` `spec_pin` advance together. Absorbs eight new proposals (0047-0054) into the conformance manifest. Two of them ship as part of the v0.12.0 cycle as textual-only acknowledgments with no code change required: **proposal 0051** (observability §8.4.1 Langfuse `trace.input` / `trace.output` implementation-surface caveat — documents that vendor SDK round-trip is required to project caller-side trace I/O updates onto the wire; the v0.11.0 (proposal 0043) caller-hook shape already matches the documented behavior) and **proposal 0053** (observability §3.4 shared-parent boundary clarification — tightens the structural-shared-parent classification to predicate the invocation span on whether at least one fan-out or parallel-branches dispatch is on the augmenter's call-stack path; behavior already matches via fixtures 034 + 039). Two more ship as implemented in this cycle: **proposal 0048** (read-symmetric metadata + queryable observer pattern docs — see *Added* below) and **proposal 0052** (implementation attribution attributes — landing in a follow-on PR of this cycle). The remaining proposals are marked `not-yet` in the conformance manifest with roadmap targets: 0047 + 0049 (v0.13.0 LLM provider hardening batch), 0050 (v0.14.0 retry & reliability batch), 0054 (per-invocation observer event drain — bundled into this v0.12.0 cycle alongside 0048 per the §9.4 accumulator-lifecycle pairing; lands in a follow-on PR).
+- **Pinned spec advanced from v0.38.0 to v0.46.0.** Submodule + `[tool.openarmature].spec_version` + `conformance.toml` `spec_pin` advance together. Absorbs eight new proposals (0047-0054) into the conformance manifest. Two ship as textual-only acknowledgments with no code change required: **proposal 0051** (observability §8.4.1 Langfuse `trace.input` / `trace.output` implementation-surface caveat — documents that vendor SDK round-trip is required to project caller-side trace I/O updates onto the wire; the v0.11.0 (proposal 0043) caller-hook shape already matches the documented behavior) and **proposal 0053** (observability §3.4 shared-parent boundary clarification — tightens the structural-shared-parent classification to predicate the invocation span on whether at least one fan-out or parallel-branches dispatch is on the augmenter's call-stack path; behavior already matches via fixtures 034 + 039). Three ship as fully implemented this cycle: **proposal 0048** (read-symmetric metadata + queryable observer pattern docs — see *Added* below), **proposal 0052** (implementation attribution attributes — see *Added* below), and **proposal 0054** (per-invocation observer event drain — see *Added* below; bundled with 0048 as the §9.4 accumulator-lifecycle pair). The remaining proposals are marked `not-yet` in the conformance manifest with roadmap targets: 0047 + 0049 (v0.13.0 LLM provider hardening batch) and 0050 (v0.14.0 retry & reliability batch).
 - **README and docs homepage refreshed around reasons-to-choose.** Replaced the 10-bullet "Why OpenArmature" feature inventory in `README.md` with 5 differentiating reasons (LLM-infused workflows to agents on one engine; crash-safe resume by contract; destination-pluggable observability with OTel + Langfuse, no SaaS lock-in; compile-time topology checks; spec + conformance). The docs homepage (`docs/index.md`) card grid carries the same five plus a sixth card retained from the previous grid for async-first / LLM-agnostic: workflows-to-agents, crash-safe, pluggable observability, bad-graphs-don't-compile, parallelism (fan-out + parallel-branches + nested correctness), async-first.
 - **Docs sweep: stale references and em-dash normalization.** Fixed three definite stale references (`spec_version='0.26.0'` in the Langfuse example output now reads `'0.38.0'`; the dangling `v0.16.1` qualifier dropped from the parallel-branches concept page; `compiled.attach_observer` corrected to `graph.attach_observer` in `non-obvious-shapes.md` for variable-name consistency with the rest of the docs). Swept em dashes out of the user-facing docs (130 instances across 17 files) per the convention set during the patterns expansion. mkdocs strict build clean; no broken intra-docs links.
 - **The checkpointing-and-migration example grows a crash-and-resume drama.** The first invoke of the v1 graph now hits a simulated transient failure inside `size_crew` (raises a `RuntimeError` on its first attempt only). The example catches `NodeException` at the `invoke()` boundary, prints what's saved on disk (`define_objective`'s position is already in `completed_positions`), then re-invokes with `resume_invocation=<id>`. The retried `size_crew` succeeds, `draft_timeline` runs, and the pipeline finishes - dramatizing the synchronous-checkpoint-by-contract reliability claim from the README pitch. The existing v1->v2 migration phase rides on top of the crash-survived checkpoint, so both reliability stories compose in one demo. Walk-through doc rewritten to cover both phases.

docs/examples/production-observability.md

@@ -1,4 +1,4 @@
-# Production observability with dual observers and timing middleware
+# Production observability with dual observers, timing middleware, and per-invocation cost rollup
 
 !!! info "Source"
     [https://github.com/LunarCommand/openarmature-python/blob/main/examples/production-observability/main.py](https://github.com/LunarCommand/openarmature-python/blob/main/examples/production-observability/main.py){target="_blank" rel="noopener"}
@@ -7,14 +7,19 @@ A single-turn lunar-mission Q&A endpoint instrumented the way you'd
 ship it: BOTH OTel and Langfuse observers attached to the same
 graph, caller hooks deriving domain-shaped `trace.input` /
 `trace.output` from State, the built-in `TimingMiddleware`
-recording per-node duration, and multi-tenant caller-supplied
-metadata propagating to both observers in one `invoke()` call.
+recording per-node duration, multi-tenant caller-supplied
+metadata propagating to both observers in one `invoke()` call, AND
+a third queryable-accumulator observer that a terminal `persist`
+node reads at request scope after synchronizing on the deliver
+loop with `drain_events_for`.
 
 ## Overview
 
-One node, one LLM call, two production-grade observability
-backends. The pipeline takes a question, calls the LLM, returns the
-answer. The interesting part is the observability wiring:
+Two nodes (`respond` then `persist`), one LLM call, three observers
+attached at compile time. The pipeline takes a question, calls the
+LLM, returns the answer, then synchronizes on the observer queue
+and rolls up token cost. The interesting part is the observability
+wiring:
 
 - `OTelObserver` attached with an `InMemorySpanExporter`
   (production swaps this for `BatchSpanProcessor` +
@@ -77,6 +82,23 @@ sees the same logical events represented two ways.
   `InMemorySpanExporter` records every Span. Production
   deployments swap each for a real exporter / SDK adapter; the
   observer call surface doesn't change.
+- **Queryable accumulator + `drain_events_for`**
+  ([queryable observer pattern](../concepts/observability.md)).
+  A third observer — `LlmUsageAccumulator` — subscribes to the
+  same event stream but only records the LLM-namespace events
+  carrying an `LlmEventPayload`. It accumulates per-invocation
+  token totals in memory, indexed by `current_invocation_id()`.
+  The terminal `persist` node calls
+  `await graph.drain_events_for(state.invocation_id, timeout=2.0)`
+  to synchronize on the deliver loop, then reads the accumulator's
+  bucket and drops it. Without the drain, the bucket might be
+  missing the most-recent LLM event's tokens (the deliver loop
+  hasn't reached them yet). The `Observer` protocol itself stays
+  a single-callable shape; the accumulator just exposes its own
+  read methods (`get_bucket` / `drop`) that the persist node knows
+  about. This is the canonical shape for per-invocation cost
+  attribution at request scope, replacing the round-trip-through-
+  State workarounds that pre-v0.12.0 deployments used.
 
 ## How to run
 
@@ -105,6 +127,7 @@ request id:  <uuid>
 feature flag:v2-canary
 
 [timing] respond: 1234.5ms (success)
+[persist] LLM usage: prompt=42, completion=38, total=80 across 1 call(s)
 answer:      The primary objective of Apollo 11 was ...
 model:       gpt-4o-mini-2024-07-18
 
@@ -133,6 +156,13 @@ Trace id=<uuid>
   `TimingMiddleware` callback as soon as the respond chain returns.
   `outcome` is `"success"` here; a `ProviderRateLimit` would surface
   as `outcome="exception"` with `exception_category="provider_rate_limit"`.
+- **`[persist] LLM usage: ...`**: emitted by the `persist` node
+  after it drains the deliver loop and reads the
+  `LlmUsageAccumulator`'s bucket for this invocation. If the drain
+  times out (slow / hung observer), the persist line is prefixed by
+  a `[persist] drain incomplete: N events still pending after 2.0s`
+  surface — the production version of that log would also flip an
+  SLO-breach metric.
 - **OTel spans block**: one line per captured span, sorted by
   start time. The relevant attributes shown are a curated subset
   for readability; the full attribute set is on each `Span` object