Skip to content

Commit cb2cf1c

Browse files
Add Langfuse observer (proposal 0031)
New openarmature.observability.langfuse subpackage maps the observer event stream onto Langfuse's native Trace + Observation data model: invocation -> Trace, node/subgraph/fan-out -> Span observation, LLM provider call -> Generation observation. The LangfuseObserver consumes the event stream and emits through a narrow LangfuseClient Protocol with four methods (trace, span, generation, update_trace). Two concrete clients: - InMemoryLangfuseClient bundled for tests and the conformance harness; captures every Trace + Observation as plain dataclass records inspectable by assertions. - A real langfuse.Langfuse() SDK instance is Protocol-compatible and drops in unchanged for production use. SDK versions whose shape diverges plug in via a small adapter the user writes; the shape is documented in examples/10-langfuse-observability. Trace id equals the framework-minted invocation_id verbatim so cross-system lookup by invocation_id finds the Langfuse Trace directly. correlation_id surfaces on both trace.metadata and every observation.metadata for cross-backend join. Trace name sources from the entry-node name; the caller-supplied invocation-label path lands in proposal 0034 (PR 4). Generation rendering follows the §8.7 contract: input/output/ request_extras appear only when disable_llm_payload=False; the §5.5.5 truncation marker is preserved verbatim as a raw string when the serialized payload exceeds payload_byte_cap. Prompt linkage follows §8.4.4 case discrimination: reads Prompt.observability_entities['langfuse_prompt'] (the field added in proposal 0033) to establish a native Prompt-entity link when the prompt's source exposes one. Backends without that reference (filesystem, in-memory) produce metadata-only linkage. Three conformance fixtures pass: 022 basic trace, 023 Generation rendering plus truncation case, 024 prompt linkage both cases. Seven new unit tests cover payload-cap validation, in-memory recorder field handling, and observation parent walking. Example 10-langfuse-observability is a runnable moon-themed demo: a lunar mission Q&A pipeline with a mock Langfuse-source prompt backend, the LangfuseObserver wired to the in-memory recorder, and a pretty-printer for the captured Trace tree at the end. Real SDK swap is a one-line constructor change. Third of 6 PRs in the v0.10.0 batch.
1 parent afbd47b commit cb2cf1c

12 files changed

Lines changed: 2209 additions & 3 deletions

File tree

docs/concepts/observability.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -586,3 +586,100 @@ appear dropped. Two workarounds:
586586
- Use `SimpleSpanProcessor` instead of `BatchSpanProcessor` in
587587
tests; it exports synchronously and is unaffected by teardown
588588
timing.
589+
590+
## Langfuse mapping (opt-in)
591+
592+
A second sibling observer maps the same `NodeEvent` stream onto
593+
Langfuse's native Trace + Observation data model — Traces at the
594+
top, Span observations for graph nodes, Generation observations for
595+
LLM calls. Use it instead of (or alongside) the OTel observer when
596+
your trace UI is Langfuse and you want first-class Generation
597+
rendering without going through Langfuse's OTLP ingest.
598+
599+
```python
600+
from openarmature.observability.langfuse import (
601+
InMemoryLangfuseClient,
602+
LangfuseObserver,
603+
)
604+
605+
client = InMemoryLangfuseClient() # or langfuse.Langfuse(...) in prod
606+
observer = LangfuseObserver(client=client)
607+
graph.attach_observer(observer)
608+
```
609+
610+
The `client` is anything matching the `LangfuseClient` Protocol —
611+
the bundled `InMemoryLangfuseClient` (used by the conformance
612+
harness, useful for unit tests), or a real `langfuse.Langfuse()`
613+
instance from the [Langfuse Python SDK](https://github.com/langfuse/langfuse-python).
614+
The Protocol declares only the methods the observer calls, so SDK
615+
versions whose shape matches drop in directly. SDK versions whose
616+
shape diverges (renamed kwargs, return-type quirks) plug in via a
617+
small adapter; see
618+
[`examples/10-langfuse-observability`](../examples/10-langfuse-observability.md)
619+
for the runnable demo plus the adapter shape.
620+
621+
### What Langfuse sees
622+
623+
- **Trace ID = invocation ID.** The Trace's `id` is the OA
624+
`invocation_id` verbatim, so cross-system lookup by invocation_id
625+
finds the Langfuse Trace directly (spec §8.4.1).
626+
- **Trace name.** Defaults to the entry-node name (spec §8.6
627+
fallback). Caller-supplied invocation labels land in PR 4
628+
(proposal 0034).
629+
- **Per-observation metadata.** Each Span / Generation carries
630+
`namespace`, `step`, `attempt_index`, optional `fan_out_index` /
631+
`branch_name`, and the `correlation_id` cross-cutting join key
632+
(spec §8.5).
633+
- **Generation fields.** LLM calls become Generation observations
634+
with `model`, `model_parameters` (the `gen_ai.request.*` request
635+
parameters lifted by inclusion per §8.4.3), `usage` (input /
636+
output / total tokens), and `metadata.finish_reason` /
637+
`system` / `response_model` / `response_id`.
638+
639+
### Payload + truncation
640+
641+
`disable_llm_payload` mirrors the OTel observer's flag — defaults
642+
to `True` for the same privacy reason. Flip to `False` to populate
643+
`generation.input` / `output` / `metadata.request_extras` from the
644+
LLM event payload.
645+
646+
```python
647+
observer = LangfuseObserver(
648+
client=client,
649+
disable_llm_payload=False,
650+
payload_byte_cap=65536,
651+
)
652+
```
653+
654+
When a payload exceeds `payload_byte_cap`, the observer emits the
655+
serialized form with the §5.5.5 truncation marker
656+
(`…[truncated, M bytes total]`) verbatim as a raw string instead of
657+
parsing back to native shape. The unparseable JSON IS the
658+
truncation signal in the Langfuse UI.
659+
660+
### Prompt linkage
661+
662+
When a Prompt's source backend exposes a Langfuse Prompt entity
663+
reference under `Prompt.observability_entities['langfuse_prompt']`,
664+
the Generation observation links to that entity natively (spec
665+
§8.4.4 case 1). Backends that don't surface a Langfuse reference
666+
(filesystem, in-memory, etc.) leave the Generation with
667+
`metadata.prompt` populated but no entity link (case 2).
668+
669+
### Composition with OTel
670+
671+
The two observers are independent §6 event consumers and can be
672+
attached together. They share the `correlation_id` as the
673+
cross-backend join key — find a slow Generation in Langfuse, search
674+
for its `correlation_id` in OTel logs, see the surrounding
675+
infrastructure activity.
676+
677+
```python
678+
otel_observer = OTelObserver(span_processor=...)
679+
langfuse_observer = LangfuseObserver(client=langfuse_client)
680+
graph.attach_observer(otel_observer)
681+
graph.attach_observer(langfuse_observer)
682+
```
683+
684+
Each observer's `disable_llm_spans` / `disable_llm_payload` flag is
685+
independent; one MAY emit while the other suppresses.
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# 10 - Langfuse observability
2+
3+
Send LLM call observability to Langfuse natively — Trace at the top,
4+
Span observations for graph nodes, Generation observations with input,
5+
output, token usage, model parameters, and a native link back to the
6+
prompt entity the call rendered from.
7+
8+
## Overview
9+
10+
A mission-briefing assistant answers questions about Apollo and Artemis
11+
missions. The pipeline fetches a versioned prompt template, renders it
12+
with the user's question, sends the rendered messages to the model,
13+
and stores the response. The Langfuse observer captures the full call
14+
shape as the graph runs.
15+
16+
The demo's prompt backend stubs a Langfuse-source by attaching a
17+
sentinel `langfuse_prompt` reference to the rendered prompt. The
18+
Generation observation reads that reference and links back to the
19+
prompt entity — exactly what you'd see in a production Langfuse
20+
dashboard threading "this generation came from prompt v7" without any
21+
manual wiring at the call site.
22+
23+
## What it teaches
24+
25+
- [`LangfuseObserver`](../concepts/observability.md#langfuse-mapping-opt-in)
26+
attaches like any other observer; nothing in the node code knows or
27+
cares about which backend is recording.
28+
- The `LangfuseClient` Protocol decouples the observer from the SDK.
29+
The bundled `InMemoryLangfuseClient` recorder is the test/demo
30+
shape; production passes a real `langfuse.Langfuse()` instance (or
31+
a thin adapter — see [Reading the output](#reading-the-output)
32+
below).
33+
- Prompt linkage through
34+
[`Prompt.observability_entities`](../concepts/prompts.md#backend-keyed-observability-entity-references):
35+
a prompt backend that exposes a Langfuse Prompt entity reference
36+
surfaces it on every Generation that renders from that prompt.
37+
Filesystem / in-memory backends without that reference work too,
38+
they just produce metadata-only linkage.
39+
- `disable_llm_payload=False` opt-in for capturing input messages +
40+
output content on Generation observations. Default-off is the
41+
privacy posture; the demo deliberately flips it.
42+
- `correlation_id` cross-cutting metadata on the Trace and every
43+
Observation — the join key if you're also running an OTel observer
44+
alongside.
45+
46+
## How to run
47+
48+
```bash
49+
uv sync --group examples
50+
LLM_API_KEY=sk-... uv run python examples/10-langfuse-observability/main.py \
51+
"what year did Apollo 11 land"
52+
```
53+
54+
The first positional arg becomes the question. The demo uses an
55+
in-memory recorder so no Langfuse account is needed.
56+
57+
## The graph
58+
59+
```mermaid
60+
flowchart TD
61+
start([start])
62+
answer[answer_briefing]
63+
stop([end])
64+
65+
start --> answer --> stop
66+
```
67+
68+
A single-node graph: fetch the prompt, render with the question, call
69+
the LLM under `with_active_prompt(...)`, store the response. The
70+
single node is deliberate — the value is in the captured Trace shape,
71+
not the graph topology.
72+
73+
## Reading the output
74+
75+
After the answer prints, the script renders the captured Langfuse
76+
Trace + Observation tree:
77+
78+
```
79+
question: what year did Apollo 11 land
80+
answer: Apollo 11 landed on the Moon on July 20, 1969 ...
81+
prompt: mission-briefing v7
82+
83+
─── captured Langfuse trace ─────────────────────────────────
84+
Trace id=01234567-89ab-...
85+
name='answer_briefing'
86+
metadata={correlation_id='...', entry_node='answer_briefing', spec_version='0.26.0'}
87+
[span] 'answer_briefing' level=DEFAULT
88+
metadata={attempt_index=0, correlation_id='...', namespace=['answer_briefing'], step=0}
89+
[generation] 'openarmature.llm.complete' level=DEFAULT
90+
metadata={correlation_id='...', finish_reason='stop', prompt={...},
91+
response_id='...', response_model='gpt-4o-mini-2024-...',
92+
system='openai'}
93+
model='gpt-4o-mini'
94+
usage=input:48 output:32 total:80
95+
prompt_entity_link='lf-prompt-mission-briefing-v7'
96+
output='Apollo 11 landed on the Moon on July 20, 1969 ...'
97+
```
98+
99+
- **Trace name = entry node name** by default. The caller-supplied
100+
invocation-label path (a per-`invoke()` argument that overrides the
101+
default) ships with proposal 0034's caller-metadata work.
102+
- **Span observation per node.** `answer_briefing` is the only node
103+
here; a multi-node graph would produce a tree of nested Span
104+
observations under the Trace.
105+
- **Generation observation per LLM call.** Carries `model`, `usage`,
106+
`output`, and the prompt-identity metadata. In a production Langfuse
107+
dashboard this is what the "Generation" detail view renders.
108+
- **`prompt_entity_link`** is the value `Prompt.observability_entities['langfuse_prompt']`
109+
carried — a sentinel string in this demo, a real Langfuse SDK Prompt
110+
object in production. When the backend doesn't surface the reference
111+
(e.g., a filesystem backend), the link is absent but the
112+
`metadata.prompt` map (name, version, label, hashes) still appears
113+
for traceability.
114+
115+
## Swapping to a real Langfuse SDK
116+
117+
The observer's `client` parameter is `LangfuseClient`-Protocol-typed,
118+
so any structurally-compatible value works:
119+
120+
```python
121+
from langfuse import Langfuse
122+
123+
client = Langfuse(
124+
public_key="pk-lf-...",
125+
secret_key="sk-lf-...",
126+
host="https://cloud.langfuse.com",
127+
)
128+
observer = LangfuseObserver(client=client, disable_llm_payload=False)
129+
```
130+
131+
If the installed SDK version's `trace` / `span` / `generation` method
132+
signatures match the Protocol exactly, this is the whole change. If
133+
they diverge (renamed kwargs, return-type quirks), wrap the SDK in a
134+
small adapter class that implements `LangfuseClient` and delegates to
135+
the SDK call-by-call. The Protocol surface is narrow — four methods —
136+
so the adapter is on the order of 40 lines.
137+
138+
For prompt linkage: in production, the
139+
`Prompt.observability_entities['langfuse_prompt']` value is the SDK's
140+
own Prompt-entity object (returned by `langfuse_client.get_prompt(...)`)
141+
rather than the sentinel string this demo uses. The observer passes
142+
that value straight through to the SDK's `generation(..., prompt=...)`
143+
argument, which is what the SDK uses to establish the native link.
144+
145+
## Composition with OTel
146+
147+
Both observers consume the same `NodeEvent` stream and can be attached
148+
together:
149+
150+
```python
151+
graph.attach_observer(OTelObserver(span_processor=batch))
152+
graph.attach_observer(LangfuseObserver(client=langfuse_client))
153+
```
154+
155+
Their `disable_llm_spans` / `disable_llm_payload` flags are
156+
independent. The `correlation_id` cross-cutting attribute is the join
157+
key — find a slow Generation in Langfuse, search for the
158+
`correlation_id` in OTel logs to see the surrounding infrastructure
159+
activity.

0 commit comments

Comments
 (0)