Render embedding observability (0059b)#198
Merged
Merged
Conversation
Render the dispatched EmbeddingEvent / EmbeddingFailedEvent on both bundled observers and wire the conformance harness, completing the 0059 embedding capability. - Add output_vectors to EmbeddingEvent (proposal 0089), populated on the success event from the response vectors; the failure event carries no output. - OTel: an openarmature.embedding.complete span parented lineage-aware under the calling node, with the gated GenAI semconv subset + the OA embedding identity attrs + payload-gated input.strings / request.extras; failure renders ERROR + error.type + an exception event. Widen the attribute_enrichers union to the embedding events. - Langfuse: a dedicated Embedding observation (asType=embedding) with model / usageDetails / the openarmature_* metadata + payload-gated input strings + output vectors; failure renders an ERROR observation. - Harness: drive embed() through a graph node so the events flow through the observer queue; un-defer fixtures 074-083 + 137 (all pass); extend the Langfuse observation-tree assertion for model + usageDetails. - conformance.toml: 0059 -> implemented. Embedding metrics (fixture 089) stay deferred under 0067.
There was a problem hiding this comment.
Pull request overview
Implements the “0059b” half of embedding observability: embedding provider calls now produce first-class observability output in both bundled observers (OTel + Langfuse), and the conformance harness runs the embedding observability fixtures end-to-end.
Changes:
- Extend
EmbeddingEventwithoutput_vectorsand populate it fromEmbeddingResponse.vectorson success. - Render embedding calls in observability backends: OTel emits
openarmature.embedding.completespans; Langfuse emits a dedicatedasType="embedding"observation (including payload gating behavior). - Update conformance harness/fixture plumbing to run embedding observability fixtures (074–083, 137) while keeping embedding-metrics (089) deferred under 0067.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_retrieval_provider.py | Updates unit tests to reflect embedding events now render, and verifies observer no-op behavior outside invocation context. |
| tests/conformance/test_observability.py | Adds embedding fixture execution path and extends Langfuse observation-tree assertions (model / usageDetails / statusMessage). |
| tests/conformance/test_fixture_parsing.py | Adjusts fixture-parsing deferrals now that embedding observability fixtures run in test_observability. |
| src/openarmature/retrieval/providers/openai.py | Populates EmbeddingEvent.output_vectors from the parsed embedding response vectors. |
| src/openarmature/observability/otel/observer.py | Adds typed embedding event handling and emits openarmature.embedding.complete spans with proper gating and error mapping. |
| src/openarmature/observability/langfuse/observer.py | Adds typed embedding event handling and emits dedicated Langfuse embedding observations with proper gating and error mapping. |
| src/openarmature/observability/langfuse/client.py | Extends the Langfuse Protocol + in-memory recorder to support an embedding observation type. |
| src/openarmature/observability/langfuse/adapter.py | Implements the adapter-side embedding() operation (including back-dated start time support). |
| src/openarmature/graph/events.py | Adds EmbeddingEvent.output_vectors field (proposal 0089) and documents its rendering contract. |
| conformance.toml | Marks proposal 0059 as implemented (since 0.16.0) and updates 0067 note to reflect remaining embedding-metrics gap. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
chris-colinsky
added a commit
that referenced
this pull request
Jul 2, 2026
* Bump spec conformance pin to v0.88.0 Advance the pinned spec submodule v0.84.0 -> v0.88.0, pulling in proposals 0090 (Cohere rerank wire), 0091 (Cohere embed wire), 0092 (embedding batch chunking), and 0093 (nullable provider usage records). Record all four as not-yet in the manifest and defer the fixtures the jump adds (retrieval 028-038, observability 139-143), so a green run still means what we implement passes. The two rerank no-usage fixtures also defer at the parse layer, since calls_rerank is unmodeled in the harness schema. Correct proposal 0089 from not-yet to partial: its embedding half (output_vectors plus the Langfuse output rendering) shipped in #198, while the rerank half rides the unshipped rerank capability (0060). Add retrieval-provider to the AGENTS.md generator so the bundled agent guide describes the embedding and rerank contracts, and regenerate the bundle. * Clarify v0.84.0 wire-fixture history in comment Reword the deferred-fixture header so the v0.84.0 line reads as when the 013-027 wire fixtures arrived, not as the current pin (now v0.88.0). Addresses a CoPilot review note on PR #203.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Renders the embedding observability for proposal 0059 (the "0059b" half),
completing the embedding capability merged in #196. The provider already
dispatched the typed
EmbeddingEvent/EmbeddingFailedEvent; both bundledobservers now render them, the conformance harness drives them, and fixtures
074-083 + 137 pass.
Changes
EmbeddingEventgainsoutput_vectors(proposal 0089), populatedon the success event from the response vectors; the failure event carries no
output.
openarmature.embedding.completespan parented lineage-awareunder the calling node, carrying the
disable_genai_semconv-gated GenAIsubset, the
openarmature.embedding.input_count/.dimensionsidentityattrs, and the
disable_provider_payload-gated.input.strings/.request.extras; the failure path renders ERROR +error.type+ anexception event. The
attribute_enrichersunion is widened to the embeddingevents.
Embeddingobservation (asType="embedding") withmodel,
usageDetails, theopenarmature_*metadata, and the payload-gatedinput strings + output vectors; the failure path renders an ERROR-level
observation.
observer queue; un-defers 074-083 + 137; extends the Langfuse
observation-tree assertion for
model+usageDetails.stays deferred under 0067 (the GenAI metric instruments are not yet wired to
the embedding event).
Tests
Follow-ups
EmbeddingEvent(un-defers 089).