Skip to content

Render embedding observability (0059b)#198

Merged
chris-colinsky merged 1 commit into
mainfrom
feature/0059b-embedding-observability
Jun 29, 2026
Merged

Render embedding observability (0059b)#198
chris-colinsky merged 1 commit into
mainfrom
feature/0059b-embedding-observability

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Summary

Renders the embedding observability for proposal 0059 (the "0059b" half),
completing the embedding capability merged in #196. The provider already
dispatched the typed EmbeddingEvent / EmbeddingFailedEvent; both bundled
observers now render them, the conformance harness drives them, and fixtures
074-083 + 137 pass.

Changes

  • Event: EmbeddingEvent gains output_vectors (proposal 0089), populated
    on the success event from the response vectors; the failure event carries no
    output.
  • OTel: an openarmature.embedding.complete span parented lineage-aware
    under the calling node, carrying the disable_genai_semconv-gated GenAI
    subset, the openarmature.embedding.input_count / .dimensions identity
    attrs, and the disable_provider_payload-gated .input.strings /
    .request.extras; the failure path renders ERROR + error.type + an
    exception event. The attribute_enrichers union is widened to the embedding
    events.
  • Langfuse: a dedicated Embedding observation (asType="embedding") with
    model, usageDetails, the openarmature_* metadata, and the payload-gated
    input strings + output vectors; the failure path renders an ERROR-level
    observation.
  • Harness: a graph-node embedding runner so the events flow through the
    observer queue; un-defers 074-083 + 137; extends the Langfuse
    observation-tree assertion for model + usageDetails.
  • conformance.toml: 0059 to implemented. Fixture 089 (embedding metrics)
    stays deferred under 0067 (the GenAI metric instruments are not yet wired to
    the embedding event).

Tests

  • Fixtures 074-083 + 137 pass (089 metrics deferred).
  • Full conformance + unit + smoke: 1487 passed. ruff + pyright clean.

Follow-ups

Render the dispatched EmbeddingEvent / EmbeddingFailedEvent on both
bundled observers and wire the conformance harness, completing the 0059
embedding capability.

- Add output_vectors to EmbeddingEvent (proposal 0089), populated on the
  success event from the response vectors; the failure event carries no
  output.
- OTel: an openarmature.embedding.complete span parented lineage-aware
  under the calling node, with the gated GenAI semconv subset + the OA
  embedding identity attrs + payload-gated input.strings / request.extras;
  failure renders ERROR + error.type + an exception event. Widen the
  attribute_enrichers union to the embedding events.
- Langfuse: a dedicated Embedding observation (asType=embedding) with
  model / usageDetails / the openarmature_* metadata + payload-gated
  input strings + output vectors; failure renders an ERROR observation.
- Harness: drive embed() through a graph node so the events flow through
  the observer queue; un-defer fixtures 074-083 + 137 (all pass); extend
  the Langfuse observation-tree assertion for model + usageDetails.
- conformance.toml: 0059 -> implemented. Embedding metrics (fixture 089)
  stay deferred under 0067.
Copilot AI review requested due to automatic review settings June 29, 2026 19:45
Comment thread src/openarmature/observability/langfuse/client.py

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements the “0059b” half of embedding observability: embedding provider calls now produce first-class observability output in both bundled observers (OTel + Langfuse), and the conformance harness runs the embedding observability fixtures end-to-end.

Changes:

  • Extend EmbeddingEvent with output_vectors and populate it from EmbeddingResponse.vectors on success.
  • Render embedding calls in observability backends: OTel emits openarmature.embedding.complete spans; Langfuse emits a dedicated asType="embedding" observation (including payload gating behavior).
  • Update conformance harness/fixture plumbing to run embedding observability fixtures (074–083, 137) while keeping embedding-metrics (089) deferred under 0067.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/unit/test_retrieval_provider.py Updates unit tests to reflect embedding events now render, and verifies observer no-op behavior outside invocation context.
tests/conformance/test_observability.py Adds embedding fixture execution path and extends Langfuse observation-tree assertions (model / usageDetails / statusMessage).
tests/conformance/test_fixture_parsing.py Adjusts fixture-parsing deferrals now that embedding observability fixtures run in test_observability.
src/openarmature/retrieval/providers/openai.py Populates EmbeddingEvent.output_vectors from the parsed embedding response vectors.
src/openarmature/observability/otel/observer.py Adds typed embedding event handling and emits openarmature.embedding.complete spans with proper gating and error mapping.
src/openarmature/observability/langfuse/observer.py Adds typed embedding event handling and emits dedicated Langfuse embedding observations with proper gating and error mapping.
src/openarmature/observability/langfuse/client.py Extends the Langfuse Protocol + in-memory recorder to support an embedding observation type.
src/openarmature/observability/langfuse/adapter.py Implements the adapter-side embedding() operation (including back-dated start time support).
src/openarmature/graph/events.py Adds EmbeddingEvent.output_vectors field (proposal 0089) and documents its rendering contract.
conformance.toml Marks proposal 0059 as implemented (since 0.16.0) and updates 0067 note to reflect remaining embedding-metrics gap.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@chris-colinsky chris-colinsky merged commit e3b3853 into main Jun 29, 2026
7 checks passed
@chris-colinsky chris-colinsky deleted the feature/0059b-embedding-observability branch June 29, 2026 20:02
chris-colinsky added a commit that referenced this pull request Jul 2, 2026
* Bump spec conformance pin to v0.88.0

Advance the pinned spec submodule v0.84.0 -> v0.88.0, pulling in
proposals 0090 (Cohere rerank wire), 0091 (Cohere embed wire), 0092
(embedding batch chunking), and 0093 (nullable provider usage records).

Record all four as not-yet in the manifest and defer the fixtures the
jump adds (retrieval 028-038, observability 139-143), so a green run
still means what we implement passes. The two rerank no-usage fixtures
also defer at the parse layer, since calls_rerank is unmodeled in the
harness schema.

Correct proposal 0089 from not-yet to partial: its embedding half
(output_vectors plus the Langfuse output rendering) shipped in #198,
while the rerank half rides the unshipped rerank capability (0060).

Add retrieval-provider to the AGENTS.md generator so the bundled agent
guide describes the embedding and rerank contracts, and regenerate the
bundle.

* Clarify v0.84.0 wire-fixture history in comment

Reword the deferred-fixture header so the v0.84.0 line reads as when the
013-027 wire fixtures arrived, not as the current pin (now v0.88.0).
Addresses a CoPilot review note on PR #203.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants