Skip to content

Complete 0067 embedding metrics#239

Merged
chris-colinsky merged 1 commit into
mainfrom
feature/0067-embedding-metrics
Jul 25, 2026
Merged

Complete 0067 embedding metrics#239
chris-colinsky merged 1 commit into
mainfrom
feature/0067-embedding-metrics

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Completes proposal 0067 (OTel GenAI metrics, observability §11) by wiring the metric instruments to the embedding event, flipping 0067 from partial to implemented. This is the last item in v0.17.0's Wave 2.

Context

0067's LLM path (fixtures 088/090/091) shipped in v0.15.0, recording two OA-namespaced instruments per LLM attempt. The embedding-call metrics were deferred because no embedding event existed yet; v0.16.0's retrieval work now dispatches EmbeddingEvent/EmbeddingFailedEvent, so this wires the same instruments to that path.

What it adds

  • observability/otel/observer.py_record_embedding_metrics(event) records the same token.usage + operation.duration histograms from the terminal EmbeddingEvent/EmbeddingFailedEvent (the operation dimension separates embeddings from chat), with operation="embeddings" and token.type="input" — an embedding call has input tokens only (EmbeddingUsage.input_tokens, no output). Duration records on every call including a failure (error.type); token.usage only when a usage record is present. Dispatched beside _handle_embedding, gated on enable_metrics and independent of disable_llm_spans.
  • Conformance — un-defers fixtures 089 (usage: token + duration) and 143 (no-usage: duration only, no token observation); marks 0067 implemented.

Rerank metrics (fixture 109) stay deferred: out of scope per the proposal.

Tests

Fixtures 089/143 plus unit coverage for embedding metrics: with usage, a reported input_tokens=0 (records a zero observation, distinct from absent usage), no-usage, and a failure (duration with error.type — the one path no fixture covers). Full suite green; ruff and pyright clean.

Wire the section 11 GenAI metric instruments to the embedding event,
flipping proposal 0067 from partial to implemented.

- otel: _record_embedding_metrics records the same token.usage +
  operation.duration histograms from the terminal EmbeddingEvent /
  EmbeddingFailedEvent, with operation="embeddings" and input-only
  tokens (an embedding call has no output tokens); duration every
  call including a failure (error.type), token only when usage is
  present.
- conformance: un-defer fixtures 089 (usage) and 143 (no-usage);
  mark 0067 implemented. Rerank metrics (109) stay deferred, out of
  scope per the proposal.
Copilot AI review requested due to automatic review settings July 25, 2026 01:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR completes proposal 0067’s remaining gap by recording OTel GenAI metric instruments for embedding calls, using the terminal EmbeddingEvent/EmbeddingFailedEvent path introduced in v0.16.0. It aligns embedding metrics behavior with the existing LLM attempt metrics (same instruments, different openarmature.gen_ai.operation).

Changes:

  • Add _record_embedding_metrics(event) to OTelObserver and invoke it for embedding success/failure events when enable_metrics is on.
  • Un-defer and run conformance fixtures 089 and 143 by extending the embedding fixture runner to capture and assert metric points + embedding-specific metric invariants.
  • Add unit tests covering embedding metrics for usage present, usage present with input_tokens=0, usage absent, and failure (error.type on duration).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
src/openarmature/observability/otel/observer.py Records token.usage (input-only) and operation.duration for embedding events under openarmature.gen_ai.operation="embeddings" when metrics are enabled.
tests/conformance/test_observability.py Enables embedding metrics fixtures (089/143) and adds an embedding-metrics invariant assertion layer over captured metric points.
tests/unit/test_observability_otel.py Adds unit coverage for embedding metrics across usage/no-usage/zero-usage/failure scenarios.
conformance.toml Marks proposal 0067 as implemented in v0.17.0 and updates the proposal note to reflect embedding metrics completion.

@chris-colinsky
chris-colinsky merged commit b875ea6 into main Jul 25, 2026
7 checks passed
@chris-colinsky
chris-colinsky deleted the feature/0067-embedding-metrics branch July 25, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants