Skip to content

Simplify validation in conformance tests#276

Merged
lmolkova merged 3 commits into
open-telemetry:mainfrom
lmolkova:refactor/conformance-expected-spans
Jul 16, 2026
Merged

Simplify validation in conformance tests#276
lmolkova merged 3 commits into
open-telemetry:mainfrom
lmolkova:refactor/conformance-expected-spans

Conversation

@lmolkova

Copy link
Copy Markdown
Member

expected_spans becomes a dict[str, int] of exact per-operation span counts, and the base Scenario.validate fails on any mismatch (conformance replays recorded cassettes, so counts are deterministic). Scenarios pin real counts here instead of re-asserting them in redundant validate overrides.

expected_spans becomes a dict[str, int] of exact per-operation span counts,
and the base Scenario.validate fails on any mismatch (conformance replays
recorded cassettes, so counts are deterministic). Scenarios pin real counts
here instead of re-asserting them in redundant validate overrides.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the GenAI conformance-test framework so scenarios declare exact per-operation span counts (by gen_ai.operation.name), and the base Scenario.validate() enforces an exact match, removing the need for redundant per-scenario span-count assertions.

Changes:

  • Change Scenario.expected_spans from a tuple of operation names to a dict[str, int] of exact counts, and make Scenario.validate() fail on any mismatch (missing or extra operations).
  • Update existing conformance scenarios across multiple instrumentations to declare the exact operation counts (and remove now-redundant validate overrides/imports).
  • Update the conformance-test authoring skill documentation to reflect the new expected_spans contract.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
util/opentelemetry-test-util-genai/src/opentelemetry/test_util_genai/conformance.py Switches span validation to exact per-operation counts and updates helper to return counts.
instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/tool_calling.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/inference.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/generate_content.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/embedding.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/tool_calling.py Updates expected span counts and removes redundant validate override/import.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/responses_stream.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/responses_conversation.py Updates expected span counts and removes redundant validate override/import.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/inference.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/embedding.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-genai-openai-agents/tests/conformance/orchestration.py Updates expected span counts and removes redundant span-count assertions in validate.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/workflow.py Updates expected span counts and removes redundant validate override/import.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/tool_calling.py Updates expected span counts and removes redundant validate override/import.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/inference.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/agent.py Updates expected span counts and removes redundant validate override/import.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/tool_calling.py Updates scenario to declare exact span operation counts.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/inference.py Updates scenario to declare exact span operation counts.
.github/skills/write-conformance-tests/SKILL.md Documents the new expected_spans: dict[str, int] behavior and guidance.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@opentelemetry-pr-dashboard

Copy link
Copy Markdown

This PR has review comments. Review suggestions, whether from maintainers or automated reviewers, aren't always correct or required. Please evaluate each comment on its merits, then make sure each thread has a clear outcome.

For example, link to the commit if you applied a suggestion, explain why it wasn't applied, or ask a follow-up question.

Automation flags a PR for human review once every review thread has a reply or is marked as resolved.

Status across open PRs is visible on the pull request dashboard.

@lmolkova lmolkova added the Skip Changelog PR does not require a changelog entry label Jul 16, 2026
@lmolkova
lmolkova added this pull request to the merge queue Jul 16, 2026
Merged via the queue into open-telemetry:main with commit 1e27b90 Jul 16, 2026
67 checks passed
@lmolkova
lmolkova deleted the refactor/conformance-expected-spans branch July 16, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Skip Changelog PR does not require a changelog entry

Development

Successfully merging this pull request may close these issues.

3 participants