Simplify validation in conformance tests#276
Conversation
expected_spans becomes a dict[str, int] of exact per-operation span counts, and the base Scenario.validate fails on any mismatch (conformance replays recorded cassettes, so counts are deterministic). Scenarios pin real counts here instead of re-asserting them in redundant validate overrides.
There was a problem hiding this comment.
Pull request overview
This PR updates the GenAI conformance-test framework so scenarios declare exact per-operation span counts (by gen_ai.operation.name), and the base Scenario.validate() enforces an exact match, removing the need for redundant per-scenario span-count assertions.
Changes:
- Change
Scenario.expected_spansfrom a tuple of operation names to adict[str, int]of exact counts, and makeScenario.validate()fail on any mismatch (missing or extra operations). - Update existing conformance scenarios across multiple instrumentations to declare the exact operation counts (and remove now-redundant validate overrides/imports).
- Update the conformance-test authoring skill documentation to reflect the new
expected_spanscontract.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| util/opentelemetry-test-util-genai/src/opentelemetry/test_util_genai/conformance.py | Switches span validation to exact per-operation counts and updates helper to return counts. |
| instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/tool_calling.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/inference.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/generate_content.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-google-genai/tests/conformance/embedding.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/tool_calling.py | Updates expected span counts and removes redundant validate override/import. |
| instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/responses_stream.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/responses_conversation.py | Updates expected span counts and removes redundant validate override/import. |
| instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/inference.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/embedding.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-genai-openai-agents/tests/conformance/orchestration.py | Updates expected span counts and removes redundant span-count assertions in validate. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/workflow.py | Updates expected span counts and removes redundant validate override/import. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/tool_calling.py | Updates expected span counts and removes redundant validate override/import. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/inference.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/conformance/agent.py | Updates expected span counts and removes redundant validate override/import. |
| instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/tool_calling.py | Updates scenario to declare exact span operation counts. |
| instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/inference.py | Updates scenario to declare exact span operation counts. |
| .github/skills/write-conformance-tests/SKILL.md | Documents the new expected_spans: dict[str, int] behavior and guidance. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
This PR has review comments. Review suggestions, whether from maintainers or automated reviewers, aren't always correct or required. Please evaluate each comment on its merits, then make sure each thread has a clear outcome. For example, link to the commit if you applied a suggestion, explain why it wasn't applied, or ask a follow-up question. Automation flags a PR for human review once every review thread has a reply or is marked as resolved. Status across open PRs is visible on the pull request dashboard. |
expected_spans becomes a dict[str, int] of exact per-operation span counts, and the base Scenario.validate fails on any mismatch (conformance replays recorded cassettes, so counts are deterministic). Scenarios pin real counts here instead of re-asserting them in redundant validate overrides.