Skip to content

Commit c2967c4

Browse files
lmolkovalmolkova
authored
Instrumentations and GenAI: Add copilot review instructions, update agents.md (#4457)
* add copilot reivew instructions for GenAI libs * up * up * expand beyond genai, create common files * up * changelog
1 parent 2caa924 commit c2967c4

7 files changed

Lines changed: 318 additions & 21 deletions

File tree

‎.github/instructions/instrumentation-genai.instructions.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
applyTo: "instrumentation-genai/**"
3+
---
4+
5+
Review rules for PRs touching `instrumentation-genai/**`. Flag violations with a link to the rule.
6+
7+
These rules are additive to
8+
[`instrumentation.instructions.md`](instrumentation.instructions.md), which applies to all
9+
instrumentation packages.
10+
11+
## 1. Scope of `instrumentation-genai/`
12+
13+
Only for:
14+
15+
- Generative AI inference providers,
16+
- Agentic frameworks,
17+
- Libraries directly supporting the above (e.g., MCP, GenAI protocols).
18+
19+
Database clients (including vector DBs used outside a GenAI-specific client) and CLI libs belong
20+
in `instrumentation/`, not here.
21+
22+
## 2. GenAI component ownership
23+
24+
See [`CONTRIBUTING.md#guideline-for-genai-instrumentations`](../../CONTRIBUTING.md#guideline-for-genai-instrumentations)
25+
for GenAI-specific maintenance expectations on top of the general
26+
[instrumentation checklist](../../CONTRIBUTING.md#guideline-for-instrumentations).
27+
28+
## 3. Telemetry and configuration via `opentelemetry-util-genai`
29+
30+
- Spans, logs, metrics, and events must go through `opentelemetry-util-genai`. Direct use of
31+
`Tracer`, `Meter`, `Logger`, or event APIs is not allowed.
32+
- Content capture, hooks, and other cross-cutting configuration are owned by the util.
33+
Instrumentations must not introduce their own env vars, settings, or hook interfaces.
34+
- Message content, prompts, and tool call arguments must only be set through the util's content
35+
capture path — never as unconditional span/log attributes.
36+
- Adding attributes to invocations produced by the util is fine.
37+
- If a capability is missing in `opentelemetry-util-genai`, land it in the util first.
38+
39+
## 4. GenAI semantic conventions
40+
41+
- Attributes, spans, events, and metrics must match the
42+
[GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai).
43+
- `gen_ai.*` attribute names must come from
44+
`opentelemetry.semconv._incubating.attributes.gen_ai_attributes`.
45+
- For attributes with a well-known value set, use the generated enum from the same module
46+
(e.g. `GenAiOutputTypeValues` for `gen_ai.output.type`) instead of string literals.
47+
48+
## 5. Tests
49+
50+
- Use recorded VCR cassettes for provider calls. No live-key-only tests; skipping on missing key
51+
is not acceptable.
52+
- Cover streaming and non-streaming variants when both exist.
53+
- For error scenarios, at minimum include: provider error / endpoint unavailable, stream
54+
interrupted by network, stream closed early by the caller.
55+
56+
## 6. Examples
57+
58+
New instrumentations must ship a minimal example under the package's `examples/`, with both a
59+
`manual/` and a `zero-code/` (auto-instrumentation) variant.
60+
61+
## 7. PR description
62+
63+
- Cover which part of the GenAI semconv the change implements or follows (when applicable) and
64+
how instrumentations should consume it.
65+
66+
See also [AGENTS.md](../../AGENTS.md) for general repo rules.

‎.github/instructions/instrumentation.instructions.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
applyTo: "{instrumentation,instrumentation-genai}/**"
3+
---
4+
5+
Review rules for PRs touching `instrumentation/**` and `instrumentation-genai/**`. Flag violations
6+
with a link to the rule.
7+
8+
## 0. Reviewer mindset
9+
10+
Review as long-term maintainer.
11+
12+
For new instrumentations, consult upstream library docs and judge:
13+
14+
- Does the library already emit its own telemetry, making this instrumentation redundant?
15+
- Is the library used widely enough to warrant a package in this repo?
16+
- Does it avoid unbounded in-memory accumulation or other side-effects?
17+
18+
For changes to existing instrumentations: prefer back-compat. Break users only for a real reason;
19+
prefer opt-in or additive. Breaking changes need explicit justification in the PR.
20+
21+
## 1. Component ownership & maintenance commitment
22+
23+
- New instrumentations must add an entry under the correct folder in
24+
[`component_owners.yml`](../component_owners.yml) in the same PR. Contributor must commit to
25+
long-term maintenance. See
26+
[Expectations from contributors](../../CONTRIBUTING.md#expectations-from-contributors) and the
27+
general [instrumentation checklist](../../CONTRIBUTING.md#guideline-for-instrumentations).
28+
29+
## 2. Semantic conventions
30+
31+
- Attribute names must come from the semconv attribute modules, not hardcoded strings. Use the
32+
module matching the namespace under `opentelemetry.semconv` (e.g. `server_attributes`,
33+
`error_attributes`, `http_attributes`, `db_attributes`, …).
34+
- For attributes with a well-known value set in semconv, use the generated enum from the same
35+
modules instead of string literals.
36+
- If a signal is not in semconv, wait until semconv lands.
37+
38+
## 3. Exception handling
39+
40+
- When catching exceptions from the underlying library to record telemetry, always re-raise the
41+
original exception unmodified.
42+
- Do not raise **new** exceptions in instrumentation/telemetry code.
43+
44+
## 4. Tests
45+
46+
- For every public API instrumented, cover sync/async variants when both exist.
47+
- Cover happy path and error scenarios.
48+
- Tests must verify exact attribute names **and value types**, checked against the semconv spec.
49+
- Test against oldest and latest supported library versions via `tests/requirements.{oldest,latest}.txt`
50+
and `{oldest,latest}` `tox.ini` factors.
51+
52+
See also [AGENTS.md](../../AGENTS.md) for general repo rules.

‎.github/instructions/util-genai.instructions.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
applyTo: "util/opentelemetry-util-genai/**"
3+
---
4+
5+
Review rules for PRs touching `util/opentelemetry-util-genai/**`. Flag violations with a link to
6+
the rule.
7+
8+
## 0. Reviewer mindset
9+
10+
Review as long-term maintainer. Every GenAI instrumentation in the repo depends on this package;
11+
churn breaks them. Default to back-compat changes. Every public addition is a long-term
12+
commitment — limit public API.
13+
14+
## 1. Semconv first
15+
16+
No code without semconv. If a signal, attribute, or operation is not in the
17+
[GenAI semconv](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/),
18+
land the semconv change first.
19+
20+
## 2. Semantic conventions
21+
22+
- Names (attributes, operations, spans) must match semconv exactly.
23+
- Use matching `opentelemetry.semconv` modules per namespace (`gen_ai`, `server`, `error`, …).
24+
Do not hardcode name strings if a constant exists.
25+
- Shared attributes must behave consistently across invocation types (same conditions, same
26+
defaults). If semconv marks an attribute for multiple invocations, set it on all.
27+
28+
## 3. API backwards compatibility
29+
30+
- Do not remove or rename public objects. Deprecate first via a docstring note pointing to the
31+
replacement (not `@deprecated` — unreliable).
32+
- Private modules and module-private objects start with `_`.
33+
- Default to internal (`_`-prefixed) unless instrumentations need it public.
34+
35+
## 4. Invocation shape
36+
37+
- `start_*()` factories must accept all sampling-relevant semconv attributes as parameters.
38+
Attributes also marked required by semconv must be required parameters (no default value).
39+
- `start_*()` factories must map 1:1 to distinct semconv operation types (inference, embeddings,
40+
tool execution, agent invocation, workflow invocation). Names must match the operation
41+
unambiguously — e.g., `create_agent` vs `invoke_agent` are distinct ops; `start_agent()` alone
42+
is ambiguous.
43+
- Each operation exposes both a factory (`start_inference(...)`) and a context-manager
44+
(`inference(...)`) form.
45+
- Never construct invocation types directly (`InferenceInvocation(...)`) — skips span creation,
46+
silent no-ops. Always use `handler.start_*()` or the context manager.
47+
48+
## 5. Exception handling
49+
50+
- When catching exceptions from the underlying library to record telemetry, always re-raise the
51+
original exception unmodified.
52+
- Do not raise **new** exceptions in utils code.
53+
54+
## 6. Performance
55+
56+
Keep the hot path tight:
57+
58+
- Avoid per-invocation allocations; do not accumulate state unboundedly.
59+
- Skip content capture when content capture is disabled.
60+
- Skip setting span attributes when the span is not recording.
61+
- Still record attributes that feed metrics — metric recording is independent of span sampling.
62+
63+
## 7. DRY
64+
65+
Do not copy-paste logic across invocation types. Extract shared helpers.
66+
67+
## 8. Tests
68+
69+
- Every new operation type or attribute change needs tests asserting exact attribute names **and
70+
value types**, checked against semconv.
71+
- Cover success (`invocation.stop()`), failure (`invocation.fail(exc)`), and conditional
72+
attribute logic.
73+
- Prefer public API in tests.
74+
75+
## 9. Documentation
76+
77+
- Docstrings for invocation types and span/event helpers must link to the corresponding semconv
78+
op.
79+
- When adding/changing attributes, update the docstring with what is set and when.
80+
81+
## 10. PR description
82+
83+
- Cover which part of the GenAI semconv the change implements or follows (when applicable) and
84+
how instrumentations should consume it.
85+
86+
See also [AGENTS.md](../../AGENTS.md) for general repo rules.

‎AGENTS.md‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,22 @@ Follow the PR scoping guidance in [CONTRIBUTING.md](CONTRIBUTING.md). Keep AI-as
1414
isolated to the requested change and never include unrelated cleanup or opportunistic improvements
1515
unless they are strictly necessary for correctness.
1616

17+
- One logical change per PR. Do not bundle multiple fixes, refactors, or features into the same
18+
PR - split them up so each can be reviewed and reverted independently.
19+
- Run the linter and the relevant tests locally and make sure they pass. See [Commands](#commands).
20+
1721
If you have been assigned an issue by the user or their prompt, please ensure that the
1822
implementation direction is agreed on with the maintainers first in the issue comments. If there are
1923
unknowns, discuss these on the issue before starting implementation. Do not forget that you cannot
20-
comment for users on issue threads on their behalf as it is against the rules of this project.
24+
comment for users on issue or pull request threads on their behalf as it is against the rules of this project.
25+
26+
## PR description
27+
28+
Follow the repo's [PR template](.github/pull_request_template.md) and fill applicable sections.
29+
Keep description short and focus on what is being changed and any gaps or concerns.
30+
31+
AI-generated analyses, long reports, or design dumps go in a relevant issue or a separate PR
32+
comment - not in the PR description.
2133

2234
## Structure
2335

@@ -55,6 +67,35 @@ uv run tox -e typecheck
5567
- Do not add `type: ignore` comments. If a type error arises, solve it properly or write a follow-up plan to address it in another PR.
5668
- Whenever applicable, all code changes should have tests that actually validate the changes.
5769

70+
## Instrumentation rules
71+
72+
Apply to packages under `instrumentation/` and `instrumentation-genai/`.
73+
74+
### Exception handling
75+
76+
- When catching exceptions from the underlying library to record telemetry, always re-raise the
77+
original exception unmodified.
78+
- Do not raise new exceptions in instrumentation/telemetry code.
79+
80+
### Semantic conventions
81+
82+
- Use the semconv attribute and metrics modules under `opentelemetry.semconv` — do not hardcode
83+
attribute or metric name strings.
84+
- For attributes with a well-known value set, use the generated enum from the same module instead
85+
of string literals.
86+
87+
### Tests
88+
89+
- For every public API instrumented, cover sync/async variants when both exist.
90+
- Cover happy path and error scenarios.
91+
- Tests must verify exact attribute names **and value types**, checked against the semconv spec.
92+
- Test against oldest and latest supported library versions via `tests/requirements.{oldest,latest}.txt`
93+
and `{oldest,latest}` `tox.ini` factors.
94+
95+
The parallel PR-review rules live in
96+
[`.github/instructions/instrumentation.instructions.md`](.github/instructions/instrumentation.instructions.md)
97+
and should be kept in sync with this section.
98+
5899
## Commit formatting
59100

60101
We appreciate it if users disclose the use of AI tools when the significant part of a commit is

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1717
([#4244](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/4244))
1818
- `opentelemetry-instrumentation-sqlite3`: Add uninstrument, error status, suppress, and no-op tests
1919
([#4335](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/4335))
20+
- Expand `AGENTS.md` with instrumentation/GenAI guidance and add PR review instructions.
21+
([#4457](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/4457))
2022

2123
### Fixed
2224

‎instrumentation-genai/AGENTS.md‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33
Instrumentation packages here wrap specific libraries (OpenAI, Anthropic, etc.) and bridge
44
them to the shared telemetry layer in `util/opentelemetry-util-genai`.
55

6+
These rules are additive to the shared instrumentation rules in the repo-root
7+
[AGENTS.md](../AGENTS.md).
8+
69
## 1. Instrumentation Layer Boundary
710

811
Do not call OpenTelemetry APIs (`tracer`, `meter`, `span`, event APIs) directly.
@@ -33,9 +36,23 @@ except Exception as exc:
3336
raise
3437
```
3538

36-
## 3. Exception Handling
39+
## 3. Semantic conventions
40+
41+
Attributes, spans, events, and metrics follow the
42+
[GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai).
43+
Do not emit signals that are not covered by semconv.
44+
45+
`gen_ai.*` attribute names and the enums for well-known values (e.g. `GenAiOutputTypeValues` for
46+
`gen_ai.output.type`) live in `opentelemetry.semconv._incubating.attributes.gen_ai_attributes`.
47+
48+
## 4. Tests
49+
50+
- Use VCR cassettes for provider calls. Do not skip tests when an API key is missing.
51+
- Cover streaming and non-streaming variants when both exist.
52+
- Cover error scenarios, at minimum: provider error / endpoint unavailable, stream interrupted by
53+
network, stream closed early by the caller.
54+
55+
## 5. Examples
3756

38-
- Do not add `raise {Error}` statements in instrumentation/telemetry code — validation belongs in
39-
tests and callers, not in the instrumentation layer.
40-
- When catching exceptions from the underlying library to record telemetry, always re-raise
41-
the original exception unmodified.
57+
New instrumentations ship a minimal example under the package's `examples/` directory, with
58+
both a `manual/` setup and a `zero-code/` (auto-instrumentation) variant.

0 commit comments

Comments
 (0)