Skip to content

fix(tracer): don't activate runtime metrics v1 if otel runtime metrics are enabled#4962

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
anna.yafi/fix-otel-runtime-metrics-dual-emission
Jun 30, 2026
Merged

fix(tracer): don't activate runtime metrics v1 if otel runtime metrics are enabled#4962
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
anna.yafi/fix-otel-runtime-metrics-dual-emission

Conversation

@anna-git

@anna-git anna-git commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Following #4803, fixes dual-emission when both DD_RUNTIME_METRICS_ENABLED=true and DD_METRICS_OTEL_ENABLED=true are set: the legacy v1 runtime metrics shouldn't start.

Fix: Pre-compute otelRuntimeMetricsShouldBeEnabled in config (option.go) and add a !otelRuntimeMetricsShouldBeEnabled guard around the legacy goroutine in newTracer().

Testing:

  • TestTracerRuntimeMetrics (all subtests) pass
  • Validated against system-tests PR #6857 (OTLP_RUNTIME_METRICS scenario): test_otel_metrics_are_present_and_attributed and test_dd_metrics_are_absent both pass

🤖 Generated with Claude Code

@anna-git anna-git added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jun 29, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jun 29, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 62.78% (-3.66%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b0ae9bf | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 29, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-30 12:07:20

Comparing candidate commit b0ae9bf in PR branch anna.yafi/fix-otel-runtime-metrics-dual-emission with baseline commit 1c9a3b0 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 321 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

@anna-git anna-git changed the title fix(tracer): suppress old runtime.go.* StatsD metrics when DD_METRICS_OTEL_ENABLED=true refactor(tracer): consolidate runtime metrics startup into newTracer with OTel/V2/legacy priority chain Jun 29, 2026
@anna-git anna-git changed the title refactor(tracer): consolidate runtime metrics startup into newTracer with OTel/V2/legacy priority chain fix(tracer): prevent dual runtime metrics emission when OTel mode is active Jun 29, 2026
@anna-git anna-git changed the title fix(tracer): prevent dual runtime metrics emission when OTel mode is active fix(tracer): guard legacy runtime metrics goroutine when OTel is active Jun 29, 2026
@anna-git anna-git marked this pull request as ready for review June 29, 2026 18:00
@anna-git anna-git requested review from a team as code owners June 29, 2026 18:00
@anna-git anna-git changed the title fix(tracer): guard legacy runtime metrics goroutine when OTel is active fix(tracer): don't activate runtime metrics v1 if otel runtime metrics are enabled Jun 30, 2026
@darccio

darccio commented Jun 30, 2026

Copy link
Copy Markdown
Member

LGTM but I think we are missing a test here. This is a regression risk, a unit test would be a safety addition. It could be TestTracerLegacyRuntimeMetricsSuppressedWhenOtelActive in metrics_otel_test.go, setting DD_RUNTIME_METRICS_ENABLED=true + DD_METRICS_OTEL_ENABLED=true, call withTestHooks, call newTracer, and assert the "Runtime metrics enabled." debug log is absent (same pattern as the on subtest in TestTracerRuntimeMetrics).

anna-git and others added 2 commits June 30, 2026 13:36
When DD_RUNTIME_METRICS_ENABLED and DD_METRICS_OTEL_ENABLED are both set,
the legacy reportRuntimeMetrics goroutine (DogStatsD, runtime.go.*) was
starting alongside the OTel path, causing dual emission.

Pre-compute otelRuntimeMetricsShouldBeEnabled once in config and use it
to guard the legacy path in newTracer().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hen OTel is active

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@anna-git anna-git force-pushed the anna.yafi/fix-otel-runtime-metrics-dual-emission branch from e2305ca to b0ae9bf Compare June 30, 2026 11:37
@anna-git

Copy link
Copy Markdown
Contributor Author

LGTM but I think we are missing a test here. This is a regression risk, a unit test would be a safety addition. It could be TestTracerLegacyRuntimeMetricsSuppressedWhenOtelActive in metrics_otel_test.go, setting DD_RUNTIME_METRICS_ENABLED=true + DD_METRICS_OTEL_ENABLED=true, call withTestHooks, call newTracer, and assert the "Runtime metrics enabled." debug log is absent (same pattern as the on subtest in TestTracerRuntimeMetrics).

Good point! Added a test that asserts legacy runtime metrics aren't enabled when OTel is active (b0ae9bf), this sets both flags, installs stub hooks, calls newTracer, and asserts legacy runtime metrics aren't started when OTel is active reading logs and boolean.

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jun 30, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-06-30 12:09:42 UTC ℹ️ Start processing command devflow:merge


2026-06-30 12:09:51 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2026-06-30 16:24:17 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

@zacharycmontoya zacharycmontoya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM requirements wise

@anna-git

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jun 30, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-06-30 16:30:30 UTC ℹ️ Start processing command /merge


2026-06-30 16:30:35 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 19m (p90).


2026-06-30 16:36:19 UTC ℹ️ MergeQueue: Readding this merge request to the queue because another merge request processed with yours failed. No action is needed from your side.


2026-06-30 17:22:59 UTCMergeQueue: The checks failed on this merge request

Tests failed on this commit ab3259f:

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jun 30, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-06-30 17:27:34 UTC ℹ️ Start processing command devflow:merge


2026-06-30 17:27:39 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 19m (p90).


2026-06-30 17:44:56 UTC ℹ️ MergeQueue: This merge request was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos mergequeue-status: done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants