Skip to content

fix(traces): keep backend as safety net for trace stats#1171

Closed
duncanista wants to merge 1 commit intomainfrom
fix-trace-stats-backend-fallback
Closed

fix(traces): keep backend as safety net for trace stats#1171
duncanista wants to merge 1 commit intomainfrom
fix-trace-stats-backend-fallback

Conversation

@duncanista
Copy link
Copy Markdown
Contributor

Summary

Fixes trace stats "disappearing" after upgrading to v95.

PR #1118 (55659d9) introduced logic to respect the Datadog-Client-Computed-Stats header from tracers. When a tracer sends this header as truthy, the extension would:

  1. Skip extension-side stats generation (correct)
  2. Set _dd.compute_stats: 0 on the trace payload, telling the backend to also skip stats (problematic)

This created a scenario where all three stats sources could be disabled simultaneously:

DD_COMPUTE_TRACE_STATS_ON_EXTENSION Datadog-Client-Computed-Stats _dd.compute_stats Ext generates? Result
false true "0" No No stats from anyone if tracer stats don't reach backend

If the tracer claims it computed stats but those stats don't actually reach the backend (tracer bug, misconfiguration, or the header is sent without stats being computed), stats vanish entirely. In v94, the backend always acted as a safety net (_dd.compute_stats: 1).

Fix

_dd.compute_stats is now determined solely by compute_trace_stats_on_extension, ignoring client_computed_stats:

DD_COMPUTE_TRACE_STATS_ON_EXTENSION Datadog-Client-Computed-Stats _dd.compute_stats Ext generates?
false false "1" No
false true "1" (was "0") No
true false "0" Yes
true true "0" No (dedup preserved)

The client_computed_stats header still correctly suppresses extension-side stats generation (the && !client_computed_stats guard on stats_generator.send()), preserving the dedup behavior from #1118 when DD_COMPUTE_TRACE_STATS_ON_EXTENSION=true.

Worst case with this fix is double-counted stats (tracer + backend) — far better than zero stats.

Test plan

  • Unit tests updated and passing (check_compute_stats_behavior covers all 4 combinations)
  • Build a layer from this branch and deploy to affected customer's Lambda
  • Verify trace stats reappear in Datadog UI

Copilot AI review requested due to automatic review settings April 7, 2026 16:47
@duncanista duncanista requested a review from a team as a code owner April 7, 2026 16:47
@duncanista duncanista force-pushed the fix-trace-stats-backend-fallback branch from cad01da to ba83898 Compare April 7, 2026 16:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a critical bug in trace stats generation where stats could disappear after upgrading to v95. The issue was introduced in PR #1118, where the extension would set _dd.compute_stats: "0" when a tracer claimed to have computed stats via the Datadog-Client-Computed-Stats header. If those tracer stats didn't reach the backend (due to tracer bugs or misconfiguration), stats would vanish entirely with no fallback.

The fix changes _dd.compute_stats to be determined solely by compute_trace_stats_on_extension, ignoring client_computed_stats. This keeps the backend as a safety net that can always compute stats if needed, while client_computed_stats continues to prevent duplicate stats from the extension (when compute_trace_stats_on_extension=true).

Changes:

  • Updated _dd.compute_stats determination logic in trace_processor.rs to depend only on compute_trace_stats_on_extension
  • Updated all four test cases and their expectations to match the new behavior
  • Bumped extension version from "94-next" to "95-next"

Reviewed changes

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

File Description
bottlecap/src/traces/trace_processor.rs Updated compute_stats logic and all test expectations
bottlecap/src/tags/lambda/tags.rs Bumped extension version to 95-next

Comment thread bottlecap/src/traces/trace_processor.rs
Copy link
Copy Markdown
Contributor

@lym953 lym953 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as a remediation of missing trace stats issue. But not sure if it will cause double counting, which was what #1118 was trying to fix. Leaving to APM Serverless team to evaluate the possibility of double counting.

@duncanista duncanista force-pushed the fix-trace-stats-backend-fallback branch from ba83898 to 33f05be Compare April 7, 2026 16:53
Comment thread bottlecap/src/traces/trace_processor.rs
…laims client-computed-stats

The Datadog-Client-Computed-Stats header should only suppress extension-side
stats generation, not backend stats. When the extension is not computing stats
itself, always set _dd.compute_stats:1 so the backend remains a fallback.
@duncanista duncanista force-pushed the fix-trace-stats-backend-fallback branch from 33f05be to 99bfcfe Compare April 7, 2026 17:44
@duncanista
Copy link
Copy Markdown
Contributor Author

Drafting due to #1172

@duncanista
Copy link
Copy Markdown
Contributor Author

Also failing internal testing.

@duncanista duncanista closed this Apr 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants