feat(errortracking): extend agenttelemetry component with error log submission#52486
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits intoJun 25, 2026
Conversation
This was referenced Jun 19, 2026
Contributor
|
🎯 Code Coverage (details) 🔗 Commit SHA: 8b76261 | Docs | Datadog PR Page | Give us feedback! |
Contributor
Go Package Import DifferencesBaseline: 26694e8
|
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-1-core
branch
from
2ce0920 to
f18b1d1
Compare
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
2 times, most recently
from
be28f36 to
53cf9d6
Compare
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-1-core
branch
2 times, most recently
from
678c519 to
056ee8f
Compare
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
from
53cf9d6 to
1b3c60a
Compare
Contributor
Gitlab CI Configuration Changes
|
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 0 | 18 | 0 | 0 |
ℹ️ Diff available in the job log.
Contributor
Files inventory check summaryFile checks results against ancestor 26694e87: Results for datadog-agent_7.82.0~devel.git.306.8b76261.pipeline.120696223-1_amd64.deb:No change detected |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: c6b6b2c ❌ Experiments with retried target crashesThis is a critical error. One or more replicates failed with a non-zero exit code. These replicates may have been retried. See Replicate Execution Details for more information.
Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_metrics_logs | memory utilization | +1.38 | [+1.12, +1.63] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | +0.19 | [+0.15, +0.23] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | +0.08 | [-0.01, +0.18] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.05 | [-0.00, +0.10] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.05 | [-0.02, +0.11] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.13 | [-0.17, -0.09] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -0.67 | [-1.74, +0.40] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 144.97MiB ≤ 154MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 575.78KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 486.42MiB ≤ 495MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 0.89MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 180.69MiB ≤ 195MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 264.19MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 339.42 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 405.19MiB ≤ 430MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.87GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 28.76 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 297.95MiB ≤ 330MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 58.89 ≤ 80 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 274.40MiB ≤ 310MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 22.92 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 284.67MiB ≤ 320MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
Replicate Execution Details
We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.
Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.
| Experiment | Variant | Replicates | Failure | Logs | Debug Dashboard |
|---|---|---|---|---|---|
| experiment_with_failures | baseline | 0 (x8) 1 (x6) | Oom killed | Debug Dashboard |
The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.
❌ Retried Normal Replicate Execution Failures (non-profiling)
| Experiment | Variant | Replicates | Failure | Debug Dashboard |
|---|---|---|---|---|
| quality_gate_idle_all_features | baseline | 8 | Oom killed | Debug Dashboard |
❌ Retried Profiling Replicate Execution Failures (ddprof)
Note: Profiling replicas may still be executing. See the debug dashboard for up to date status.
| Experiment | Variant | Replicates | Failure | Debug Dashboard |
|---|---|---|---|---|
| quality_gate_idle_all_features | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_idle_all_features | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_logs | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_metrics_logs | comparison | 10 | Oom killed | Debug Dashboard |
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
Contributor
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
13 successful checks with minimal change (< 2 KiB)
|
Base automatically changed from
pducolin/coat-errortracking/stack-1-core
to
main
June 22, 2026 12:14
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
from
1b3c60a to
f473840
Compare
dd-octo-sts
Bot
added
internal
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
from
f473840 to
732215e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 732215ec26
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
from
732215e to
ac130e5
Compare
pducolin
added
qa/no-code-change
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
from
70b773d to
af32141
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…ubmission - Add SubmitErrorLog(ErrorLog) to the component interface - Add bounded errLogsCh channel, non-blocking SubmitErrorLog(), and flushErrortracking() scheduler job - Add errortracking_sender.go: converts ErrorLog to dd-go Log with stack symbolization - Add logs_payload.go: builds LogsPayload batches for HTTP POST - Refactor sender.go: extract shared sendPayloadBody() helper reused by metrics and logs paths
- Remove stale pkg/util/log/setup indirect dep from agenttelemetry/fx go.mod - Add missing replace rule for pkg/util/log/setup in agenttelemetry/impl go.mod - Drop unused //comp/logs/agent/config from agenttelemetry impl test BUILD target - Drop unused //pkg/util/log/errortracking and //slog from log/setup test BUILD target
Git worktrees under .worktrees/ were not excluded from Bazel's directory scanning, causing gazelle to find duplicate Go package targets and generate incorrect BUILD.bazel files.
pducolin
force-pushed
the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
from
af32141 to
9b343ec
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
louis-cqrl
approved these changes
Jun 24, 2026
chouquette
approved these changes
Jun 24, 2026
fabbing
approved these changes
Jun 24, 2026
KevinFairise2
approved these changes
Jun 24, 2026
s-alad
approved these changes
Jun 24, 2026
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
pducolin/coat-errortracking/stack-2-agenttelemetry
branch
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jun 29, 2026
…52487) > **Stacked PR 3/4** — part of [#50607](#50607). Builds on [stack-2 #52486](#52486). ### What does this PR do? Adds `/api/v2/apmtelemetry` support to fakeintake so E2E tests can assert on error-tracking payloads forwarded by the agent, add CI boilerplate to run agent telemetry e2e tests. - New `AgentTelemetryLogAggregator` parses agent-logs request type from the shared `/api/v2/apmtelemetry` endpoint. - `GetAgentTelemetryLogs()` client method fetches and returns all received log records - Register `/api/v2/apmtelemetry` route in `server/serverstore/parser.go` so the /fakeintake/payloads query path works ### Stack 1. [stack-1 #52485] Core `pkg/util/log/errortracking/` package 2. [stack-2 #52486] agenttelemetry component extensions + sender refactor 3. **This PR** — e2e test boilerplate 4. [stack-4] config, log-setup wiring, agent run command Co-authored-by: paola.ducolin <paola.ducolin@datadoghq.com>
github-actions Bot
pushed a commit
to nothingtosurprise/datadog-agent
that referenced
this pull request
Jun 29, 2026
…ataDog#52487) > **Stacked PR 3/4** — part of [DataDog#50607](DataDog#50607). Builds on [stack-2 DataDog#52486](DataDog#52486). ### What does this PR do? Adds `/api/v2/apmtelemetry` support to fakeintake so E2E tests can assert on error-tracking payloads forwarded by the agent, add CI boilerplate to run agent telemetry e2e tests. - New `AgentTelemetryLogAggregator` parses agent-logs request type from the shared `/api/v2/apmtelemetry` endpoint. - `GetAgentTelemetryLogs()` client method fetches and returns all received log records - Register `/api/v2/apmtelemetry` route in `server/serverstore/parser.go` so the /fakeintake/payloads query path works ### Stack 1. [stack-1 DataDog#52485] Core `pkg/util/log/errortracking/` package 2. [stack-2 DataDog#52486] agenttelemetry component extensions + sender refactor 3. **This PR** — e2e test boilerplate 4. [stack-4] config, log-setup wiring, agent run command Co-authored-by: paola.ducolin <paola.ducolin@datadoghq.com> e882170
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jul 3, 2026
…52488) > **Stacked PR 4/4** — part of [#50607](#50607). Builds on [stack-3 #52487](#52487). ### What does this PR do? Wires the error-tracking handler into the agent and validates the full pipeline end-to-end: - **Config defaults** — adds `buffer_size`, `flush_interval`, `bouncer_window` to `common_settings.go` - **Log setup** — wires `errortracking.Handler` into `buildSlogLogger` as a synchronous sibling handler - **Agent run command** — installs an `fx.Lifecycle` hook that starts/stops the handler - **E2E test suite** (`test/new-e2e/tests/agent-telemetry/`) — provisions a real AWS EC2 VM via Pulumi, installs the agent with a Python check that logs errors and raises an exception. Asserts that error log payloads reach the internal telemetry intake. Tests both `errortracking.enabled: true` and the default disabled state. - **CI** — extends `.on_agenttelemetry_or_e2e_changes` with `test/new-e2e/tests/agent-telemetry/**/*` ### Stack 1. [stack-1 #52485] Core `pkg/util/log/errortracking/` package 2. [stack-2 #52486] agenttelemetry component extensions + sender refactor 3. [stack-3 #52487] FakeIntake aggregator/client + CI scaffolding 4. **This PR** — agent wiring + E2E test Co-authored-by: paola.ducolin <paola.ducolin@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Extends
comp/core/agenttelemetryto buffer and forwardErrorLogentries produced by the handler in stack 1:SubmitErrorLog(ErrorLog)toComponent(def layer)errLogsChchannel;SubmitErrorLogdoes a non-blocking send (drops on overflow, never blocks the hot path)flushErrortracking()job — no new goroutines, no new lifecycleerrortracking_sender.go: convertsErrorLog→ dd-goLogviaenrichErrorLog(), appends commit SHA after scrubbing (to avoid the scrubber stripping it), wraps inLogsPayloadsender.go: extractssendPayloadBody()helper reused by both the metrics-payload path and the new logs pathThe handler is not yet wired, so no customer facing changes yet.
Stack
pkg/util/log/errortracking/package