[Debugger] Add system-test coverage for the debugger guardrails by dudikeleti · Pull Request #7287 · DataDog/system-tests

dudikeleti · 2026-07-09T13:24:07Z

Motivation

Add focused system-test coverage for the debugger guardrails required for GA.

Changes

Add Test_Debugger_Evaluation_Timeout: expensive when evaluation (ReDoS regex, large collection filter) must be aborted and surfaced as an evaluation-error snapshot (populated evaluationErrors[], no captured data), not a normal snapshot.
Add Test_Debugger_Snapshot_Guardrails: completed snapshots are trimmed to ≤1 MB, and capture exceeding the time budget reports notCapturedReason=timeout.
Extend test_default_max_length to assert notCapturedReason=stringLength, completing default capture-limit reason coverage.
Extract shared captures_contain_data helper into debugger.utils (reused by the condition-error tests).
All new guardrail rows are marked missing_feature across every tracer; enable per-language as support lands. The regex fixture only exercises backtracking engines (linear engines like Go stay disabled via manifests).

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

…reasons and evaluation timeout behavior, with new rows disabled in manifests until tracer support is ready.

…re timeout reporting. 1MB snapshot cap and notCapturedReason=timeout behavior.

…ves the setup-time negative snapshot wait so evaluation-timeout assertions rely on collected scenario output instead of a fixed sleep.

github-actions · 2026-07-09T13:25:16Z

CODEOWNERS have been resolved as:

tests/debugger/test_debugger_guardrails.py                              @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_capture_timeout_reason.json           @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_evaluation_timeout_collection_filter.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_evaluation_timeout_regex.json         @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_snapshot_size_cap.json                @DataDog/debugger @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/debugger/test_debugger_condition_errors.py                        @DataDog/debugger @DataDog/system-tests-core
tests/debugger/test_debugger_probe_snapshot.py                          @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils.py                                                 @DataDog/debugger @DataDog/system-tests-core

datadog-datadog-prod-us1-2 · 2026-07-09T13:28:13Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 142 Pipeline jobs failed

Testing the test | System Tests (java, dev) / End-to-end #1 / akka-http 1

🧪 1 Test failed

tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails.test_capture_timeout_reports_reason[akka-http] from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status within 30s\nProbes did not reach EMITTING status within 10s\nSnapshot was not received within 30s&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status within 30s\nProbes did not reach EMITTING status within 10s\nSnapshot was not received within 30s&#39; = &lt;built-in method join of str object at 0x7f95bd42b720&gt;([&#39;Probes did not reach INSTALLED status within 30s&#39;, &#39;Probes did not reach EMITTING status within 10s&#39;, &#39;Snapshot was not received within 30s&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f95bd42b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status within 30s&#39;, &#39;Probes did not reach EMITTING status within 10s&#39;, &#39;Snapshot was not received within 30s&#39;] = &lt;tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails object at 0x7f95b4f25ca0&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails object at 0x7f95b4f25ca0&gt;

    def test_capture_timeout_reports_reason(self) -&gt; None:
        &#34;&#34;&#34;A capture exceeding the tracer&#39;s time budget identifies the value it could not finish.
    
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.0 1

🧪 1 Test failed

tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails.test_capture_timeout_reports_reason[apache-mod-7.0]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status within 30s&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status within 30s&#39; = &lt;built-in method join of str object at 0x7f02ae82b720&gt;([&#39;Probes did not reach INSTALLED status within 30s&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f02ae82b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status within 30s&#39;] = &lt;tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails object at 0x7f02a4d69f40&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails object at 0x7f02a4d69f40&gt;

    def test_capture_timeout_reports_reason(self) -&gt; None:
        &#34;&#34;&#34;A capture exceeding the tracer&#39;s time budget identifies the value it could not finish.
    
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.0-zts 1

🧪 1 Test failed

tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails.test_capture_timeout_reports_reason[apache-mod-7.0-zts]

from system_tests_suite

AssertionError: assert &#39;Probes did not reach INSTALLED status within 30s&#39; is None
 &#43;  where &#39;Probes did not reach INSTALLED status within 30s&#39; = &lt;built-in method join of str object at 0x7f1cb382b720&gt;([&#39;Probes did not reach INSTALLED status within 30s&#39;])
 &#43;    where &lt;built-in method join of str object at 0x7f1cb382b720&gt; = &#39;\n&#39;.join
 &#43;    and   [&#39;Probes did not reach INSTALLED status within 30s&#39;] = &lt;tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails object at 0x7f1cac1339e0&gt;.setup_failures

self = &lt;tests.debugger.test_debugger_guardrails.Test_Debugger_Snapshot_Guardrails object at 0x7f1cac1339e0&gt;

    def test_capture_timeout_reports_reason(self) -&gt; None:
        &#34;&#34;&#34;A capture exceeding the tracer&#39;s time budget identifies the value it could not finish.
    
...

View all 142 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 2 jobs - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 2c71ee0 | Docs | Datadog PR Page | Give us feedback!}

…upported `notCapturedReason`.

datadog-datadog-prod-us1-2 · 2026-07-10T15:45:39Z

Bits couldn’t fix the CI

🟢 Investigated · 🔴 Fix failed · ⚪ Validate · ⚪ Ready

The newly enabled tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_capture_timeout_reports_reason guardrail was running for tracer manifests where debugger capture-timeout reporting is still unsupported. Added explicit missing_feature entries for that test in manifests/dotnet.yml, manifests/golang.yml, manifests/java.yml, manifests/nodejs.yml, manifests/php.yml, manifests/python.yml, and manifests/ruby.yml.

View in Datadog | Reviewed commit 2c71ee0

…t relying on runner timing.

dudikeleti added 4 commits July 9, 2026 12:00

Introduce initial debugger guardrail coverage for capture incomplete …

04e1fb1

…reasons and evaluation timeout behavior, with new rows disabled in manifests until tracer support is ready.

Adds debugger guardrail coverage for snapshot size trimming and captu…

3ef4275

…re timeout reporting. 1MB snapshot cap and notCapturedReason=timeout behavior.

Adds the guardrail RFC link to the new debugger test classes and remo…

1d74e93

…ves the setup-time negative snapshot wait so evaluation-timeout assertions rely on collected scenario output instead of a fixed sleep.

Address review feedback on the debugger guardrail tests

9f01c86

dudikeleti added 2 commits July 10, 2026 16:58

fix ci

8b0d7f8

Validate the established truncated flag instead of requiring an uns…

e09ad0a

…upported `notCapturedReason`.

dudikeleti changed the title ~~Dudik/debugger/guardrails tests~~ [Debugger] Add system-test coverage for the debugger guardrails Jul 10, 2026

dudikeleti added 2 commits July 10, 2026 18:10

validate timeout reporting on the intended captured collection withou…

b234701

…t relying on runner timing.

Enable debugger capture-timeout coverage across tracers

2c71ee0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Debugger] Add system-test coverage for the debugger guardrails#7287

[Debugger] Add system-test coverage for the debugger guardrails#7287
dudikeleti wants to merge 8 commits into
mainfrom
dudik/debugger/guardrails-tests

dudikeleti commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 9, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 10, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

dudikeleti commented Jul 9, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 9, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 10, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bits couldn’t fix the CI

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

datadog-datadog-prod-us1-2 Bot commented Jul 9, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 10, 2026 •

edited by datadog-prod-us1-6 Bot

Loading