[Debugger] Add system-test coverage for the debugger guardrails

manifests/dotnet.yml

@@ -654,6 +654,8 @@ manifest:
   tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language::test_expression_language_nulls_true: bug (DEBUG-2618)
   ? tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language::test_expression_language_string_operations
   : bug (DEBUG-2560)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin: bug (DEBUG-4637)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin_Default_On: missing_feature (Temporarily disabled; enable from v3.42.0 later)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation: bug (DEBUG-4637)

manifests/golang.yml

@@ -1074,6 +1074,8 @@ manifest:
   tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_recursion_inlined: irrelevant (Test for specific bug in dotnet)
   tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
   tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language: missing_feature (feature not implemented)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin: missing_feature
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin_Default_On: irrelevant (Scoped to dotnet rollout in this PR)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation: missing_feature

manifests/java.yml

@@ -3041,6 +3041,8 @@ manifest:
         spring-boot-undertow: v1.38.0
         spring-boot-wildfly: v1.38.0
         uds-spring-boot: v1.33.0
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin:
     - weblog_declaration:
         "*": missing_feature

manifests/nodejs.yml

@@ -1612,6 +1612,8 @@ manifest:
   tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language::test_expression_language_instance_of:
     - weblog_declaration:
         express4-typescript: bug (DEBUG-3715)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin:
     - weblog_declaration:
         "*": *ref_5_83_0

manifests/php.yml

@@ -651,6 +651,8 @@ manifest:
   tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language::test_expression_language_nulls_true: '>=1.16.0'
   ? tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language::test_expression_language_string_operations
   : '>=1.16.0'
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin: missing_feature (code origin entry spans not captured in PHP-FPM; RC toggle works but no spans to verify)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin_Default_On: irrelevant (Scoped to dotnet rollout in this PR)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation:

manifests/python.yml

@@ -1223,6 +1223,8 @@ manifest:
     - weblog_declaration:
         "*": missing_feature
         *flask: v2.11.0
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin:
     - weblog_declaration:
         "*": missing_feature

manifests/ruby.yml

@@ -1778,6 +1778,8 @@ manifest:
   tests/debugger/test_debugger_expression_language.py::Test_Debugger_Expression_Language::test_expression_language_hash_operations:
     - declaration: missing_feature (Hash length not implemented)
       component_version: <=2.22.0
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Evaluation_Timeout: missing_feature (Debugger evaluation timeout guardrail not implemented yet)
+  tests/debugger/test_debugger_guardrails.py::Test_Debugger_Snapshot_Guardrails::test_snapshot_size_cap: missing_feature (Debugger snapshot size guardrail not implemented yet)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin: missing_feature
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Code_Origin_Default_On: irrelevant (Scoped to dotnet rollout in this PR)
   tests/debugger/test_debugger_inproduct_enablement.py::Test_Debugger_InProduct_Enablement_Dynamic_Instrumentation: missing_feature

tests/debugger/test_debugger_condition_errors.py

@@ -7,40 +7,10 @@
 import time
 
 import tests.debugger.utils as debugger
+from tests.debugger.utils import captures_contain_data
 from utils import context, features, scenarios
 
 
-def _capture_point_contains_data(point: object) -> bool:
-    """Return True iff a single capture point (entry / return / one line entry) contains data."""
-    if not isinstance(point, dict):
-        return False
-    for key in ("arguments", "locals", "staticFields"):
-        if point.get(key):
-            return True
-    return bool(point.get("throwable"))
-
-
-def _captures_contain_data(captures: object) -> bool:
-    """Return True iff the ``captures`` field of a snapshot contains any captured probe data.
-
-    A snapshot whose ``captures`` is missing, ``None``, or only contains empty
-    ``entry`` / ``return`` / ``lines`` sub-structures (i.e. no captured arguments,
-    locals, static fields, or throwable) returns False -- it is an empty captures
-    container and acceptable as part of an evaluation-error snapshot.
-    """
-    if not isinstance(captures, dict):
-        return False
-    for key, value in captures.items():
-        # `entry` and `return` map directly to a capture-point dict.
-        # `lines` maps line-numbers to capture-point dicts, so descend one level.
-        if key == "lines" and isinstance(value, dict):
-            if any(_capture_point_contains_data(v) for v in value.values()):
-                return True
-        elif _capture_point_contains_data(value):
-            return True
-    return False
-
-
 class _ConditionTestBase(debugger.BaseDebuggerTest):
     """Shared base for the condition-error compliance tests."""
 
@@ -214,7 +184,7 @@ def test_runtime_condition_error_emits_error_only_snapshot(self) -> None:
             )
 
             captures = snap.get("captures")
-            assert not _captures_contain_data(captures), (
+            assert not captures_contain_data(captures), (
                 f"The probe emitted a snapshot whose ``captures`` field contains captured "
                 f"data ({captures!r}); an eval-error snapshot must have empty captures "
                 f"because the condition was not successfully evaluated."

tests/debugger/test_debugger_guardrails.py

@@ -0,0 +1,219 @@
+# Unless explicitly stated otherwise all files in this repository are licensed under the the Apache License Version 2.0.
+# This product includes software developed at Datadog (https://www.datadoghq.com/).
+# Copyright 2021 Datadog, Inc.
+
+import json
+from typing import Any
+
+import tests.debugger.utils as debugger
+from utils import context, features, rfc, scenarios, slow, weblog
+
+
+MAX_SNAPSHOT_BYTES = 1024 * 1024
+# `^(a+)+$` against a non-matching input backtracks exponentially (~2^n), so a few dozen
+# characters already blow past any evaluation-time budget. Keep it small so the value fits
+# comfortably in the request URL.
+REDOS_INPUT_LENGTH = 50
+GUARDRAILS_RFC = "https://docs.google.com/document/d/1OhCH3SMuS_B4Ickays94GpqDlqKcc9b9gLos1T85F-Q/edit?usp=sharing"
+
+
+@rfc(GUARDRAILS_RFC)
+@features.debugger_expression_language
+@scenarios.debugger_probes_snapshot
+@slow
+class Test_Debugger_Evaluation_Timeout(debugger.BaseDebuggerTest):
+    """RFC guardrail: expensive expression evaluation must stop before normal snapshot creation.
+
+    A conforming tracer aborts the expensive ``when`` evaluation and surfaces the failure as an
+    evaluation-error snapshot (non-empty ``evaluationErrors[]`` with no captured user data),
+    exactly like a runtime condition error -- it must not produce a normal, fully-captured snapshot.
+    """
+
+    def _convert_to_line_probe_if_needed(self, probe: dict[str, Any], method: str) -> None:
+        if context.library.name not in ("nodejs", "ruby"):
+            return
+
+        where = probe["where"]
+        where.pop("methodName", None)
+        where["typeName"] = None
+        where["sourceFile"] = "ACTUAL_SOURCE_FILE"
+        where["lines"] = self.method_and_language_to_line_number(method, context.library.name)
+
+    def _setup_evaluation_timeout(self, probes_name: str, method: str, request_path: str) -> None:
+        self.initialize_weblog_remote_config()
+
+        probes = debugger.read_probes(probes_name)
+        for probe in probes:
+            probe["id"] = debugger.generate_probe_id("log")
+            self._convert_to_line_probe_if_needed(probe, method)
+
+        self.set_probes(probes)
+        self.send_rc_probes()
+        if not self.wait_for_all_probes(statuses=["INSTALLED"], timeout=30):
+            self.setup_failures.append("Probes did not reach INSTALLED status within 30s")
+
+        self.weblog_responses.append(weblog.get(request_path, timeout=15))
+        # A conforming tracer emits an evaluation-error snapshot; wait for it to land before
+        # collect(). Tracers that (wrongly) emit nothing just hit the timeout and fail the
+        # snapshot-presence assertion in the test with a clear message.
+        self.wait_for_all_snapshots(timeout=10)
+
+    def setup_evaluation_timeout_regex(self) -> None:
+        self._setup_evaluation_timeout(
+            "probe_evaluation_timeout_regex",
+            "StringOperations",
+            f"/debugger/expression/strings?strValue={'a' * REDOS_INPUT_LENGTH}!",
+        )
+
+    def test_evaluation_timeout_regex(self) -> None:
+        self._assert_evaluation_timeout_snapshot()
+
+    def setup_evaluation_timeout_collection_filter(self) -> None:
+        self._setup_evaluation_timeout(
+            "probe_evaluation_timeout_collection_filter",
+            "SnapshotLimits",
+            "/debugger/snapshot/limits?collectionSize=100000",
+        )
+
+    def test_evaluation_timeout_collection_filter(self) -> None:
+        self._assert_evaluation_timeout_snapshot()
+
+    def _assert_evaluation_timeout_snapshot(self) -> None:
+        self.collect()
+        self.assert_setup_ok()
+        self.assert_rc_state_not_error()
+        self.assert_all_weblog_responses_ok()
+
+        for probe_id in self.probe_ids:
+            snapshots = self.probe_snapshots.get(probe_id, [])
+            assert snapshots, (
+                "Probe emitted no snapshot; a conforming tracer must surface the aborted "
+                "expression evaluation via a probe result carrying evaluationErrors[]."
+            )
+
+            envelope = snapshots[0]
+            snapshot = envelope.get("debugger", {}).get("snapshot") or envelope.get("debugger.snapshot") or {}
+
+            evaluation_errors = snapshot.get("evaluationErrors") or []
+            assert evaluation_errors, (
+                "Evaluation-timeout snapshot has an empty evaluationErrors[]; the tracer must "
+                "report that expression evaluation was aborted."
+            )
+
+            captures = snapshot.get("captures")
+            assert not debugger.captures_contain_data(captures), (
+                f"Evaluation-timeout snapshot leaked captured data ({captures!r}); a conforming "
+                "tracer must stop before normal snapshot creation."
+            )
+
+
+@rfc(GUARDRAILS_RFC)
+@features.debugger_line_probe
+@scenarios.debugger_probes_snapshot
+@slow
+class Test_Debugger_Snapshot_Guardrails(debugger.BaseDebuggerTest):
+    """RFC guardrails for completed snapshot size and capture-time budget."""
+
+    SNAPSHOT_SIZE_COLLECTION_ITEMS = 300_000
+    CAPTURE_TIMEOUT_COLLECTION_ITEMS = 1_000_000
+
+    def _setup_snapshot_guardrail(self, probes_name: str, collection_size: int) -> None:
+        self.initialize_weblog_remote_config()
+
+        probes = debugger.read_probes(probes_name)
+        for probe in probes:
+            probe["id"] = debugger.generate_probe_id("log")
+            if "methodName" in probe["where"]:
+                del probe["where"]["methodName"]
+            probe["where"]["lines"] = self.method_and_language_to_line_number("SnapshotLimits", context.library.name)
+            probe["where"]["sourceFile"] = "ACTUAL_SOURCE_FILE"
+            probe["where"]["typeName"] = None
+
+        self.set_probes(probes)
+        self.send_rc_probes()
+        if not self.wait_for_all_probes(statuses=["INSTALLED"], timeout=30):
+            self.setup_failures.append("Probes did not reach INSTALLED status within 30s")
+
+        self.weblog_responses.append(
+            weblog.get(f"/debugger/snapshot/limits?collectionSize={collection_size}", timeout=30)
+        )
+        if not self.wait_for_all_probes(statuses=["EMITTING"], timeout=10):
+            self.setup_failures.append("Probes did not reach EMITTING status within 10s")
+        if not self.wait_for_all_snapshots(timeout=30):
+            self.setup_failures.append("Snapshot was not received within 30s")
+
+    def setup_snapshot_size_cap(self) -> None:
+        self._setup_snapshot_guardrail("probe_snapshot_size_cap", self.SNAPSHOT_SIZE_COLLECTION_ITEMS)
+
+    def test_snapshot_size_cap(self) -> None:
+        _, snapshot = self._get_single_snapshot()
+        serialized_size = len(json.dumps(snapshot, separators=(",", ":")).encode())
+        assert serialized_size <= MAX_SNAPSHOT_BYTES, (
+            f"Snapshot should be trimmed to <= {MAX_SNAPSHOT_BYTES} bytes, got {serialized_size} bytes"
+        )
+
+        large_collection = self._get_captured_local(snapshot, "largeCollection")
+        assert self._is_partially_captured(large_collection, self.SNAPSHOT_SIZE_COLLECTION_ITEMS), (
+            "Expected the oversized snapshot to contain only part of largeCollection after trimming, "
+            f"got: {large_collection!r}"
+        )
+
+    def setup_capture_timeout_reports_reason(self) -> None:
+        self._setup_snapshot_guardrail("probe_capture_timeout_reason", self.CAPTURE_TIMEOUT_COLLECTION_ITEMS)
+
+    def test_capture_timeout_reports_reason(self) -> None:
+        """A capture exceeding the tracer's time budget identifies the value it could not finish.
+
+        The probe allows twice as many collection items as the fixture contains, so the
+        collection-size limit cannot cause the truncation. Tracers use budgets between 50 and
+        250 ms, so we assert the reported reason instead of elapsed time.
+        """
+        _, snapshot = self._get_single_snapshot()
+        large_collection = self._get_captured_local(snapshot, "largeCollection")
+        assert large_collection.get("notCapturedReason") == "timeout", (
+            "largeCollection should have notCapturedReason='timeout' when capture exceeds the tracer's "
+            f"time budget, got: {large_collection!r}"
+        )
+
+    def _get_single_snapshot(self) -> tuple[dict[str, Any], dict[str, Any]]:
+        self.collect()
+        self.assert_setup_ok()
+        self.assert_rc_state_not_error()
+        self.assert_all_probes_are_emitting()
+        self.assert_all_weblog_responses_ok()
+
+        for probe_id in self.probe_ids:
+            snapshots = self.probe_snapshots.get(probe_id, [])
+            assert len(snapshots) == 1, f"Expected exactly 1 snapshot for {probe_id}, got {len(snapshots)}"
+            envelope = snapshots[0]
+            snapshot = envelope.get("debugger", {}).get("snapshot") or envelope.get("debugger.snapshot")
+            assert isinstance(snapshot, dict), f"Snapshot data not found in expected format for {probe_id}"
+            return envelope, snapshot
+
+        raise AssertionError("No probe IDs were registered")
+
+    def _get_captured_local(self, snapshot: dict[str, Any], variable_name: str) -> dict[str, Any]:
+        captures = snapshot.get("captures", {})
+        lines = captures.get("lines", {})
+        assert isinstance(lines, dict), f"Expected line captures to be a dict, got: {lines!r}"
+        assert len(lines) == 1, f"Expected one line capture, got: {lines!r}"
+
+        line_data = next(iter(lines.values()))
+        locals_data = line_data.get("locals", {})
+        assert variable_name in locals_data, f"{variable_name!r} is missing from snapshot locals"
+        value = locals_data[variable_name]
+        assert isinstance(value, dict), f"Expected {variable_name!r} to be a captured object, got: {value!r}"
+        return value
+
+    def _is_partially_captured(self, value: object, expected_items: int) -> bool:
+        if not isinstance(value, dict):
+            return False
+
+        if value.get("notCapturedReason") == "payloadTooLarge" or value.get("pruned") is True:
+            return True
+
+        elements = value.get("elements") or value.get("entries")
+        if isinstance(elements, list) and len(elements) < expected_items:
+            return True
+
+        return any(self._is_partially_captured(child, expected_items) for child in value.values())

tests/debugger/test_debugger_probe_snapshot.py

@@ -381,6 +381,7 @@ def test_default_max_length(self):
         assert len(string_value) <= self.DEFAULT_MAX_LENGTH, (
             f"longString should have length {self.DEFAULT_MAX_LENGTH}, got: {len(string_value)}"
         )
+        assert long_string.get("truncated") is True, "longString should be marked as truncated"
 
     def setup_log_line_snapshot_debug_track(self):
         self.use_debugger_endpoint = True