Skip to content

chore(debugger): implement Phase 1 guardrail foundations#18806

Open
P403n1x87 wants to merge 3 commits into
mainfrom
chore/debugger-guardrails-phase-1
Open

chore(debugger): implement Phase 1 guardrail foundations#18806
P403n1x87 wants to merge 3 commits into
mainfrom
chore/debugger-guardrails-phase-1

Conversation

@P403n1x87

@P403n1x87 P403n1x87 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Description

Addresses the Phase 1 gaps from the Live Debugger Circuit-Breakers RFC:

Configurable time budgets

  • Add DD_DYNAMIC_INSTRUMENTATION_EVALUATION_TIMEOUT_MS (default 10 ms) for condition and template expression evaluation wall-time budget
  • Add DD_DYNAMIC_INSTRUMENTATION_CAPTURE_TIMEOUT_MS (default 150 ms) to make the existing hardcoded 200 ms capture budget configurable
  • Register both keys in supported-configurations.json and regenerate _supported_configurations.py

Probe-entry skip on repeated evaluation errors

  • Add _error_throttled_until timestamp to ProbeConditionMixin; when in the future, _eval_condition returns immediately without evaluating the condition
  • Evaluation overruns are treated as errors and fed into condition_error_limiter (same path as DDExpressionEvaluationError), so sustained slow conditions eventually trigger the same probe-entry skip as repeated runtime errors
  • Track _eval_duration_ms on Signal and _capture_duration_ms / _template_eval_duration_ms on Snapshot for observability

Canonical RFC metric names and reason codes (Appendix B)

  • Replace old skip / encoder.buffer.full metrics with the canonical dynamic_instrumentation.guardrails.* namespace
  • SKIP_RATE split into SKIP_RATE_GLOBAL and SKIP_RATE_PROBE to map to rateLimitGlobal / rateLimitProbe reason codes
  • New skip reasons: evaluationErrorThrottled, budgetExceededInvocation, queueFull
  • New distribution metrics: evaluation.duration (condition / template) and capture.duration (with truncated tag)
  • Remove unused _probe_type helper and _PROBE_TYPE_ATTR constant from collector; inline type(signal.probe).__name__ at call sites

Refs: DEBUG-5460

@P403n1x87 P403n1x87 added changelog/no-changelog A changelog entry is not required for this PR. Dynamic Instrumentation Dynamic Instrumentation/Live Debugger labels Jun 30, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/internal/settings/dynamic_instrumentation.py                    @DataDog/debugger-python
supported-configurations.json                                           @DataDog/apm-sdk-capabilities-python @DataDog/apm-python

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

🚦 13 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741589-d2b8243-musllinux_1_2_aarch64, 1]   View in Datadog   GitLab

View all 13 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2328a93 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 30, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-30 12:28:34

Comparing candidate commit eed9ba0 in PR branch chore/debugger-guardrails-phase-1 with baseline commit a819241 in branch main.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 615 metrics, 10 unstable metrics.

scenario:iast_aspects-re_search_aspect

  • 🟥 execution_time [+21.933µs; +29.042µs] or [+7.587%; +10.047%]

scenario:iastaspects-strip_aspect

  • 🟥 execution_time [+69.646µs; +77.816µs] or [+22.060%; +24.648%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+86.300µs; +97.414µs] or [+20.067%; +22.651%]

scenario:span-start

  • 🟥 execution_time [+1.216ms; +1.388ms] or [+7.851%; +8.961%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+160.478ns; +192.117ns] or [+7.475%; +8.949%]

Addresses the Phase 1 gaps from the Live Debugger Circuit-Breakers RFC:

**Configurable time budgets**
- Add `DD_DYNAMIC_INSTRUMENTATION_EVALUATION_TIMEOUT_MS` (default 10 ms) for
  condition and template expression evaluation wall-time budget
- Add `DD_DYNAMIC_INSTRUMENTATION_CAPTURE_TIMEOUT_MS` (default 150 ms) to make
  the existing hardcoded 200 ms capture budget configurable
- Register both keys in `supported-configurations.json` and regenerate
  `_supported_configurations.py`

**Probe-entry skip on repeated evaluation errors**
- Add `_error_throttled_until` timestamp to `ProbeConditionMixin`; when in the
  future, `_eval_condition` returns immediately without evaluating the condition
- Evaluation overruns are treated as errors and fed into `condition_error_limiter`
  (same path as `DDExpressionEvaluationError`), so sustained slow conditions
  eventually trigger the same probe-entry skip as repeated runtime errors
- Track `_eval_duration_ms` on `Signal` and `_capture_duration_ms` /
  `_template_eval_duration_ms` on `Snapshot` for observability

**Canonical RFC metric names and reason codes (Appendix B)**
- Replace old `skip` / `encoder.buffer.full` metrics with the canonical
  `dynamic_instrumentation.guardrails.*` namespace
- `SKIP_RATE` split into `SKIP_RATE_GLOBAL` and `SKIP_RATE_PROBE` to map to
  `rateLimitGlobal` / `rateLimitProbe` reason codes
- New skip reasons: `evaluationErrorThrottled`, `budgetExceededInvocation`,
  `queueFull`
- New distribution metrics: `evaluation.duration` (condition / template) and
  `capture.duration` (with `truncated` tag)
- Remove unused `_probe_type` helper and `_PROBE_TYPE_ATTR` constant from
  collector; inline `type(signal.probe).__name__` at call sites
@P403n1x87
P403n1x87 marked this pull request as ready for review June 30, 2026 14:27
@P403n1x87
P403n1x87 requested review from a team as code owners June 30, 2026 14:27
@P403n1x87
P403n1x87 requested a review from rachelyangdog June 30, 2026 14:27
@P403n1x87
P403n1x87 force-pushed the chore/debugger-guardrails-phase-1 branch from eed9ba0 to c6f0f64 Compare June 30, 2026 14:27
@P403n1x87
P403n1x87 requested a review from emmettbutler June 30, 2026 14:27

@emmettbutler emmettbutler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-debugger changes look good

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 45 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 45 shortest)
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR. Dynamic Instrumentation Dynamic Instrumentation/Live Debugger

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants