Skip to content

Add APM test agent trace decoding - #12089

Open
PerfectSlayer wants to merge 1 commit into
bbujon/smoke-tests-decoded-spansfrom
bbujon/smoke-tests-test-agent-decoding
Open

Add APM test agent trace decoding#12089
PerfectSlayer wants to merge 1 commit into
bbujon/smoke-tests-decoded-spansfrom
bbujon/smoke-tests-test-agent-decoding

Conversation

@PerfectSlayer

@PerfectSlayer PerfectSlayer commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

This PR adds APM test agent trace decoding.

Motivation

The goal is to retrieve the full traces captured by the APM test agent in order to evaluate them during smoke tests.
Unlike our mock backend, using the test agent will avoid testing against ourselves and our decoders.

Additional Notes

Contributor Checklist

Jira ticket: APMLP-1247

@PerfectSlayer
PerfectSlayer requested a review from a team as a code owner July 28, 2026 09:58
@PerfectSlayer PerfectSlayer added the type: feature Enhancements and improvements label Jul 28, 2026
@PerfectSlayer
PerfectSlayer requested a review from amarziali July 28, 2026 09:58

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The new JSON decoder leaves v0.4 meta_struct entries as base64 strings, although the shared DecodedSpan contract and existing v0.4 decoder expose their decoded JSON values. Any smoke test that inspects AppSec or other structured span metadata will receive the wrong type or fail when casting it, so this path needs a custom decode before shipping.

Open Bits AI session

🤖 Datadog Autotest · Commit 0bfd4d7 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

public long getDuration() {
return this.duration;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Decode base64 meta_struct payloads

Smoke tests inspecting AppSec or other structured span metadata will see the wrong type and can fail or silently miss the metadata they are intended to validate.

Assertion details
  • Input: A /test/traces JSON response containing v0.4 meta_struct, for example meta_struct:{"appsec":"<base64 of JSON {"triggers":3}>"}. The test-agent wire representation uses binary meta_struct values, and JSON encoding represents those byte arrays as base64 strings.
  • Expected: getMetaStruct() returns the decoded structured value, matching SpanV04 and the DecodedSpan contract, so callers can inspect nested maps and lists.
  • Actual: SpanJson declares metaStruct as Map<String,Object> and returns it directly. Moshi therefore exposes the base64 text as a String instead of decoding the bytes and parsing the JSON payload.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0bfd4d7e74

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +36 to +37
@Json(name = "meta_struct")
Map<String, Object> metaStruct;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Decode binary meta_struct entries before exposing them

When a trace contains meta_struct, each v0.4 entry is a MessagePack-encoded binary payload (see TraceMapperV0_4.MetaStructWriter), which /test/traces represents as a base64 JSON string. Deserializing directly into Map<String, Object> therefore exposes strings instead of the structured objects returned by the existing v0.4 decoder, so AppSec/LLM smoke tests that inspect or cast these values will fail or evaluate the wrong data. Decode each base64 value and unpack its MessagePack payload; the current synthetic test should also use the wire representation rather than an already-decoded object.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not apply here. From Claude:

Thanks — this is the right instinct in general (in the raw msgpack v0.4/v1 wire formats, meta_struct values are binary MessagePack blobs, which is exactly why SpanV04/SpanV1 decode them). But it doesn't apply to the JSON this decoder consumes: the dd-apm-test-agent decodes meta_struct server-side before serving /test/traces and /test/session/traces, so the JSON already carries nested objects, not base64.

From ddapm_test_agent/trace.py at the pinned v1.44.0:
meta_struct: NotRequired[Dict[str, Dict[str, Any]]] # decoded dicts, not strings

def _parse_meta_struct(value):
return {key: msgpack.unpackb(val_bytes) for key, val_bytes in value.items()}
_parse_meta_struct is applied via _flexible_decode_meta_struct(payload) inside decode_v04(...), and handle_session_traces / handle_test_traces simply web.json_response(traces) those already-decoded traces. Source: https://github.com/DataDog/dd-apm-test-agent/blob/v1.44.0/ddapm_test_agent/trace.py#L363-L367

So typing meta_struct as base64 Map<String,String> and re-decoding would actually break against the real agent (the value is a nested JSON object, not a string). Keeping Map<String,Object> and letting the value deserialize directly is correct here. Not making this change; verified against the agent source (and the existing metaStructEmptyWhenAbsentAndAMapWhenPresent fixture uses the real nested shape).

@datadog-datadog-prod-us1-2

This comment has been minimized.

@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from 0bfd4d7 to d14912e Compare July 28, 2026 14:17
@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.97 s 13.88 s [-0.1%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent 12.86 s 12.96 s [-1.6%; -0.0%] (maybe better)
startup:petclinic:appsec:Agent 16.90 s 16.76 s [-0.1%; +1.8%] (no difference)
startup:petclinic:iast:Agent 16.87 s 17.03 s [-1.8%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 16.66 s 16.78 s [-1.7%; +0.3%] (no difference)
startup:petclinic:sca:Agent 16.99 s 16.89 s [-0.4%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 16.17 s 16.23 s [-1.2%; +0.5%] (no difference)

Commit: fa902990 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@PerfectSlayer
PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from d14912e to fa90299 Compare July 29, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: testing Testing type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant