Skip to content

Commit 5b3377b

Browse files
joeyzhao2018claude
andauthored
durable: extract trace context from checkpoints (#774)
* initial working version * some updates and fixes * remove debugging ENV * update comments * refactor away the AI's complex extractor * clean up AI complicated stuff... * rearrange and enhancements * build: support dd-trace package override and filter publish layer check - build_layers.sh: add DD_TRACE_PACKAGE, DD_TRACE_COMMIT, and DD_TRACE_COMMIT_BRANCH env vars to override the dd-trace dependency used inside the layer (useful for git-bisect / branch testing). The resolved spec is forwarded to docker as a build-arg. - Dockerfile: accept an optional `dd_trace_package` build-arg. When set, rewrite package.json's dd-trace dep to the override spec before yarn install. The rewrite is gated on the build-arg being non-empty, so default builds are unchanged. - publish_layers.sh: only require layer zips for the layers in LAYERS, rather than unconditionally requiring all 4 entries in LAYER_PATHS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * revert some unrelated changes * revert irrelavant changes * lint * revert dockerfile * revert dockerfile * refactor(durable-execution): drop root span, force datadog-only checkpoint extraction Two changes to align with the dd-trace-js cross-invocation tracecontext propagation branch (joey/cross-invocation-tracecontext-propagation): 1. Remove createDurableExecutionRootSpan. This integration no longer constructs a synthetic aws.durable-execution root span — the canonical entry point for a durable execution is the first aws.durable.execute span created by the dd-trace-js plugin. The Lambda wrapper just extracts the saved trace context and lets the aws.lambda span (and downstream plugin spans) anchor to it. The removed function had no production caller in this repo; only spec-only references existed. 2. Add TracerWrapper.extractDatadogOnly and use it for the _datadog_{N} checkpoint extraction path. Checkpoints are written by dd-trace-js in Datadog style only (regardless of DD_TRACE_PROPAGATION_STYLE_INJECT), so we must extract them with a matching forced-datadog propagator. Built by shadowing tracePropagationStyle.extract = ['datadog'] on a sibling propagator via the live text_map propagator's constructor, so we don't bind to a dd-trace internal module path. Upstream customer-event headers continue to use the standard extract since they come from arbitrary services with arbitrary propagation styles. * refactor(durable-execution): drop upstream-headers extraction path We don't currently expect tracecontext to be carried on the customer event that triggers a durable execution, so the upstream-headers branch had no real input to operate on. Drop the call site (which referenced the deleted findUpstreamHeaders helper and was a compile error after the partial removal) along with the corresponding test and the now-unused `standard` return mock in the test helper. The extractor now follows a single path: pull the latest _datadog_{N} checkpoint payload and hand it to TracerWrapper.extractDatadogOnly. * simplify AI code * remove isDurableExecutionReplay * further cleanup * simplify the extractor part * further simplify * stype consistency * lint * lint * add DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED env explanation in README.md --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 655f5f2 commit 5b3377b

6 files changed

Lines changed: 180 additions & 0 deletions

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ Besides the environment variables supported by dd-trace-js, the datadog-lambda-j
3030
| DD_COLD_START_TRACE_SKIP_LIB | optionally skip creating Cold Start Spans for a comma-separated list of libraries. Useful to limit depth or skip known libraries. | `./opentracing/tracer` |
3131
| DD_CAPTURE_LAMBDA_PAYLOAD | [Captures incoming and outgoing AWS Lambda payloads][1] in the Datadog APM spans for Lambda invocations. | `false` |
3232
| DD_CAPTURE_LAMBDA_PAYLOAD_MAX_DEPTH | Determines the level of detail captured from AWS Lambda payloads, which are then assigned as tags for the `aws.lambda` span. It specifies the nesting depth of the JSON payload structure to process. Once the specified maximum depth is reached, the tag's value is set to the stringified value of any nested elements beyond this level. <br> For example, given the input payload: <pre>{<br> "lv1" : {<br> "lv2": {<br> "lv3": "val"<br> }<br> }<br>}</pre> If the depth is set to `2`, the resulting tag's key is set to `function.request.lv1.lv2` and the value is `{\"lv3\": \"val\"}`. <br> If the depth is set to `0`, the resulting tag's key is set to `function.request` and value is `{\"lv1\":{\"lv2\":{\"lv3\": \"val\"}}}` | `10` |
33+
| DD_DURABLE_CROSS_INVOCATION_TRACING_ENABLED | For AWS Durable functions, the tracer creates extra checkpoints named `_datadog_{N}` to propagate trace context across function invocations, keeping spans from multiple invocations in one intact trace for each durable execution. | `true` |
3334

3435

3536
## Lambda Profiling Beta

src/trace/context/extractor.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { XrayService } from "../xray-service";
55
import {
66
AppSyncEventTraceExtractor,
77
CustomTraceExtractor,
8+
DurableExecutionEventTraceExtractor,
89
EventBridgeEventTraceExtractor,
910
EventBridgeSQSEventTraceExtractor,
1011
HTTPEventTraceExtractor,
@@ -81,6 +82,9 @@ export class TraceContextExtractor {
8182
private getTraceEventExtractor(event: any): EventTraceExtractor | undefined {
8283
if (!event || typeof event !== "object") return;
8384

85+
if (EventValidator.isDurableExecutionEvent(event))
86+
return new DurableExecutionEventTraceExtractor(this.tracerWrapper);
87+
8488
const headers = event.headers ?? event.multiValueHeaders;
8589
if (headers !== null && typeof headers === "object") {
8690
return new HTTPEventTraceExtractor(this.tracerWrapper, this.config.decodeAuthorizerContext);
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
import { DurableExecutionEventTraceExtractor } from "./durable-execution";
2+
import { TracerWrapper } from "../../tracer-wrapper";
3+
4+
function makeTracerWrapper(extractReturn: any = null): TracerWrapper {
5+
return {
6+
extract: jest.fn().mockReturnValue(extractReturn),
7+
} as unknown as TracerWrapper;
8+
}
9+
10+
describe("DurableExecutionEventTraceExtractor", () => {
11+
beforeEach(() => {
12+
jest.clearAllMocks();
13+
});
14+
15+
it("extracts checkpoint headers via the standard propagator", () => {
16+
const executionArn = "arn:aws:lambda:us-east-2:123456789012:function:demo:$LATEST/durable-execution/demo/abc";
17+
18+
const checkpointHeaders = {
19+
"x-datadog-trace-id": "149750110124521191",
20+
"x-datadog-parent-id": "987654321012345678",
21+
"x-datadog-sampling-priority": "1",
22+
};
23+
24+
const event = {
25+
DurableExecutionArn: executionArn,
26+
CheckpointToken: "t-1",
27+
InitialExecutionState: {
28+
Operations: [
29+
{
30+
Id: "op-1",
31+
Name: "_datadog_0",
32+
Status: "SUCCEEDED",
33+
StepDetails: {
34+
Result: JSON.stringify(checkpointHeaders),
35+
},
36+
},
37+
],
38+
},
39+
};
40+
41+
const sentinelContext = { sentinel: true };
42+
const tracerWrapper = makeTracerWrapper(sentinelContext);
43+
const extractor = new DurableExecutionEventTraceExtractor(tracerWrapper);
44+
const context = extractor.extract(event);
45+
46+
// Checkpoint headers are Datadog-style; the default extract list includes
47+
// `datadog`, so the standard extract path picks them up.
48+
expect(tracerWrapper.extract).toHaveBeenCalledWith(checkpointHeaders);
49+
expect(context).toBe(sentinelContext);
50+
});
51+
52+
it("returns null when no checkpoint exists", () => {
53+
const tracerWrapper = makeTracerWrapper();
54+
const extractor = new DurableExecutionEventTraceExtractor(tracerWrapper);
55+
56+
const context = extractor.extract({
57+
DurableExecutionArn: "arn:aws:lambda:us-east-2:123:function:demo",
58+
CheckpointToken: "t-empty",
59+
InitialExecutionState: { Operations: [] },
60+
});
61+
62+
expect(context).toBeNull();
63+
expect(tracerWrapper.extract).not.toHaveBeenCalled();
64+
});
65+
});
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
/**
2+
* Durable Execution Trace Extractor — Checkpoint Approach
3+
*
4+
* Strategy:
5+
* 1. Look for trace context in the latest `_datadog_{N}` checkpoint.
6+
* 2. If no trace checkpoint exists, return null and let the default extraction
7+
* path create the context.
8+
*
9+
* The extracted context becomes the parent of the `aws.lambda` span (and any
10+
* downstream spans created by dd-trace-js, including `aws.durable.execute`).
11+
* Therefore all `aws.lambda` spans will be anchored to the first
12+
* `aws.durable.execute` span for a durable execution.
13+
*
14+
* Checkpoint data will be written by the dd-trace-js plugin in Datadog style
15+
* (`x-datadog-*`). Extraction goes through the standard `TracerWrapper.extract`
16+
* path, which honors `DD_TRACE_PROPAGATION_STYLE_EXTRACT`. The default extract
17+
* list (`datadog, tracecontext, baggage`) already includes `datadog`. Customers
18+
* who override the extract list MUST keep `datadog` in it.
19+
*/
20+
21+
import { logDebug } from "../../../utils";
22+
import { SpanContextWrapper } from "../../span-context-wrapper";
23+
import { TracerWrapper } from "../../tracer-wrapper";
24+
import { EventTraceExtractor } from "../extractor";
25+
26+
const TRACE_CHECKPOINT_NAME_PREFIX = "_datadog_";
27+
28+
interface CheckpointOperation {
29+
Name?: string;
30+
Payload?: string;
31+
StepDetails?: { Result?: string };
32+
}
33+
34+
interface DurableExecutionEventShape {
35+
DurableExecutionArn?: string;
36+
InitialExecutionState?: { Operations?: CheckpointOperation[] };
37+
}
38+
39+
function parseTraceCheckpointNumber(name: unknown): number | null {
40+
if (typeof name !== "string") return null;
41+
42+
if (!name.startsWith(TRACE_CHECKPOINT_NAME_PREFIX)) return null;
43+
const suffix = name.slice(TRACE_CHECKPOINT_NAME_PREFIX.length);
44+
const n = Number.parseInt(suffix, 10);
45+
if (Number.isNaN(n) || String(n) !== suffix) return null;
46+
return n;
47+
}
48+
49+
/**
50+
* Find the highest-numbered `_datadog_{N}` checkpoint in the event and return
51+
* its parsed header dict.
52+
*
53+
* Each invocation that changes trace context saves a new checkpoint with N+1;
54+
* the one with the highest N is the most recent. Headers are written by the
55+
* dd-trace-js plugin via `tracer.inject(span, 'http_headers', headers)` so the
56+
* payload is a standard HTTP-style header dict.
57+
*
58+
*/
59+
function findLatestCheckpointHeaders(event: DurableExecutionEventShape): Record<string, string> | null {
60+
const operations = event.InitialExecutionState?.Operations;
61+
if (!operations || operations.length === 0) return null;
62+
63+
let best: { number: number; op: CheckpointOperation } | null = null;
64+
for (const op of operations) {
65+
const n = parseTraceCheckpointNumber(op?.Name);
66+
if (n === null) continue;
67+
if (best === null || n > best.number) {
68+
best = { number: n, op };
69+
}
70+
}
71+
if (best === null) return null;
72+
73+
const raw = best.op.Payload ?? best.op.StepDetails?.Result;
74+
if (!raw || typeof raw !== "string") return null;
75+
try {
76+
const parsed = JSON.parse(raw);
77+
if (parsed && typeof parsed === "object") {
78+
return parsed as Record<string, string>;
79+
}
80+
} catch (e) {
81+
logDebug(`Failed to parse trace checkpoint payload: ${e}`);
82+
}
83+
return null;
84+
}
85+
86+
export class DurableExecutionEventTraceExtractor implements EventTraceExtractor {
87+
constructor(private tracerWrapper: TracerWrapper) {}
88+
89+
extract(event: unknown): SpanContextWrapper | null {
90+
const e = event as DurableExecutionEventShape | undefined;
91+
if (!e?.DurableExecutionArn) {
92+
logDebug("No DurableExecutionArn in event");
93+
return null;
94+
}
95+
96+
const checkpointHeaders = findLatestCheckpointHeaders(e);
97+
if (checkpointHeaders) {
98+
logDebug("Extracting trace context from durable checkpoint");
99+
return this.tracerWrapper.extract(checkpointHeaders);
100+
}
101+
102+
logDebug("No durable trace context found; deferring to default extraction");
103+
return null;
104+
}
105+
}

src/trace/context/extractors/index.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,4 @@ export { SNSSQSEventTraceExtractor } from "./sns-sqs";
99
export { StepFunctionEventTraceExtractor } from "./step-function";
1010
export { LambdaContextTraceExtractor } from "./lambda-context";
1111
export { CustomTraceExtractor } from "./custom";
12+
export { DurableExecutionEventTraceExtractor } from "./durable-execution";

src/utils/event-validator.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,4 +74,8 @@ export class EventValidator {
7474
static isKinesisStreamEvent(event: any): event is KinesisStreamEvent {
7575
return Array.isArray(event.Records) && event.Records.length > 0 && event.Records[0].kinesis !== undefined;
7676
}
77+
78+
static isDurableExecutionEvent(event: any): boolean {
79+
return typeof event.DurableExecutionArn === "string";
80+
}
7781
}

0 commit comments

Comments
 (0)