Skip to content

Commit 9f9d797

Browse files
lym953claude
andcommitted
test(oom): bump python OOM function to 512 MB to fix flaky metric assertion
The `integration-suite: [oom]` python case failed ~1 run in 3 with `out_of_memory` count 0 (all other runtimes passing). Datadog logs show the python function emitted no telemetry at all on failing runs — no logs and no metric — while the same function emits both on passing runs. At 256 MB the python3.13 runtime plus the `datadog_lambda` handler shim leave too little headroom, so when the function OOMs the kernel OOM-killer intermittently takes the extension too before it can run its end-of-invocation flush, dropping both logs and the enhanced metric. Bumping the python function to 512 MB keeps the extension alive to detect and flush. Detection paths are unchanged: the function still hits its memory cap and emits `MemoryError`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 4b5a7e6 commit 9f9d797

1 file changed

Lines changed: 12 additions & 11 deletions

File tree

  • integration-tests/lib/stacks

integration-tests/lib/stacks/oom.ts

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -53,19 +53,20 @@ export class Oom extends cdk.Stack {
5353
const dotnetLayer = getDefaultDotnetLayer(this);
5454
const rubyLayer = getDefaultRubyLayer(this);
5555

56-
// 256 MB (not the customer's 192 MB from #1237) so the bottlecap
57-
// extension has memory headroom to survive when the function process
58-
// OOMs. At 192 MB the kernel OOM-killer often picks the extension
59-
// instead of the function runtime (Lambda surfaces this as the
60-
// `Extension.Crash` error type), and a dead extension can't emit the
61-
// OOM metric. With 256 MB the function runtime's RSS dominates and
62-
// kernel reliably kills it; the extension survives to detect/flush.
63-
// The detection paths under test are unchanged — the functions still
64-
// hit `max_memory_used == memory_size` in PlatformReport and still
65-
// emit runtime-specific OOM error log lines.
56+
// 256 MB (not the customer's 192 MB from #1237) to give the extension
57+
// headroom to survive the function's OOM. If memory is too tight the
58+
// kernel OOM-killer takes the extension instead (Lambda reports
59+
// `Extension.Crash`), and a dead extension emits neither logs nor the OOM
60+
// metric. Detection paths are unchanged: functions still hit
61+
// `max_memory_used == memory_size` and emit runtime-specific OOM log lines.
6662
const oomMemorySize = 256;
6763
const oomTimeout = cdk.Duration.seconds(30);
6864

65+
// python3.13 + the `datadog_lambda` handler shim need more slack: at
66+
// 256 MB the extension was intermittently killed before its flush,
67+
// dropping the metric (~1 run in 3). 512 MB keeps it alive.
68+
const pythonOomMemorySize = 512;
69+
6970
// Node case A — V8 heap exhaustion (log-line path).
7071
const nodeV8FunctionName = `${id}-node-v8-heap-lambda`;
7172
const nodeV8Function = new lambda.Function(this, nodeV8FunctionName, {
@@ -122,7 +123,7 @@ export class Oom extends cdk.Stack {
122123
code: lambda.Code.fromAsset('./lambda/oom-python'),
123124
functionName: pythonFunctionName,
124125
timeout: oomTimeout,
125-
memorySize: oomMemorySize,
126+
memorySize: pythonOomMemorySize,
126127
environment: {
127128
...defaultDatadogEnvVariables,
128129
DD_SERVICE: pythonFunctionName,

0 commit comments

Comments
 (0)