Skip to content

fix(metrics): hold back cold-start invocation metric until durable tag is known#1301

Draft
lym953 wants to merge 1 commit into
mainfrom
yiming.luo/durable-metric-tag-cold-start
Draft

fix(metrics): hold back cold-start invocation metric until durable tag is known#1301
lym953 wants to merge 1 commit into
mainfrom
yiming.luo/durable-metric-tag-cold-start

Conversation

@lym953

@lym953 lym953 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Overview

On the sandbox's first (cold-start) invocation in On-Demand mode, the aws.lambda.enhanced.invocations metric is almost always missing the durable_function:true tag.

Root cause: two independent event producers feed the same single-threaded Processor actor with no ordering guarantee between them:

  • The Extensions API's /next long-poll drives on_invoke_event, unbuffered.
  • The Telemetry API's platform.initStart event (which sets durable_function:true via set_durable_function_tag) is delivered through a buffered subscription.

In On-Demand mode, /next's Invoke event consistently reaches the processor first, so the invocation metric was emitted before durable_function:true was known, and the tag was missed on invocation #1. This was confirmed empirically against real Lambda functions (0/3 cold starts tagged in Managed Instance mode, ~100% missing in On-Demand mode) and against AWS docs — the durable execution ARN is never exposed via the Extensions API, so a synchronous ARN-based detection at invoke time isn't possible.

Fix: hold back the invocation metric for the sandbox's very first On-Demand invocation (pending_first_invocation_metric: Option<i64>) until platform.initStart has had a chance to set the durable tag, then flush it. Fallback flush points on the next invoke and on shutdown ensure the metric is never dropped if platform.initStart never arrives. No new ProcessorCommand variant or timer is needed.

Testing

  • Added two unit tests in processor.rs:
    • test_on_invoke_event_cold_start_holds_invocation_metric_until_init_start — verifies the metric is held back on on_invoke_event and flushed with durable_function:true once on_platform_init_start reports a durable runtime.
    • test_on_invoke_event_flushes_pending_metric_if_init_start_never_arrives — verifies the fallback flush on a subsequent on_invoke_event if platform.initStart never shows up.
  • cargo fmt clean.
  • cargo clippy --all-targets -- -D warnings clean (no new warnings).
  • Full lifecycle::invocation:: test suite (214 tests) passes.

…g is known

In On-Demand mode, the Extensions API's `/next` poll (driving on_invoke_event)
consistently beats the buffered Telemetry API's platform.initStart (which sets
the durable_function tag), so the first invocation's aws.lambda.enhanced.invocations
metric almost always missed durable_function:true.

Hold the sandbox's first invocation metric in a pending field and flush it once
platform.initStart has set the durable tag, with fallbacks on the next invoke or
shutdown so the metric is never dropped if initStart never arrives.
@datadog-datadog-prod-us1-2

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant