fix(metrics): hold back cold-start invocation metric until durable tag is known#1301
Draft
lym953 wants to merge 1 commit into
Draft
fix(metrics): hold back cold-start invocation metric until durable tag is known#1301lym953 wants to merge 1 commit into
lym953 wants to merge 1 commit into
Conversation
…g is known In On-Demand mode, the Extensions API's `/next` poll (driving on_invoke_event) consistently beats the buffered Telemetry API's platform.initStart (which sets the durable_function tag), so the first invocation's aws.lambda.enhanced.invocations metric almost always missed durable_function:true. Hold the sandbox's first invocation metric in a pending field and flush it once platform.initStart has set the durable tag, with fallbacks on the next invoke or shutdown so the metric is never dropped if initStart never arrives.
This comment has been minimized.
This comment has been minimized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
On the sandbox's first (cold-start) invocation in On-Demand mode, the
aws.lambda.enhanced.invocationsmetric is almost always missing thedurable_function:truetag.Root cause: two independent event producers feed the same single-threaded
Processoractor with no ordering guarantee between them:/nextlong-poll driveson_invoke_event, unbuffered.platform.initStartevent (which setsdurable_function:trueviaset_durable_function_tag) is delivered through a buffered subscription.In On-Demand mode,
/next's Invoke event consistently reaches the processor first, so the invocation metric was emitted beforedurable_function:truewas known, and the tag was missed on invocation #1. This was confirmed empirically against real Lambda functions (0/3 cold starts tagged in Managed Instance mode, ~100% missing in On-Demand mode) and against AWS docs — the durable execution ARN is never exposed via the Extensions API, so a synchronous ARN-based detection at invoke time isn't possible.Fix: hold back the invocation metric for the sandbox's very first On-Demand invocation (
pending_first_invocation_metric: Option<i64>) untilplatform.initStarthas had a chance to set the durable tag, then flush it. Fallback flush points on the next invoke and on shutdown ensure the metric is never dropped ifplatform.initStartnever arrives. No newProcessorCommandvariant or timer is needed.Testing
processor.rs:test_on_invoke_event_cold_start_holds_invocation_metric_until_init_start— verifies the metric is held back onon_invoke_eventand flushed withdurable_function:trueonceon_platform_init_startreports a durable runtime.test_on_invoke_event_flushes_pending_metric_if_init_start_never_arrives— verifies the fallback flush on a subsequenton_invoke_eventifplatform.initStartnever shows up.cargo fmtclean.cargo clippy --all-targets -- -D warningsclean (no new warnings).lifecycle::invocation::test suite (214 tests) passes.