Skip to content

Fix durable execution conformance issues found via cross-SDK testing#2473

Merged
GarrettBeatty merged 4 commits into
devfrom
durable-execution-conformance-fixes
Jul 13, 2026
Merged

Fix durable execution conformance issues found via cross-SDK testing#2473
GarrettBeatty merged 4 commits into
devfrom
durable-execution-conformance-fixes

Conversation

@GarrettBeatty

Copy link
Copy Markdown
Contributor

Summary

Fixes five behavioral bugs in the Amazon.Lambda.DurableExecution operations, surfaced by running the multi-language durable-execution conformance suite against the .NET SDK and comparing the emitted execution history / results against the JS, Python, and Java reference SDKs.

All five were confirmed as divergences from the reference SDKs (one matches a known open bug in the Python SDK, issue #279).

Fixes

  1. ConcurrentOperation — missing ParentId on parent CONTEXT checkpoints. Parallel/Map did not set ParentId on their START/SUCCEED checkpoints (every other operation type does). This dropped the parent linkage for a nested parallel/map — its child events lost their ParentId.

  2. CallbackOperation — missing ParentId on the callback START. Nested callbacks (e.g. the inner callback created by WaitForCallbackAsync) lost their ParentId.

  3. WaitForConditionOperationStepStarted only emitted on the first poll. Now emits a fresh StepStarted before every poll iteration, matching Python/JS/Java. On resume the persisted status is READY/PENDING, so each poll gets its own START; the only skip is when the status is already STARTED (crash-recovery, where the original START is authoritative).

  4. WaitForConditionOperationSTART checkpoint was awaited instead of fire-and-forget. Awaiting it forced two back-to-back synchronous flushes (START then terminal FAIL) with no suspension when the check throws on the first attempt, which could hang the invocation. Now fire-and-forget, mirroring StepOperation and the reference SDKs. Replay correctness depends only on the terminal RETRY/SUCCEED/FAIL checkpoints.

  5. ConcurrentOperation/BatchResultTotalCount counted never-dispatched branches. Never-dispatched branches (skipped by a CompletionConfig short-circuit before they ever started) are now excluded from the user-facing All list, so TotalCount = started(in-flight) + succeeded + failed, matching the JS SDK (same bug as Python Support for Binary Data using Razor Pages #279). A dispatched-then-bailed (genuinely in-flight) branch still appears as Started. The completion policy still reasons over the declared branch count (so ToleratedFailurePercentage and the early-stop / min-successful signal are unchanged), and the persisted checkpoint summary still records every declared unit for replay/reconstruct. The replay and reconstruct paths were updated to keep the reconstructed All consistent with the fresh run.

Tests

Updated the affected ParallelOperationTests and MapOperationTests to reflect the corrected never-dispatched semantics. Dispatched-then-bailed (in-flight) cases are deliberately unchanged. All 72 parallel + map operation unit tests pass.

Validated end-to-end against the conformance harness: all implementable .NET conformance tests pass across step, wait, callback, child, invoke, parallel, wait_for_callback, and wait_for_condition operations.

Draft: opening for review of the SDK behavior changes. Three conformance cases (per-operation custom serdes for child/parallel/invoke) remain unimplementable in .NET and are tracked separately as a capability gap, not addressed here.

Fixes five behavioral bugs in the DurableExecution operations, surfaced by
running the multi-language conformance suite against the .NET SDK and comparing
against the JS/Python/Java reference SDKs.

1. ConcurrentOperation: emit ParentId on the parent CONTEXT START/SUCCEED
   checkpoints. Every other operation type sets ParentId; parallel/map did not,
   which dropped the parent linkage for a nested parallel/map (its child events
   lost their ParentId).

2. CallbackOperation: emit ParentId on the callback START checkpoint. Nested
   callbacks (e.g. the inner callback of a WaitForCallback) lost their ParentId.

3. WaitForConditionOperation: emit a fresh StepStarted before every poll
   iteration (not just the first). On resume the persisted status is
   READY/PENDING, so each poll now gets its own START, matching the
   Python/JS/Java SDKs. The only skip is when status is already STARTED
   (crash-recovery: the original START is authoritative).

4. WaitForConditionOperation: make the START checkpoint fire-and-forget instead
   of awaiting it. Awaiting forced two back-to-back synchronous flushes
   (START then terminal FAIL) with no suspension when the check throws on the
   first attempt, which could hang the invocation. Mirrors StepOperation and the
   reference SDKs; replay correctness relies only on the terminal
   RETRY/SUCCEED/FAIL checkpoints.

5. ConcurrentOperation/BatchResult: exclude never-dispatched branches from the
   user-facing All list so TotalCount = started(in-flight) + succeeded + failed,
   matching the JS SDK (same bug tracked in Python issue #279). A
   dispatched-then-bailed (genuinely in-flight) branch still appears as Started.
   The completion policy still reasons over the declared branch count (so
   ToleratedFailurePercentage and the early-stop signal are unchanged), and the
   persisted checkpoint summary still records every declared unit for
   replay/reconstruct. Replay/reconstruct paths updated to keep the reconstructed
   All consistent with the fresh run.

Updated the affected Parallel/Map operation unit tests to reflect the corrected
never-dispatched semantics (dispatched-then-bailed cases are unchanged).
READY/PENDING replays now emit a fresh START each poll iteration to match
the spec and Python/JS/Java SDKs; only a STARTED-status replay skips it.
The two replay tests still asserted the old start-once behavior and failed
CI's Assert.DoesNotContain(..."START"). Flip them to Assert.Contains.
The service now re-emits a fresh START per poll iteration (READY/PENDING
replays re-emit START to match the spec and Python/JS/Java SDKs). The two
integration tests still asserted a single StepStarted per polling op and
failed CI (Expected: 1, Actual: 3 / expected a single element). Assert one
START per SUCCEED across iterations instead, mirroring the unit-test fix in
69c03f3.
Documents the WaitForCondition per-iteration START behavior change in 6ec8d3f.
@GarrettBeatty GarrettBeatty marked this pull request as ready for review July 13, 2026 17:37
@GarrettBeatty GarrettBeatty requested review from a team as code owners July 13, 2026 17:37
@GarrettBeatty GarrettBeatty requested review from normj and philasmar July 13, 2026 17:37
@GarrettBeatty GarrettBeatty merged commit df20caa into dev Jul 13, 2026
5 checks passed
@GarrettBeatty GarrettBeatty deleted the durable-execution-conformance-fixes branch July 13, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants