How should waitForCallback be retried durably after callback failure or heartbeat timeout?

[yuya-takeyama]

Hi, I am trying to understand the intended pattern for using waitForCallback with an external worker that may fail after accepting the callback work.

Problem

I want to run an external async task through waitForCallback and make the workflow eventually complete according to a retry policy, even if the external system fails after the callback ID has already been submitted.

From reading the docs and SDK source, it looks like waitForCallback's retryStrategy only retries the submitter step. If the external system later reports failure with SendDurableExecutionCallbackFailure, or stops sending heartbeats and the callback hits heartbeatTimeout, the callback promise fails, but a new callback operation is not automatically created.

That means I currently need to write my own retry loop around waitForCallback.

Minimal example

async function runExternalTask(context: DurableContext): Promise<Result> {
  let attempt = 0;

  while (true) {
    attempt += 1;

    try {
      return await context.waitForCallback<Result>(
        "external-task",
        async callbackId => {
          await submitTaskToExternalSystem({ callbackId });
        },
        {
          timeout: { minutes: 5 },
          heartbeatTimeout: { seconds: 30 },
          retryStrategy: (error, attemptCount) => ({
            shouldRetry: attemptCount < 20,
            delay: { seconds: 1 },
          }),
        },
      );
    } catch (error) {
      if (!isRetryableCallbackError(error) || attempt >= 20) {
        throw error;
      }

      await context.wait({ seconds: 1 });
    }
  }
}

Question / feature request

Is there a recommended durable pattern for retrying waitForCallback itself after callback failure or heartbeat timeout?

Ideally, if the external system crashes after accepting the callback work, I would like the SDK to provide a way to retry the whole callback operation according to a retry policy, creating a new callback ID and re-running the submitter durably.

The main thing I am looking for is a durable, SDK-supported way to express: "if the external worker fails or stops heartbeating, retry the external task according to this policy."

[SilanHe]

Hello, we are in the process of implementing the following feature which I believe would help with this use case: aws/aws-durable-execution-docs#140 (comment)

In the problem statement, there is a pattern that you can use in the meantime.

[yuya-takeyama]

Thanks! That makes sense.

The withRetry helper proposed in #140 looks like exactly what I was looking for.

Closing this as answered.