Skip to content

[Bug] interceptors-opentelemetry breaks workflow determinism when handleQuery/validateUpdate are called #2023

Description

@orr-a-security

What are you really trying to do?

Use @temporalio/interceptors-opentelemetry to capture spans from long-lived Temporal workflows in our tracing backend, with workflow.uuid4()-generated child workflow ids passed to startChild().

Describe the bug

With the stock OTel workflow interceptors installed, workflow.uuid4() calls made after any handled query or update-validation return a different value on replay than what was recorded. If that uuid appears in a recorded command (child workflow id, activity arg, etc.), replay fails:

[TMPRL1100] Nondeterminism error:
Child workflow id of scheduled event 'child-…-'
does not match child workflow id of command 'child-…-'

The workflow runs fine as long as it stays in sticky cache. The error only surfaces when the workflow is evicted and has to replay from history on a fresh worker — typically hours later, after a pod roll.

Minimal Reproduction

export async function Parent(): Promise<string> {
  let tickCount = 0;
  let queryCount = 0;

  setHandler(tickUpdate, () => {tickCount++;});
  setHandler(noopQuery, () => ++queryCount);

  // wait for update #1
  await condition(() => tickCount >= 1);

  // here need to call a query (will advance the PRNG)

  // wait for update #2
  await condition(() => tickCount >= 2);

  // PRNG position is shifted by any queries handled between tick #1 and tick #2.
  const id = uuid4();

  // Replay never re-invokes the query handler, so its PRNG isn't advanced by those
  // bytes; `id` ends up different than what was recorded. Because it feeds into a
  // recorded command, replay fails with TMPRL1100.
  await startChild(Child, {
    workflowId: `repro-child-${workflowInfo().workflowId}-${id}`,
    taskQueue: workflowInfo().taskQueue,
  });

  return id;
}

Environment/Versions

  • OS / arch: macOS arm64 and Linux amd64/arm64 (Node 20 & 25). No platform dependency.
  • SDK: reproduces on @temporalio/@1.16.1 (latest, what hello-world pins to) and @temporalio/@1.14.1 (where we first saw it in production). @opentelemetry/sdk-trace-base@1.30.1 (transitive).
  • Temporal server: embedded via TestWorkflowEnvironment.createLocal() in the repro; originally observed against self-hosted Temporal on Kubernetes.

Additional context

This issue may be related to #1790

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions