world-postgres: intended model for horizontal scaling? Seeing the same step execute on multiple replicas

[isimisi]

We run several app replicas against one shared Postgres, and we keep seeing the same step run on more than one replica at the same time. We can't tell from the docs whether that's expected (and we should make our steps idempotent) or something we're getting wrong. Before we design around it, we'd like to know what the intended model is.

Per #64, there's no external runner and WDK is a client-side library backed by a world. So on a self-hosted setup the graphile-worker runner is embedded in each app instance, and scaling the app horizontally scales the runners with it. That's the situation the questions below are about.

Environment

• @workflow/world-postgres@4.2.0, graphile-worker, one shared Postgres.
• Several app replicas (a Kubernetes Deployment behind an HPA, so multiple pods). Each boots the world with createWorld(...) and starts the runner. We haven't changed anything in the queue or the runner; we use world-postgres as published. Our HTTP framework integration is custom, but the world, the runner, and the enqueue path are stock.

What we see

Within a single run, one step runs on two pods at once:

• The step's attempt counter reaches 2 or 3, and step_started is written 2 or 3 times for the same (run_id, correlation_id).
• Application logs show the step body running on two pods with overlapping time windows: same input, and the same external side effect performed twice.
• Only one step_completed lands (the other writer's completion is rejected), so the event log looks clean. But the side effect ran twice, and so did the streamed output. In our case the visible result is a durable output stream with a second, dangling copy of the step's events.

As far as we can tell, the concurrency behavior sits in world-postgres, not in any framework adapter. The runner (run()) starts once per process, and across processes there is no shared execution dedup. The only guards are in-process: the inflightWorkflowRuns serialization keyed by workflow:${runId}, and the completedMessages / inflightMessages caches. None of those cross pods. The step_started entity guard (notInArray(status, terminalStepStatuses)) still matches a step that another pod has already set to running, so a second replica re-enters and runs the body. We're not suggesting that as the thing to change; we point at it only to show where the behavior comes from. The same multi-pod setup was already reported for a different symptom in #2039 ("three HPA-scaled pods against the same Postgres World"), so we don't think this is specific to our integration.

Prior art we read first

• @workflow/world-postgres can insert duplicate step_created events under concurrent workers #2039: duplicate step_created under concurrent workers, fixed with the entity-creation unique index. That's event-level, not execution-level.
• @workflow/world-postgres: late concurrent step_started can land after step_completed in the event log (non-atomic entity UPDATE + event INSERT) #2331 and Fix Postgres step lifecycle event ordering #2714: late concurrent step_started landing after step_completed; Fix Postgres step lifecycle event ordering #2714 puts the lifecycle update and its event write in one transaction for log consistency. That's ordering, not execution dedup.
• Idempotency follow-ups: atomic keyed start(), post-completion dedupe, and attribute-based run lookup #2376: the idempotency roadmap (atomic keyed start(), post-completion dedupe).
• Concurrency Controls #301: the user-facing concurrency lock primitive.
• Docker image deployment with Next.js: Single image inside Next.js or multiple images for web / workflow runner respectively? #64: the embedded-runner deployment model.

None of them say what guarantee step execution has when the workers are horizontally scaled.

Questions

1. Is world-postgres meant to run as several concurrent embedded runners (horizontally scaled replicas against one Postgres), or is a single worker process the intended model?
2. If several workers are supported, what is the guarantee for step execution across them: exactly-once, or at-least-once by design (so side effects are expected to be idempotent)?
3. If a single worker is the intended model, is that documented anywhere, and what do you recommend for HA and scaling?

Impact

Right now we can't scale the app horizontally without steps running more than once, which pushes us toward a single worker instance for otherwise stateless work. That's a scaling ceiling and a single point of failure, so we want to match the intended model before building around it.

[Keanottesen]

+1