What's the recommended shape for chat orchestrators with variable step count (up to ~99) and total runtime that may exceed 800s? Per-step gap growth makes per-LLM-round splits unviable

[shtefcs]

Problem

Using DurableAgent with the Workflow SDK in development mode results in 7-12 seconds of overhead per workflow step, making even simple agent interactions take 2-4 minutes instead of 10-15 seconds. This makes development and testing of DurableAgent-based applications effectively impossible.

We are building a multi-agent AI orchestration platform with 10 agents, 56+ tools, and complex agent chaining (web research -> coding -> deployment). The original implementation using direct streamText calls completes in 10-30 seconds. The same functionality wrapped in DurableAgent takes 3-12 minutes due to per-step overhead.

Environment

• workflow: 4.2.1
• @workflow/ai: 4.1.1
• @workflow/core: 4.2.1
• @workflow/world-postgres: 4.1.0
• Next.js: 16.1.6 (Turbopack)
• Node.js: 22.x
• Platform: WSL2 on Windows, 48GB RAM
• Database: Neon Postgres (US East)

Measured Latency

Every POST /.well-known/workflow/v1/flow request takes 7-12 seconds:

POST flow 200 in 9.1s  (compile: 2ms, render: 9.1s)
POST flow 200 in 7.9s  (compile: 2ms, render: 7.9s)
POST flow 200 in 6.9s  (compile: 3ms, render: 6.9s)
POST flow 200 in 9.7s  (compile: 2ms, render: 9.7s)
POST flow 200 in 7.1s  (compile: 4ms, render: 7.1s)
POST flow 200 in 7.6s  (compile: 3ms, render: 7.6s)
POST flow 200 in 8.0s  (compile: 4ms, render: 8.0s)

The compile: 2-4ms confirms route compilation is cached. The render: 7-12s is pure SDK processing time.

A simple general agent query (1 LLM call, no tools) takes 3.5 minutes total. A coding agent session with 15 tool calls takes 8-12 minutes.

With the original direct streamText approach (no workflow), the same queries complete in 10-30 seconds.

Root Cause Analysis

We traced the issue through the SDK source code and identified three compounding factors:

1. Full event log replay on every flow request

In @workflow/core/dist/runtime/helpers.js, getAllWorkflowRunEvents() loads ALL events from the database on every flow request via paginated queries. The SDK developers have acknowledged this is suboptimal with a TODO comment at line 231-233:

"TODO: we're currently loading all the data with resolveRef behaviour. We need to update this to lazyload the data from the world instead so that we can optimize and make the event log loading much faster and memory efficient"

For a workflow with 30 completed steps, this means 30+ event records loaded from the database on every single flow request, even though only the latest step result is needed to resume.

2. Linear growth of step processing time

This matches the behavior reported in #1315, where step-to-step overhead grows linearly with completed step count (+0.62s per minute of runtime). Our DurableAgent workflows with 15-30 tool calls accumulate significant replay overhead as the workflow progresses.

3. Queue round-trip latency

Each step requires a full round-trip through the queue system:

1. Flow handler suspends at step boundary
2. Step queued via graphile-worker (or Vercel Queue)
3. Step handler executes (the actual useful work: 1-5s for LLM calls)
4. Step re-queues the flow
5. Flow handler replays entire event log again

This matches the queue delay observations in #1160 where steps spend 4-5s in queue even when execution takes less than 200ms.

Impact on DurableAgent Development

DurableAgent splits each LLM call and each tool execution into separate workflow steps. A typical coding agent session involves:

• 1 routing step
• 1 prepareAgentConfig step
• 5-15 doStreamStep + tool execution cycles (LLM call + tool call per cycle)
• 1 processAgentResult step
• 1 verify loop step
• 1 finalization step

That is 10-20 steps minimum. At 7-12s overhead per step, the workflow overhead alone (excluding actual LLM/tool execution time) is 70-240 seconds. This makes development iteration painfully slow.

What We Have Tried

• Local World: Faster step-to-step (in-memory queue) but still has 2-3s per step overhead from VM replay. Also causes OOM crashes on long sessions and loses state on server restart.
• Postgres World with remote Neon DB: 7-12s per step due to network latency multiplied by event count.
• Reducing step count: Combined post-agent steps from 5 to 3. Helps marginally but the fundamental per-step overhead remains.
• Adjusting worker concurrency and pool size: No meaningful impact since the bottleneck is event replay, not worker scheduling.

Expected Behavior

Development mode should be usable for testing DurableAgent workflows. A reasonable target would be less than 500ms of SDK overhead per step, making a 20-step workflow add less than 10 seconds of total overhead.

Suggested Improvements

1. Incremental event loading: Instead of loading ALL events on every flow request, maintain a checkpoint and load only events since the last processed step. The event log is append-only, so this is safe.

2. In-process step execution for Local World: Instead of routing steps through HTTP (flow -> POST step -> step executes -> POST flow), execute step functions directly in the same process for Local World. This eliminates the queue round-trip entirely.

3. Event log pagination with cursor caching: Cache the cursor position between flow requests so subsequent calls skip already-replayed events.

4. Resume-from-checkpoint: Instead of replaying the entire workflow from the beginning, snapshot the VM state after each step and resume from the snapshot. This would make step processing O(1) instead of O(n).

Related Issues

• Workflow performance: step processing time grows with completed step count #1315 - Step processing time grows linearly with completed step count
• Significant step queue times (~4-5s) observed on Vercel World #1160 - Significant step queue times (4-5s) observed on Vercel World
• Concurrency Controls #301 - Discussion on concurrency controls and event log size limits

[VaguelySerious]

in development mode

What does this mean?

Either way, the behavior is odd, and it's hard for us to reproduce. It looks like your agent might also have gone the wrong way when doing root cause analysis:

Full event log replay on every flow request

Yes, but this takes <=100ms normally

Linear growth of step processing time

Yes, but under 100 steps in a run, this is negligible

Queue round-trip latency

Similarly sub 100ms between steps.

---

The real issue is you're seeing 10s flow endpoints, and it has nothing to do with any of the above. Can you share logs of network requests that the flow route is making? If you run your deployment with DEBUG=workflow:, you get additional logs.

Also, what region are you in? Are you deploying to Vercel or something else?

[shtefcs]

Posting actual data and full architecture this time, sorry the earlier theories were guesses. We have DEBUG=workflow:* on, and we are 3 patch versions behind on stable.

Our setup

• workflow 4.2.1
• @workflow/ai 4.1.1
• @workflow/core 4.2.1
• ai 6.0.141
• Next.js 16.1.6
• WORKFLOW_TARGET_WORLD=vercel
• Vercel deployment, region iad1, US East
• Neon Postgres database, also US East

We use the SDK for two distinct cases. One is a long running voting campaign workflow that runs for hours and benefits from durability. The other is the chat orchestrator (the one this issue is about), which is interactive, typically finishes in 5 to 30 seconds, and drives a streaming UI. The chat orchestrator is what we are seeing slow.

Our chat orchestrator shape

The workflow function (orchestrateDurable) does this in order:

1. await initOrchestration() as a "use step": loads chat history, user secrets, connected integrations, user prefs, memory recall via Hindsight HTTP, last context. Six awaits, currently sequential.
2. await routeQueryStep() as a "use step": one LLM call to classify the query and pick an agent sequence.
3. await writeExecutionId() as a "use step": writes one custom chunk to getWritable.
4. for each agent in the sequence:
   • await setupBrowserContextStep() if browser agent
   • await prepareAgentConfig() as a "use step": model selection, system prompt build, tool list, message preparation, Gemini signature handling, validation. Heavy step.
   • await writeDataChunk() as a "use step" if sandbox: writes one custom chunk
   • construct DurableAgent at workflow level
   • await agent.stream({ messages, writable, prepareStep, onStepFinish, onAbort, ... })
   • inside agent.stream the SDK runs doStreamStep per LLM call and one step per tool execute
   • prepareStep callback runs before each LLM call. We do compaction, doom loop detection, error injection, task plan re-injection from a use step (loadTaskPlanInStep), context alerts. It calls saveIncrementalMessages as a use step on every LLM round
   • onStepFinish callback runs after each LLM call. We extract gateway cost from providerMetadata, call deductCreditsInStep as a use step, update lastKnownBalance, set creditsDepleted flag if circuit breaker tripped
   • await processAgentResult() as a "use step" after agent.stream returns
5. await finalizeOrchestration() as a "use step"
6. await sendFinalChunksAndClose() as a "use step"

For a single LLM call chat with one tool call, that is roughly:

• 5 explicit pre and post steps from us
• agent.stream internal steps: doStreamStep + tool.execute + writeToolOutputToUI + doStreamStep + closeStream = 5 SDK steps
• onStepFinish triggers deductCreditsInStep, so add 1 to 2 nested step calls
• Total per chat: roughly 12 to 15 step boundaries

For a coding agent with 15 tool calls, that scales to 30 to 45 boundaries because each tool gets its own SDK step.

Tool definitions look like this (about 126 of them):

async function durableWebSearch(input: any, ctx: any) {
  "use step";
  const mod = await import("@/lib/tools/web");
  return mod.webSearchTool.execute?.(input, ctx);
}

Each tool wrapper is its own step function so DurableAgent persists each tool result independently.

Observed timing

Test chat: a single tool call (cryptoGetPrices), one agent (general), one user message. Total wall clock 107.7 seconds.

35 framework events total. Events POST timings are fine and match what you said earlier:

step events: 39, 41, 41, 42, 48, 48, 49, 49, 60, 73, 124, 188, 1429 ms (one outlier)
flow events: 40, 43, 46, 47, 49, 50, 51, 59, 61, 70, 84, 87, 90, 128, 135, 231 ms
Total framework HTTP overhead: 3.9 seconds across the whole run

What we cannot explain are the silent gaps between consecutive framework events:

gap before next event after start: 6.0 s
gap mid run: 32.5 s   ← single biggest, sits between two adjacent flow + step events with no app logs in between
gap: 12.2 s
gap: 19.4 s
gap: 14.2 s
plus 7 sub second gaps

Histogram of all 35 inter event gaps:

• 0 to 100 ms: 1
• 100 to 500 ms: 14
• 500 ms to 1 s: 8
• 1 to 5 s: 7
• 5 to 15 s: 3
• over 15 s: 2

The three biggest gaps account for 64 of 107 seconds. Five biggest gaps account for 84 seconds, which is 78 percent of the chat.

In those silent windows the Vercel CLI logs show neither additional events POST nor any application level log line. With --expand we still only see framework HTTP wrapper lines for those periods.

What we want to understand

1. When events POST reports 200 OK in 84 ms and the next event does not arrive for 32 seconds, what is happening on the SDK side during those 32 seconds? Is the workflow worker actively executing user code (in which case those 32 seconds are inside our async function), or is the run queued waiting for a worker to be assigned?

2. Is a step count of 12 to 15 per chat, growing to 30 to 45 for tool heavy coding agent runs, considered high for DurableAgent? Is there documented guidance on coarsening step boundaries, or a target step count?

3. We are about to add per function timing logs around every "use step" function and around agent.stream entry and exit, so we can attribute every second to a specific phase in our code. Is there anything internal to the SDK that we can enable beyond DEBUG=workflow:* that surfaces step execution timing on the worker side, the way DEBUG=workflow:world-vercel:http surfaces the events POST timings?

4. Can the same DurableAgent run benefit from any kind of concurrency control or warm worker reuse on Vercel? Or does each step in a single workflow run risk landing on a different worker, paying repeated module init?

We will follow this up with another comment once our timing instrumentation is in and we have run a fresh chat with line by line timing inside our orchestrator. Anything specific you want us to capture, let us know.

For comparison, we keep an older non workflow path (plain streamText inside a Next.js route, manual loop with onStepFinish, no workflow runtime, no use step boundaries, single Lambda) that takes 5 to 10 seconds for the same chat. That is the baseline we are trying to understand the gap against.

[shtefcs]

Follow-up with cleaner measurements. We removed DEBUG=workflow:*. Your hint that it wasn't the cause was right, gap shape is unchanged.

Setup: WORKFLOW_TARGET_WORLD=vercel, iad1 preview, workflow@4.2.1 + @workflow/ai@4.1.1 + @workflow/core@4.2.1. Project automatio-ai-9, team automatio-ai, branch integration/agent-browser+durable.

One agent-browser chat: run wrun_01KQ2AXJWG1XR94G097QK7DDMQ, 77 steps, 175 s wall clock. Numbers from workflow inspect steps --backend=vercel.

Position	Avg gap between consecutive steps
Steps 1 to 25	961 ms
Steps 26 to 51	1524 ms
Steps 52 to 77	2135 ms

Step bodies are fast: single-digit ms for our DB writes, ~250 ms per tool, 1 to 3 s for doStreamStep (expected LLM time). Inter-step gaps account for 117 s of 175 s and grow linearly with step index. Pattern repeats across runs.

That is 15 to 20x higher than the <=100 ms baseline you mentioned earlier, so one of two things must be true:

1. We are misusing the SDK. Our hot per-tool sequence has 3 step boundaries: SDK's writeToolOutputToUI + our deductCreditsInStep + our saveIncrementalMessages. Our prepareAgentConfig step returns ~100 to 300 KB (message array + tool schemas). Step count per chat is 77 to 97. Are any of these known to inflate per-step cost on the managed backend, and is there guidance for chat workloads with this many tool calls?

2. ~1 to 2 s is the actual baseline in our setup, and <=100 ms applies elsewhere (different backend tier, region, smaller state). What distinguishes your <=100 ms case from ours?

Run ID + project + team above are everything you need to look it up. We want to keep "use workflow" for our long-running voting campaigns where durability matters. We just need to know whether chat-style runs belong here.

[shtefcs]

Hi @VaguelySerious, following up on my April 25 comment with a focused architectural question rather than re-litigating timing numbers.

What we're trying to ship

A chat endpoint that:

1. Streams chunks to the browser in real time via getWritable()
2. Survives Lambda termination mid-chat (resume via getRun(runId).getReadable({startIndex}))
3. Honors stop-button cancellations via getRun(runId).cancel()
4. Most importantly: can run beyond the 800s Lambda ceiling for coding/browser agents that legitimately need 15 to 30 minutes (cloning a repo, building a Next.js app, running a 99-tool browser automation)

The first three are working. The fourth is what we have spent a month trying to deliver and what this issue is now about.

The shapes we tried (and why each fails)

V0 baseline was plain streamText in a Next.js route, capped at 800s. Works for short chats, dies for long ones.

We tried three rewrites:

V1: V0 wrapped in one big "use step" inside "use workflow". SDK gives us durability, SDK-native cancel/resume, no Redis bridge. But the whole chat is one step, so still 800s ceiling. Ready to ship as cleanup but does not deliver the actual prize.

V2: DurableAgent with per-tool steps. Every LLM call and every tool execute is its own step. 77 steps in 175 seconds for one browser automation. Per-step gaps from workflow inspect steps: 961ms, 1524ms, 2135ms (the data in my April 25 follow-up). 15-20x higher than the "<=100ms normally" you mentioned.

Toy hybrid with manual per-LLM-round splits. Plain streamText with stopWhen: stepCountIs(1) so each call is one LLM round plus its tool batch. Wrapped each round in "use step". Browser-only test: 16-23 steps, 36-60s wall clock. Better than V2 but still 1-2s per-step gap, growing linearly. For a coding agent at 99 rounds: about 3 minutes of pure SDK gap overhead before any actual LLM time.

The shape that should logically work (each LLM round is its own step, each gets fresh 800s budget) is the same shape #1315 documents as O(N) replay cost per step, where you confirmed the architecture was not built for high N yet.

What our minimal repro shape looks like

To eliminate the "is this code-fault or SDK-fault" doubt, here is a stripped-down version we tested. No business logic, just streamText plus step boundaries, about 80 lines:

async function runRoundStep(opts: {
  messages: ModelMessage[];
  modelId: string;
  systemPrompt: string;
  toolNames: string[];
  isFirst: boolean;
}): Promise<{
  newMessages: ModelMessage[];
  finishReason: string | undefined;
}> {
  "use step";
  const { streamText, stepCountIs } = await import("ai");
  const { gateway } = await import("@ai-sdk/gateway");
  const { getWritable } = await import("workflow");
  const writable = getWritable<UIMessageChunk>();

  // Reconstruct tools INSIDE step from name strings (avoids passing
  // live tool objects across step boundary, per @okomarov's advice
  // in #1315).
  const tools = await loadTools(opts.toolNames);

  const result = streamText({
    model: gateway.languageModel(opts.modelId),
    system: opts.systemPrompt,
    messages: opts.messages,
    tools,
    stopWhen: stepCountIs(1),
  });

  for await (const chunk of result.toUIMessageStream({
    sendStart: opts.isFirst,
    sendFinish: false,
  })) {
    const writer = writable.getWriter();
    try { await writer.write(chunk); } finally { writer.releaseLock(); }
  }

  return {
    newMessages: (await result.response).messages as ModelMessage[],
    finishReason: (await result.finishReason) as string | undefined,
  };
}

export async function orchestrateHybrid(params: {
  messages: ModelMessage[];
  modelId: string;
  systemPrompt: string;
  toolNames: string[];
}) {
  "use workflow";

  let messages = params.messages;
  for (let round = 0; round < 99; round++) {
    const r = await runRoundStep({
      messages,
      modelId: params.modelId,
      systemPrompt: params.systemPrompt,
      toolNames: params.toolNames,
      isFirst: round === 0,
    });
    messages = [...messages, ...r.newMessages];
    if (r.finishReason !== "tool-calls") break;
  }
}

Step inputs are minimal serializable primitives. Tools rebuilt inside the step from name strings so we do not drag function callbacks across the boundary, per @okomarov's advice in #1315.

Even at this minimum shape, the per-step gap was 1-2s growing to 3.5s across 23 steps on Vercel iad1. Tool execution inside step bodies was about 250ms each. Step body for doStreamStep was 1-3s (expected LLM time). The gap between consecutive step events is what we cannot explain. Same pattern @okomarov reported.

The architectural question

Is per-LLM-round step boundary the correct pattern for chat orchestrators with variable round count up to about 99, where total runtime may exceed 800s?

Three possible answers, any of which lets us plan:

A. "Yes, that is correct, wait for the event-replay fix." Then we land V1 (no >800s but durable, no Redis) and revisit when you ship the fix. Is there a quarter we can plan around for that work?

B. "No, use a different pattern." What pattern? Sub-workflows per agent so each child has bounded N? Cron-driven batching like @okomarov went to? Something else specific to chat or streaming workloads?

C. "We can repro your 1-2s gap, try X." Concrete knobs we have not explored: regional placement is already iad1 plus Neon US East; we use the managed world (WORKFLOW_TARGET_WORLD=vercel); we are on 4.2.1 across the workflow packages. Anything specific likely to move 1-2s gaps to <100ms?

If the answer is "we do not know yet", that is fine, we will plan for A.

What we have ready to share

If it helps narrow toward C: I can extract the toy hybrid above into a standalone repro repo (no business logic, no auth, just one tool and one agent) and run it on a Vercel preview. Should reproduce 1-2s gaps in under 50 lines. Want me to share that?

Run IDs from the April 25 measurements are still queryable via workflow inspect steps --backend=vercel --project=automatio-ai-9 --team=automatio-ai.

Renaming this issue

Original title was dev-mode-specific. The actual question is broader (both dev and prod, both DurableAgent and plain streamText with "use step", fundamentally about chat workload fit). Title updated to reflect that.

[okomarov]

@shtefcs One important difference I see against their suggested: https://vercel.com/kb/guide/building-a-slack-agent-with-durable-workflows is the stream and durable agent are created in the 'workflow' not the 'step'.

[shtefcs]

@okomarov good catch, that's a real difference. Confirmed by reading vercel-labs/community-agent-template which the KB article points at. Their pattern: DurableAgent and getWritable() constructed at workflow level, tools have "use step" on their execute function, no manual round loop in the workflow function. The toy snippet I posted does the opposite (getWritable() and streamText inside the step, manual round loop in the workflow function). That's wrong.

One nuance though: we did try the workflow-level pattern earlier (our V2). It used DurableAgent constructed in the workflow function with getWritable() at workflow level, and tool wrappers like:

async function durableWebSearch(input, ctx) {
  "use step";
  const mod = await import("@/lib/tools/web");
  return mod.webSearchTool.execute?.(input, ctx);
}

passed to DurableAgent's tools. This matches the template shape. We still measured 77 steps in 175s with gaps growing 961ms → 1524ms → 2135ms (the data in our April 25 follow-up). So your placement fix is real, but the gap growth at high N persists with the official template shape too, same finding as your #1315.

Going to try the template pattern again now that we have a smaller per-agent surface, see if the absolute number changes.

Thanks for the pointer.

[VaguelySerious]

Hi @shtefcs agent, it does still look like the performance you're seeing is a bit lower than what we're expecting for an agent running in IAD1. One of the best ways to debug this is to go to your run in Vercel observability, click on the "Logs" button, then for a later or slow /flow and /step route requests, inspect the external network requests made. If you paste them here, I'll help you take a look.

However, directionally it sounds like your agent has enough steps that you'll run into replay limitations either way. Those issues will be resolved once #1338 lands as a v5 beta release. We're trying hard to finalize this over the coming 1-2 weeks

[shtefcs]

Hi @shtefcs agent, it does still look like the performance you're seeing is a bit lower than what we're expecting for an agent running in IAD1. One of the best ways to debug this is to go to your run in Vercel observability, click on the "Logs" button, then for a later or slow /flow and /step route requests, inspect the external network requests made. If you paste them here, I'll help you take a look.

However, directionally it sounds like your agent has enough steps that you'll run into replay limitations either way. Those issues will be resolved once #1338 lands as a v5 beta release. We're trying hard to finalize this over the coming 1-2 weeks

Hey, I appreciate your time trying to help. I am honestly lost where to look for those requests.

I went inside of Observability tab to search for workflows and this is what I see.

If I go trough the Workflow dashboard for that specific Workflow, click on Logs, I get this:

[VaguelySerious]

If I go trough the Workflow dashboard for that specific Workflow, click on Logs, I get this:

@shtefcs Exactly on that page, if you click at any of the /flow or /step invocations, a sidebar panel will open with a list of external network requests. If you could send a screenshot of those request lists for one of the flow and step invocations each, maybe we'll find some anomalies that could be used for debugging

[shtefcs]

If I go trough the Workflow dashboard for that specific Workflow, click on Logs, I get this:

@shtefcs Exactly on that page, if you click at any of the /flow or /step invocations, a sidebar panel will open with a list of external network requests. If you could send a screenshot of those request lists for one of the flow and step invocations each, maybe we'll find some anomalies that could be used for debugging

It was easier to record a video than screenshot to show all requests, since its scrollable.

Please let me know if I can help further with debugging.

Worflow.debuging.Automatio.-.Cap.2026-04-29.at.14.40.08.mp4

Also this in case its useful: https://gist.github.com/shtefcs/cde97f7f591424ab287e722326433418

[shtefcs]

hey @VaguelySerious was my video/gist data useful for you?

I also saw that you merged #1338 to master. Is that mean we could/should expect the issue fixed? Should I go upgrade and test it?