Skip to content

[WORKFLOW SDK BUG] Activity payloads >4 MiB raise RESOURCE_EXHAUSTED on the workflow gRPC channel #1084

@tezizzm

Description

@tezizzm

Expected Behavior

Exporting DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES (added in #1024) should raise the
gRPC message-size limit for the workflow channel the same way it does for the Dapr
API channel, so workflows whose activity payloads (or accumulated history) exceed the
4 MiB default keep running instead of failing.

Actual Behavior

Long-running workflows whose activity results exceed gRPC's 4 MiB default fail with:

RESOURCE_EXHAUSTED: Sent message larger than max (X vs. 4194304)

The failure is on the worker stream between dapr-ext-workflow and the durabletask
service inside daprd. Once the orchestration history (input + accumulated activity
results) crosses 4 MiB, the worker can't ship the next event and the workflow stalls.

DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES does not help, because #1024 only threaded it
through the Dapr API channel (dapr.clients.grpc.client.DaprClient and the
dapr-ext-grpc callback server). It does not cover the workflow channel — these
sites construct their gRPC channels with no channel_options:

  • ext/dapr-ext-workflow/dapr/ext/workflow/workflow_runtime.pyworker.TaskHubGrpcWorker(...)
  • ext/dapr-ext-workflow/dapr/ext/workflow/dapr_workflow_client.pyclient.TaskHubGrpcClient(...)
  • ext/dapr-ext-workflow/dapr/ext/workflow/aio/dapr_workflow_client.pyaioclient.AsyncTaskHubGrpcClient(...)

So the durabletask channel falls back to the gRPC default regardless of the env var.

Secondary problem: #1024 only sets grpc.max_receive_message_length when reading the
env var. Workflow activity results flow in both directions (worker → daprd carries
activity output too), so the workflow fix needs to be symmetric on send + receive.

Steps to Reproduce the Problem

  1. Author a workflow with an activity that returns a payload approaching 4 MiB (or
    several activities whose accumulated results cross 4 MiB) — e.g. a long agentic loop
    where each step's tool output is stored in workflow history.
  2. Export DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES to a value above 4 MiB (e.g.
    33554432).
  3. Run the workflow under dapr run.
    RESOURCE_EXHAUSTED: Sent message larger than max (X vs. 4194304) — the raised limit
    is ignored because it never reaches the workflow channel.

Reproduced on every release from 1.17.0 through 1.18.0rc1 / workflow-v1.17.5; the
instantiation sites are identical across them.

Proposed fix

Mirror the #1024 pattern in ext/dapr-ext-workflow/: add an optional
max_grpc_message_length: Optional[int] = None kwarg to the three constructors
(WorkflowRuntime, sync DaprWorkflowClient, async DaprWorkflowClient), resolve
kwarg → settings.DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES → gRPC default, and pass
channel_options setting both grpc.max_send_message_length and
grpc.max_receive_message_length (symmetric). No new env var; the durabletask classes
already accept channel_options. Related: #1023 / #1024. Happy to open the PR — it's
already drafted.

Release Note

RELEASE NOTE: FIX Workflow gRPC channels now honor DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES (and a new max_grpc_message_length kwarg) for activity payloads larger than 4 MiB.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions