Expected Behavior
Exporting DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES (added in #1024) should raise the
gRPC message-size limit for the workflow channel the same way it does for the Dapr
API channel, so workflows whose activity payloads (or accumulated history) exceed the
4 MiB default keep running instead of failing.
Actual Behavior
Long-running workflows whose activity results exceed gRPC's 4 MiB default fail with:
RESOURCE_EXHAUSTED: Sent message larger than max (X vs. 4194304)
The failure is on the worker stream between dapr-ext-workflow and the durabletask
service inside daprd. Once the orchestration history (input + accumulated activity
results) crosses 4 MiB, the worker can't ship the next event and the workflow stalls.
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES does not help, because #1024 only threaded it
through the Dapr API channel (dapr.clients.grpc.client.DaprClient and the
dapr-ext-grpc callback server). It does not cover the workflow channel — these
sites construct their gRPC channels with no channel_options:
ext/dapr-ext-workflow/dapr/ext/workflow/workflow_runtime.py → worker.TaskHubGrpcWorker(...)
ext/dapr-ext-workflow/dapr/ext/workflow/dapr_workflow_client.py → client.TaskHubGrpcClient(...)
ext/dapr-ext-workflow/dapr/ext/workflow/aio/dapr_workflow_client.py → aioclient.AsyncTaskHubGrpcClient(...)
So the durabletask channel falls back to the gRPC default regardless of the env var.
Secondary problem: #1024 only sets grpc.max_receive_message_length when reading the
env var. Workflow activity results flow in both directions (worker → daprd carries
activity output too), so the workflow fix needs to be symmetric on send + receive.
Steps to Reproduce the Problem
- Author a workflow with an activity that returns a payload approaching 4 MiB (or
several activities whose accumulated results cross 4 MiB) — e.g. a long agentic loop
where each step's tool output is stored in workflow history.
- Export
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES to a value above 4 MiB (e.g.
33554432).
- Run the workflow under
dapr run.
RESOURCE_EXHAUSTED: Sent message larger than max (X vs. 4194304) — the raised limit
is ignored because it never reaches the workflow channel.
Reproduced on every release from 1.17.0 through 1.18.0rc1 / workflow-v1.17.5; the
instantiation sites are identical across them.
Proposed fix
Mirror the #1024 pattern in ext/dapr-ext-workflow/: add an optional
max_grpc_message_length: Optional[int] = None kwarg to the three constructors
(WorkflowRuntime, sync DaprWorkflowClient, async DaprWorkflowClient), resolve
kwarg → settings.DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES → gRPC default, and pass
channel_options setting both grpc.max_send_message_length and
grpc.max_receive_message_length (symmetric). No new env var; the durabletask classes
already accept channel_options. Related: #1023 / #1024. Happy to open the PR — it's
already drafted.
Release Note
RELEASE NOTE: FIX Workflow gRPC channels now honor DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES (and a new max_grpc_message_length kwarg) for activity payloads larger than 4 MiB.
Expected Behavior
Exporting
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES(added in #1024) should raise thegRPC message-size limit for the workflow channel the same way it does for the Dapr
API channel, so workflows whose activity payloads (or accumulated history) exceed the
4 MiB default keep running instead of failing.
Actual Behavior
Long-running workflows whose activity results exceed gRPC's 4 MiB default fail with:
The failure is on the worker stream between
dapr-ext-workflowand the durabletaskservice inside daprd. Once the orchestration history (input + accumulated activity
results) crosses 4 MiB, the worker can't ship the next event and the workflow stalls.
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTESdoes not help, because #1024 only threaded itthrough the Dapr API channel (
dapr.clients.grpc.client.DaprClientand thedapr-ext-grpccallback server). It does not cover the workflow channel — thesesites construct their gRPC channels with no
channel_options:ext/dapr-ext-workflow/dapr/ext/workflow/workflow_runtime.py→worker.TaskHubGrpcWorker(...)ext/dapr-ext-workflow/dapr/ext/workflow/dapr_workflow_client.py→client.TaskHubGrpcClient(...)ext/dapr-ext-workflow/dapr/ext/workflow/aio/dapr_workflow_client.py→aioclient.AsyncTaskHubGrpcClient(...)So the durabletask channel falls back to the gRPC default regardless of the env var.
Secondary problem: #1024 only sets
grpc.max_receive_message_lengthwhen reading theenv var. Workflow activity results flow in both directions (worker → daprd carries
activity output too), so the workflow fix needs to be symmetric on send + receive.
Steps to Reproduce the Problem
several activities whose accumulated results cross 4 MiB) — e.g. a long agentic loop
where each step's tool output is stored in workflow history.
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTESto a value above 4 MiB (e.g.33554432).dapr run.RESOURCE_EXHAUSTED: Sent message larger than max (X vs. 4194304)— the raised limitis ignored because it never reaches the workflow channel.
Reproduced on every release from 1.17.0 through 1.18.0rc1 / workflow-v1.17.5; the
instantiation sites are identical across them.
Proposed fix
Mirror the #1024 pattern in
ext/dapr-ext-workflow/: add an optionalmax_grpc_message_length: Optional[int] = Nonekwarg to the three constructors(
WorkflowRuntime, syncDaprWorkflowClient, asyncDaprWorkflowClient), resolvekwarg → settings.DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES → gRPC default, and passchannel_optionssetting bothgrpc.max_send_message_lengthandgrpc.max_receive_message_length(symmetric). No new env var; the durabletask classesalready accept
channel_options. Related: #1023 / #1024. Happy to open the PR — it'salready drafted.
Release Note
RELEASE NOTE: FIX Workflow gRPC channels now honor
DAPR_GRPC_MAX_INBOUND_MESSAGE_SIZE_BYTES(and a newmax_grpc_message_lengthkwarg) for activity payloads larger than 4 MiB.