Add support for async workflow activities

[seherv]

Description

Workflow activities can now be async, and the runtime will automatically dispatch them to the event loop. Sync activities are still dispatched to the thread pool. The user-facing API remains exactly the same.

Basic benchmarks

Ran all of these on an M3 Pro (Python 3.13). The async activities take a 100KB array of random data, do asyncio.sleep(1) and return the zeroes unmodified.

1. Fan-out from one workflow

Activities	Async
100	1.03 s
1 000	1.24 s
10 000	11.03 s

2. Fan-out from multiple workflows × 3 activities

Workflows × activities	Async
100 × 3	1.09 s
1 000 × 3	3.60 s
10 000 × 3	39.51 s

3. Variable payload (100 workflows × 3 activities)

Payload	Async
1 KB	3.23 s
100 KB	3.64 s
1 MB	8.80 s

Issue reference

Closes #834, #897 and #975

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation

[seherv]

This nesting got hard to follow, I needed to refactor this file to understand the logic better.

[codecov]

Codecov Report

❌ Patch coverage is 87.95411% with 63 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.07%. Comparing base (bffb749) to head (3f80ed6).
⚠️ Report is 143 commits behind head on main.

Files with missing lines	Patch %	Lines
...-workflow/dapr/ext/workflow/_durabletask/worker.py	54.92%	32 Missing ⚠️
...workflow/tests/test_async_activity_registration.py	92.72%	12 Missing ⚠️
...kflow/dapr/ext/workflow/_durabletask/aio/client.py	50.00%	11 Missing ⚠️
...ext-workflow/dapr/ext/workflow/workflow_runtime.py	93.61%	3 Missing ⚠️
...ests/durabletask/test_async_dispatch_regression.py	93.18%	3 Missing ⚠️
...rkflow/tests/durabletask/test_activity_executor.py	93.33%	1 Missing ⚠️
.../tests/durabletask/test_activity_executor_async.py	97.61%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1053      +/-   ##
==========================================
- Coverage   86.63%   83.07%   -3.56%     
==========================================
  Files          84      151      +67     
  Lines        4473    15294   +10821     
==========================================
+ Hits         3875    12705    +8830     
- Misses        598     2589    +1991     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

ext/dapr-ext-workflow/dapr/ext/workflow/_durabletask/worker.py:678

• This adds an extra registry lookup (get_activity) and inspect.iscoroutinefunction check on the hot dispatch path, but _ActivityExecutor._resolve() will look up the activity again during execution. Consider plumbing activity_fn (and/or an is_async flag decided at registration time) through to the executor so each activity work item only does one lookup in total.

                                activity_handler,
                                work_item.activityRequest,
                                stub,
                                work_item.completionToken,
                            )

[Copilot]

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated 7 comments.

[seherv]

Improving benchmarks and regression tests before submitting for review again

[acroca]

Thanks for working on this! I agree with the general direction, async activity support is something we've wanted for a while (#834, #897, #975), and keeping the user-facing API unchanged is the right approach.

I'd like to share my opinion on the benchmarking side, though. I think the sync-vs-async comparison is valuable in the context of this PR, to justify the why behind the change, but not as something we keep and maintain in the codebase. Concretely:

• The benchmark suite adds ~1,100 lines (benchmarks/, _bench_harness.py), which is more code than the feature itself, and the comparison is essentially a one-off: once async dispatch is merged, it has served its purpose.
• _bench_harness.py sits inside dapr/ext/workflow/, so it would ship to every user as part of the package.
• The regression tests in test_async_dispatch_regression.py mostly assert sync-vs-async ratios, which again is the one-off comparison, and they depend on the harness.

My suggestion: run the benchmarks locally and paste the results + methodology into the PR description. Then the harness, the benchmarks/ directory, and the ratio-based tests can be dropped from the PR.

That said, I do think we should keep one simple throughput regression test so the async path can't silently regress, something like running 1000 activities that each take 1 second and asserting the batch completes in well under 10 seconds (ideally it's ~1s, so a 10x bound leaves plenty of headroom for CI noise). That single assertion catches the failure mode we actually care about, async activities accidentally getting serialized, without needing the full harness, and it can be written directly against the runtime in a few dozen lines.

Thoughts?

[sicoyle]

few comments so far. I'm almost half way through the files/changes :)

[seherv]

Your comments have convinced me that the repo is not the right place for benchmark code. It was very useful to validate everything myself, but I think our users only need to see a summary of the results and their applications are the real stress tests anyway.

I'll remove the bench code and simplify the regression tests to validate only the async side of things.

[sicoyle]

went through all the files - few comments/questions pls

[sicoyle]

do we need this comment?

[seherv]

Honestly I like using AGENTS.md as a development log of sorts to not waste time rediscovering old issues and rereading the code base. That's just a personal preference though, not sure how much we should let this file grow

[sicoyle]

what else is left to continue iterating on the async work?

[seherv]

The worker still consumes gRPC synchronously in a separate thread. We should use grpc.aio instead and replace lots of code that use that thread with simple awaits.

Also the workflow runtime is still sync-first and has its own event loop, and I've had to apply a few patches to make everything run on the same event loop as the user's application. Being able to do async with WorkflowRuntime() would avoid having to use those patches.

I'll create issues for both 😄