Describe the issue

We saw this in our company development setup. We are using Hatchet 0.84 with long lived async Python workers.

Our workers stream LLM output to clients as many small SSE chunks. Each chunk calls ctx.aio_put_stream(chunk). This means PutStreamEvent runs at high frequency inside each task run, and across many concurrent task runs.

ctx.aio_put_stream currently sends each stream event by calling the synchronous put_stream path in a thread:

await asyncio.to_thread(self.put_stream, data)

put_stream then sends PutStreamEvent with the synchronous gRPC stub.

When many tasks stream at the same time, those calls can keep the shared thread pool busy. The runner also uses that thread pool to read assigned actions from its queue. When the pool is busy, new task runs can take a long time to start.

In our development setup, we saw warnings like this:

THE TIME TO START THE TASK RUN IS TOO LONG, THE EVENT LOOP MAY BE BLOCKED

Environment

• Hatchet: 0.84
• SDK: Python SDK
• Worker type: long lived async Python workers
• Workload: LLM streaming, many small chunks per run, many concurrent runs

Expected behavior

ctx.aio_put_stream should not use the shared thread pool for each stream event.

It should send PutStreamEvent with grpc.aio, the async gRPC client. This keeps stream sending async and avoids making task start depend on thread pool availability.

The stream index should still be assigned before sending the event, so stream chunks keep their existing order.

Code to reproduce, logs, or screenshots

After seeing this in our development setup, I tried to reproduce it with self contained local checks.

First, I ran a microbenchmark with a local fake gRPC server. It compares the current path with an async gRPC path:

await asyncio.to_thread(sync_stub.PutStreamEvent, request)
await aio_stub.PutStreamEvent(request)

The async gRPC path avoided filling the shared thread pool.

Second, I ran a local Hatchet engine worker test. It started 200 concurrent runs, and each run sent 175 stream chunks with ctx.aio_put_stream.

In that test:

• Current path: 185 task start warnings
• Async gRPC path: 0 task start warnings
• Stream chunks still arrived in order