Skip to content

Commit ea93c78

Browse files
committed
Fix pdb / breakpoint() hang in workflow code (#1104)
When debug_mode=True (or TEMPORAL_DEBUG=1), breakpoint() inside workflow code now opens an interactive pdb prompt -- including from a sandboxed workflow run under pytest. Four pieces: - Inline dispatch on the asyncio main thread (via loop.call_soon to avoid nesting inside the dispatch task's __step() and tripping Python 3.14's task-entry validation). - breakpoint removed from the sandbox's invalid builtins so the call reaches the worker hook. Nothing else is relaxed. - A Pdb subclass that lands at the workflow's own frame, suspends sandbox checks during each REPL interaction, and overrides q/Ctrl-D to continue the workflow instead of failing it with BdbQuit. - A defensive sys.breakpointhook that raises a clear RuntimeError when breakpoint() is called from a workflow worker thread without debug_mode, replacing the previous silent hang. When debug_mode is not set, the worker's dispatch and sandbox config are unchanged. Adds a README subsection on debugging workflows and five tests at tests/worker/test_breakpoint_hang.py. Verified on Python 3.13 and 3.14. Closes #1104.
1 parent 7ea54e6 commit ea93c78

3 files changed

Lines changed: 523 additions & 45 deletions

File tree

README.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ informal introduction to the features and their implementation.
8282
- [Customizing the Sandbox](#customizing-the-sandbox)
8383
- [Passthrough Modules](#passthrough-modules)
8484
- [Invalid Module Members](#invalid-module-members)
85+
- [Debugging Workflows with `breakpoint()` / `pdb`](#debugging-workflows-with-breakpoint--pdb)
8586
- [Known Sandbox Issues](#known-sandbox-issues)
8687
- [Global Import/Builtins](#global-importbuiltins)
8788
- [Sandbox is not Secure](#sandbox-is-not-secure)
@@ -1241,6 +1242,79 @@ my_worker = Worker(..., workflow_runner=SandboxedWorkflowRunner(restrictions=my_
12411242

12421243
See the API for more details on exact fields and their meaning.
12431244

1245+
##### Debugging Workflows with `breakpoint()` / `pdb`
1246+
1247+
Setting `debug_mode=True` on the `Worker` (or `TEMPORAL_DEBUG=1` in the environment) routes workflow activations
1248+
onto the asyncio main thread instead of a worker thread pool. This lets `breakpoint()` and `pdb.set_trace()`
1249+
inside workflow code open an interactive REPL — without it, pdb hangs because its `input()` call would run on a
1250+
thread that does not own the controlling TTY.
1251+
1252+
A minimal runnable example:
1253+
1254+
```python
1255+
import asyncio
1256+
from datetime import timedelta
1257+
1258+
from temporalio import workflow
1259+
from temporalio.client import Client
1260+
from temporalio.worker import Worker
1261+
1262+
1263+
@workflow.defn
1264+
class DebugMeWorkflow:
1265+
@workflow.run
1266+
async def run(self) -> str:
1267+
x = 42
1268+
breakpoint() # interactive pdb prompt opens at this line
1269+
return f"x was {x}"
1270+
1271+
1272+
async def main() -> None:
1273+
client = await Client.connect("localhost:7233")
1274+
async with Worker(
1275+
client,
1276+
task_queue="debug-me",
1277+
workflows=[DebugMeWorkflow],
1278+
debug_mode=True,
1279+
):
1280+
result = await client.execute_workflow(
1281+
DebugMeWorkflow.run,
1282+
id="debug-me-wf",
1283+
task_queue="debug-me",
1284+
task_timeout=timedelta(minutes=10), # see caveat below
1285+
)
1286+
print(result)
1287+
1288+
1289+
if __name__ == "__main__":
1290+
asyncio.run(main())
1291+
```
1292+
1293+
Run with `python debug_me.py`, or under pytest with `pytest -s` (the `-s` flag disables pytest's stdin
1294+
capture). At the `(Pdb)` prompt you'll land at the line where `breakpoint()` was called, with workflow
1295+
locals in scope. Try `p x`, `n`, `c`, `q`.
1296+
1297+
**Quitting cleanly.** Typing `q` or hitting Ctrl-D continues the workflow rather than raising `BdbQuit`
1298+
(which would fail the workflow task). To genuinely abort, kill the outer process with Ctrl-C.
1299+
1300+
Two caveats when pausing at a breakpoint inside a workflow:
1301+
1302+
1. **Workflow task timeout.** Temporal expires a workflow task after ~10 seconds by default. If you sit at the
1303+
`(Pdb)` prompt longer than that, the server reassigns the task and your workflow replays from the start when
1304+
you continue — re-hitting the breakpoint. Pass `task_timeout=timedelta(minutes=N)` to `execute_workflow` /
1305+
`start_workflow` to give yourself debugging headroom:
1306+
1307+
```python
1308+
await client.execute_workflow(MyWorkflow.run, ..., task_timeout=timedelta(minutes=10))
1309+
```
1310+
1311+
2. **Deterministic replay.** Workflows are deterministic and replay from history; any wall-clock pause violates
1312+
that contract. For post-mortem debugging without these caveats, use the [Replayer](#replayer) on a recorded
1313+
history instead of live debugging.
1314+
1315+
A `breakpoint()` call from workflow code without `debug_mode` enabled raises a `RuntimeError` with a pointer to
1316+
this section, so the failure mode is loud rather than a silent hang.
1317+
12441318
##### Known Sandbox Issues
12451319

12461320
Below are known sandbox issues. As the sandbox is developed and matures, some may be resolved.

0 commit comments

Comments
 (0)