Skip to content

Commit a36b8ec

Browse files
committed
[Observability] Log interrupted processing tasks on unexpected worker death
When a worker drops off the cluster unexpectedly (e.g., due to an OOM kill), the scheduler tracks the processing_keys but previously did not log them to the console. This change surfaces exactly which tasks were interrupted, significantly improving debugging provenance for cluster hangs and memory crashes.
1 parent cf508b9 commit a36b8ec

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

distributed/scheduler.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5714,6 +5714,12 @@ async def remove_worker(
57145714
f"Removing worker {ws.address!r} caused the cluster to lose scattered "
57155715
f"data, which can't be recovered: {lost_keys} ({stimulus_id=})"
57165716
)
5717+
if not expected and processing_keys:
5718+
logger.warning(
5719+
f"Worker {ws.address!r} dropped unexpectedly. "
5720+
f"Interrupting {len(processing_keys)} processing tasks: "
5721+
f"{processing_keys} ({stimulus_id=})"
5722+
)
57175723

57185724
event_msg = {
57195725
"action": "remove-worker",

0 commit comments

Comments
 (0)