AsyncDeleter deadlocks on full disk: crashes on boot with misleading :enoent, never reclaims trash

[erik-the-implementer]

Summary

Electric.AsyncDeleter cannot start when its trash directory's filesystem is full (or otherwise un-writable), and because AsyncDeleter is the process responsible for emptying the trash, a full disk becomes a self-reinforcing deadlock: the disk is full → AsyncDeleter crashes on boot → trash is never reclaimed → the disk stays full. On a multi-tenant deployment this takes the whole stack down and the affected tenant cannot recover on its own.

To make matters worse, the surfaced error is misleading — it reports :enoent on a path that the same function tried to create two lines earlier, so the real cause (ENOSPC) is hidden.

Root cause

In init/1 (packages/sync-service/lib/electric/async_deleter.ex):

trash_dir = trash_dir!(stack_id)
File.mkdir_p(trash_dir)          # return value is ignored

state = %__MODULE__{
  stack_id: stack_id,
  interval_ms: Keyword.get(opts, :cleanup_interval_ms, @default_cleanup_interval_ms),
  pending: File.ls!(trash_dir)   # raises here
}

File.mkdir_p/1's return value is discarded. When the filesystem is full, mkdir_p fails with {:error, :enospc} and the directory is never created, so the immediately following File.ls!(trash_dir) raises File.Error with reason: :enoent (because the dir doesn't exist) — masking the real ENOSPC.

Since AsyncDeleter is a child of the stack supervisor, this crash propagates as :failed_to_start_child and brings down the entire stack on boot.

Observed failure

Seen on a multi-tenant deployment when /var/electric hit 100% usage:

{:shutdown,
 {:failed_to_start_child, Electric.AsyncDeleter,
  {%File.Error{reason: :enoent,
     path: "/var/electric/<stack_id>/shapes/.electric_trash/<stack_id>",
     action: "list directory"},
   [{File, :ls!, 1, [file: ~c"lib/file.ex", line: 1700]},
    {Electric.AsyncDeleter, :init, 1, [file: ~c"lib/electric/async_deleter.ex", line: 84]},
    ...]}}}

df -h on the host at the time:

/dev/nvme1n1   20G   20G   20K 100% /var/electric

The stack went into a crash/cleanup/retry loop and the tenant could not initialize until the volume was freed manually.

Why this is especially bad

AsyncDeleter is the GC mechanism for batched deletes. The one condition under which it's most needed — a full disk with reclaimable trash sitting in the trash dir — is exactly the condition under which it refuses to start, so it can never reclaim that trash. Recovery requires out-of-band manual deletion of */shapes/.electric_trash/* (and any */trash/*).

Suggested fix

1. Don't ignore File.mkdir_p/1's result — handle the error and fail with the real reason (or log it) instead of letting a later File.ls! raise a misleading :enoent.
2. Don't hard-crash init on a transient/full-disk condition. Two reasonable options:
   • Treat a non-listable/absent trash dir as "nothing pending" (pending: []) and let the periodic cleanup retry, so the process can at least boot and start reclaiming once space frees up; or
   • Make the boot resilient (e.g. File.ls/1 with a fallback) so a full disk doesn't take down the whole stack.

Either way, the key goal is to break the deadlock so that AsyncDeleter can come up and do its job (freeing trash) precisely when the disk is full.

Environment

• Observed against sync-service at commit 12a6ee0 (the relevant init/1 code is unchanged across recent revisions).
• Multi-tenant embedded usage (each tenant is a stack).