DotNext.Net.Cluster 6.3.0, .NET 10. RaftCluster (TCP) with a hand-assigned WriteAheadLog as AuditTrail over a SimpleStateMachine, one cluster per pod. Membership and restart leadership are already solved with a persistent configuration storage (per #207); this question is purely about WAL durability.
I expose a synchronous storage API, so a write must be durable before it returns, or at minimum survive a graceful (SIGTERM) restart. Completion is detected with a waiter completed inside SimpleStateMachine.ApplyAsync: I await ReplicateAsync(...) then await the waiter, so a write returns once the entry is applied.
- With
FlushInterval = TimeSpan.Zero (flush on every commit in the background) this is reliable: a node restarted from the same WAL recovers everything via InitializeAsync. But it fsyncs on every commit.
- With a positive
FlushInterval (to batch flushes for throughput), a graceful shutdown loses the un-snapshotted tail. A single committed entry written then immediately followed by DisposeAsync is not recovered after a restart, even when I call FlushAsync() before disposing the cluster and the log. A longer run (e.g. 200 writes) is recovered, because the periodic flusher and snapshots cover it.
Questions:
- Is there a supported way to make a committed write durable on return — awaiting the flush, not just the apply — so the caller knows it is on stable storage?
- With a hand-assigned
WriteAheadLog, what is the correct flush-on-graceful-shutdown sequence so a positive FlushInterval does not lose the most recent committed-but-unflushed entries? Is FlushAsync() the right call, and where must it run relative to RaftCluster.StopAsync() and disposal?
- Or is
FlushInterval = TimeSpan.Zero the only mode that guarantees a committed entry survives a restart, with any positive interval being best-effort (up to one interval lost on a stop)?
Goal: one cluster per pod that can batch flushes for throughput yet never lose a committed write across a graceful restart. Happy to provide a minimal repro.
DotNext.Net.Cluster 6.3.0, .NET 10.
RaftCluster(TCP) with a hand-assignedWriteAheadLogasAuditTrailover aSimpleStateMachine, one cluster per pod. Membership and restart leadership are already solved with a persistent configuration storage (per #207); this question is purely about WAL durability.I expose a synchronous storage API, so a write must be durable before it returns, or at minimum survive a graceful (SIGTERM) restart. Completion is detected with a waiter completed inside
SimpleStateMachine.ApplyAsync: Iawait ReplicateAsync(...)then await the waiter, so a write returns once the entry is applied.FlushInterval = TimeSpan.Zero(flush on every commit in the background) this is reliable: a node restarted from the same WAL recovers everything viaInitializeAsync. But it fsyncs on every commit.FlushInterval(to batch flushes for throughput), a graceful shutdown loses the un-snapshotted tail. A single committed entry written then immediately followed byDisposeAsyncis not recovered after a restart, even when I callFlushAsync()before disposing the cluster and the log. A longer run (e.g. 200 writes) is recovered, because the periodic flusher and snapshots cover it.Questions:
WriteAheadLog, what is the correct flush-on-graceful-shutdown sequence so a positiveFlushIntervaldoes not lose the most recent committed-but-unflushed entries? IsFlushAsync()the right call, and where must it run relative toRaftCluster.StopAsync()and disposal?FlushInterval = TimeSpan.Zerothe only mode that guarantees a committed entry survives a restart, with any positive interval being best-effort (up to one interval lost on a stop)?Goal: one cluster per pod that can batch flushes for throughput yet never lose a committed write across a graceful restart. Happy to provide a minimal repro.