fix: gracefully handle out of disk space failures (#3425)#3908
Open
iho wants to merge 1 commit into
Open
Conversation
When disk fills during chain writes, avoid committing a DB tip that points at incomplete MMR state, and surface a clear DiskFull error instead of opaque I/O failures. - Detect ENOSPC / StorageFull via is_out_of_disk_space helpers - Harden save_via_temp_file: fsync parent dir, clean up temp on error, leave original file intact if the write fails - AppendOnlyFile::flush keeps buffer/rewind backup until write+sync succeed so discard() can roll back after a failed flush - Sync PMMR backends before committing the DB batch on extension commit; discard extension if sync fails - Add store::Error::DiskFull and chain::Error::DiskFull with operator-facing guidance (free space, avoid --clean unless needed) Tests: store disk_full suite + existing store/chain regression tests.
ardocrat
reviewed
Jul 11, 2026
| // heed/lmdb may surface ENOSPC as a map full / I/O error string. | ||
| let lower = msg.to_lowercase(); | ||
| if lower.contains("map full") | ||
| || lower.contains("mdb_map_full") |
Contributor
There was a problem hiding this comment.
map full means it should be resized, not necessary it can be full disk reason, need to check disk space on specific error instead of parsing its messages
ardocrat
reviewed
Jul 11, 2026
| return true; | ||
| } | ||
| let msg = err.to_string().to_lowercase(); | ||
| msg.contains("no space left") || msg.contains("disk full") || msg.contains("not enough space") |
Contributor
There was a problem hiding this comment.
We can no handle all possible messages here, so need to check disc space exactly to decide if it was a reason
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #3425: out-of-disk-space during chain writes could leave the node in a state where restart failed with
failed to find head hashand operators resorted togrin --clean.Changes
store::is_out_of_disk_space/map_io_err) forErrorKind::StorageFulland UnixENOSPC.save_via_temp_file): fsync temp file (existing), fsync parent dir after rename, remove temp on failure so the original file is preserved.AppendOnlyFile::flush: do not clear the rewind backup / buffer until write+sync succeed; log a clear error if the write fails.store::Error::DiskFullandchain::Error::DiskFullwith guidance to free space and avoid wiping chain data unless recovery fails.Related: #3352 (fsync/leaf_set durability notes), unmerged #3266 (checkpoint idea — not implemented here).
Testing
cargo test -p grin_store(includes newdisk_fulltests)cargo test -p grin_chain --libcargo test -p grin_chain --test mine_simple_chainCompatibility
Closes #3425