fix: clean old txhashset snapshot zips on every zip_read#3888
Conversation
Keep only the snapshot for the header currently being served and remove other txhashset_snapshot_*.zip files whenever a peer requests a zip. Previously cleanup ran only when creating a new archive and retained 24h of multi-GB files, which could fill disks on low-end nodes. Addresses mimblewimble#3647.
| Err(_) => continue, | ||
| }; | ||
| // Match `txhashset_snapshot_<hash>.zip` only. | ||
| if !name.starts_with(&prefix) || !name.ends_with(".zip") { |
There was a problem hiding this comment.
The comment says .zip, but this matches any text between the prefix and .zip.
Generated snapshot names use a 12-character hex hash. Could we restrict cleanup to that exact format?
| File::create(&unrelated).unwrap(); | ||
| File::create(¬_zip).unwrap(); | ||
|
|
||
| let removed = txhashset::cleanup_old_txhashset_zips(db_root, &keep); |
There was a problem hiding this comment.
Could this also exercise zip_read() with the current archive already present and one stale archive beside it? The original regression was specifically that the reuse path skipped cleanup, while this test only covers the helper directly.
| } | ||
| match fs::remove_file(&path) { | ||
| Ok(()) => { | ||
| debug!("cleanup_old_txhashset_zips: removed {:?}", path); |
There was a problem hiding this comment.
zip_read() already logs the total number removed. Could this per file message be trace! or be dropped to avoid duplicate debug output during a rollover?
| } | ||
| } | ||
|
|
||
| /// Remove other `txhashset_snapshot_*.zip` files in `data_dir`, keeping only |
There was a problem hiding this comment.
Could we shorten this? The key detail here is that cleanup runs before both reuse and creation
2 participants
Summary
txhashset_snapshot_*.zipfiles (Clean old txhashset_snapshot_*.zip periodically #3647).zip_read(reuse and create), remove other snapshot zips and keep only the one for the requested header.cleanup_old_txhashset_zips+ unit test.Test plan
cargo test -p grin_chain --test test_txhashset -- --test-threads=1(2 passed)