Speed up chain compaction, harden PIBD validation, crash-safe compact#3906
Open
iho wants to merge 7 commits into
Open
Speed up chain compaction, harden PIBD validation, crash-safe compact#3906iho wants to merge 7 commits into
iho wants to merge 7 commits into
Conversation
- write_tmp_pruned: prune_pos is ordered, so compare against the head of the remaining slice instead of scanning it for every element. Rewriting the PMMR hash/data files was O(elements x prune_count), which could take minutes on low-end hardware after a large prune set accumulated. - init_output_pos_index: binary search for the first header that can contain the first missing output and stop once the last missing entry is indexed, instead of walking every header from genesis to head on each compaction. Helps with mimblewimble#3872.
Segment root validation only checked hashes that participate in reconstructing the segment root. Apply, however, writes every hash in the segment into the local PMMR. A peer could therefore pass a segment with a valid root proof while poisoning intermediate hashes; the corruption only showed up at full state validation as a bad root, with no peer attribution. After the merkle proof succeeds, verify every carried hash is bound to the proof-validated root: either consumed during reconstruction, or a child of an already bound parent that hashes to that parent. Legitimate dense segments from uncompacted peers (pre-mimblewimble#3820 style) still pass when their extra hashes are consistent; only inconsistent or unbound hashes are rejected early as UnverifiableHash. Adds store segment tests covering tampered intermediate hashes and the uncompacted-peer dense format.
iho
changed the title
A hard kill between replacing hash/data files and flushing the prune list left compacted files paired with a stale prune list, producing wrong shifts and a bad root on restart. Compaction now: 1. Writes pruned hash/data temps and staged prune/leaf bitmaps 2. Fsyncs those temps, then writes a compact journal marker 3. Only then renames staging files into place and clears the journal On PMMRBackend::new, a present journal rolls renames forward; temps without a journal are discarded as abandoned prep. Variable-size size files are rebuilt on re-open when inconsistent with data, matching the existing open path. Adds tests for successful cleanup, roll-forward recovery, and orphan temp discard.
iho
changed the title
Reproduce slow compaction and hard-kill mid-compact (issue mimblewimble#3872): - compact_crash_stress example: prepare large pruned PMMR, compact with timing, verify root, kill-test that SIGKILLs a child paused at a phase - Test hooks in check_compact (env-gated, no-op in production): ready file + pause at hash_tmp / data_tmp / journal / applied - etc/qemu-compact-stress/run.sh: native, qemu-user (TCG), or Docker linux/amd64 with 0.5 CPU / 512MB (QEMU on arm hosts) - Manual CI workflow for optional long runs
Testnet is usually the easier real-world repro (and often heavier than mainnet even on a desktop): high spend churn and first compact after PIBD rewrites almost the entire hash/data files. Add prune_pct to prepare/kill-test, --testnet-like runner preset (large leaf count + ~92% pruned), and first vs second compact timing so the post-PIBD vs incremental gap is visible.
- Docker --weak: use host platform (not forced amd64 under QEMU), export /usr/local/cargo/bin, source cargo env, fail clearly if cargo missing - Progress logs during prepare/compact so --testnet-like is not silent - Clarify that harness modes never need a manual kill; only real grin --testnet does for optional full-node recovery experiments
Owner JSON-RPC helper to start a no-TUI testnet node, wait for sync, force compact_chain with wall timing, and optional mid-compact SIGKILL for journal recovery. Uses /v2/owner (v1 routes are not registered).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related fixes for low-end hardware reliability and post-PIBD stability, plus a stress harness for kill-during-compact.
Compaction speed (#3872)
Avoid quadratic scan when rewriting PMMR files.
write_tmp_prunedcheckedprune_pos.containsfor every element against the whole remaining prune list. Both are sorted, so this was O(elements × prune_count). Now only the head of the remaining slice is compared: O(elements + prune_count).Bound the header walk in
init_output_pos_index. When any output was missing from the index, the rebuild walked every height from genesis to head and kept walking after all missing outputs were placed. It now binary-searches for the first relevant header and breaks once the last missing entry is indexed.PIBD segment hash validation (PR #3820 fallout)
Segment root validation only checked hashes used to rebuild the segment root, while apply writes every carried hash into the local PMMR. A peer could pass a valid root proof with poisoned intermediate hashes; failure only appeared at full state validation as a bad root (PIBD restart loop).
After the merkle proof succeeds, every carried hash must be bound to that root (consumed during reconstruction, or a child of a bound parent that hashes correctly). Dense segments from uncompacted peers still pass when consistent; unbound/wrong hashes fail early as
UnverifiableHash.Crash-safe compaction
A hard kill between replacing hash/data and flushing the prune list left compacted files with a stale prune list → wrong shifts → bad root on restart. Compaction now stages all new files, fsyncs, writes a journal marker, then renames. On open, a present journal rolls renames forward; temps without a journal are discarded as abandoned prep.
QEMU / weak-VPS stress harness
etc/qemu-compact-stress/+store/examples/compact_crash_stress.rs:SIGKILLthe compact process athash_tmp/data_tmp/journaland reopenqemu-user(TCG), Dockerlinux/amd64with 0.5 CPU / 512MB (QEMU on arm hosts)check_compact(no-op unless set)Manual workflow:
.github/workflows/compact-stress.yaml(workflow_dispatch).Testing
cargo test -p grin_store -p grin_core -p grin_chainsegment_carried_hash_verificationcompact_clears_journal,compact_journal_recovery_roll_forward,compact_orphan_temps_discardedspend_in_fork_and_compact./etc/qemu-compact-stress/run.sh --all-phases(native smoke)