feat: SPV in DET#90
Closed
pauldelucia wants to merge 235 commits into
Closed
Conversation
* split out key wallet functionality from rust-dash-core * work on splitting out the key wallet * more work * more work * more work * commit * commit * Delete NETWORK_HANDLING_FIXES.md * more work * more * more
* feat: add chainlock to inv * add chainlock / islock stuff; request chainlocks we see in inv * bloom no work * compact filters * dash-spv crate * feat: implement BIP158 filter matching and comprehensive SPV monitoring - Replace placeholder filter_matches_scripts with real BIP158 GCS implementation - Add comprehensive integration test framework with Docker support - Implement network monitoring for ChainLocks and InstantLocks with signature verification - Enhance masternode engine with proper block header feeding and state management - Add watch item persistence and improved transaction discovery - Increase filter search range from 50 to 1000 blocks for better coverage - Enable X11 hashing and BLS signature verification in dependencies - Add proper error handling and logging throughout the sync pipeline 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * "Add Improved Network Message Handling and Block Processing - Ping and Pong Handling: Added mechanisms to send periodic pings and handle incoming pings/pongs, enhancing network reliability. - Block Processing: Implemented functions to process new block hashes immediately and manage block headers and filters effectively. - Filter Headers and Filters: Added logic to handle CFHeaders and CFilter network messages and check them against watch items. - Logging Enhancements: Improved logging for better traceability, including filter matches and network message receipt. - Error Handling: Strengthened error handling for network messages and block processing errors. This update enhances network responsiveness and block synchronization, enabling better SPV client performance." * "fix: Update Regtest network constants and genesis block" * feat: add batch header loading and reverse index to storage - Add get_header_height_by_hash() method for O(1) hash-to-height lookups - Add get_headers_batch() method for efficient bulk header loading - Implement reverse index in both disk and memory storage - Add as_any_mut() trait for storage downcasting - Leverage existing segmented file structure for batch operations These optimizations enable efficient masternode sync by reducing individual storage reads from millions to thousands. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * perf: optimize masternode sync header feeding by 1000x Replace inefficient strategy that fed ALL 2.2+ million headers individually with selective feeding of only required headers: - Use reverse index for O(1) hash-to-height lookups - Feed only target, base, and quorum block hashes - Use batch loading for recent header ranges (~1000 headers) - Eliminate "Feeding 2278524 block headers" bottleneck Performance improvement: ~2.2M individual reads → ~1K batch operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add modern terminal UI with real-time status display Implement a status bar showing sync progress at the bottom of the terminal: - Headers count and filter headers count - Latest ChainLock height and peer count - Network name (Dash/Testnet/Regtest) - Updates every 100ms without interfering with log output Features: - Uses crossterm for cross-platform terminal control - RAII cleanup with TerminalGuard - Logs stream normally above persistent status bar - Optional --no-terminal-ui flag to disable 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: integrate terminal UI with SPV client Add comprehensive terminal UI integration to the SPV client: - enable_terminal_ui() and get_terminal_ui() methods - Real-time status updates after network connections - Status updates after header processing and ChainLock events - update_status_display() method with storage data integration - Proper shutdown sequence ensuring storage persistence - Network configuration getter for UI display The client now displays live sync progress including header counts from storage, peer connections, and ChainLock heights. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add terminal UI support to CLI and improve logging CLI improvements: - Add --no-terminal-ui flag to disable status bar - Proper terminal UI initialization timing - Network name display integration - Remove unused Arc import Logging improvements: - Fix log level handling in init_logging() - Improve tracing-subscriber configuration - Remove thread IDs for cleaner output The CLI now provides a modern terminal experience with optional real-time status display alongside streaming logs. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: minor improvements to sync modules Small enhancements to header and filter sync: - Improve logging and error handling - Better progress reporting during sync operations - Consistent formatting across sync modules These changes support the terminal UI integration and provide better visibility into sync progress. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * remove redundant chainlock storage * adjust how blocks are fed to masternode engine to avoid redundant block submissions * reduce verbose logging * adds batch of tests, some that should've been commited earlier * p2p: connect to multiple nodes, multiple threads * fixup network constants * fix: correct genesis_block static values for mainnet * feat: implement UTXO tracking and wallet functionality * refactor: resolve cargo check warnings * feat: improve network architecture and multi-peer management - Add thread-safe Mutex wrapper around BufReader to prevent race conditions - Implement sticky peer selection for sync consistency during operations - Increase peer count limits (2-5 peers) for better network resilience - Add single-peer message routing for sync operations requiring consistency - Improve connection error handling and peer disconnection detection - Add timeout-based message receiving to prevent indefinite blocking - Reduce log verbosity for common sync messages to improve readability 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: enhance sync system with robust coordination and recovery - Add comprehensive sync state management with timeout detection - Implement overlapping header handling for improved sync reliability - Add coordinated message routing between sync managers and main client - Enhance filter sync with batch processing and progress tracking - Add sync timeout detection and recovery mechanisms - Improve masternode sync coordination and state management - Add detailed sync progress logging and error handling - Implement proper chain validation during sync operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: implement centralized message routing and coordination - Add centralized network message handling to prevent race conditions - Implement message routing between monitoring loop and sync operations - Add comprehensive sync timeout detection and recovery mechanisms - Enhance filter sync coordination with monitoring loop management - Add detailed documentation for network message architecture - Improve sync progress reporting and status updates - Reduce debug noise from transaction input checking - Add sync_and_check_filters_with_monitoring method for better coordination 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * test: add comprehensive sync testing and verification utilities - Add filter header verification test for chain validation - Enhance multi-peer test with better error handling and timeouts - Add checksum utility for data integrity verification - Improve consensus encoding with better error messages - Add test infrastructure for sync coordination scenarios 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct sync state management to prevent premature completion Remove premature finish_sync() calls that were marking header and filter header synchronization as complete immediately after starting. The sync should only be marked as finished when handle_*_message() returns false, indicating actual sync completion. - Remove finish_sync() calls after starting header sync - Remove finish_sync() calls after starting filter header sync - Add sync_state_mut() accessor for proper state management - Add proper sync completion in client message handlers This fixes the issue where sync would complete with 0 headers because the sync state was marked as finished before any headers were processed. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: add proper sync state completion handling in client Add logic to properly finish sync state when header and filter header synchronization actually completes, rather than when it starts. - Call finish_sync() when handle_headers_message() returns false - Call finish_sync() when handle_cfheaders_message() returns false - Add debug logging to track message processing flow This ensures sync state accurately reflects the actual synchronization progress and completion status. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * debug: add comprehensive logging to header sync manager Add detailed debug and info logging to track header synchronization flow and help diagnose sync issues: - Log when handle_headers_message() is called with header count - Log sync state (syncing_headers flag) for debugging - Log when headers sync is ignored due to inactive state - Log when empty headers response indicates sync completion - Log when syncing_headers flag is set during sync start This logging helps identify whether sync issues are due to: - Messages not reaching the handler - Incorrect sync state management - Empty responses from peers - Premature sync completion 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * improve: enhance network error handling for checksum failures Add graceful handling of checksum validation failures to prevent connection drops when corrupted messages are received: - Catch InvalidChecksum errors and log them as warnings - Skip corrupted messages instead of failing the entire connection - Add special detection for all-zeros checksum corruption - Return None (no message) instead of connection error This prevents the connection from being dropped when individual messages are corrupted, allowing sync to continue with subsequent valid messages. Particularly important for handling version message corruption during handshake. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add comprehensive project documentation Add CLAUDE.md with detailed project overview, architecture, and development guidance covering: - Project overview and architecture description - Core modules and design patterns - Development commands (build, test, run) - Key concepts (sync coordination, storage, validation) - Testing strategy and organization - Development workflow and best practices - Current project status and roadmap This documentation provides essential context for understanding the codebase structure and development practices. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve header sync state management issues Key issues fixed: - Removed duplicate sync state tracking between SyncState and HeaderSyncManager - Fixed race condition where HeaderSyncManager.syncing_headers could get out of sync - Added is_syncing() method to HeaderSyncManager for proper state checking - Removed premature finish_sync() calls in client message handling - Simplified state management to use HeaderSyncManager as the single source of truth The header sync now properly: 1. Sets syncing_headers=true when starting sync 2. Processes incoming headers when syncing_headers=true 3. Clears syncing_headers=false when empty headers received (sync complete) 4. Avoids dual state management that was causing race conditions 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve critical race condition in header sync timing The headers sync was completing immediately with 0 headers because of a race condition where: 1. sync_to_tip() sends getheaders requests and returns immediately 2. "Sync completed\!" was logged before monitoring loop started 3. Headers responses arrived before monitoring loop was active to process them Fixed by: - Starting monitoring loop concurrently with sync operations (not after) - Adding 100ms delay to ensure monitoring loop initializes before sync starts - Clarifying log messages to indicate sync is asynchronous - Headers will now be properly received and processed by active monitoring loop 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve race condition by coordinating sync with monitoring loop The root cause was that sync_to_tip() sent network requests before monitor_network() started listening, causing headers responses to be dropped. Solution: - Modified monitor_network() to initiate sync requests after it starts listening - Added prepare_sync() method to set up sync state without sending requests - Changed sync_to_tip() to only prepare state, not send network requests - Ensures monitoring loop is active before any network requests are sent This eliminates the race condition and ensures headers responses are properly received and processed by the monitoring loop. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: implement interleaved header and filter header sync This commit enables proper interleaved synchronization where filter headers are automatically requested as soon as new block headers are received and stored. Key changes: - Modified handle_headers_message() in SyncManager to automatically trigger filter header requests when new headers are received and filters are enabled - Added proper filter header sync state management to coordinate with the existing header sync process - Enhanced CFHeaders message processing with better logging and error handling - Added is_syncing_filter_headers() method to FilterSyncManager for state checking - Updated client startup to initialize filter header sync when needed This fixes the issue where filter headers were not being downloaded despite being enabled, ensuring the sync process follows the proper pattern: 1. Request headers 2. Receive headers and store them 3. Immediately request filter headers for the new blocks 4. Receive and process filter headers 5. Repeat until sync is complete 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: ensure filter header requests are sent even when sync is active The previous implementation would skip sending filter header requests when filter header sync was already active, assuming it would handle them automatically. However, this caused filter headers to never be requested for new block ranges. This fix ensures that filter header requests are always sent for new block ranges, regardless of the current sync state, while still maintaining proper state management. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: remove hardcoded 10000 height limit in filter header sync Replace hardcoded search limits with dynamic header tip height lookups in: - store_filter_headers() method - download_filter_header_for_block() method - download_and_check_filter() method This fixes filter header sync failures when blockchain height exceeds 10,000 blocks, where the system could verify filter headers but then fail to store them due to the hardcoded search limit. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * improve: enhance network monitoring resilience during peer disconnections Add intelligent reconnection handling in client monitoring loop: - Detect when all peers disconnect during monitoring - Wait up to 5 seconds for potential reconnection - Resume monitoring gracefully when peers reconnect - Provide clear logging of connection state changes This prevents monitoring loop crashes when network connectivity is unstable. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: improve storage concurrency safety in header storage Replace individual lock acquisitions with atomic operations: - Acquire write locks for cached_tip_height and header_hash_index together - Update both atomically to prevent race conditions - Release locks before background save operations to avoid deadlocks This prevents inconsistencies between tip height cache and reverse index during concurrent header storage operations. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * improve: add adaptive timeout handling for header sync Implement peer-aware timeout handling: - Use 5-second timeout when no peers are connected (faster failure detection) - Use 10-second timeout when peers are available (normal operation) - Reset sync state when no peers available to allow clean restart - Provide clear error messaging for connection failures This improves sync reliability when network connectivity is intermittent. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: simplify filter header sync coordination logic Remove redundant manual filter header requests: - Trust FilterSyncManager's automatic batch progression - Remove fallback manual requests that could cause duplicates - Rely on handle_cfheaders_message to request next batches - Simplify sync coordination between headers and filter headers This reduces complexity and prevents potential race conditions in filter header synchronization. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: auto-trigger masternode sync after header sync completion When header synchronization completes in the monitoring loop, automatically start masternode synchronization if it's enabled. This fixes the issue where ChainLock verification would fail with "NoMasternodeLists" error because masternode sync was only triggered during manual sync_all() calls, not during continuous monitoring. The fix adds automatic coordination between header sync completion and masternode sync startup, ensuring the masternode list is populated for ChainLock signature verification. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * improve: enhance dash-spv CLAUDE.md with debugging and implementation details - Add specific test execution commands for debugging async code - Include storage architecture details (segmented storage, file organization) - Add async architecture patterns (trait objects, message passing, state machines) - Provide debugging and troubleshooting guidance with common commands - Document debug data locations and network debugging tips 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve CFilter message processing and add comprehensive debug logging - Enhanced CFilter message handling in handle_network_message to properly process received filters - Added comprehensive debug logging to trace filter sync coordination and height lookup - Improved error handling for height lookup failures with fallback to regular filter processing - Fixed issue where filters were being downloaded but not actually processed due to height lookup failures - Added automatic filter downloading trigger after filter header sync completion - Made system more robust to timing issues by processing filters as regular checks when sync coordination fails This resolves the issue where "we are requesting & downloading cfilters now, but not actually processing them" 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve storage layer race condition in segmented eviction This commit fixes a critical race condition in the DiskStorageManager where get_tip_height() could return heights for which get_header() would fail. Issue: The "Next batch stop header not found" error occurred when: 1. get_tip_height() returned a height from cached tip 2. get_header() failed because the segment was evicted to background worker 3. The background save was still in progress asynchronously Solution: Make segment eviction synchronous when dirty segments need saving: - evict_oldest_segment() now calls save_segment_to_disk() directly - evict_oldest_filter_segment() now calls save_filter_segment_to_disk() directly - Ensures data consistency between cached tip heights and retrievable data Root cause: Async segment saving created a gap where tip height was updated immediately but underlying data might not be retrievable due to background persistence timing. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve UTXO serialization and balance calculation issues - Fix UTXO serialization format mismatch by switching from bincode to JSON - Resolve Amount subtraction panic by using signed integer arithmetic for balance changes - Add comprehensive balance tracking with real-time updates and reporting - Implement AddressBalance struct with custom serialization for dashcore::Amount - Add get_address_balance() and get_all_balances() methods for wallet functionality - Track both UTXO creation (outputs) and spending (inputs) with proper balance updates - Clear corrupted UTXO data that was stored in incompatible bincode format 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve filter header sync storage consistency issue - Add fallback logic when calculated stop header height is not found in storage - Implement graceful degradation by falling back to tip header when intermediate heights are missing due to segmented storage gaps or cached tip inconsistencies - Apply fix to all three locations: main sync, timeout recovery, and initial sync - Add detailed debug logging to identify storage inconsistency issues - Prevents "Next batch stop header not found" errors during filter sync This resolves the issue where filter sync would fail when the cached tip height doesn't match actual available headers in segmented storage, particularly at height boundaries between segments. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: implement exclusive peer connection mode and adjust peer discovery logic * fix: resolve WatchItem deserialization issue with earliest_height field Fixes SPV client startup failure with error: 'Failed to deserialize watch items: invalid type: null, expected u32' The issue was in WatchItem deserialization where Option<u32> earliest_height was being double-wrapped when handling null values. Changed from: earliest_height = Some(map.next_value()?) to: earliest_height = map.next_value()? This properly handles null values as None for the Option<u32> type. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add ISLock message support to network message parsing Add support for parsing "isdlock" network messages as ISLock message type. This enables proper handling of InstantSend Lock messages in the Dash network protocol, which are used for InstantSend transaction locking. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: enhance ProTx parsing logic for BasicBLS version and platform fields - Import existing ProviderMasternodeType enum to avoid duplication - Add ProTxVersion enum for LegacyBLS (1) and BasicBLS (2) versions - Extend ProviderUpdateServicePayload with conditional fields: - mn_type field for BasicBLS version - platform_node_id, platform_p2p_port, platform_http_port for Evo masternodes - Implement version validation in consensus_decode - Add conditional parsing logic matching C++ SERIALIZE_METHODS pattern - Include comprehensive block parsing tests for both ProUpServTx and ProRegTx - Tests validate successful parsing of real mainnet blocks with ProTx transactions This resolves the "unknown special transaction type: 41851" errors by properly handling conditional field serialization based on ProTx version and masternode type. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: complete overhaul of filter processing architecture for better concurrency and reliability This commit implements a major architectural refactor of the SPV client's filter processing system, moving from a complex synchronous coordination model to a clean asynchronous background processing approach that improves performance, reliability, and maintainability. ## Key Architectural Changes ### 1. Filter Processing Thread Architecture - **REMOVED**: Complex `FilterSyncState` coordination mechanism between monitoring loop and sync operations - **ADDED**: Dedicated background `FilterProcessor` thread that handles all CFilter message processing - **BENEFIT**: Eliminates race conditions and simplifies the message handling flow **Before**: Monitoring loop had to coordinate with active sync operations, route CFilter messages based on expected ranges, and track sync progress with counters and state flags. **After**: All CFilter messages are sent to a dedicated processing thread that handles watch item matching, block requests, and storage operations independently. ### 2. Simplified Filter Sync Workflow - **REMOVED**: Complex pipelined processing with timeout coordination and batch management - **ADDED**: Simple batch request sending with automatic background processing - **REMOVED**: 400+ lines of complex sync coordination and timeout handling code - **ADDED**: Clean separation between request sending and response processing **Before**: `sync_filters_coordinated()` was 200+ lines with complex pipelining, timeout management, and coordination between request sending and response handling. **After**: Filter sync simply sends batch requests; all processing happens automatically in background when CFilter messages arrive. ### 3. Improved Startup and Peer Management - **ADDED**: Defer all sync operations until at least one peer is connected - **ADDED**: `initial_sync_started` flag to prevent duplicate sync initiation - **BENEFIT**: Prevents sending protocol messages to empty peer lists and improves connection stability ### 4. Post-Sync Header Handling - **ADDED**: `handle_post_sync_headers()` method to process headers received after main sync completes - **ADDED**: Automatic filter header and filter requests for new blocks - **BENEFIT**: Ensures continuous operation and real-time block processing after initial sync ### 5. Enhanced MnListDiff Processing - **UPDATED**: `handle_mnlistdiff_message()` to accept network manager parameter - **ADDED**: Better error handling and logging for masternode list updates - **BENEFIT**: Improved masternode sync reliability and debugging ## Technical Implementation Details ### Filter Processor Thread ```rust // New architecture: spawn dedicated processing thread let (filter_processor, watch_item_updater) = FilterSyncManager::spawn_filter_processor( watch_items, network_message_sender, processing_thread_requests ); ``` The processing thread: - Receives CFilter messages via bounded channel - Matches filters against current watch items - Automatically requests blocks for matches - Updates statistics and handles storage operations - Receives watch item updates dynamically ### Simplified Message Handling ```rust // Old: Complex coordination logic if sync_state.active && filter_in_expected_range { // Route to sync operation } else { // Process as regular filter } // New: Simple delegation filter_processor.send(cfilter)?; ``` ### Robust Startup Sequence ```rust // Wait for peer connections before starting sync if \!initial_sync_started && self.network.peer_count() > 0 { // Start header sync // Start filter header sync // Mark as started } ``` ## Code Quality Improvements ### Dependencies - **ADDED**: `hex = "0.4"` dependency for test utilities and debugging ### Constants Extraction - **ADDED**: Network timing constants in `constants.rs`: - `DNS_DISCOVERY_DELAY: Duration::from_secs(10)` - `MESSAGE_POLL_INTERVAL: Duration::from_millis(10)` - `MESSAGE_RECEIVE_TIMEOUT: Duration::from_millis(100)` - **BENEFIT**: Eliminates magic numbers and makes timeouts configurable ### Logging and Debugging - **REMOVED**: Excessive debug logging and emoji-heavy output - **SIMPLIFIED**: CFilter processing logs to focus on essential information - **IMPROVED**: More structured and production-ready logging patterns ### Error Handling - **IMPROVED**: Better error propagation in sync manager methods - **ADDED**: Proper error handling for channel operations and background thread communication - **ENHANCED**: More descriptive error messages for debugging ## Performance and Reliability Benefits ### Concurrency Improvements - **BEFORE**: Single-threaded processing with complex state coordination - **AFTER**: Multi-threaded with dedicated filter processing thread - **RESULT**: Better CPU utilization and reduced blocking operations ### Memory Management - **REDUCED**: Eliminated complex state tracking structures - **SIMPLIFIED**: Cleaner object lifecycles and reduced memory overhead - **IMPROVED**: Better resource cleanup and error recovery ### Network Efficiency - **ENHANCED**: More reliable peer connection management - **IMPROVED**: Better handling of network timeouts and disconnections - **OPTIMIZED**: Reduced redundant protocol message sending ## Testing and Validation - **MAINTAINED**: All existing test compatibility - **IMPROVED**: Better testability with cleaner separation of concerns - **ENHANCED**: More predictable behavior for integration testing ## Migration Impact - **BREAKING**: Internal architecture changes (external API unchanged) - **COMPATIBLE**: All existing watch item and filter functionality preserved - **IMPROVED**: Better performance and reliability for existing use cases This refactor addresses several production issues: 1. Race conditions in filter sync coordination 2. Complex timeout and retry logic 3. Poor separation of concerns in message handling 4. Inefficient single-threaded processing 5. Unreliable startup sequence with network timing issues The new architecture is more maintainable, performant, and robust while preserving all existing functionality and improving the user experience. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: address over-reading issue in coinbase payload decoding for version 1 * debug: log block hash for blocks the fail deser * feat: implement request timeout handling and tracking for network messages * feat: add blocks_processed counter to SpvStats with logging - Add blocks_processed field to SpvStats struct for tracking processed blocks vs requested - Increment counter in process_new_block when block processing completes successfully - Display both blocks_requested and blocks_processed in sync status logging - Enables monitoring of block processing performance and completion rates 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add MNHF Signal transaction support in special transaction handling * feat: implement in-memory UTXO cache with disk persistence and address indexing * feat: implement asynchronous block processing with dedicated worker and task handling * feat: integrate storage access for current blockchain tip height retrieval * feat: clean up whitespace and improve code readability in mod.rs * refactor: modularize SPV client and add wallet management functionality This commit performs a major refactoring of the Dash SPV client to improve code organization, maintainability, and add comprehensive wallet management capabilities. Key Changes: - Split monolithic client code into focused modules: - block_processor.rs: Async block processing with dedicated worker thread - consistency.rs: Wallet consistency validation and recovery - wallet_utils.rs: Safe wallet operations with error handling - message_handler.rs: Network message processing logic - filter_sync.rs: Compact filter synchronization coordinator - status_display.rs: UI and progress reporting - watch_manager.rs: Watch item management - Added wallet integration: - Wallet-based UTXO tracking instead of direct storage manipulation - Address balance calculation through wallet - Wallet consistency checking and recovery mechanisms - Automatic wallet synchronization with watch items - Improved error handling: - Comprehensive error recovery in block processing - Safe UTXO operations with fallback behavior - Better error categorization and logging - Enhanced statistics tracking: - Separated filters_matched from blocks_with_relevant_transactions - More granular tracking of sync operations - Fixed transaction processing bugs: - Proper handling of multiple inputs from same address - Correct balance change calculations - Added test coverage for transaction calculation edge cases - Code quality improvements: - Reduced code duplication through helper methods - Better separation of concerns - More testable architecture This refactoring maintains backward compatibility while providing a cleaner architecture for future enhancements and easier maintenance. * feat: implement filter sync tracking and progress reporting * feat: enhance filter synchronization with flow control and request tracking * refactor: de-duplicate filter header chain verification logic - Extract duplicate height calculation logic into calculate_batch_start_height() helper - Consolidate repeated hash-to-height lookups into get_batch_height_range() helper - Simplify handle_overlapping_headers() from 104 to 70 lines using new helpers - Remove 3 instances of identical saturating_sub calculations - Remove 4 instances of nearly identical block hash lookup patterns - Fix missing ClientConfig fields in network tests Net result: ~22 lines of code reduction through elimination of duplication Improves maintainability by centralizing common logic in reusable helpers 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add CFHeader gap detection and auto-restart functionality * refactor: remove redundant CLSig and ISLock structs * fix: build failure re CLSig and ISLock messages * fix: add NetworkExt import to multiple files * refactor: simplify WatchManager usage by removing instance creation * fix: change logging initialization to use try_init with error handling * feat: implement handshake timeout mechanism with message polling * fix: add terminal size check before drawing status bar * fix: update filter segment paths to use the correct directory * fix: improve error handling for directory creation in disk module * fix: ensure proper sync state handling during masternode synchronization --------- Co-authored-by: Claude <noreply@anthropic.com>
* ffi work * fixes
Replaces bls-signatures with blsful library from dashpay/agora-blsful. Updates error handling to remove BLSError and adds comprehensive tests. Changes include: - Updated dependency to dashpay/agora-blsful (rev: 5f017aa) - Removed unused bitvec dependency - Added debug logging for key deserialization failures - Improved error messages with parsed key count and format info - Fixed all test code to use SerializationFormat enum 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
feat: migrate from bls-signatures to blsful library
* feat(network): add SendDsq message for CoinJoin queue notification - Add SendDsq(bool) network message variant - Implement serialization/deserialization for SendDsq - Add tests for SendDsq message encoding/decoding - This message allows peers to notify whether they want to receive CoinJoin queue messages * feat(blockdata): implement Dash-specific genesis block parameters - Add dash_genesis_tx() function with proper Dash genesis transaction - Update genesis_block() to use Dash-specific merkle roots for each network - Implement correct ChainHash constants for mainnet, testnet, devnet, regtest - Fix tests to expect Dash genesis values instead of Bitcoin values - Use explicit merkle root values instead of calculating from transaction The genesis block parameters match Dash Core implementation: - Mainnet: block 00000ffd590b1485b3caadc19b22e6379c733355108f107a430458cdf3407ab6 - Testnet: block 00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c - Devnet: uses Bitcoin genesis transaction for compatibility - Regtest: calculates merkle root from genesis transaction 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(network): fix SendDsq message serialization * fix: remove bitcoin_genesis_tx and unify all networks to use dash_genesis_tx - Remove inappropriate bitcoin_genesis_tx function that was providing Bitcoin genesis data - Update devnet to use dash_genesis_tx like all other networks (mainnet, testnet, regtest) - Update devnet merkle root to match Dash genesis transaction - Update devnet ChainHash constant to match new block hash - Update devnet test expectations to use Dash genesis values - All networks now consistently use Dash-specific genesis data 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: make genesis transaction wtxid test deterministic Replace multiple wtxid assertions with single deterministic value check. The test was previously accepting three different wtxid values which could hide potential mismatches. Now validates the actual deterministic wtxid value produced by the Dash genesis coinbase transaction. - Remove problematic multi-value assertion that accepted 3 different wtxid values - Use single assert_eq\! with the actual deterministic wtxid value - Ensures test fails properly if wtxid doesn't match expected value - Improves test reliability and debugging capability The genesis coinbase transaction wtxid is deterministic and should always be: babeaa0bf3af03c0f12d94da95c7f28168be22087a16fb207e7abda4ae654ee3 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: quantum <quantum@pastas-Mac-Studio.local> Co-authored-by: Claude <noreply@anthropic.com>
- Fix unsafe block warnings in siphash24.rs by properly wrapping unsafe operations - Improve hash utility functions with better variable naming ($generator instead of $gen) - Remove unused import in internal_macros.rs for bincode feature These changes address compiler warnings about unsafe operations and improve code clarity without changing functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
fix(hashes): improve unsafe block warnings and macro cleanup
…ter error handling - Implement standard FromStr trait for Address parsing - Keep explicit to_string() method for backward compatibility - Add from_string() convenience method - Improve error handling by using explicit Error::Bip32 wrapping instead of generic Into - Fix AccountDerivation to derive private keys first, then convert to public - Update tests to import FromStr trait where needed These changes make the key-wallet API more idiomatic while maintaining compatibility.
refactor(key-wallet): improve API with standard FromStr trait and bet…
…atting - Remove unnecessary clone() in basic_usage.rs example - Change Into<u32> to From<KeyDerivationType> for more idiomatic Rust - Remove unnecessary lifetime annotation on &str IntoDerivationPath impl - Use matches! macro instead of match for boolean checks - Fix extend() call to not need a reference - Apply cargo fmt to improve code formatting consistency These changes follow Rust best practices and improve code maintainability without affecting functionality.
refactor(key-wallet): improve code quality with clippy fixes and formatting
Remove hardcoded localhost addresses from mainnet and testnet seed lists. These were causing unnecessary connection attempts during initial sync
- Add DetailedSyncProgress struct with performance metrics - Headers per second, bytes per second, ETA calculation - Sync stage tracking (Connecting, Querying, Downloading, etc.) - Add SyncStage enum for granular progress states - Add filter_sync_available field to track peer capabilities - Add supports_compact_filters() helper to PeerInfo - Add progress channel to DashSpvClient for real-time updates - Add is_filter_sync_available() method to check peer support This enables better progress reporting and performance monitoring for SPV sync operations.
- Add intermediate handshake states for better debugging: - VersionReceivedVerackSent: Version received, verack sent - VerackReceived: Verack received from peer - Add tracking flags for version_received, verack_received, version_sent - Improve logging throughout handshake process - Better error messages with handshake state information This makes handshake debugging easier and provides clearer state transitions during peer connection establishment.
- Reduce header sync timeout from 10s to 4s for faster failure detection - Change status update interval from 5s to 500ms for smoother progress - Add detailed logging for sync process including locator info - Improve empty locator handling for genesis sync These changes provide more responsive sync behavior and better real-time progress feedback.
- Enhanced status display with better formatting and metrics - Improved network connection handling and error recovery - Updated network constants for better peer management - Enhanced multi-peer connection logic with better peer selection - Improved filter sync with better error handling - General sync module improvements for reliability These changes collectively improve the stability and performance of the SPV client's network and synchronization operations.
Change header sync timeout from 4 seconds to 500 milliseconds for more responsive timeout detection during sync operations.
- Add Swift SDK package with comprehensive Dash Core functionality - Implement SPV client FFI bindings with async/await support - Add HD wallet integration via key-wallet-ffi - Create example iOS app demonstrating wallet and SPV features - Include persistent wallet storage and transaction management - Add network FFI bindings for Swift integration - Update CLAUDE.md with build instructions and project structure - Include build scripts for iOS targets (arm64, x86_64) - Add comprehensive documentation and implementation plans 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add env_logger dependency for better debugging - Add FFIDetailedSyncProgress struct with comprehensive sync metrics - Add FFISyncStage enum for granular sync state tracking - Add sync-specific callbacks for progress and completion events - Add test_sync function for verifying sync functionality - Update cbindgen configuration to exclude internal callback types - Enhance FFI client with sync progress channels and callbacks - Update tests to handle new sync progress features This enables FFI consumers to track detailed sync progress including headers per second, ETA, sync stages, and real-time updates.
- Add detailed project overview and structure - Include build commands for Rust and FFI targets - Add test commands and environment variables - Document development commands (linting, formatting, docs) - Include key features (Dash-specific and architecture) - Add code style guidelines and constraints - Document Git workflow and current status - Add security considerations and API stability notes - Include known limitations This provides comprehensive guidance for Claude Code when working with the rust-dashcore repository.
- Simplify DashSPVFFI.swift by removing empty namespace enum - Update dash_spv_ffi.h with new FFI types matching Rust implementation - Add FFISyncStage, FFIDetailedSyncProgress types - Update function signatures for enhanced sync tracking - Add invalidArgument error case to DashSDKError - Provides better error handling for invalid inputs - Includes helpful error messages and recovery suggestions These changes align the Swift SDK with the updated Rust FFI layer and improve error handling capabilities.
Add documentation file to provide Claude Code with context and guidance when working with the Swift SDK package. This helps maintain consistency and understanding of the SDK structure.
- Add EnhancedSyncProgressView for detailed sync progress display - Add SettingsView for app configuration - Add ModelContainerHelper for SwiftData management - Update project configuration and build scripts - Improve wallet services with better error handling - Enhanced UI with platform-specific colors and layouts - Add comprehensive documentation: - DEBUG_SUMMARY.md for debugging tips - IOS_APP_SETUP_GUIDE.md for setup instructions - LINKING_FIX.md for resolving linking issues - SPM_LINKING_SOLUTION.md for Swift Package Manager setup - Add DashSPVFFI.xcframework support - Update build phase script for better library management - Improve wallet models with better persistence support This provides a more complete example app demonstrating HD wallet functionality with SPV sync capabilities and improved user experience.
Remove libdash_spv_ffi.a from version control. Binary artifacts should be built locally and not committed to the repository.
- Build and copy key_wallet_ffi libraries for both simulator and device - Create symlinks for both dash_spv_ffi and key_wallet_ffi libraries - Default symlinks point to simulator versions for development - Improve output messages with detailed library information This enables the iOS example app to use both SPV and wallet FFI functionality with proper library management.
- Add mempool_filter module for transaction filtering - Implement mempool transaction storage and callbacks - Add mempool transaction types and event handling - Update storage layer to support mempool transactions - Add FFI bindings for mempool events - Add Swift types for mempool transactions This enables SPV clients to track and filter mempool transactions, providing real-time transaction updates before blockchain confirmation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Enhance block processor for better balance updates - Update FFI wallet bindings with improved error handling - Add immature balance tracking to Swift Balance model - Improve persistent wallet management with better error handling - Add comprehensive balance calculations in WalletManager 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add SPV client verification methods and retry management - Implement comprehensive mempool support in Swift SDK - Add WatchStatusView for monitoring address watch status - Enhance WalletService with improved SPV integration - Update FFI bridge with new callback handlers - Improve error handling and state management - Add support for mempool transactions in example app - Update build script with improved error handling 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- transaction_processor_test.rs: Transaction relevance detection, output matching, and statistics - utxo_test.rs: UTXO management, spendability rules, and serialization - wallet_state_test.rs: State persistence, concurrent access, and rollback handling - utxo_rollback_test.rs: Rollback manager, snapshot creation, and reorganization handling Note: Tests are currently commented out in mod.rs due to API visibility issues that need resolution 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…ests - error_handling_test.rs: Network, storage, and validation error scenarios - error_recovery_integration_test.rs: End-to-end error recovery scenarios - error_types_test.rs: Error type conversions and formatting Tests cover error propagation, recovery mechanisms, and graceful degradation across all layers Note: Some tests have compilation issues due to trait method mismatches that need resolution 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Document the test enhancement effort including: - 163 passing tests across bloom, validation, and chain modules - Test implementation status for all modules - Known issues with client and wallet module tests - Recommendations for future improvements 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Document that wallet test modules are provided but currently commented out due to API visibility issues that need resolution 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Add explicit checks in validate_chain_basic and validate_chain_full to respect ValidationMode::None and always return Ok() without performing any validation checks 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…files The data/mainnet/mod.rs file referenced 21 JSON files but only terminal_block_2000000.json exists. This caused compile-time errors with include_str\! macros. Fixed by: - Removing references to non-existent files - Adding unit tests to validate all JSON files parse correctly - Note: This file appears to be auto-generated but unused in compilation The actual terminal block loading happens in src/sync/terminal_block_data/mainnet.rs which already correctly references only the existing file. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fix unsafe CString usage in test_client.rs by using CStr::from_ptr - Add error handling for missing masternode fields in fetch_terminal_blocks.py - Remove unnecessary f-string prefixes in fetch_terminal_blocks.py - Clarify error message in chain_tip.rs for single active tip - Add MAX_PENDING_CHAINLOCKS limit to prevent unbounded queue growth - Remove unnecessary header clone in orphan_pool.rs - Fix incorrectly prefixed storage parameter in filters.rs 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Removed DashSPVFFI target from Package.swift - DashSPVFFI module now provided by unified SDK in dashpay-ios - Updated SwiftDashCoreSDK to have no dependencies - Added comments explaining standalone build limitations SwiftDashCoreSDK now relies on the unified SDK's DashSPVFFI module, which is available when used as a dependency in dashpay-ios but not for standalone builds. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
When no peers support compact filters, the sequential sync manager now properly transitions to the next phase instead of getting stuck. This fixes the issue where masternode lists weren't being synced to the chain tip. Changes: - Check return value of start_sync_headers and transition if it returns false - Add current_phase_needs_execution to detect phases that need execution after transition - Modified check_timeout to execute pending phases before checking for timeouts This ensures Platform SDK can fetch quorum public keys at recent heights by keeping masternode lists synced to the chain tip. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
- Move diff count increment to success case only to ensure accurate tracking - Add debug logging for diff completion checks and state tracking - Handle storage height exceeding tip gracefully by adjusting to available data - Fix phase completion logic to verify target height is actually reached - Re-start masternode sync if it reports complete but hasn't reached target - Add masternode engine state logging after applying diffs This improves sync reliability when dealing with partial syncs, phase transitions, and edge cases where the requested height exceeds available data. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.