Skip to content

With FlatInTrie, --FlatDb.ImportFromPruningTrieState=true triggers on partway synced DB #11418

Closed
#11422
Closed
With FlatInTrie, --FlatDb.ImportFromPruningTrieState=true triggers on partway synced DB#11418
#11422
Assignees
asdacap
Labels
stability
@yorickdowne

Description

@yorickdowne
yorickdowne
opened on Apr 29, 2026

Description

When syncing a FlatInTrie, if --FlatDb.ImportFromPruningTrieState=true is also specified, Nethermind will attempt to import from PruningTrieState if the sync is interrupted and Nethermind is restarted, leading to a restart loop.

Steps to Reproduce

  1. Sync a full node FlatInTrie with --FlatDb.ImportFromPruningTrieState=true specified as well
  2. Restart Nethermind before sync completes
  3. Observe logs

Full command line used:

/nethermind/nethermind --Init.WebSocketsEnabled true --Network.DiscoveryPort 30303 --Network.P2PPort 30303 --HealthChecks.Enabled true --HealthChecks.UIEnabled true --JsonRpc.Enabled true --JsonRpc.Host 0.0.0.0 --JsonRpc.Port 8545 --JsonRpc.WebSocketsPort 8546 --JsonRpc.EngineHost 0.0.0.0 --JsonRpc.EnginePort 8551 --JsonRpc.JwtSecretFile=/var/lib/nethermind/ee-secret/jwtsecret --Metrics.Enabled true --Metrics.ExposeHost 0.0.0.0 --Metrics.ExposePort 6060 --log info --data-dir /var/lib/nethermind --config mainnet --Sync.AncientBodiesBarrier=0 --Sync.AncientReceiptsBarrier=0 --FlatDb.Enabled=true --FlatDb.ImportFromPruningTrieState=true --FlatDb.Layout=FlatInTrie

Desktop (please complete the following information):
Please provide the following information regarding your setup:

  • Operating System: Linux
  • Version: 1.37.1+706ad499
  • Installation Method: Docker
  • Consensus Client: Nimbus beacon node v26.3.1-102be7-stateofus

Additional context

Restarting Nethermind without --FlatDb.ImportFromPruningTrieState=true will allow the sync to continue. DB does not get corrupted afaict

Logs

Logs after first restart

execution-1  | Enabling Nethermind FlatDB with Layout FlatInTrie
execution-1  | Nethermind full node without history expiry
execution-1  | Using pruning parameters:
execution-1  |  --Sync.AncientBodiesBarrier=0 --Sync.AncientReceiptsBarrier=0
execution-1  | Using FlatDB parameters:
execution-1  | --FlatDb.Enabled=true --FlatDb.ImportFromPruningTrieState=true --FlatDb.Layout=FlatInTrie
execution-1  | 29 Apr 21:41:16 | Nethermind is starting up
execution-1  | 29 Apr 21:41:16 | Version: 1.37.1+706ad499
execution-1  | 29 Apr 21:41:16 | Loading embedded plugins
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Consensus.Clique.CliquePlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.EthStats.EthStatsPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Flashbots.Flashbots
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.HealthChecks.HealthChecksPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Hive.HivePlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Init.Snapshot.SnapshotPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Merge.AuRa.AuRaMergePlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Merge.Plugin.MergePlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Optimism.OptimismPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Shutter.ShutterPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Taiko.TaikoPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.UPnP.Plugin.UPnPPlugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.Xdc.XdcPlugin
execution-1  | 29 Apr 21:41:16 | Loading 14 assemblies from /nethermind/plugins
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Optimism
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.ExternalSigner.Plugin
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.ExternalSigner.Plugin
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Merge.Plugin
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Api
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.EthStats
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.HealthChecks
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Taiko
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.JsonRpc.TraceStore
execution-1  | 29 Apr 21:41:16 |   Found plugin type Nethermind.JsonRpc.TraceStore
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Init
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Consensus.AuRa
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Flashbots
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Shutter
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Init.Snapshot
execution-1  | 29 Apr 21:41:16 | Loading assembly Nethermind.Merge.AuRa
execution-1  | 29 Apr 21:41:16 | Loading configuration from /nethermind/configs/mainnet.json
execution-1  | 29 Apr 21:41:16 | BaseDbPath: /var/lib/nethermind/nethermind_db/mainnet
execution-1  | 29 Apr 21:41:16 | LogDirectory: /var/lib/nethermind/logs
execution-1  | 29 Apr 21:41:16 | KeyStoreDirectory: /var/lib/nethermind/keystore
execution-1  | 29 Apr 21:41:16 | Configuration complete
execution-1  | 29 Apr 21:41:16 | RocksDB: v10.4.2
execution-1  | 29 Apr 21:41:16 | Loading ChainSpec from embedded resources
execution-1  | 29 Apr 21:41:17 | Detected 18 plugins
execution-1  | 29 Apr 21:41:17 |   HealthChecks by Nethermind     Enabled
execution-1  | 29 Apr 21:41:17 |   Clique by Nethermind           Disabled
execution-1  | 29 Apr 21:41:17 |   AuRa by Nethermind             Disabled
execution-1  | 29 Apr 21:41:17 |   Ethash by Nethermind           Enabled
execution-1  | 29 Apr 21:41:17 |   Optimism by Nethermind         Disabled
execution-1  | 29 Apr 21:41:17 |   Shutter by Nethermind          Disabled
execution-1  | 29 Apr 21:41:17 |   Taiko by Nethermind            Disabled
execution-1  | 29 Apr 21:41:17 |   AuRaMerge by Nethermind        Disabled
execution-1  | 29 Apr 21:41:17 |   Merge by Nethermind            Enabled
execution-1  | 29 Apr 21:41:17 |   Hive by Nethermind             Disabled
execution-1  | 29 Apr 21:41:17 |   Clef signer by Nethermind      Disabled
execution-1  | 29 Apr 21:41:17 |   EthStats by Nethermind         Disabled
execution-1  | 29 Apr 21:41:17 |   Flashbots by Nethermind        Disabled
execution-1  | 29 Apr 21:41:17 |   NethDev by Nethermind          Disabled
execution-1  | 29 Apr 21:41:17 |   Snapshot by Nethermind         Disabled
execution-1  | 29 Apr 21:41:17 |   TraceStore by Nethermind       Disabled
execution-1  | 29 Apr 21:41:17 |   UPnP by Nethermind             Disabled
execution-1  | 29 Apr 21:41:17 |   Xdc by Nethermind              Disabled
execution-1  | 29 Apr 21:41:17 | Setting up memory allowances
execution-1  | 29 Apr 21:41:17 |   Memory hint:         1024 MB
execution-1  | 29 Apr 21:41:17 |   General memory:        32 MB
execution-1  | 29 Apr 21:41:17 |   Peers memory:          50 MB
execution-1  | 29 Apr 21:41:17 |   Netty memory:         134 MB
execution-1  | 29 Apr 21:41:17 |   Mempool memory:        81 MB
execution-1  | 29 Apr 21:41:17 |   Fast blocks memory:    72 MB
execution-1  | 29 Apr 21:41:17 |   Trie memory:          130 MB
execution-1  | 29 Apr 21:41:17 |   DB memory:            522 MB
execution-1  | 29 Apr 21:41:17 | Loading Ckzg trusted setup from file /nethermind/kzg_trusted_setup.txt
execution-1  | 29 Apr 21:41:17 | CPU: Intel(R) Xeon(R) CPU E3-1225 v6 @ 3.30GHz (4C4T)
execution-1  | 29 Apr 21:41:17 | Using http://ipv4.icanhazip.com to get external ip
execution-1  | 29 Apr 21:41:17 | Initializing 3 plugins
execution-1  | 29 Apr 21:41:17 |   HealthChecks by Nethermind
execution-1  | 29 Apr 21:41:17 |   HealthChecks by Nethermind initialized in 34ms
execution-1  | 29 Apr 21:41:17 |   Ethash by Nethermind
execution-1  | 29 Apr 21:41:17 |   Ethash by Nethermind initialized in 0ms
execution-1  | 29 Apr 21:41:17 |   Merge by Nethermind
execution-1  | 29 Apr 21:41:17 | EngineUrl specified. EnginePort 8551 EngineHost 0.0.0.0. Additional JSON RPC URL 'http://localhost:8551' has engine module enabled. skipping...
execution-1  | 29 Apr 21:41:18 | Pivot block has been set based on data from db. Pivot block number: 24988316, hash: 0x8ee142aeb800f8e619d82f39cf096d48d940dfe43d79ffa41ec1a3fa7c393982
execution-1  | 29 Apr 21:41:18 | Start block loaded from HEAD - 0 (0xd4e567...cb8fa3)
execution-1  | 29 Apr 21:41:18 | Numbers resolved, level = 24988464, header = 24988464, body = 24988464
execution-1  | 29 Apr 21:41:18 | Beacon Numbers resolved, level = 0, header = 24988496, body = 0
execution-1  | 29 Apr 21:41:18 | Loading fork choice info
execution-1  | 29 Apr 21:41:18 | Block tree initialized, last processed is 0 (0xd4e567...cb8fa3), best queued is 24988464, best known is 24988464, lowest inserted header 8182685, lowest sync inserted block number 24988317
execution-1  | 29 Apr 21:41:18 | Client started with TTD: 58750000000000000000000, TTD reached: True, Terminal Block Number , FinalTotalDifficulty: 58750003716598352816469
execution-1  | 29 Apr 21:41:18 |   Merge by Nethermind initialized in 612ms
execution-1  | 29 Apr 21:41:18 | Started monitoring for the group: nethermind, instance: Mainnet
execution-1  | 29 Apr 21:41:18 | System.Diagnostics.Metrics disabled
execution-1  | 29 Apr 21:41:18 | Setting 307 MB of block cache to account
execution-1  | 29 Apr 21:41:18 | Setting 716 MB of block cache to storage
execution-1  | 29 Apr 21:41:18 | Detected 22 GB of available memory. Applying large memory State Db config.
execution-1  | 29 Apr 21:41:18 | No current state db key scheme.
execution-1  | 29 Apr 21:41:18 | Current state is StateId { BlockNumber = -1, StateRoot = 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 }
execution-1  | 29 Apr 21:41:18 | Copying state 0 (0xd4e567...cb8fa3) with state root 0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544
execution-1  | 29 Apr 21:41:18 | Starting import
execution-1  | 29 Apr 21:41:18 | Ingest thread started
execution-1  | 29 Apr 21:41:18 | Ingest thread started
execution-1  | 29 Apr 21:41:18 | Flat Import               100.00 % [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿ ] nodes:        0
execution-1  | 29 Apr 21:41:18 | Ingest thread started
execution-1  | 29 Apr 21:41:18 | Ingest thread started
execution-1  | 29 Apr 21:41:18 | Step StepWrapper              failed after 378ms Nethermind.Trie.TrieException: Missing node is not expected
execution-1  |    at Nethermind.State.Flat.Importer.Visitor.VisitMissingNode(TreePathContextWithStorage& nodeContext, ValueHash256& nodeHash) in /nethermind/src/Nethermind/Nethermind.State.Flat/Importer.cs:line 177
execution-1  |    at Nethermind.State.Flat.Importer.Visitor.Nethermind.Trie.ITreeVisitor<Nethermind.Trie.TreePathContextWithStorage>.VisitMissingNode(TreePathContextWithStorage& nodeContext, ValueHash256& nodeHash)
execution-1  |    at Nethermind.Trie.PatriciaTree.<Accept>g__TryGetRootRef|70_1[TNodeContext](TrieNode& rootRef, <>c__DisplayClass70_0`1&) in /nethermind/src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 1073
execution-1  |    at Nethermind.Trie.PatriciaTree.Accept[TNodeContext](ITreeVisitor`1 visitor, Hash256 rootHash, VisitingOptions visitingOptions, Hash256 storageAddr, Hash256 storageRoot) in /nethermind/src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 1097
execution-1  |    at Nethermind.State.Flat.Importer.<>c__DisplayClass11_0.<Copy>b__0() in /nethermind/src/Nethermind/Nethermind.State.Flat/Importer.cs:line 64
execution-1  |    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
execution-1  | --- End of stack trace from previous location ---
execution-1  |    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
execution-1  |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
execution-1  | --- End of stack trace from previous location ---
execution-1  |    at Nethermind.State.Flat.Importer.Copy(StateId to, CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.State.Flat/Importer.cs:line 83
execution-1  |    at Nethermind.Init.Steps.ImportFlatDb.Execute(CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/ImportFlatDb.cs:line 62
execution-1  |    at Nethermind.Init.Steps.EthereumStepsManager.StepWrapper.StartExecute(IEnumerable`1 dependentSteps, CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 281
execution-1  |    at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(StepWrapper stepWrapper, Dictionary`2 stepBaseTypeMap, CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 130
execution-1  |    at Nethermind.Init.Steps.EthereumStepsManager.InitializeAll(CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 51
execution-1  |    at Nethermind.Runner.Ethereum.EthereumRunner.Start(CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Runner/Ethereum/EthereumRunner.cs:line 25
execution-1  |    at Program.<>c__DisplayClass0_0.<<<Main>$>g__RunAsync|2>d.MoveNext() in /nethermind/src/Nethermind/Nethermind.Runner/Program.cs:line 198
execution-1  | 29 Apr 21:41:18 | A critical error has occurred Nethermind.Trie.TrieException: Missing node is not expected
execution-1  |    at Nethermind.State.Flat.Importer.Visitor.VisitMissingNode(TreePathContextWithStorage& nodeContext, ValueHash256& nodeHash) in /nethermind/src/Nethermind/Nethermind.State.Flat/Importer.cs:line 177
execution-1  |    at Nethermind.State.Flat.Importer.Visitor.Nethermind.Trie.ITreeVisitor<Nethermind.Trie.TreePathContextWithStorage>.VisitMissingNode(TreePathContextWithStorage& nodeContext, ValueHash256& nodeHash)
execution-1  |    at Nethermind.Trie.PatriciaTree.<Accept>g__TryGetRootRef|70_1[TNodeContext](TrieNode& rootRef, <>c__DisplayClass70_0`1&) in /nethermind/src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 1073
execution-1  |    at Nethermind.Trie.PatriciaTree.Accept[TNodeContext](ITreeVisitor`1 visitor, Hash256 rootHash, VisitingOptions visitingOptions, Hash256 storageAddr, Hash256 storageRoot) in /nethermind/src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 1097
execution-1  |    at Nethermind.State.Flat.Importer.<>c__DisplayClass11_0.<Copy>b__0() in /nethermind/src/Nethermind/Nethermind.State.Flat/Importer.cs:line 64
execution-1  |    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
execution-1  | --- End of stack trace from previous location ---
execution-1  |    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
execution-1  |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
execution-1  | --- End of stack trace from previous location ---
execution-1  |    at Nethermind.State.Flat.Importer.Copy(StateId to, CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.State.Flat/Importer.cs:line 83
execution-1  |    at Nethermind.Init.Steps.ImportFlatDb.Execute(CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/ImportFlatDb.cs:line 62
execution-1  |    at Nethermind.Init.Steps.EthereumStepsManager.StepWrapper.StartExecute(IEnumerable`1 dependentSteps, CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 281
execution-1  |    at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(StepWrapper stepWrapper, Dictionary`2 stepBaseTypeMap, CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 130
execution-1  |    at Nethermind.Init.Steps.EthereumStepsManager.InitializeAll(CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 51
execution-1  |    at Nethermind.Runner.Ethereum.EthereumRunner.Start(CancellationToken cancellationToken) in /nethermind/src/Nethermind/Nethermind.Runner/Ethereum/EthereumRunner.cs:line 25
execution-1  |    at Program.<>c__DisplayClass0_0.<<<Main>$>g__RunAsync|2>d.MoveNext() in /nethermind/src/Nethermind/Nethermind.Runner/Program.cs:line 198
execution-1  | 29 Apr 21:41:18 | Nethermind is shutting down... Please wait until all activities are stopped.
execution-1  | 29 Apr 21:41:18 | Disposing DB State0
execution-1  | 29 Apr 21:41:18 | Disposing DB Flat
execution-1  | 29 Apr 21:41:18 | Disposing DB BlobTransactions
execution-1  | 29 Apr 21:41:18 | Disposing DB Bloom
execution-1  | 29 Apr 21:41:18 | Disposing DB BlockAccessLists
execution-1  | 29 Apr 21:41:18 | Disposing DB BadBlocks
execution-1  | 29 Apr 21:41:18 | Disposing DB BlockInfos
execution-1  | 29 Apr 21:41:18 | Disposing DB BlockNumbers
execution-1  | 29 Apr 21:41:18 | Disposing DB Headers
execution-1  | 29 Apr 21:41:18 | Disposing DB Blocks
execution-1  | 29 Apr 21:41:18 | Disposing DB Metadata
execution-1  | 29 Apr 21:41:18 | All DBs closed
execution-1  | 29 Apr 21:41:18 | Ethereum runner stopped
execution-1  | 29 Apr 21:41:18 | Nethermind is shut down
execution-1 exited with code 1 (restarting)

Metadata

Metadata

Assignees

Labels

stability

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions