With `FlatInTrie`, a restart during `Snap State Ranges` greatly increases RAM usage and OOMs

[yorickdowne]

Description

I'll preface this by saying that these tests take hours, and I am not certain of where this bug is, or even whether this is a bug. That said.

Sync a FlatInTrie full node, and while it is in stage Snap State Ranges, restart it. Where previously RSS memory was flat, it will now grow steadily and eventually OOM the machine. Where previous FlatInTrie syncs (but history expired, not full) succeeded on machines with 24 GiB RAM total, this OOMs on 48 GiB.

Here's a Grafana chart

Attempt to sync around 08:00. Note flat RSS. Restart around 10:00. Note increasing RSS and saw tooth pattern because OOM. Snap State Ranges does not complete as seen in logs.

Wipe the DB and try to sync again at around 14:15. Do not restart. Note RSS stays steady and does not increase for the duration of this observation, about 3 hours. Snap State Ranges does complete and moves on to Snap Large storage left

Steps to Reproduce
In order to replicate the behavior, please provide a detailed list of steps:

1. Sync a FlatInTrie full node. Not tested: Does this also happen with Flat, does this also happen with a node with history expiry
2. While the node is in stage Snap State Ranges, restart it. Not tested: Does this also happen in other stages. How long does the node have to be in State Ranges. Tested were 1 hour and 2 hours before restarting.
3. Observe RSS memory growing steadily after the restart; but staying steady without a restart

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: Linux
• Version: 1.37.1+706ad499
• Installation Method: Docker
• Consensus Client: Lodestar v1.42.0/443e0a4

Additional context

Full command line: /nethermind/nethermind --Init.WebSocketsEnabled true --Network.DiscoveryPort 30303 --Network.P2PPort 30303 --HealthChecks.Enabled true --HealthChecks.UIEnabled true --JsonRpc.Enabled true --JsonRpc.Host 0.0.0.0 --JsonRpc.Port 8545 --JsonRpc.WebSocketsPort 8546 --JsonRpc.EngineHost 0.0.0.0 --JsonRpc.EnginePort 8551 --JsonRpc.JwtSecretFile=/var/lib/nethermind/ee-secret/jwtsecret --Metrics.Enabled true --Metrics.ExposeHost 0.0.0.0 --Metrics.ExposePort 6060 --log info --data-dir /var/lib/nethermind --config mainnet --Sync.AncientBodiesBarrier=0 --Sync.AncientReceiptsBarrier=0 --FlatDb.Enabled=true --FlatDb.Layout=FlatInTrie

[yorickdowne]

Keeping the test going without restart, it is now at Old Receipts and RSS memory has remained stable. The restart does cause an issue and I am more comfortable calling this a bug. I won't be able to test all the combinations of Flat/FlatInTrie and full/history expiry that the node is capable of, to see whether the issue is limited to particular combinations.

Another Grafana snapshot. This sync has been running since 14:15, RSS memory increased when Snap finished, and has been steady during each stage.