fix: bound SeenPayloadEnvelopeInput cache size to prevent OOM#9489
fix: bound SeenPayloadEnvelopeInput cache size to prevent OOM#9489nflaig wants to merge 2 commits into
SeenPayloadEnvelopeInput cache size to prevent OOM#9489Conversation
nflaig
left a comment
nflaig
left a comment
There was a problem hiding this comment.
@lodekeeper can you review please, that should partially cover #9073
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f42a368a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| root: props.blockRootHex, | ||
| daOutOfRange, | ||
| }); | ||
| this.pruneToMaxSize(); |
There was a problem hiding this comment.
When range-sync responses are cached out of order or the cache is already full with look-ahead slots, this insertion-time prune can immediately evict the lower-slot PayloadEnvelopeInput that cacheByRangeResponses() just seeded; that same function then does seenPayloadEnvelopeInputCache.get(envelopeBlockRootHex) for downloaded envelopes and throws Missing PayloadEnvelopeInput even though the batch added it moments earlier. This breaks Gloas range sync for full/look-ahead caches, so the cap needs to avoid pruning entries needed by the active batch or run after the batch has built its local payloadEnvelopes map.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed — this is real. Walked the call chain:
cacheByRangeResponsesatdownloadByRange.ts:212-222seedsseenPayloadEnvelopeInputCache.add({...})for every gloas block in the batch.- Each
add()now ends withthis.pruneToMaxSize()(line 127), which sorts by slot ASC and evicts the lowest entries. - Then at
downloadByRange.ts:230the same function looks up viaseenPayloadEnvelopeInputCache.get(envelopeBlockRootHex)— ifundefined, throwsMissing PayloadEnvelopeInput for block ${blockRootHex}(line 234).
Failure scenario: cache at cap with high-slot look-ahead (e.g., batch B already seeded slots 1100-1132), then batch A's responses arrive out-of-order for slots 1000-1032. Each add({slot: 1000...}) brings size to cap+1; pruneToMaxSize() evicts the lowest = slot 1000 (the entry just added). The next add({slot: 1001}) does the same — evicts slot 1001 just after adding. After the seed loop, none of batch A's entries survive. The subsequent .get(envelopeBlockRootHex) returns undefined → throw, breaks range sync.
This matches the design comment on SeenBlockInput at seenGossipBlockInput.ts:89-94:
"If pruneToMaxSize() evicts blocks from the batch currently being processed, those blocks may not yet be persisted to the database, causing getBlockByRoot() to fail when async event handlers try to look them up."
SeenBlockInput sidesteps this by pruning only after prune() and onFinalized() (known synchronization points), not on insertion. Moving prune to insertion fixes the stalled-finality OOM but reintroduces the in-flight-eviction class.
Two fixes, either works:
1. skipPrune flag for batch seeding (cleanest, preserves single OOM-safe trigger point):
add(props, options: {skipPrune?: boolean} = {}) {
// ... existing add body ...
if (!options.skipPrune) this.pruneToMaxSize();
return input;
}Make pruneToMaxSize public; cacheByRangeResponses calls add({skipPrune: true}) during seeding and seenPayloadEnvelopeInputCache.pruneToMaxSize() once after the lookup loop.
2. Local seed-map fallback in cacheByRangeResponses (smaller surface, no cache contract change):
const seededByRoot = new Map<RootHex, PayloadEnvelopeInput>();
for (const blockInput of updatedBatchBlocks.values()) {
if (!blockInput.hasBlock() || !isForkPostGloas(blockInput.forkName)) continue;
seededByRoot.set(blockInput.blockRootHex, seenPayloadEnvelopeInputCache.add({...}));
}
// later:
const payloadInput = seededByRoot.get(envelopeBlockRootHex) ?? seenPayloadEnvelopeInputCache.get(envelopeBlockRootHex);Resilient to eviction but doesn't prevent the wasted cache churn during the batch.
Lean toward option 1 — matches the SeenBlockInput intent and avoids the per-insert prune cost during the batch.
|
this happened on |
|
I made my claude open this because it was debugging the issue anyways, but we knew this was not correct and had an issue here #9073 already, this just aligns it with our pruning we have for block inputs |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a cache size limit (MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE) to SeenPayloadEnvelopeInput to prevent unbounded cache growth and potential out-of-memory (OOM) issues during non-finality. It implements a pruneToMaxSize method that evicts the oldest entries (lowest slots) when the cache exceeds the limit, and adds corresponding unit tests. Review feedback highlights a correctness issue where a newly inserted block with a lower slot (e.g., during a reorg or sync) could be immediately evicted upon insertion, and suggests excluding the newly added block root from pruning. Additionally, feedback points out a performance inefficiency in sorting the entire cache on every insertion, recommending an
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| root: props.blockRootHex, | ||
| daOutOfRange, | ||
| }); | ||
| this.pruneToMaxSize(); |
There was a problem hiding this comment.
To prevent the newly added entry from being immediately evicted if its slot is lower than the existing cache entries (e.g., during a reorg or sync of an older fork), we should exclude it from pruning. Pass props.blockRootHex to pruneToMaxSize so it can be safely skipped.
| this.pruneToMaxSize(); | |
| this.pruneToMaxSize(props.blockRootHex); |
There was a problem hiding this comment.
Applied in #9490 — pruneToMaxSize(props.blockRootHex) here. The just-inserted entry is now filtered out of eviction candidates inside pruneToMaxSize.
| root: props.blockRootHex, | ||
| daOutOfRange, | ||
| }); | ||
| this.pruneToMaxSize(); |
There was a problem hiding this comment.
Pass props.blockRootHex to pruneToMaxSize to prevent the newly added entry from being immediately evicted if its slot is lower than the existing cache entries.
| this.pruneToMaxSize(); | |
| this.pruneToMaxSize(props.blockRootHex); |
There was a problem hiding this comment.
Same — applied in #9490 alongside the matching change in add.
| private pruneToMaxSize(): void { | ||
| let itemsToDelete = this.payloadInputs.size - MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE; | ||
| if (itemsToDelete <= 0) { | ||
| return; | ||
| } | ||
|
|
||
| const sorted = [...this.payloadInputs.values()].sort((a, b) => a.slot - b.slot); | ||
| let deletedCount = 0; | ||
| for (const input of sorted) { | ||
| this.evictPayloadInput(input); | ||
| deletedCount++; | ||
| if (--itemsToDelete <= 0) { | ||
| break; | ||
| } | ||
| } | ||
|
|
||
| this.logger?.debug("SeenPayloadEnvelopeInput.pruneToMaxSize evicted entries", { | ||
| deletedCount, | ||
| maxSize: MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE, | ||
| }); | ||
| } |
There was a problem hiding this comment.
There are two issues with the current implementation of pruneToMaxSize:
-
Correctness: If we are adding an older block (e.g., during a reorg or sync), its slot might be lower than the existing entries. If the cache is full, it will be immediately evicted upon insertion, which will break subsequent queries for this block root during the import pipeline. We should support an
excludeRootparameter to prevent this. -
Efficiency: In the steady state,
itemsToDeleteis almost always1. Copying and sorting the entire cache array ($O(N \log N)$) on every single insertion is inefficient. We can optimize the common case ofitemsToDelete === 1by doing a single pass ($O(N)$) to find the minimum slot entry without any array allocation or sorting.
private pruneToMaxSize(excludeRoot?: RootHex): void {
let itemsToDelete = this.payloadInputs.size - MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE;
if (itemsToDelete <= 0) {
return;
}
let deletedCount = 0;
if (itemsToDelete === 1) {
let minInput: PayloadEnvelopeInput | undefined;
for (const input of this.payloadInputs.values()) {
if (input.blockRootHex === excludeRoot) {
continue;
}
if (minInput === undefined || input.slot < minInput.slot) {
minInput = input;
}
}
if (minInput !== undefined) {
this.evictPayloadInput(minInput);
deletedCount = 1;
}
} else {
const sorted = [...this.payloadInputs.values()]
.filter((input) => input.blockRootHex !== excludeRoot)
.sort((a, b) => a.slot - b.slot);
for (const input of sorted) {
this.evictPayloadInput(input);
deletedCount++;
if (--itemsToDelete <= 0) {
break;
}
}
}
this.logger?.debug("SeenPayloadEnvelopeInput.pruneToMaxSize evicted entries", {
deletedCount,
maxSize: MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE,
});
}There was a problem hiding this comment.
Applied in #9490 with the structure you suggested: excludeRoot filter + O(N) fast path for itemsToDelete === 1 (the per-insertion case) and the sort-based path retained for multi-item evictions. Also added a regression test that fills the cache with high-slot entries (range-sync look-ahead state), inserts a low-slot entry, and asserts the new entry survives while the previously-lowest existing entry is evicted.
Note: this only covers the single-add case — the multi-add batch-seed scenario Codex flagged in discussion_r3380314898 (cacheByRangeResponses seeding 32 slots back-to-back, later iterations evicting earlier same-batch entries) is not addressed by excludeRoot alone since the exclude only covers the entry being added in the current call. That class needs a skipPrune flag plumbed through batch seeding, called out in the #9490 PR description.
lodekeeper
left a comment
lodekeeper
left a comment
There was a problem hiding this comment.
Thanks for the quick fix on concern #1 from #9073. Cap + insertion-time prune correctly bounds the cache during stalled finality, sizing matches SeenBlockInput's MAX_BLOCK_INPUT_CACHE_SIZE convention, and the unit test cleanly locks down the basic cap + lowest-slot eviction behavior. Concerns #2 (prune on processExecutionPayload failure) and #4 (prune on gossip rejection) noted as out-of-scope.
One blocking observation echoing Codex's P1 (discussion_r3380314898, replied in detail at discussion_r3380358420): the insertion-time prune can evict entries cacheByRangeResponses just seeded inside the same batch when range-sync responses arrive out-of-order. With cap=96 and look-ahead full of high-slot entries, an earlier-batch's low-slot adds become the lowest in the cache and are evicted by the very next add()'s prune — the subsequent seenPayloadEnvelopeInputCache.get(envelopeBlockRootHex) at downloadByRange.ts:230 then throws Missing PayloadEnvelopeInput. SeenBlockInput's design comment (seenGossipBlockInput.ts:89-94) explicitly warns against this and only prunes at prune() / onFinalized() synchronization points.
Would lean toward fix option 1 (add({skipPrune: true}) flag + public pruneToMaxSize(), called explicitly from cacheByRangeResponses after its lookup loop) — preserves the OOM-safe trigger point this PR introduces, matches SeenBlockInput intent, avoids per-insert prune cost during batches.
Minor non-blocking:
- Port the multi-paragraph design comment from
seenGossipBlockInput.ts:73-98toseenPayloadEnvelopeInput.tsso the sizing + eviction reasoning is documented at the new cache. - Add a regression test for the cap-full + out-of-order batch-seed scenario once the fix lands.
The sequential catch-up OOM (the scenario in the PR description) is unaffected by the in-flight issue — adds always happen at the high-slot end, so eviction always hits old entries. PR ships value as-is for that case. Happy to approve after the in-flight-eviction fix.
|
Acked Codex's automated review banner. The P1 inline ( |
|
Re Gemini's correctness + perf observations on |
|
| Benchmark suite | Current: b55c4b1 | Previous: b00d95d | Ratio |
|---|---|---|---|
| runFastConfirmationRules vc:600000 bc:1200 eq:0 | 68.992 us/op | 21.117 us/op | 3.27 |
| runFastConfirmationRules vc:600000 bc:96 eq:1000 | 17.129 us/op | 4.6900 us/op | 3.65 |
Full benchmark results
| Benchmark suite | Current: b55c4b1 | Previous: b00d95d | Ratio |
|---|---|---|---|
| getPubkeys - index2pubkey - req 1000 vs - 250000 vc | 1.2612 ms/op | 991.33 us/op | 1.27 |
| getPubkeys - validatorsArr - req 1000 vs - 250000 vc | 41.554 us/op | 39.086 us/op | 1.06 |
| BLS verify - blst | 668.98 us/op | 718.67 us/op | 0.93 |
| BLS verifyMultipleSignatures 3 - blst | 1.3872 ms/op | 1.2966 ms/op | 1.07 |
| BLS verifyMultipleSignatures 8 - blst | 2.2123 ms/op | 2.0583 ms/op | 1.07 |
| BLS verifyMultipleSignatures 32 - blst | 7.0152 ms/op | 6.5330 ms/op | 1.07 |
| BLS verifyMultipleSignatures 64 - blst | 13.471 ms/op | 12.728 ms/op | 1.06 |
| BLS verifyMultipleSignatures 128 - blst | 26.333 ms/op | 24.933 ms/op | 1.06 |
| BLS deserializing 10000 signatures | 645.93 ms/op | 617.27 ms/op | 1.05 |
| BLS deserializing 100000 signatures | 6.4227 s/op | 6.2171 s/op | 1.03 |
| BLS verifyMultipleSignatures - same message - 3 - blst | 810.83 us/op | 763.76 us/op | 1.06 |
| BLS verifyMultipleSignatures - same message - 8 - blst | 944.52 us/op | 871.80 us/op | 1.08 |
| BLS verifyMultipleSignatures - same message - 32 - blst | 1.5788 ms/op | 1.5161 ms/op | 1.04 |
| BLS verifyMultipleSignatures - same message - 64 - blst | 2.4941 ms/op | 2.3563 ms/op | 1.06 |
| BLS verifyMultipleSignatures - same message - 128 - blst | 4.1894 ms/op | 3.9695 ms/op | 1.06 |
| BLS aggregatePubkeys 32 - blst | 18.016 us/op | 17.318 us/op | 1.04 |
| BLS aggregatePubkeys 128 - blst | 64.142 us/op | 61.503 us/op | 1.04 |
| getSlashingsAndExits - default max | 48.517 us/op | 54.141 us/op | 0.90 |
| getSlashingsAndExits - 2k | 458.17 us/op | 410.85 us/op | 1.12 |
| proposeBlockBody type=full, size=empty | 788.52 us/op | 1.6291 ms/op | 0.48 |
| isKnown best case - 1 super set check | 166.00 ns/op | 164.00 ns/op | 1.01 |
| isKnown normal case - 2 super set checks | 168.00 ns/op | 160.00 ns/op | 1.05 |
| isKnown worse case - 16 super set checks | 169.00 ns/op | 160.00 ns/op | 1.06 |
| validate api signedAggregateAndProof - struct | 1.5628 ms/op | 1.4784 ms/op | 1.06 |
| validate gossip signedAggregateAndProof - struct | 1.5749 ms/op | 1.4630 ms/op | 1.08 |
| batch validate gossip attestation - vc 640000 - chunk 32 | 113.78 us/op | 109.15 us/op | 1.04 |
| batch validate gossip attestation - vc 640000 - chunk 64 | 98.263 us/op | 94.521 us/op | 1.04 |
| batch validate gossip attestation - vc 640000 - chunk 128 | 90.933 us/op | 88.802 us/op | 1.02 |
| batch validate gossip attestation - vc 640000 - chunk 256 | 92.392 us/op | 87.446 us/op | 1.06 |
| bytes32 toHexString | 304.00 ns/op | 280.00 ns/op | 1.09 |
| bytes32 Buffer.toString(hex) | 171.00 ns/op | 172.00 ns/op | 0.99 |
| bytes32 Buffer.toString(hex) from Uint8Array | 240.00 ns/op | 230.00 ns/op | 1.04 |
| bytes32 Buffer.toString(hex) + 0x | 169.00 ns/op | 169.00 ns/op | 1.00 |
| Return object 10000 times | 0.21980 ns/op | 0.20940 ns/op | 1.05 |
| Throw Error 10000 times | 3.3939 us/op | 3.2741 us/op | 1.04 |
| toHex | 87.620 ns/op | 87.330 ns/op | 1.00 |
| Buffer.from | 80.488 ns/op | 78.893 ns/op | 1.02 |
| shared Buffer | 53.926 ns/op | 51.680 ns/op | 1.04 |
| fastMsgIdFn sha256 / 200 bytes | 1.4840 us/op | 1.4510 us/op | 1.02 |
| fastMsgIdFn h32 xxhash / 200 bytes | 155.00 ns/op | 152.00 ns/op | 1.02 |
| fastMsgIdFn h64 xxhash / 200 bytes | 204.00 ns/op | 205.00 ns/op | 1.00 |
| fastMsgIdFn sha256 / 1000 bytes | 4.7630 us/op | 4.6410 us/op | 1.03 |
| fastMsgIdFn h32 xxhash / 1000 bytes | 246.00 ns/op | 243.00 ns/op | 1.01 |
| fastMsgIdFn h64 xxhash / 1000 bytes | 250.00 ns/op | 247.00 ns/op | 1.01 |
| fastMsgIdFn sha256 / 10000 bytes | 41.766 us/op | 41.387 us/op | 1.01 |
| fastMsgIdFn h32 xxhash / 10000 bytes | 1.2890 us/op | 1.2730 us/op | 1.01 |
| fastMsgIdFn h64 xxhash / 10000 bytes | 847.00 ns/op | 815.00 ns/op | 1.04 |
| enrSubnets - fastDeserialize 64 bits | 742.00 ns/op | 729.00 ns/op | 1.02 |
| enrSubnets - ssz BitVector 64 bits | 266.00 ns/op | 261.00 ns/op | 1.02 |
| enrSubnets - fastDeserialize 4 bits | 100.00 ns/op | 103.00 ns/op | 0.97 |
| enrSubnets - ssz BitVector 4 bits | 270.00 ns/op | 260.00 ns/op | 1.04 |
| prioritizePeers score -10:0 att 32-0.1 sync 2-0 | 232.53 us/op | 200.52 us/op | 1.16 |
| prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 | 239.55 us/op | 236.67 us/op | 1.01 |
| prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 | 345.29 us/op | 350.49 us/op | 0.99 |
| prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 | 616.52 us/op | 597.09 us/op | 1.03 |
| prioritizePeers score 0:0 att 64-1 sync 4-1 | 735.12 us/op | 690.15 us/op | 1.07 |
| array of 16000 items push then shift | 1.3153 us/op | 1.2340 us/op | 1.07 |
| LinkedList of 16000 items push then shift | 7.1010 ns/op | 7.5300 ns/op | 0.94 |
| array of 16000 items push then pop | 65.200 ns/op | 67.508 ns/op | 0.97 |
| LinkedList of 16000 items push then pop | 6.0510 ns/op | 5.9020 ns/op | 1.03 |
| array of 24000 items push then shift | 1.9463 us/op | 1.8409 us/op | 1.06 |
| LinkedList of 24000 items push then shift | 6.8150 ns/op | 7.3250 ns/op | 0.93 |
| array of 24000 items push then pop | 93.296 ns/op | 93.709 ns/op | 1.00 |
| LinkedList of 24000 items push then pop | 6.1880 ns/op | 6.0140 ns/op | 1.03 |
| intersect bitArray bitLen 8 | 4.8400 ns/op | 4.6740 ns/op | 1.04 |
| intersect array and set length 8 | 30.322 ns/op | 28.898 ns/op | 1.05 |
| intersect bitArray bitLen 128 | 25.043 ns/op | 23.685 ns/op | 1.06 |
| intersect array and set length 128 | 512.95 ns/op | 494.10 ns/op | 1.04 |
| bitArray.getTrueBitIndexes() bitLen 128 | 1.0220 us/op | 1.0240 us/op | 1.00 |
| bitArray.getTrueBitIndexes() bitLen 248 | 1.8360 us/op | 1.7990 us/op | 1.02 |
| bitArray.getTrueBitIndexes() bitLen 512 | 3.8130 us/op | 3.6810 us/op | 1.04 |
| Full columns - reconstruct all 6 blobs | 115.35 us/op | 122.97 us/op | 0.94 |
| Full columns - reconstruct half of the blobs out of 6 | 63.807 us/op | 72.095 us/op | 0.89 |
| Full columns - reconstruct single blob out of 6 | 81.923 us/op | 33.913 us/op | 2.42 |
| Half columns - reconstruct all 6 blobs | 402.11 ms/op | 384.20 ms/op | 1.05 |
| Half columns - reconstruct half of the blobs out of 6 | 202.91 ms/op | 192.78 ms/op | 1.05 |
| Half columns - reconstruct single blob out of 6 | 73.368 ms/op | 68.649 ms/op | 1.07 |
| Full columns - reconstruct all 10 blobs | 211.29 us/op | 221.10 us/op | 0.96 |
| Full columns - reconstruct half of the blobs out of 10 | 102.45 us/op | 155.64 us/op | 0.66 |
| Full columns - reconstruct single blob out of 10 | 36.966 us/op | 31.734 us/op | 1.16 |
| Half columns - reconstruct all 10 blobs | 670.68 ms/op | 620.38 ms/op | 1.08 |
| Half columns - reconstruct half of the blobs out of 10 | 332.18 ms/op | 315.38 ms/op | 1.05 |
| Half columns - reconstruct single blob out of 10 | 71.641 ms/op | 68.014 ms/op | 1.05 |
| Full columns - reconstruct all 20 blobs | 584.99 us/op | 1.3090 ms/op | 0.45 |
| Full columns - reconstruct half of the blobs out of 20 | 173.75 us/op | 265.59 us/op | 0.65 |
| Full columns - reconstruct single blob out of 20 | 33.497 us/op | 30.209 us/op | 1.11 |
| Half columns - reconstruct all 20 blobs | 1.3163 s/op | 1.2662 s/op | 1.04 |
| Half columns - reconstruct half of the blobs out of 20 | 656.22 ms/op | 625.85 ms/op | 1.05 |
| Half columns - reconstruct single blob out of 20 | 71.685 ms/op | 67.361 ms/op | 1.06 |
| Set add up to 64 items then delete first | 2.2512 us/op | 2.0634 us/op | 1.09 |
| OrderedSet add up to 64 items then delete first | 3.5886 us/op | 3.2745 us/op | 1.10 |
| Set add up to 64 items then delete last | 2.2988 us/op | 2.0492 us/op | 1.12 |
| OrderedSet add up to 64 items then delete last | 3.4908 us/op | 3.2300 us/op | 1.08 |
| Set add up to 64 items then delete middle | 2.2791 us/op | 2.0607 us/op | 1.11 |
| OrderedSet add up to 64 items then delete middle | 5.1105 us/op | 4.7182 us/op | 1.08 |
| Set add up to 128 items then delete first | 4.4043 us/op | 5.8737 us/op | 0.75 |
| OrderedSet add up to 128 items then delete first | 6.5487 us/op | 6.3611 us/op | 1.03 |
| Set add up to 128 items then delete last | 4.1305 us/op | 5.6867 us/op | 0.73 |
| OrderedSet add up to 128 items then delete last | 6.2545 us/op | 5.7545 us/op | 1.09 |
| Set add up to 128 items then delete middle | 4.0994 us/op | 5.7045 us/op | 0.72 |
| OrderedSet add up to 128 items then delete middle | 12.302 us/op | 11.538 us/op | 1.07 |
| Set add up to 256 items then delete first | 8.1082 us/op | 12.355 us/op | 0.66 |
| OrderedSet add up to 256 items then delete first | 12.030 us/op | 12.344 us/op | 0.97 |
| Set add up to 256 items then delete last | 8.1933 us/op | 11.489 us/op | 0.71 |
| OrderedSet add up to 256 items then delete last | 12.396 us/op | 11.506 us/op | 1.08 |
| Set add up to 256 items then delete middle | 7.7895 us/op | 11.358 us/op | 0.69 |
| OrderedSet add up to 256 items then delete middle | 40.040 us/op | 36.032 us/op | 1.11 |
| runFastConfirmationRules vc:100000 bc:96 eq:0 | 3.5180 us/op | 3.2240 us/op | 1.09 |
| runFastConfirmationRules vc:600000 bc:96 eq:0 | 12.143 us/op | 11.029 us/op | 1.10 |
| runFastConfirmationRules vc:1000000 bc:96 eq:0 | 14.078 us/op | 10.653 us/op | 1.32 |
| runFastConfirmationRules vc:600000 bc:320 eq:0 | 31.423 us/op | 12.950 us/op | 2.43 |
| runFastConfirmationRules vc:600000 bc:1200 eq:0 | 68.992 us/op | 21.117 us/op | 3.27 |
| runFastConfirmationRules vc:600000 bc:96 eq:1000 | 17.129 us/op | 4.6900 us/op | 3.65 |
| runFastConfirmationRules vc:600000 bc:96 eq:10000 | 8.2370 us/op | 16.200 us/op | 0.51 |
| runFastConfirmationRules vc:600000 bc:96 eq:300000 | 12.619 us/op | 21.254 us/op | 0.59 |
| pass gossip attestations to forkchoice per slot | 2.5419 ms/op | 2.4957 ms/op | 1.02 |
| forkChoice updateHead vc 100000 bc 64 eq 0 | 400.32 us/op | 390.71 us/op | 1.02 |
| forkChoice updateHead vc 600000 bc 64 eq 0 | 2.3878 ms/op | 2.3807 ms/op | 1.00 |
| forkChoice updateHead vc 1000000 bc 64 eq 0 | 4.0720 ms/op | 3.9193 ms/op | 1.04 |
| forkChoice updateHead vc 600000 bc 320 eq 0 | 2.4383 ms/op | 2.3979 ms/op | 1.02 |
| forkChoice updateHead vc 600000 bc 1200 eq 0 | 2.5378 ms/op | 2.5837 ms/op | 0.98 |
| forkChoice updateHead vc 600000 bc 7200 eq 0 | 3.0823 ms/op | 3.1510 ms/op | 0.98 |
| forkChoice updateHead vc 600000 bc 64 eq 1000 | 2.7757 ms/op | 2.3902 ms/op | 1.16 |
| forkChoice updateHead vc 600000 bc 64 eq 10000 | 2.9584 ms/op | 2.5109 ms/op | 1.18 |
| forkChoice updateHead vc 600000 bc 64 eq 300000 | 7.2290 ms/op | 6.6832 ms/op | 1.08 |
| computeDeltas 1400000 validators 0% inactive | 13.207 ms/op | 12.337 ms/op | 1.07 |
| computeDeltas 1400000 validators 10% inactive | 12.671 ms/op | 11.393 ms/op | 1.11 |
| computeDeltas 1400000 validators 20% inactive | 11.556 ms/op | 10.589 ms/op | 1.09 |
| computeDeltas 1400000 validators 50% inactive | 8.6206 ms/op | 8.2496 ms/op | 1.04 |
| computeDeltas 2100000 validators 0% inactive | 20.100 ms/op | 18.780 ms/op | 1.07 |
| computeDeltas 2100000 validators 10% inactive | 19.284 ms/op | 17.384 ms/op | 1.11 |
| computeDeltas 2100000 validators 20% inactive | 17.579 ms/op | 15.855 ms/op | 1.11 |
| computeDeltas 2100000 validators 50% inactive | 10.598 ms/op | 9.3154 ms/op | 1.14 |
| altair processAttestation - 250000 vs - 7PWei normalcase | 2.4268 ms/op | 2.3047 ms/op | 1.05 |
| altair processAttestation - 250000 vs - 7PWei worstcase | 3.5742 ms/op | 3.1455 ms/op | 1.14 |
| altair processAttestation - setStatus - 1/6 committees join | 107.87 us/op | 98.543 us/op | 1.09 |
| altair processAttestation - setStatus - 1/3 committees join | 216.98 us/op | 198.62 us/op | 1.09 |
| altair processAttestation - setStatus - 1/2 committees join | 292.14 us/op | 284.71 us/op | 1.03 |
| altair processAttestation - setStatus - 2/3 committees join | 420.07 us/op | 374.49 us/op | 1.12 |
| altair processAttestation - setStatus - 4/5 committees join | 531.09 us/op | 827.45 us/op | 0.64 |
| altair processAttestation - setStatus - 100% committees join | 619.82 us/op | 615.06 us/op | 1.01 |
| altair processBlock - 250000 vs - 7PWei normalcase | 2.9963 ms/op | 4.6754 ms/op | 0.64 |
| altair processBlock - 250000 vs - 7PWei normalcase hashState | 18.767 ms/op | 14.480 ms/op | 1.30 |
| altair processBlock - 250000 vs - 7PWei worstcase | 21.825 ms/op | 24.062 ms/op | 0.91 |
| altair processBlock - 250000 vs - 7PWei worstcase hashState | 47.006 ms/op | 49.052 ms/op | 0.96 |
| phase0 processBlock - 250000 vs - 7PWei normalcase | 1.5339 ms/op | 1.4936 ms/op | 1.03 |
| phase0 processBlock - 250000 vs - 7PWei worstcase | 18.771 ms/op | 16.467 ms/op | 1.14 |
| altair processEth1Data - 250000 vs - 7PWei normalcase | 312.24 us/op | 286.24 us/op | 1.09 |
| getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16 | 3.8400 us/op | 4.1260 us/op | 0.93 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220 | 22.403 us/op | 22.548 us/op | 0.99 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43 | 6.8620 us/op | 8.4470 us/op | 0.81 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19 | 3.7060 us/op | 5.0570 us/op | 0.73 |
| getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021 | 97.944 us/op | 103.46 us/op | 0.95 |
| getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778 | 1.4181 ms/op | 1.4849 ms/op | 0.96 |
| getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 | 1.8829 ms/op | 1.8140 ms/op | 1.04 |
| getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 | 1.8479 ms/op | 1.7497 ms/op | 1.06 |
| getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 | 3.8749 ms/op | 3.5424 ms/op | 1.09 |
| getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 | 2.0895 ms/op | 2.0522 ms/op | 1.02 |
| getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 | 4.1651 ms/op | 4.1913 ms/op | 0.99 |
| Tree 40 250000 create | 381.15 ms/op | 343.25 ms/op | 1.11 |
| Tree 40 250000 get(125000) | 102.89 ns/op | 92.156 ns/op | 1.12 |
| Tree 40 250000 set(125000) | 1.0562 us/op | 966.11 ns/op | 1.09 |
| Tree 40 250000 toArray() | 10.543 ms/op | 14.899 ms/op | 0.71 |
| Tree 40 250000 iterate all - toArray() + loop | 9.5217 ms/op | 15.257 ms/op | 0.62 |
| Tree 40 250000 iterate all - get(i) | 37.575 ms/op | 40.287 ms/op | 0.93 |
| Array 250000 create | 2.1148 ms/op | 2.1522 ms/op | 0.98 |
| Array 250000 clone - spread | 678.71 us/op | 671.27 us/op | 1.01 |
| Array 250000 get(125000) | 0.30400 ns/op | 0.29400 ns/op | 1.03 |
| Array 250000 set(125000) | 0.30300 ns/op | 0.29400 ns/op | 1.03 |
| Array 250000 iterate all - loop | 58.519 us/op | 56.916 us/op | 1.03 |
| phase0 afterProcessEpoch - 250000 vs - 7PWei | 54.052 ms/op | 39.136 ms/op | 1.38 |
| Array.fill - length 1000000 | 2.2327 ms/op | 2.3832 ms/op | 0.94 |
| Array push - length 1000000 | 9.9741 ms/op | 9.4883 ms/op | 1.05 |
| Array.get | 0.20652 ns/op | 0.19947 ns/op | 1.04 |
| Uint8Array.get | 0.24510 ns/op | 0.22782 ns/op | 1.08 |
| phase0 beforeProcessEpoch - 250000 vs - 7PWei | 15.496 ms/op | 15.719 ms/op | 0.99 |
| altair processEpoch - mainnet_e81889 | 345.41 ms/op | 301.69 ms/op | 1.14 |
| mainnet_e81889 - altair beforeProcessEpoch | 34.697 ms/op | 38.035 ms/op | 0.91 |
| mainnet_e81889 - altair processJustificationAndFinalization | 8.4910 us/op | 6.9870 us/op | 1.22 |
| mainnet_e81889 - altair processInactivityUpdates | 4.5781 ms/op | 6.4626 ms/op | 0.71 |
| mainnet_e81889 - altair processRewardsAndPenalties | 17.830 ms/op | 19.574 ms/op | 0.91 |
| mainnet_e81889 - altair processRegistryUpdates | 531.00 ns/op | 529.00 ns/op | 1.00 |
| mainnet_e81889 - altair processSlashings | 135.00 ns/op | 139.00 ns/op | 0.97 |
| mainnet_e81889 - altair processEth1DataReset | 130.00 ns/op | 126.00 ns/op | 1.03 |
| mainnet_e81889 - altair processEffectiveBalanceUpdates | 1.7013 ms/op | 6.2076 ms/op | 0.27 |
| mainnet_e81889 - altair processSlashingsReset | 709.00 ns/op | 669.00 ns/op | 1.06 |
| mainnet_e81889 - altair processRandaoMixesReset | 1.4470 us/op | 1.4200 us/op | 1.02 |
| mainnet_e81889 - altair processHistoricalRootsUpdate | 131.00 ns/op | 134.00 ns/op | 0.98 |
| mainnet_e81889 - altair processParticipationFlagUpdates | 421.00 ns/op | 448.00 ns/op | 0.94 |
| mainnet_e81889 - altair processSyncCommitteeUpdates | 107.00 ns/op | 106.00 ns/op | 1.01 |
| mainnet_e81889 - altair afterProcessEpoch | 42.633 ms/op | 40.367 ms/op | 1.06 |
| capella processEpoch - mainnet_e217614 | 1.0806 s/op | 828.08 ms/op | 1.30 |
| mainnet_e217614 - capella beforeProcessEpoch | 61.302 ms/op | 55.504 ms/op | 1.10 |
| mainnet_e217614 - capella processJustificationAndFinalization | 8.0920 us/op | 6.3620 us/op | 1.27 |
| mainnet_e217614 - capella processInactivityUpdates | 13.261 ms/op | 15.115 ms/op | 0.88 |
| mainnet_e217614 - capella processRewardsAndPenalties | 102.72 ms/op | 90.777 ms/op | 1.13 |
| mainnet_e217614 - capella processRegistryUpdates | 4.3850 us/op | 4.3120 us/op | 1.02 |
| mainnet_e217614 - capella processSlashings | 130.00 ns/op | 126.00 ns/op | 1.03 |
| mainnet_e217614 - capella processEth1DataReset | 125.00 ns/op | 125.00 ns/op | 1.00 |
| mainnet_e217614 - capella processEffectiveBalanceUpdates | 16.752 ms/op | 16.862 ms/op | 0.99 |
| mainnet_e217614 - capella processSlashingsReset | 691.00 ns/op | 660.00 ns/op | 1.05 |
| mainnet_e217614 - capella processRandaoMixesReset | 1.4620 us/op | 1.2900 us/op | 1.13 |
| mainnet_e217614 - capella processHistoricalRootsUpdate | 125.00 ns/op | 124.00 ns/op | 1.01 |
| mainnet_e217614 - capella processParticipationFlagUpdates | 416.00 ns/op | 417.00 ns/op | 1.00 |
| mainnet_e217614 - capella afterProcessEpoch | 112.71 ms/op | 107.00 ms/op | 1.05 |
| phase0 processEpoch - mainnet_e58758 | 324.48 ms/op | 326.98 ms/op | 0.99 |
| mainnet_e58758 - phase0 beforeProcessEpoch | 64.362 ms/op | 70.634 ms/op | 0.91 |
| mainnet_e58758 - phase0 processJustificationAndFinalization | 6.6400 us/op | 6.3660 us/op | 1.04 |
| mainnet_e58758 - phase0 processRewardsAndPenalties | 15.627 ms/op | 15.682 ms/op | 1.00 |
| mainnet_e58758 - phase0 processRegistryUpdates | 2.2220 us/op | 2.1860 us/op | 1.02 |
| mainnet_e58758 - phase0 processSlashings | 224.00 ns/op | 126.00 ns/op | 1.78 |
| mainnet_e58758 - phase0 processEth1DataReset | 125.00 ns/op | 124.00 ns/op | 1.01 |
| mainnet_e58758 - phase0 processEffectiveBalanceUpdates | 871.55 us/op | 850.63 us/op | 1.02 |
| mainnet_e58758 - phase0 processSlashingsReset | 819.00 ns/op | 856.00 ns/op | 0.96 |
| mainnet_e58758 - phase0 processRandaoMixesReset | 1.4190 us/op | 1.2180 us/op | 1.17 |
| mainnet_e58758 - phase0 processHistoricalRootsUpdate | 130.00 ns/op | 129.00 ns/op | 1.01 |
| mainnet_e58758 - phase0 processParticipationRecordUpdates | 1.2230 us/op | 1.2140 us/op | 1.01 |
| mainnet_e58758 - phase0 afterProcessEpoch | 33.380 ms/op | 32.358 ms/op | 1.03 |
| phase0 processEffectiveBalanceUpdates - 250000 normalcase | 1.0216 ms/op | 950.62 us/op | 1.07 |
| phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 | 1.2399 ms/op | 1.5048 ms/op | 0.82 |
| altair processInactivityUpdates - 250000 normalcase | 10.701 ms/op | 12.079 ms/op | 0.89 |
| altair processInactivityUpdates - 250000 worstcase | 10.702 ms/op | 12.019 ms/op | 0.89 |
| phase0 processRegistryUpdates - 250000 normalcase | 2.5450 us/op | 2.0470 us/op | 1.24 |
| phase0 processRegistryUpdates - 250000 badcase_full_deposits | 139.77 us/op | 134.22 us/op | 1.04 |
| phase0 processRegistryUpdates - 250000 worstcase 0.5 | 60.612 ms/op | 67.706 ms/op | 0.90 |
| altair processRewardsAndPenalties - 250000 normalcase | 13.546 ms/op | 16.522 ms/op | 0.82 |
| altair processRewardsAndPenalties - 250000 worstcase | 13.467 ms/op | 15.444 ms/op | 0.87 |
| phase0 getAttestationDeltas - 250000 normalcase | 5.4326 ms/op | 5.3498 ms/op | 1.02 |
| phase0 getAttestationDeltas - 250000 worstcase | 5.6356 ms/op | 5.4497 ms/op | 1.03 |
| phase0 processSlashings - 250000 worstcase | 59.873 us/op | 61.963 us/op | 0.97 |
| altair processSyncCommitteeUpdates - 250000 | 10.542 ms/op | 12.389 ms/op | 0.85 |
| BeaconState.hashTreeRoot - No change | 163.00 ns/op | 162.00 ns/op | 1.01 |
| BeaconState.hashTreeRoot - 1 full validator | 72.326 us/op | 87.518 us/op | 0.83 |
| BeaconState.hashTreeRoot - 32 full validator | 872.01 us/op | 854.28 us/op | 1.02 |
| BeaconState.hashTreeRoot - 512 full validator | 7.3210 ms/op | 9.4566 ms/op | 0.77 |
| BeaconState.hashTreeRoot - 1 validator.effectiveBalance | 103.51 us/op | 106.83 us/op | 0.97 |
| BeaconState.hashTreeRoot - 32 validator.effectiveBalance | 1.5614 ms/op | 1.5574 ms/op | 1.00 |
| BeaconState.hashTreeRoot - 512 validator.effectiveBalance | 14.294 ms/op | 21.127 ms/op | 0.68 |
| BeaconState.hashTreeRoot - 1 balances | 92.151 us/op | 82.518 us/op | 1.12 |
| BeaconState.hashTreeRoot - 32 balances | 723.49 us/op | 758.08 us/op | 0.95 |
| BeaconState.hashTreeRoot - 512 balances | 5.0499 ms/op | 8.6228 ms/op | 0.59 |
| BeaconState.hashTreeRoot - 250000 balances | 163.87 ms/op | 151.00 ms/op | 1.09 |
| aggregationBits - 2048 els - zipIndexesInBitList | 19.822 us/op | 21.631 us/op | 0.92 |
| regular array get 100000 times | 22.522 us/op | 23.204 us/op | 0.97 |
| wrappedArray get 100000 times | 22.413 us/op | 23.188 us/op | 0.97 |
| arrayWithProxy get 100000 times | 11.235 ms/op | 17.273 ms/op | 0.65 |
| ssz.Root.equals | 21.137 ns/op | 76.051 ns/op | 0.28 |
| byteArrayEquals | 20.902 ns/op | 22.729 ns/op | 0.92 |
| Buffer.compare | 8.6450 ns/op | 9.6110 ns/op | 0.90 |
| processSlot - 1 slots | 9.2520 us/op | 12.009 us/op | 0.77 |
| processSlot - 32 slots | 2.5336 ms/op | 2.1689 ms/op | 1.17 |
| getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei | 3.5990 ms/op | 5.4772 ms/op | 0.66 |
| getCommitteeAssignments - req 1 vs - 250000 vc | 1.6752 ms/op | 1.6515 ms/op | 1.01 |
| getCommitteeAssignments - req 100 vs - 250000 vc | 3.4498 ms/op | 3.3508 ms/op | 1.03 |
| getCommitteeAssignments - req 1000 vs - 250000 vc | 3.7088 ms/op | 3.6222 ms/op | 1.02 |
| findModifiedValidators - 10000 modified validators | 715.88 ms/op | 775.91 ms/op | 0.92 |
| findModifiedValidators - 1000 modified validators | 459.59 ms/op | 460.31 ms/op | 1.00 |
| findModifiedValidators - 100 modified validators | 263.44 ms/op | 298.57 ms/op | 0.88 |
| findModifiedValidators - 10 modified validators | 230.80 ms/op | 236.47 ms/op | 0.98 |
| findModifiedValidators - 1 modified validators | 208.44 ms/op | 164.84 ms/op | 1.26 |
| findModifiedValidators - no difference | 180.95 ms/op | 195.14 ms/op | 0.93 |
| migrate state 1500000 validators, 3400 modified, 2000 new | 3.8512 s/op | 3.4140 s/op | 1.13 |
| RootCache.getBlockRootAtSlot - 250000 vs - 7PWei | 3.8800 ns/op | 3.6500 ns/op | 1.06 |
| state getBlockRootAtSlot - 250000 vs - 7PWei | 405.82 ns/op | 402.63 ns/op | 1.01 |
| computeProposerIndex 100000 validators | 1.4294 ms/op | 1.3639 ms/op | 1.05 |
| getNextSyncCommitteeIndices 1000 validators | 2.9767 ms/op | 2.8767 ms/op | 1.03 |
| getNextSyncCommitteeIndices 10000 validators | 26.135 ms/op | 25.611 ms/op | 1.02 |
| getNextSyncCommitteeIndices 100000 validators | 86.807 ms/op | 91.556 ms/op | 0.95 |
| computeProposers - vc 250000 | 562.20 us/op | 554.37 us/op | 1.01 |
| computeEpochShuffling - vc 250000 | 40.946 ms/op | 39.615 ms/op | 1.03 |
| getNextSyncCommittee - vc 250000 | 9.5469 ms/op | 10.744 ms/op | 0.89 |
| nodejs block root to RootHex using toHex | 99.338 ns/op | 94.457 ns/op | 1.05 |
| nodejs block root to RootHex using toRootHex | 60.211 ns/op | 53.106 ns/op | 1.13 |
| nodejs fromHex(blob) | 828.46 us/op | 918.76 us/op | 0.90 |
| nodejs fromHexInto(blob) | 663.30 us/op | 649.99 us/op | 1.02 |
| nodejs block root to RootHex using the deprecated toHexString | 539.52 ns/op | 522.61 ns/op | 1.03 |
| nodejs byteArrayEquals 32 bytes (block root) | 26.228 ns/op | 26.142 ns/op | 1.00 |
| nodejs byteArrayEquals 48 bytes (pubkey) | 37.992 ns/op | 38.124 ns/op | 1.00 |
| nodejs byteArrayEquals 96 bytes (signature) | 38.007 ns/op | 34.435 ns/op | 1.10 |
| nodejs byteArrayEquals 1024 bytes | 47.877 ns/op | 41.547 ns/op | 1.15 |
| nodejs byteArrayEquals 131072 bytes (blob) | 1.8245 us/op | 1.7793 us/op | 1.03 |
| browser block root to RootHex using toHex | 148.00 ns/op | 146.10 ns/op | 1.01 |
| browser block root to RootHex using toRootHex | 132.68 ns/op | 132.50 ns/op | 1.00 |
| browser fromHex(blob) | 1.6247 ms/op | 1.5964 ms/op | 1.02 |
| browser fromHexInto(blob) | 667.74 us/op | 618.55 us/op | 1.08 |
| browser block root to RootHex using the deprecated toHexString | 374.99 ns/op | 342.53 ns/op | 1.09 |
| browser byteArrayEquals 32 bytes (block root) | 28.086 ns/op | 27.324 ns/op | 1.03 |
| browser byteArrayEquals 48 bytes (pubkey) | 39.590 ns/op | 38.543 ns/op | 1.03 |
| browser byteArrayEquals 96 bytes (signature) | 74.305 ns/op | 71.361 ns/op | 1.04 |
| browser byteArrayEquals 1024 bytes | 757.83 ns/op | 727.06 ns/op | 1.04 |
| browser byteArrayEquals 131072 bytes (blob) | 95.358 us/op | 92.597 us/op | 1.03 |
by benchmarkbot/action
|
Treating this Nothing actionable here. If the same benchmark trips again on the next CI run we should look at whether the runner is contended; otherwise it's noise. |
A `pruneToMaxSize()` on insertion could evict the entry just added when it is the lowest slot (e.g. an older block during a reorg/sync), so its own lookup would then miss. Pass the inserted root as `excludeRoot` so it is never the one evicted. Also simplify `pruneToMaxSize` to a single lowest-slot eviction: it runs on every insert, so the cache is at most one over the cap.
|
@lodekeeper please re-review and approve if it looks good to you |
lodekeeper
left a comment
lodekeeper
left a comment
There was a problem hiding this comment.
LGTM. The excludeRoot plumbing through add / addFromBid plus the simplified single-lowest-slot eviction in pruneToMaxSize matches what I had in #9490 (will close that as superseded). The while-loop guard around the eviction is a nice defensive touch — even though in steady state the cache can only ever be one over cap, the loop survives any future call path that adds multiple entries before pruning.
The updated comment on MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE is sharper now — calling out the prepareNextSlot bail during catch-up explains why none of the existing pruners run in the OOM scenario.
New unit test cleanly locks down the single-add reorg/older-fork scenario (lowest-slot insertion survives, previously-lowest is evicted).
Reminder that the multi-add batch-seed scenario Codex flagged in discussion_r3380314898 (out-of-order cacheByRangeResponses seeding 32 slots back-to-back, later iterations evicting earlier same-batch entries) is not covered by excludeRoot alone — needs caller-side skipPrune plumbing or a local seed-map fallback in cacheByRangeResponses. Worth a follow-up if mainnet exposes it via lodestar_seen_payload_envelope_input_pruned or the downstream Missing PayloadEnvelopeInput throw.
| * Called on every insertion so the cache is at most one over the cap. `excludeRoot` is never | ||
| * evicted, so an add can't remove itself when it is the lowest slot (e.g. an older block in a reorg). | ||
| */ | ||
| private pruneToMaxSize(excludeRoot?: RootHex): void { |
There was a problem hiding this comment.
@lodekeeper can you confirm whether this exclude root is even needed, in what scenario can this be relevant with our current code?
There was a problem hiding this comment.
Walked the 5 add / addFromBid call sites for the scenario where the just-inserted entry's slot is the strict minimum in the cache (the only case where excludeRoot actually fires):
| Caller | Slot of insert | Hits excludeRoot? |
|---|---|---|
chain.ts:445 (addFromBid at init) |
anchor block slot | No — cache is empty at init, no eviction can fire |
gossipHandlers.ts:191 (gossip block) |
gossip-window slot | Yes, but only when finalized_slot < head - 96 (stalled finality) AND the block is on a non-canonical fork at a slot lower than every cache entry — validateGossipBlock lets that range through |
downloadByRange.ts:214 (range sync seed) |
batch epoch slot | Yes for out-of-order batch completion (Codex P1) — but excludeRoot only protects the single entry being added in this call; sibling adds earlier in the same batch can still get evicted by the next add. Partial coverage. |
downloadByRoot.ts:119 (parent root sync) |
parent slot | Yes for deep-reorg / fork-recovery where the parent is significantly older than head, edge case |
api/impl/beacon/blocks/index.ts:116 (publish API) |
near-head slot | No — validator-published blocks are at or near current slot |
So it's reachable but only in three narrow regimes: stalled-finality gossip on a low-slot non-canonical fork, out-of-order range-sync batches, and deep-reorg parent fetches.
One extra wrinkle worth flagging: even when excludeRoot fires, the protection only lasts until the next add() call — that next add doesn't know to exclude the previous entry, so it becomes the new lowest and gets evicted. So excludeRoot is at most a 1-tick reprieve. For the gossip case in particular, the gap between gossipHandlers.ts:191 (block add) and the matching envelope lookup at gossipHandlers.ts:1141 (PAYLOAD_ENVELOPE_INPUT_MISSING throw) can easily span several other gossip adds in a busy network, in which case excludeRoot doesn't actually save the lookup.
Verdict: defensible to keep — costs almost nothing, plugs the immediate self-eviction path that the unit test exercises. But fair to drop too if you'd rather simplify; the OOM scenario this PR addresses (sequential catch-up) never enters the excludeRoot branch since adds are always at the high end of the slot range. If we drop it I'd suggest a brief comment on pruneToMaxSize noting that a single low-slot insert can be self-evicting, so callers shouldn't assume add() always succeeds.
If you want a stronger guarantee for the gossip-block → envelope path specifically, the real fix is either (a) skip-prune on add and prune at a clock-tick boundary, or (b) the local seed-map fallback in cacheByRangeResponses Codex suggested — both follow-up scope.
|
merged into |
|
there was a I implemented #9509 as an alternative |
|
@twoeths thanks, confirmed from the Discord log. The same root ( So yes, #9489 as-is is unsafe for range-sync batches. The I do not think we should merge #9489 in its current form. #9509’s direction looks like the right immediate devnet fix: prune after successful import / fork-choice progression rather than during batch seeding, and avoid evicting non- One caveat: #9509 does not fully cover the original “cache grows while no blocks import / no fork-choice progression” OOM shape from #9073, so I think that should remain a separate follow-up instead of reintroducing unsafe insert-time eviction. |
**Motivation** - We only call `pruneBelowParent()` when the node is synced, ie in "PrepareNextSlot", this may cause OOM when the node is syncing and unfinality network as in `glamsterdam-devnet-5` **Description** - call `pruneBelowParent()` after `importBlock()` this is an alternative to #9489 Created with the help of Claude Co-authored-by: twoeths <twoeths@users.noreply.github.com>
3 participants
Motivation
SeenPayloadEnvelopeInput(added in #8962) has no max-size cap. Its only eviction paths arepruneFinalized(on finalization) andpruneBelowParent(on chain progression), so during extended non-finality — or a catch-up where no blocks import — neither fires and the cache grows unbounded. Each entry holds a payload envelope plus up toNUMBER_OF_COLUMNSdata-column sidecars, so this can OOM the node.This was hit on a Gloas devnet: a node restarting ~250 slots behind caught up via unknown-block-by-root sync that never connected to fork choice (
parentInForkChoice=false, 0 imports, same roots re-downloaded), and the payload-envelope cache grew until the JS heap hit its--max-old-space-sizelimit and the process OOM-crashed in a loop.This is concern #1 of #9073 (
pruneToMaxSize()cap missing — unbounded growth during non-finality).Description
MAX_PAYLOAD_ENVELOPE_INPUT_CACHE_SIZE = (MAX_LOOK_AHEAD_EPOCHS + 1) * SLOTS_PER_EPOCH, mirroringSeenBlockInput'sMAX_BLOCK_INPUT_CACHE_SIZEso the range-sync look-ahead window stays resident.pruneToMaxSize()that evicts the lowest-slot entries first (oldest / furthest behind the head).add/addFromBid) rather than only after finalization. Insertion is the only event guaranteed to fire while the cache is growing, so the cap holds even when finality is stalled — the exact OOM condition. AnonFinalized-only trigger would not have prevented it.pruneFinalized/pruneBelowParentnever run.Concerns #2 (prune on
processExecutionPayloadfailure) and #4 (prune on gossip rejection) from #9073 are not addressed here.Relates to #9073
AI Assistance Disclosure
Root cause was diagnosed from a node's debug logs and the fix drafted with assistance from Claude; reviewed and verified by me (biome, unit tests, type-check).