Skip to content

db/snapshotsync: resume caplin state dump per-type to avoid overlapping files#21901

Merged
domiwei merged 5 commits into
mainfrom
sudeepdino008/caplin-state-per-type-resume
Jun 24, 2026
Merged

db/snapshotsync: resume caplin state dump per-type to avoid overlapping files#21901
domiwei merged 5 commits into
mainfrom
sudeepdino008/caplin-state-per-type-resume

Conversation

@sudeepdino008

Copy link
Copy Markdown
Member

Problem

When a new Caplin beacon-state snapshot type is added to an already-populated datadir (e.g. the GLOAS BuilderPendingPayments table), seg publishable starts failing with overlapping and/or short snapshot files.

Root cause is in how state snapshots are dumped:

  • BlocksAvailable() reports the minimum to across all state types. A newly added type has no files, so the global available slot collapses to 0 (or to that one lagging type's progress).
  • DumpCaplinState used that single global from for every type and looped outer over types, inner over slot ranges with no per-type resume and no skip-if-exists.

So a restart re-dumped already-caught-up types from the lagging floor, slicing their existing base file (e.g. 000000-007150) into overlapping 50k sub-files, while the genuinely-new type never reliably caught up to head. Observed on a sepolia snapshotter:

  • PendingDepositsDump — clean layout plus spurious 000000-000050 … 001400-001450 files overlapping its 000000-007150 base.
  • BuilderPendingPayments — only reached 2.7M, never the head (10.4M).

Fix

Make the state dump resume per-type from each type's own coverage:

  • Add blocksAvailableForType — last visible segment to for a single type.
  • Add planStateDump — pure function that, given each type's availability and the target to, produces the per-type [from,to) dump jobs. A caught-up type yields no jobs (so its base file is never re-sliced); a new type still starts from genesis.
  • DumpCaplinState now builds per-type availability and dumps the plan; the redundant global fromSlot parameter is dropped (callers updated).

This also makes dump order deterministic (sorted by type name) instead of relying on Go map iteration order.

Tests

  • TestPlanStateDumpResumesPerType — covered types are not re-dumped, lagging types resume at their own availability, new/empty types start at 0.
  • TestBlocksAvailableForType — last-segment semantics.

Written test-first (Red → Green); the initial failing version surfaced that a shared fromSlot floor would leave a genesis gap for brand-new types, which is why the parameter was removed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes Caplin beacon-state snapshot dumping so it resumes per snapshot type rather than using a single global “from” slot, preventing already-covered types from being re-dumped into overlapping sub-files when a new snapshot type appears in an existing datadir.

Changes:

  • Add per-type availability lookup (blocksAvailableForType) and a deterministic per-type dump planner (planStateDump), then rework DumpCaplinState to execute that plan.
  • Remove the redundant fromSlot argument from DumpCaplinState and update call sites.
  • Add focused tests for per-type resume planning and per-type availability semantics.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
db/snapshotsync/caplin_state_snapshots.go Introduces per-type availability + planned dump jobs; updates DumpCaplinState to avoid overlapping files and to dump deterministically.
db/snapshotsync/caplin_state_dump_test.go Adds unit tests for per-type resume planning and blocksAvailableForType last-segment semantics.
cmd/capcli/cli.go Updates CLI call site to the new DumpCaplinState signature.
cl/antiquary/state_antiquary.go Updates antiquary call site to the new DumpCaplinState signature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cl/antiquary/state_antiquary.go
Comment thread db/snapshotsync/caplin_state_snapshots.go Outdated
@yperbasis yperbasis added ErigonDB Caplin Caplin: Consensus Layer, Beacon API labels Jun 22, 2026

@domiwei domiwei left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left two correctness comments around per-type resume availability. Both are about avoiding new permanent gaps/overlaps in partial snapshot layouts.

Comment thread db/snapshotsync/caplin_state_snapshots.go Outdated
Comment thread db/snapshotsync/caplin_state_snapshots.go Outdated
@sudeepdino008 sudeepdino008 requested a review from domiwei June 24, 2026 05:52
@domiwei domiwei added this pull request to the merge queue Jun 24, 2026
Merged via the queue into main with commit 270a20b Jun 24, 2026
94 checks passed
@domiwei domiwei deleted the sudeepdino008/caplin-state-per-type-resume branch June 24, 2026 09:12
AskAlexSharov pushed a commit that referenced this pull request Jul 1, 2026
…per-type to avoid overlapping files (#21901) (#22124)

Cherry-pick of #21901 to `performance`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Caplin Caplin: Consensus Layer, Beacon API ErigonDB

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants