Skip to content

feat(vc): add committer init-from-snapshot bootstrap mode#561

Draft
sridhar-panigrahi wants to merge 2 commits into
hyperledger:mainfrom
sridhar-panigrahi:feat/init-from-snapshot
Draft

feat(vc): add committer init-from-snapshot bootstrap mode#561
sridhar-panigrahi wants to merge 2 commits into
hyperledger:mainfrom
sridhar-panigrahi:feat/init-from-snapshot

Conversation

@sridhar-panigrahi
Copy link
Copy Markdown

⚠️ Stacked on top of #516. This PR's diff includes the StateImporter commit from that PR — please review #516 first or alongside this one. Once #516 merges, this PR will rebase onto main and the diff will collapse to only the bootstrap-mode files listed below.

What this PR adds

This is the missing seam between the Exporter CLI (hyperledger/fabric-x#97, hyperledger/fabric-x#180) and the StateImporter primitives in #516. It gives the committer a one-shot bootstrap mode so an operator can take a genesis-data file produced by the migration tool and use it to initialise a fresh Fabric-X state database.

fabric peer snapshot
      │
      │  fxmigrate export   (hyperledger/fabric-x#97)
      ▼
genesis-data file
      │
      │  committer init-from-snapshot   (this PR)
      │    └─► StateImporter             (#516)
      ▼
populated VC state database
      │
      │  fxmigrate verify   (hyperledger/fabric-x#180)
      ▼
integrity report

This addresses deliverable #3 of the LFX mentorship issue LF-Decentralized-Trust-Mentorships/mentorship-program#65 ("Bootstrap-from-snapshot feature in the Fabric-X committer").

Files

File Purpose
service/vc/snapshot_loader.go Streaming reader for the genesis-data binary format documented in hyperledger/fabric-x-rfcs#5. Defines SnapshotHeader and SnapshotData. SnapshotData implements the StateIterator interface from #516 so it plugs straight into ImportNamespaceState without an adapter layer.
service/vc/snapshot_loader_test.go Eight unit tests covering the binary parser end-to-end: header round-trip, full entry stream, empty entries, binary value bytes, missing file, empty header, empty namespace, and a truncated entry to verify that a partial write surfaces as a non-nil Err() rather than a clean EOF.
service/vc/bootstrap.go Orchestrator. BootstrapFromSnapshot opens the snapshot, creates a StateImporter, runs InitSchema → CreateNamespace → ImportNamespaceState → SetBlockHeight → VerifyImport, and cross-checks the imported row count against the header's declared count to catch a truncated snapshot before the operator starts the committer for real.
cmd/committer/init_cmd.go The user-facing subcommand: committer init-from-snapshot --config <vc.yaml> --snapshot-data <genesis.bin>. Sibling to start and healthcheck. One-shot: imports the snapshot and exits.
cmd/committer/main.go Registers the new subcommand on the root.

Operator workflow

# 1. Migrate state out of the source Fabric peer:
fxmigrate export \
  --snapshot ./peer/snapshots/completed/mychannel/100 \
  --channel  mychannel \
  --namespace token \
  --output   genesis.bin

# 2. Provision a fresh Fabric-X DB and bootstrap it:
committer init-from-snapshot \
  --config       vc-config.yaml \
  --snapshot-data genesis.bin

# 3. Cross-check before going live:
fxmigrate verify \
  --genesis  genesis.bin \
  --snapshot ./peer/snapshots/completed/mychannel/100

# 4. Start the committer normally — it will resume at block N+1:
committer start vc --config vc-config.yaml

Design choices worth flagging for the reviewer

  • Independent format parser, not a cross-repo import. The genesis-data format is a contract documented in fabric-x-rfcs#5; the producer (fxmigrate, in hyperledger/fabric-x) and the consumer (this committer) implement it independently. Avoids a Go module dependency between fabric-x-committer and fabric-x flowing in either direction. Open to revisiting if you'd prefer one canonical implementation.
  • Subcommand sibling, not a flag on start. Bootstrap is one-shot initialisation, not a service mode. Matches the structure used by healthcheck and avoids overloading start vc with semantics that change behaviour mid-flight.
  • Single namespace per snapshot file. That is what the genesis-data format already supports; multi-namespace would be a format change and should be addressed in the RFC first.
  • No idempotency / re-run guard. CreateNamespace fails if the target namespace already exists; surfacing the conflict to the operator is preferable to silently merging into existing state on a re-run.
  • No transaction-ID history migration. Consistent with the RFC's decision (security/format incompatibility); only state and block height are bootstrapped.

What's deliberately deferred

  • End-to-end integration tests against a real Postgres — both the bootstrap orchestrator and the CLI surface tests need a running DB. I'd like to add them on this branch in a follow-up commit once the design is acked, rather than bloat the initial review.
  • Policy import. The current genesis-data format from fxmigrate doesn't carry namespace policies; once the format is extended, BootstrapFromSnapshot can call ImportPolicy from this same orchestrator with no change to the CLI surface.

Testing

  • go build ./... — clean.
  • go vet ./... — clean.
  • golangci-lint run --new-from-rev=upstream/main — no findings on the files added by this PR.
  • go test ./service/vc/... -run TestOpenSnapshotData — all eight unit tests pass.

Related work

…al sources

Adds a StateImporter to the VC service's database layer that enables
bulk-loading trusted state into the committer's database, bypassing the
normal MVCC validation pipeline.

This is the foundational component needed for the Fabric to Fabric-X
ledger migration tool. The migration tool's bootstrap mode will use this
importer to populate the state database from a Fabric peer snapshot.

- Uses PostgreSQL COPY protocol (pgx.CopyFrom) for high-throughput bulk insert
- StateIterator interface enables streaming import without loading entire
  snapshots into memory
- ImportPolicy writes to __meta namespace so the coordinator's existing
  recoverPolicyManagerFromStateDB() works after import
- VerifyImport provides post-import integrity checks (row counts + block height)
- 14 unit tests covering all operations, edge cases, and a multi-namespace
  end-to-end scenario

Signed-off-by: Shridhar Panigrahi <sridharpanigrahi2006@gmail.com>
Wires together the StateImporter from hyperledger#516 and the genesis-data file
format from fabric-x-rfcs#5 to give the committer a one-shot bootstrap
mode for migrating ledger state from a Fabric peer snapshot.

The full migration pipeline now spans three repos:

  fabric peer snapshot
        |
        |  fxmigrate export   (hyperledger/fabric-x#97)
        v
  genesis-data file
        |
        |  committer init-from-snapshot   (this PR)
        |    -> StateImporter             (hyperledger#516)
        v
  populated VC state database
        |
        |  fxmigrate verify   (hyperledger/fabric-x#180)
        v
  integrity report

What this PR adds:

  * service/vc/snapshot_loader.go - streaming reader for the
    genesis-data binary format documented in fabric-x-rfcs#5. Defines
    SnapshotHeader and SnapshotData; SnapshotData implements the
    StateIterator interface from hyperledger#516 so it can be passed straight to
    ImportNamespaceState without an adapter layer.

  * service/vc/snapshot_loader_test.go - eight unit tests covering the
    binary parser end-to-end: header round-trip, full entry stream,
    empty entries, binary value bytes, missing file, empty header,
    empty namespace, and a truncated entry to verify that a partial
    write surfaces as a non-nil Err() rather than a clean EOF.

  * service/vc/bootstrap.go - orchestrator. BootstrapFromSnapshot opens
    the snapshot, creates a StateImporter, runs the full
    InitSchema -> CreateNamespace -> ImportNamespaceState ->
    SetBlockHeight -> VerifyImport sequence, and cross-checks the
    imported row count against the header's declared count to catch a
    truncated or partially-written snapshot file before the operator
    starts the committer for real.

  * cmd/committer/init_cmd.go - the user-facing subcommand:

        committer init-from-snapshot --config <vc.yaml> \
          --snapshot-data <genesis.bin>

    Sibling to start and healthcheck. One-shot: imports the snapshot
    and exits.

  * cmd/committer/main.go - registers the new subcommand on the root.

Design choices worth flagging:

  * Independent format parser. The genesis-data format is a contract
    documented in fabric-x-rfcs#5; both the producer (fxmigrate) and
    the consumer (this committer) implement it independently. This
    avoids a cross-repo Go module dependency between fabric-x-committer
    and fabric-x.

  * Subcommand sibling, not a flag on start. Bootstrap is a one-shot
    initialisation, not a service mode. Keeping it as a separate verb
    matches the way healthcheck is structured and avoids overloading
    start vc with semantics that change behaviour mid-flight.

  * Single namespace per snapshot file - that is what the genesis-data
    format already supports; multi-namespace would be a format change
    and should be addressed in the RFC first.

  * No idempotency / re-run guard. CreateNamespace fails if the target
    namespace already exists; surfacing the conflict to the operator
    is preferable to silently merging into existing state.

Testing:

Unit tests for the binary parser are included and pass locally. The
end-to-end integration tests against a real Postgres (full
BootstrapFromSnapshot exercise + CLI surface tests) are deliberately
deferred to keep this PR focused and reviewable; happy to add them in
a follow-up commit on this branch once the design is acked.

Related work:

  * RFC: hyperledger/fabric-x-rfcs#5
  * Migration tool (Exporter CLI): hyperledger/fabric-x#97
  * Migration tool (verify command): hyperledger/fabric-x#180
  * Mentorship issue: LF-Decentralized-Trust-Mentorships/mentorship-program#65

Signed-off-by: Shridhar Panigrahi <sridharpanigrahi2006@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant