feat(vc): add committer init-from-snapshot bootstrap mode#561
Draft
sridhar-panigrahi wants to merge 2 commits into
Draft
feat(vc): add committer init-from-snapshot bootstrap mode#561sridhar-panigrahi wants to merge 2 commits into
sridhar-panigrahi wants to merge 2 commits into
Conversation
…al sources Adds a StateImporter to the VC service's database layer that enables bulk-loading trusted state into the committer's database, bypassing the normal MVCC validation pipeline. This is the foundational component needed for the Fabric to Fabric-X ledger migration tool. The migration tool's bootstrap mode will use this importer to populate the state database from a Fabric peer snapshot. - Uses PostgreSQL COPY protocol (pgx.CopyFrom) for high-throughput bulk insert - StateIterator interface enables streaming import without loading entire snapshots into memory - ImportPolicy writes to __meta namespace so the coordinator's existing recoverPolicyManagerFromStateDB() works after import - VerifyImport provides post-import integrity checks (row counts + block height) - 14 unit tests covering all operations, edge cases, and a multi-namespace end-to-end scenario Signed-off-by: Shridhar Panigrahi <sridharpanigrahi2006@gmail.com>
Wires together the StateImporter from hyperledger#516 and the genesis-data file format from fabric-x-rfcs#5 to give the committer a one-shot bootstrap mode for migrating ledger state from a Fabric peer snapshot. The full migration pipeline now spans three repos: fabric peer snapshot | | fxmigrate export (hyperledger/fabric-x#97) v genesis-data file | | committer init-from-snapshot (this PR) | -> StateImporter (hyperledger#516) v populated VC state database | | fxmigrate verify (hyperledger/fabric-x#180) v integrity report What this PR adds: * service/vc/snapshot_loader.go - streaming reader for the genesis-data binary format documented in fabric-x-rfcs#5. Defines SnapshotHeader and SnapshotData; SnapshotData implements the StateIterator interface from hyperledger#516 so it can be passed straight to ImportNamespaceState without an adapter layer. * service/vc/snapshot_loader_test.go - eight unit tests covering the binary parser end-to-end: header round-trip, full entry stream, empty entries, binary value bytes, missing file, empty header, empty namespace, and a truncated entry to verify that a partial write surfaces as a non-nil Err() rather than a clean EOF. * service/vc/bootstrap.go - orchestrator. BootstrapFromSnapshot opens the snapshot, creates a StateImporter, runs the full InitSchema -> CreateNamespace -> ImportNamespaceState -> SetBlockHeight -> VerifyImport sequence, and cross-checks the imported row count against the header's declared count to catch a truncated or partially-written snapshot file before the operator starts the committer for real. * cmd/committer/init_cmd.go - the user-facing subcommand: committer init-from-snapshot --config <vc.yaml> \ --snapshot-data <genesis.bin> Sibling to start and healthcheck. One-shot: imports the snapshot and exits. * cmd/committer/main.go - registers the new subcommand on the root. Design choices worth flagging: * Independent format parser. The genesis-data format is a contract documented in fabric-x-rfcs#5; both the producer (fxmigrate) and the consumer (this committer) implement it independently. This avoids a cross-repo Go module dependency between fabric-x-committer and fabric-x. * Subcommand sibling, not a flag on start. Bootstrap is a one-shot initialisation, not a service mode. Keeping it as a separate verb matches the way healthcheck is structured and avoids overloading start vc with semantics that change behaviour mid-flight. * Single namespace per snapshot file - that is what the genesis-data format already supports; multi-namespace would be a format change and should be addressed in the RFC first. * No idempotency / re-run guard. CreateNamespace fails if the target namespace already exists; surfacing the conflict to the operator is preferable to silently merging into existing state. Testing: Unit tests for the binary parser are included and pass locally. The end-to-end integration tests against a real Postgres (full BootstrapFromSnapshot exercise + CLI surface tests) are deliberately deferred to keep this PR focused and reviewable; happy to add them in a follow-up commit on this branch once the design is acked. Related work: * RFC: hyperledger/fabric-x-rfcs#5 * Migration tool (Exporter CLI): hyperledger/fabric-x#97 * Migration tool (verify command): hyperledger/fabric-x#180 * Mentorship issue: LF-Decentralized-Trust-Mentorships/mentorship-program#65 Signed-off-by: Shridhar Panigrahi <sridharpanigrahi2006@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR adds
This is the missing seam between the Exporter CLI (hyperledger/fabric-x#97, hyperledger/fabric-x#180) and the StateImporter primitives in #516. It gives the committer a one-shot bootstrap mode so an operator can take a
genesis-datafile produced by the migration tool and use it to initialise a fresh Fabric-X state database.This addresses deliverable #3 of the LFX mentorship issue LF-Decentralized-Trust-Mentorships/mentorship-program#65 ("Bootstrap-from-snapshot feature in the Fabric-X committer").
Files
service/vc/snapshot_loader.goSnapshotHeaderandSnapshotData.SnapshotDataimplements theStateIteratorinterface from #516 so it plugs straight intoImportNamespaceStatewithout an adapter layer.service/vc/snapshot_loader_test.goErr()rather than a clean EOF.service/vc/bootstrap.goBootstrapFromSnapshotopens the snapshot, creates aStateImporter, runsInitSchema → CreateNamespace → ImportNamespaceState → SetBlockHeight → VerifyImport, and cross-checks the imported row count against the header's declared count to catch a truncated snapshot before the operator starts the committer for real.cmd/committer/init_cmd.gocommitter init-from-snapshot --config <vc.yaml> --snapshot-data <genesis.bin>. Sibling tostartandhealthcheck. One-shot: imports the snapshot and exits.cmd/committer/main.goOperator workflow
Design choices worth flagging for the reviewer
hyperledger/fabric-x) and the consumer (this committer) implement it independently. Avoids a Go module dependency betweenfabric-x-committerandfabric-xflowing in either direction. Open to revisiting if you'd prefer one canonical implementation.start. Bootstrap is one-shot initialisation, not a service mode. Matches the structure used byhealthcheckand avoids overloadingstart vcwith semantics that change behaviour mid-flight.CreateNamespacefails if the target namespace already exists; surfacing the conflict to the operator is preferable to silently merging into existing state on a re-run.What's deliberately deferred
BootstrapFromSnapshotcan callImportPolicyfrom this same orchestrator with no change to the CLI surface.Testing
go build ./...— clean.go vet ./...— clean.golangci-lint run --new-from-rev=upstream/main— no findings on the files added by this PR.go test ./service/vc/... -run TestOpenSnapshotData— all eight unit tests pass.Related work