feat(cmd): add catalog syncer#552
Conversation
🦋 Changeset detectedLatest commit: 8c3acdd The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
ajaskolski
force-pushed
the
feat-add-catalog-syncer
branch
from
67f6f3a to
a411cc2
Compare
ajaskolski
force-pushed
the
feat-add-catalog-syncer
branch
2 times, most recently
from
0e63b5e to
b031140
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR introduces catalog service integration for datastore operations, enabling syncing and merging of deployment data with a remote catalog service. The implementation provides transactional guarantees and introduces a breaking change to the EnvDir.MergeMigrationDataStore method signature, which now requires context and optional catalog parameters.
Key changes:
- Added
SyncDataStoreToCatalogandMergeDataStoreToCatalogfunctions for catalog synchronization with transactional semantics - Updated
MergeMigrationDataStoresignature to support catalog operations (breaking change) - Enhanced CLI to auto-detect and use catalog configuration when available
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| datastore/catalog_syncer.go | New functions for syncing and merging datastores to catalog with transaction support |
| engine/cld/domain/envdir.go | Updated MergeMigrationDataStore with new signature and added catalog sync capability |
| engine/cld/legacy/cli/commands/migration.go | Enhanced CLI to detect and use catalog configuration for datastore operations |
| engine/cld/domain/envdir_test.go | Updated tests to use new MergeMigrationDataStore signature |
| datastore/catalog_syncer_test.go | Comprehensive unit tests for catalog sync/merge operations |
| datastore/catalog_syncer_integration_test.go | Integration tests validating real catalog service interactions |
| datastore/mock_*.go | Generated mock implementations for testing |
| .changeset/loud-spiders-end.md | Changeset documenting breaking changes and new features |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ajaskolski
force-pushed
the
feat-add-catalog-syncer
branch
from
b031140 to
cf3bdda
Compare
| // | ||
| // This function is the inverse of LoadDataStoreFromCatalog - while that function reads from | ||
| // catalog to local, this function writes from local to catalog. | ||
| func SyncDataStoreToCatalog(ctx context.Context, localDS DataStore, catalog CatalogStore) error { |
There was a problem hiding this comment.
I wonder if we should offer this functionality as a changeset/operation instead. Since everything is supposed to run through CI, it won't be possible for anyone to execute this as a command-line utility except in emergency situations.
There was a problem hiding this comment.
Ye you might be right, I have actually written cmd in migration.go in CLD where datastore merge is so I thought its around same spot. Hmm but yea people won't be able to use it manually unless emergency.
There was a problem hiding this comment.
discussed externally to use it in cmd in the CLD and have reusable workflow to move data to the catalog
There was a problem hiding this comment.
added cmd for that in that PR so might be reused for workflow
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ajaskolski
force-pushed
the
feat-add-catalog-syncer
branch
from
51947bb to
60ce1e3
Compare
ajaskolski
force-pushed
the
feat-add-catalog-syncer
branch
from
60ce1e3 to
c758d12
Compare
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ajaskolski
force-pushed
the
feat-add-catalog-syncer
branch
from
c758d12 to
8c3acdd
Compare
|
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## chainlink-deployments-framework@0.61.0 ### Minor Changes - [#552](#552) [`32b13c5`](32b13c5) Thanks [@ajaskolski](https://github.com/ajaskolski)! - feat: add catalog service integration for datastore operations Features: - Add catalog service support for datastore management as alternative to local file storage - Add `MergeMigrationDataStoreCatalog` method for catalog-based datastore persistence - Existing `MergeMigrationDataStore` method continues to work for file-based storage (no breaking changes) - Add unified `MergeDataStoreToCatalog` function for both initial migration and ongoing merge operations - All catalog operations are transactional to prevent data inconsistencies - Add `DatastoreType` configuration option (`file`/`catalog`) in domain.yaml to control storage backend - Add new CLI command `datastore sync-to-catalog` for initial migration from file-based to catalog storage in CI - Add `SyncDataStoreToCatalog` method to sync entire local datastore to catalog - CLI automatically selects the appropriate merge method based on domain.yaml configuration - Catalog mode does not modify local files - all updates go directly to the catalog service Configuration: - Set `datastore: catalog` in domain.yaml to enable catalog mode - Set `datastore: file` or omit the setting to use traditional file-based storage - CLI commands automatically detect the configuration and use the appropriate storage backend - [#549](#549) [`3e33b93`](3e33b93) Thanks [@jkongie](https://github.com/jkongie)! - Improve JD Memory client to be aligned with the Job Distributor implementation --------- Co-authored-by: app-token-issuer-engops[bot] <144731339+app-token-issuer-engops[bot]@users.noreply.github.com>
This pull request introduces a new integration for catalog-based datastore management, allowing the system to use a remote catalog service for data persistence instead of local file storage. The changes are designed to be non-breaking, providing configuration options to select the desired backend and new CLI commands for migration and synchronization. Additionally, several mock implementations are added to support testing of the new functionality.
Catalog Service Integration & Data Management
MergeDataStoreToCatalog) for both initial migration and ongoing merge operations, ensuring atomic updates and preventing data inconsistencies. [1] [2]datastore sync-to-catalogfor migrating all local datastore data to the catalog service, supporting CI and operational workflows.Testing & Mock Implementations
datastore/mock_address_ref_store_test.go[1]datastore/mock_chain_metadata_store_test.go[2]datastore/mock_contract_metadata_store_test.go[3]https://smartcontract-it.atlassian.net/browse/CLD-782