Feature/xmin sync#64
Merged
Merged
Conversation
…lication Implements core xmin-based sync functionality: - XminReader: Reads changes from source using PostgreSQL xmin system column for efficient incremental change detection (Closes #55) - SyncState: Persists high-water mark xmin values per table to enable resumable incremental syncs (Part of #55) - ChangeWriter: Applies changes to target using INSERT ON CONFLICT DO UPDATE (upsert) pattern for efficient batched updates (Closes #56) - Reconciler: Detects and removes orphaned rows in target that no longer exist in source through primary key comparison (Closes #57) - Writer utilities: get_table_columns, get_primary_key_columns, row_to_values helpers for type-safe value extraction (Closes #58) - SyncDaemon: Orchestrates continuous sync cycles with configurable intervals and reconciliation scheduling (Closes #59) Features: - Batched operations with PostgreSQL parameter limit handling - Type-safe value conversion for common PostgreSQL types - URL sanitization to prevent credential leakage in state files - Comprehensive unit tests for all components Related to #60 (sync command integration) Related to #61 (integration tests) Related to #62 (documentation)
Adds a new 'xmin-sync' CLI command that provides an alternative to logical replication for continuous synchronization between PostgreSQL databases. CLI options: - --source: Source database URL - --target: Target database URL - --schema: Schema to sync (default: public) - --tables: Comma-separated list of tables to sync (default: all) - --interval: Sync interval in seconds (default: 60) - --reconcile-interval: Delete detection interval (default: 3600) - --batch-size: Rows per batch (default: 1000) - --state-file: Custom path for sync state persistence - --once: Run single sync cycle then exit (useful for cron) - --no-reconcile: Skip delete detection Features: - Graceful shutdown via Ctrl+C - Persistent state for resume capability - Progress logging with table and row counts - Error reporting for partial failures Closes #60
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
xmin-Based Incremental Sync with Full Delete Reconciliation (Accelerated 1-Day Plan)
Author: Taariq
Date: 2025-12-08
Status: Proposal
Related Issues: N/A (to be created)
Executive Summary
This proposal introduces an xmin-based incremental sync mode for
database-replicatorthat enables continuous replication from any PostgreSQL source to SerenDB without requiringwal_level=logicalon the source database.Key Benefits:
Trade-offs: