Skip to content

feat(sync): add support for paginated synchronization (time-based)#1965

Open
arthurcurry7 wants to merge 2 commits into
Nozbe:masterfrom
arthurcurry7:master
Open

feat(sync): add support for paginated synchronization (time-based)#1965
arthurcurry7 wants to merge 2 commits into
Nozbe:masterfrom
arthurcurry7:master

Conversation

@arthurcurry7
Copy link
Copy Markdown

Description

This PR implements support for time-based pagination in the synchronization protocol. Currently, WatermelonDB attempts to pull all remote changes in a single request, which can lead to memory overhead and UI freezes when dealing with very large datasets.

Key Changes:

  • Pagination Flag: Added experimentalHasMore?: boolean to the SyncPullResult type.
  • Sync Loop: Refactored the synchronize orchestration logic to loop through multiple pages as long as experimentalHasMore is true.
  • Atomic Checkpointing: To ensure data consistency, the lastPulledAt timestamp is only updated in the local database after all pages have been successfully pulled and applied. This ensures that a partial sync can be safely resumed from the beginning without missing records.
  • Hook Support: Updated onWillApplyRemoteChanges and onDidPullChanges to trigger for every page, allowing for more granular progress tracking in the UI.

Testing

  • Added comprehensive unit tests in src/sync/impl/__tests__/synchronize.test.js:
    • can pull changes in pages: Verifies multiple page application and sequential timestamp usage.
    • calls sync hooks for every page: Verifies hook lifecycle during pagination.
    • does not update lastPulledAt if paginated sync fails midway: Verifies atomic checkpointing behavior.
  • Verified that all 80 existing test suites pass.

Types

  • Updated both Flow (src/sync/index.js) and TypeScript (src/sync/index.d.ts) definitions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant