Background
Current snapshot parallelism depends on splitting tables into multiple sub tasks.It introduces duplicated runtime components for the same logical snapshot job, such as redundant pipeline, sinker, and related worker/task setup per sub task. As snapshot parallelism scales up, this increases tokio task scheduling overhead, resource usage, and internal coordination cost.
At the same time, snapshot monitor and extractor concurrency are not managed through a unified model, which makes table-level execution, completion, and future extension harder to maintain.
Proposed Solution
- remove multi-task fan-out for snapshot execution
- keep one runtime snapshot task and move scheduling inside that task
- move table scheduling into snapshot extractor
- unify monitor, progress, checkpoint, and completion handling in a single task flow
- reduce redundant pipeline/sinker instances and worker/task creation for snapshot parallelism
- support table-level and chunk-level parallelism through a clearer internal dispatch model
Expected Benefits
- simpler snapshot execution model
- lower runtime overhead by reducing redundant pipeline/sinker instances and tokio task creation
- more consistent monitor and checkpoint behavior
- clearer lifecycle management for snapshot execution
- easier maintenance and extension of snapshot parallelism
Background
Current snapshot parallelism depends on splitting tables into multiple sub tasks.It introduces duplicated runtime components for the same logical snapshot job, such as redundant pipeline, sinker, and related worker/task setup per sub task. As snapshot parallelism scales up, this increases tokio task scheduling overhead, resource usage, and internal coordination cost.
At the same time, snapshot monitor and extractor concurrency are not managed through a unified model, which makes table-level execution, completion, and future extension harder to maintain.
Proposed Solution
Expected Benefits