feat: Phase 3b — ParquetMergePlanner, scheduler extension, downloader stub#6357
Open
g-talbot wants to merge 1 commit intogtt/parquet-merge-pipelinefrom
Open
feat: Phase 3b — ParquetMergePlanner, scheduler extension, downloader stub#6357g-talbot wants to merge 1 commit intogtt/parquet-merge-pipelinefrom
g-talbot wants to merge 1 commit intogtt/parquet-merge-pipelinefrom
Conversation
4 tasks
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
e96a920 to
9926093
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
3b171a0 to
84c6dd3
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
9926093 to
abf93e1
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
84c6dd3 to
a23011c
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
abf93e1 to
acc5099
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
3 times, most recently
from
ef391c9 to
90c5589
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
acc5099 to
49176b0
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
90c5589 to
49c6c19
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
49176b0 to
17135dc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
49c6c19 to
93a0a20
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
17135dc to
5a5ee74
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
93a0a20 to
3a91a31
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
5a5ee74 to
bde30af
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
3a91a31 to
956a6e5
Compare
…der stub (Phase 3b) Phase 3 pipeline integration, second PR: - ParquetMergePlanner: receives ParquetNewSplits, groups by CompactionScope, invokes ParquetMergePolicy::operations(), dispatches to scheduler. Handles startup seeding of immature splits, deduplication via known_split_ids, RunFinalizeMergePolicyAndQuit for cold-window finalization. 6 tests. - MergeSchedulerService extension: ScheduleParquetMerge message with shared merge_semaphore for global concurrency control across Tantivy and Parquet merges. Feature-gated behind cfg(feature = "metrics"). Existing scheduler tests unaffected. - ParquetMergeSplitDownloader stub: minimal Actor + Handler<ParquetMergeTask> for the scheduler and planner to reference. Full download implementation comes in PR 3c. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline
branch
from
bde30af to
6e709a0
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
956a6e5 to
4bde5b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #6352 (Phase 3a).
ParquetNewSplits, groups byCompactionScope, invokesParquetMergePolicy::operations(), dispatches to scheduler. Handles startup seeding of immature splits, deduplication viaknown_split_ids,RunFinalizeMergePolicyAndQuitfor cold-window finalization.ScheduleParquetMergehandler with sharedmerge_semaphorefor global concurrency control across Tantivy and Parquet merges. Feature-gated behindcfg(feature = "metrics").Handler<ParquetMergeTask>for the scheduler and planner to reference. Full download implementation in Phase 3c.Test plan
metricsfeature🤖 Generated with Claude Code