fix(storage): stabilize recluster candidate scheduling by zhyass · Pull Request #19995 · databendlabs/databend

zhyass · 2026-06-11T06:36:59Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Reworks linear Fuse recluster candidate scheduling so RECLUSTER FINAL can make bounded progress across large or continuously-mutated tables without repeatedly restarting candidate selection from the table head.

Add statement-scoped FINAL carry over fixed scan ranges. Carried candidate windows are validated against each fresh snapshot, and retryable commit conflicts reuse the same scan state instead of restarting the FINAL loop from the table head.
Split recluster planning into candidate-window probing and task materialization. Segment windows are disjoint, task candidates are ranked across windows, and recluster can accumulate work up to max_tasks while keeping repack-only work bounded.
Make task selection more memory-aware by applying the effective recluster memory threshold, splitting hotspot work by budget, skipping runaway high-level blocks, and preserving bounded two-level grouping for mature FINAL levels.
Reduce planning overhead by reusing the segment pruner, decoding block metas with bounded parallelism, caching block metas, avoiding unnecessary cluster-stat clones, and loading HLL data only for remained blocks that need it.
Deprecate row_avg_depth_threshold for recluster decisions while keeping the legacy option unsettable for old tables.

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8413751b6b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-11T08:32:44Z

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cddea20d2e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3251b51f1e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6fbdc363b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-11T16:31:09Z

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dcc779eccd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-18T03:59:12Z

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 63838ed097

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

github-actions · 2026-06-23T06:43:04Z

Docker Image for PR

tag: pr-19995-e2c393c-1782196814

note: this image tag is only available for internal use.

chatgpt-codex-connector · 2026-06-24T04:32:16Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: bd0a9fb308

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

github-actions · 2026-06-24T05:54:50Z

Docker Image for PR

tag: pr-19995-577ec58-1782280313

note: this image tag is only available for internal use.

github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Jun 11, 2026

zhyass marked this pull request as draft June 11, 2026 06:47

zhyass force-pushed the add_option branch from fb260e0 to 244fbac Compare June 11, 2026 08:11

zhyass marked this pull request as ready for review June 11, 2026 08:13

zhyass added the ci-cloud Build docker image for cloud test label Jun 11, 2026

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/recluster.rs Outdated

zhyass force-pushed the add_option branch from 8413751 to a8221c7 Compare June 11, 2026 08:26

zhyass removed the ci-cloud Build docker image for cloud test label Jun 11, 2026

zhyass requested review from dantengsky, sundy-li and zhang2014 June 11, 2026 09:37

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 11, 2026

zhyass force-pushed the add_option branch from 394becc to cddea20 Compare June 11, 2026 12:51

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 11, 2026

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/mutation/mutator/recluster_mutator.rs Outdated

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/mutation/mutator/recluster_mutator.rs Outdated

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/mutation/mutator/recluster_mutator.rs

zhyass force-pushed the add_option branch from e6fbdc3 to 9a0e994 Compare June 11, 2026 16:25

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 11, 2026

zhyass marked this pull request as draft June 14, 2026 17:16

zhyass marked this pull request as ready for review June 14, 2026 17:16

chatgpt-codex-connector Bot reviewed Jun 14, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/operations/mutation/mutator/recluster_mutator.rs Outdated

Comment thread src/query/storages/fuse/src/operations/recluster.rs

zhyass changed the title ~~fix: select disjoint segment windows and fill task budget across them~~ fix(storage): stabilize recluster candidate scheduling Jun 17, 2026

This comment was marked as outdated.

Sign in to view

fix

c580679

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 17, 2026

zhyass added 3 commits June 18, 2026 10:45

fix

634e6cd

Merge remote-tracking branch 'root/main' into add_option

fc1f6a1

fix

63838ed

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 18, 2026

zhyass added 2 commits June 18, 2026 20:55

fix

33b8953

fix

522fbf8

zhyass marked this pull request as draft June 22, 2026 15:06

zhyass marked this pull request as ready for review June 22, 2026 15:06

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 22, 2026

zhyass added 2 commits June 23, 2026 13:39

fix

6070bc3

Merge remote-tracking branch 'root/main' into add_option

e49f7b6

zhyass added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 23, 2026

databendlabs deleted a comment from github-actions Bot Jun 23, 2026

zhyass added 2 commits June 24, 2026 12:24

fix

f1addb1

Merge remote-tracking branch 'root/main' into add_option

bd0a9fb

sundy-li approved these changes Jun 24, 2026

View reviewed changes

Uh oh!

Conversation

zhyass commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Type of change

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Jun 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

chatgpt-codex-connector Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

Docker Image for PR

Uh oh!

chatgpt-codex-connector Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Docker Image for PR

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhyass commented Jun 11, 2026 •

edited

Loading