Skip to content

feat(fuse): add hash-distributed partition writes - #20214

Closed
KKould wants to merge 20 commits into
databendlabs:mainfrom
KKould:feat/fuse-hash-write-distribution
Closed

feat(fuse): add hash-distributed partition writes#20214
KKould wants to merge 20 commits into
databendlabs:mainfrom
KKould:feat/fuse-hash-write-distribution

Conversation

@KKould

@KKould KKould commented Jul 28, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Draft stacked on #20143.

  • Add an Iceberg-compatible bucket(count, value) partition transform.
  • Add WRITE_DISTRIBUTION_MODE = hash for partitioned Fuse tables.
  • For eligible INSERT SELECT plans, compose the existing EvalScalar -> GlobalHash -> Sort physical operators before DistributedInsertSelect.
  • Reuse the existing Fuse append, compact, partition split, serialization, and commit paths without adding new execution processors.

The goal is to reduce fragmented blocks when unordered input repeatedly crosses physical partition boundaries.

Current scope

  • Hash write distribution currently optimizes INSERT SELECT when no schema cast is required and all partition source columns are present.
  • Other INSERT sources and unsupported INSERT SELECT shapes fall back to the existing append path.
  • bucket(count, value) controls physical partition cardinality. It does not configure the number of execution nodes or pipeline lanes.
  • This draft intentionally keeps the execution change small while a common physical table-write sink remains future work.

Tests

  • Unit Test
    • CARGO_BUILD_JOBS=2 env REGENERATE_GOLDENFILES=1 cargo test -p databend-common-functions --test it test_hash
    • CARGO_BUILD_JOBS=2 cargo check -p databend-query
  • Logic Test - coverage added in 09_0054_partition_by.test, execution pending
  • Benchmark Test
  • No Test - Pair with the reviewer to explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

KKould added 20 commits July 13, 2026 22:12
# Conflicts:
#	src/query/ast/src/parser/statement.rs
#	src/query/service/src/interpreters/interpreter_table_rename_column.rs
#	src/query/service/src/interpreters/interpreter_table_show_create.rs
#	src/query/sql/src/planner/binder/ddl/table.rs
#	src/query/storages/common/table_meta/src/table/table_keys.rs
#	src/query/storages/fuse/src/fuse_table.rs
#	src/query/storages/fuse/src/operations/append.rs
#	src/query/storages/fuse/src/operations/common/meta/mutation_log.rs
#	src/query/storages/fuse/src/operations/common/processors/transform_block_writer.rs
#	src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs
#	src/query/storages/fuse/src/operations/common/processors/transform_serialize_block.rs
#	src/query/storages/fuse/src/operations/recluster.rs
#	src/query/storages/fuse/src/pruning/fuse_pruner.rs
# Conflicts:
#	src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs
- sqllogictest: verify SET OPTIONS partition_by rejection, MODIFY COLUMN
  partition-key rejection, unrelated-filter no-pruning, DELETE+compact
  boundary preservation, and CTAS partition boundary on initial write
- unit: partition_values/same_partition fallback cases (None stats,
  mismatched key id, min!=max prefix)
- unit: PartitionPruner::should_keep returns true conservatively when
  segment has no cluster statistics
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jul 28, 2026
@KKould

KKould commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

Superseded by #20216, recreated from the updated main branch with only the hash-distributed partition write commit.

@KKould KKould closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant