File tree Expand file tree Collapse file tree
common/src/main/scala/org/apache/comet
docs/source/user-guide/latest Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -400,6 +400,9 @@ object CometConf extends ShimCometConf {
400400 " which can improve performance for wide tables while still providing " +
401401 " reasonable distribution." )
402402 .intConf
403+ .checkValue(
404+ v => v >= 0 ,
405+ " The maximum number of columns to hash for round robin partitioning must be non-negative." )
403406 .createWithDefault(0 )
404407
405408 val COMET_EXEC_SHUFFLE_COMPRESSION_CODEC : ConfigEntry [String ] =
Original file line number Diff line number Diff line change @@ -87,7 +87,8 @@ impossible to match Spark's exact partition assignments.
8787Instead of true round-robin assignment, Comet implements round-robin as hash partitioning on ALL columns. This
8888achieves the same semantic goals:
8989
90- - ** Even distribution** : Rows are distributed evenly across partitions
90+ - ** Even distribution** : Rows are distributed evenly across partitions (as long as the hash varies sufficiently -
91+ in some cases there could be skew)
9192- ** Deterministic** : Same input always produces the same partition assignments (important for fault tolerance)
9293- ** No semantic grouping** : Unlike hash partitioning on specific columns, this doesn't group related rows together
9394
You can’t perform that action at this time.
0 commit comments