Skip to content

Commit 5fa8dc7

Browse files
micheal-oanishshri-db
authored andcommitted
[SPARK-56021][SS] Increase AutoSnapshotRepair default maxChangeFileReplay threshold from 50 to 500
### What changes were proposed in this pull request? Increasing the default from 50 to 500. ### Why are the changes needed? 50 is too conservative. Users would prefer to spend more time repairing the pipeline than having the pipeline completely down. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing tests ### Was this patch authored or co-authored using generative AI tooling? Claude-4.5-opus Closes #54843 from micheal-o/auto_repair_max_changefile_increase. Authored-by: micheal-o <micheal.okutubo@gmail.com> Signed-off-by: Anish Shrigondekar <anish.shrigondekar@databricks.com>
1 parent 523921a commit 5fa8dc7

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

  • sql/catalyst/src/main/scala/org/apache/spark/sql/internal

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2717,7 +2717,7 @@ object SQLConf {
27172717
.version("4.1.0")
27182718
.intConf
27192719
.checkValue(k => k > 0, "Must allow at least 1 change file replay")
2720-
.createWithDefault(50)
2720+
.createWithDefault(500)
27212721

27222722
val STATE_STORE_INSTANCE_METRICS_REPORT_LIMIT =
27232723
buildConf("spark.sql.streaming.stateStore.numStateStoreInstanceMetricsToReport")

0 commit comments

Comments
 (0)