Skip to content

Commit e3a5c3b

Browse files
authored
CI: Limit parallelism (apache#1764)
For the tests, we want to limit parallelism to avoid creating 1-row Parquet files.
1 parent c6e6903 commit e3a5c3b

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

tests/conftest.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2433,6 +2433,8 @@ def spark() -> "SparkSession":
24332433
spark = (
24342434
SparkSession.builder.appName("PyIceberg integration test")
24352435
.config("spark.sql.session.timeZone", "UTC")
2436+
.config("spark.sql.shuffle.partitions", "1")
2437+
.config("spark.default.parallelism", "1")
24362438
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
24372439
.config("spark.sql.catalog.integration", "org.apache.iceberg.spark.SparkCatalog")
24382440
.config("spark.sql.catalog.integration.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")

0 commit comments

Comments
 (0)