test: disable Comet shuffle in pyarrow UDF pytest session

CometSparkSessionExtensions.isCometLoaded short-circuits the whole
extension (returning false; no rules registered) when
spark.comet.exec.shuffle.enabled is true but spark.shuffle.manager is
not Comet's manager. The pytest conftest only sets the basic Comet
configs, so this guard fired and CometScanRule never ran. The plan
stayed vanilla Parquet, the rewrite chain never had a Comet columnar
producer to match, and every [accelerated] assertion that checks for
CometMapInBatch failed.

These tests do not exercise shuffle, so disable Comet shuffle in the
session. Comet's scan and exec rules then run normally and the rewrite
fires.

Diagnoses the wholesale PyArrow UDF Spark 4.0 CI failure on #4234.

Author: andygrove

spark/src/test/resources/pyspark/test_pyarrow_udf.py

@@ -63,6 +63,12 @@ def spark():
         .config("spark.plugins", "org.apache.spark.CometPlugin")
         .config("spark.comet.enabled", "true")
         .config("spark.comet.exec.enabled", "true")
+        # spark.comet.exec.shuffle.enabled defaults to true, and
+        # CometSparkSessionExtensions.isCometLoaded refuses to register Comet's rules
+        # at all when shuffle is on but spark.shuffle.manager is not the Comet manager.
+        # These tests do not need Comet shuffle, so disable it explicitly to keep
+        # Comet's scan and exec rules active without configuring shuffle.
+        .config("spark.comet.exec.shuffle.enabled", "false")
         .config("spark.memory.offHeap.enabled", "true")
         .config("spark.memory.offHeap.size", "2g")
         .getOrCreate()