fix(ci): skip Spark 4.1 Python data source probe in local SQL tests

Spark 4.1's DataSourceManager probes for Python data sources during
query analysis by spawning a python3 worker. The CI amd64/rust
container has no python3, so the probe is skipped there. On a developer
machine that has python3 the worker can hang indefinitely, since the
JVM-side read has no idle timeout by default, stalling suites such as
GlobalTempViewSuite.

Point PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON at a nonexistent
interpreter so the probe is skipped, matching CI. The value is
overridable for developers who want to run the Python-dependent suites.

Author: andygrove

dev/ci/spark-sql-tests/README.md

@@ -78,6 +78,15 @@ PASS/FAIL summary is printed at the end.
 | `SPARK_REF`        | `v4.1.1`                                  | Git ref checked out for the Spark sources. |
 | `SBT_MEM`          | `4096`                                    | sbt heap size in MB. |
 | `LC_ALL`           | `C.UTF-8`                                 | Locale for the sbt run. Use `en_US.UTF-8` on macOS if `C.UTF-8` is unavailable. |
+| `PYSPARK_PYTHON`   | a nonexistent path                        | Python interpreter for Spark. The default skips Spark 4.1's Python data source probe, which can hang on machines that have `python3`. Export a real interpreter to run the Python-dependent suites. |
+
+> **Note on Python:** Spark 4.1 probes for Python data sources during query
+> analysis by spawning a Python worker. The CI `amd64/rust` container has no
+> `python3`, so the probe is skipped. On a developer machine that has `python3`
+> the worker can hang indefinitely (the JVM-side read has no idle timeout),
+> stalling suites such as `GlobalTempViewSuite`. `run.sh` therefore points
+> `PYSPARK_PYTHON` / `PYSPARK_DRIVER_PYTHON` at a nonexistent path by default so
+> the probe is skipped, matching CI.
 
 ## How it works
 

dev/ci/spark-sql-tests/run.sh

@@ -48,6 +48,10 @@ Environment variables:
   SPARK_REF           Git ref for the Spark sources (default: v$SPARK_VERSION).
   SBT_MEM             sbt heap size in MB (default: 4096).
   LC_ALL              Locale for the sbt run (default: C.UTF-8; use en_US.UTF-8 on macOS).
+  PYSPARK_PYTHON      Python interpreter for Spark. Defaults to a nonexistent
+                      path so Spark 4.1's Python data source probe is skipped
+                      (it can hang on machines that have python3). Export a
+                      real interpreter to run the Python-dependent suites.
 EOF
 }
 
@@ -139,13 +143,25 @@ for m in "${modules_to_run[@]}"; do
   # Stale Parquet cache workaround (mirrors spark_sql_test.yml).
   rm -rf "$maven_repo/org/apache/parquet"
 
+  # Spark 4.1's DataSourceManager probes for Python data sources during query
+  # analysis by spawning a Python worker. The CI amd64/rust container has no
+  # python3, so the probe is skipped there. On a developer machine that does
+  # have python3 (every macOS install does) the worker can hang indefinitely:
+  # the JVM-side read has no idle timeout by default, so suites such as
+  # GlobalTempViewSuite stall forever instead of failing fast. Point PySpark at
+  # a nonexistent interpreter so the probe is skipped, matching CI. A developer
+  # who wants the Python suites can export PYSPARK_PYTHON themselves.
+  no_python="/nonexistent/comet-disable-python-datasources"
+
   (
     cd "$COMET_SPARK_DIR" || exit 1
     NOLINT_ON_COMPILE=true \
     ENABLE_COMET=true \
     ENABLE_COMET_ONHEAP=true \
     ENABLE_COMET_LOG_FALLBACK_REASONS=false \
     SERIAL_SBT_TESTS=1 \
+    PYSPARK_DRIVER_PYTHON="${PYSPARK_DRIVER_PYTHON:-$no_python}" \
+    PYSPARK_PYTHON="${PYSPARK_PYTHON:-$no_python}" \
       build/sbt -Dsbt.log.noformat=true -mem "$SBT_MEM" \
         'set Global / concurrentRestrictions := Seq(Tags.limit(Tags.ForkedTestGroup, 1))' \
         "$sbt_args"