revert: default JVM UDF codegen dispatcher back to disabled

Flip spark.comet.exec.scalaUDF.codegen.enabled back to false and restore
the experimental, disabled-by-default language across the regex/codegen
docs and CometConf strings. With this default, the regex family (java
engine path) and the DateFormat dispatcher fall back to Spark unless the
user explicitly opts in.

This keeps the engine=rust JVM-dispatcher fallthrough behavior introduced
separately on this branch; only the codegen-enabled-by-default change is
reverted.

Author: andygrove

docs/source/user-guide/latest/compatibility/regex.md

@@ -20,21 +20,21 @@ under the License.
 # Regular Expressions
 
 Comet provides two regexp engines for evaluating regular expressions: a **Rust engine** that uses the Rust
-[`regex`] crate natively, and a **Java engine** that runs Spark's own `doGenCode` for the
+[`regex`] crate natively, and an experimental **Java engine** that runs Spark's own `doGenCode` for the
 expression inside Comet's Arrow-direct codegen dispatcher (the same dispatcher used by Comet's
 `ScalaUDF` codegen path). The engine is selected with `spark.comet.exec.regexp.engine`, which accepts:
 
-- `java` (default) — route through the Java engine for full Spark compatibility. Uses the codegen
-  dispatcher (`spark.comet.exec.scalaUDF.codegen.enabled`, also default `true`); if that flag is
-  disabled, regex expressions fall back to Spark with an explanatory message.
+- `java` (default) — route through the Java engine for full Spark compatibility. Requires
+  `spark.comet.exec.scalaUDF.codegen.enabled=true`; otherwise regex expressions fall back to Spark with
+  an explanatory message.
 - `rust` — run the Rust engine when an expression has a native implementation. Setting this is itself
   the opt-in for the semantic differences between Java and Rust regex (no separate `allowIncompatible`
   flag needed). Expressions without a native Rust implementation (`regexp_extract`,
   `regexp_extract_all`, `regexp_instr`) fall through to the Java engine so users still get Comet
   acceleration with full Spark semantics.
 
-With pure defaults (`engine=java`, `scalaUDF.codegen.enabled=true`), all regex expressions run on
-the Comet path with full Spark compatibility.
+The codegen dispatcher is experimental and disabled by default. With pure defaults
+(`engine=java`, `scalaUDF.codegen.enabled=false`), all regex expressions fall back to Spark.
 
 ## Disabling Comet for individual regex expressions
 
@@ -54,7 +54,7 @@ the engine selector:
 
 ## Choosing an engine
 
-|                      | Rust engine                                                                                                         | Java engine (default)                                                                                               |
+|                      | Rust engine                                                                                                         | Java engine (experimental, default)                                                                                 |
 | -------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
 | **Compatibility**    | Differs from Java regex (see below)                                                                                 | 100% compatible with Spark                                                                                          |
 | **Feature coverage** | `rlike`, `regexp_replace`, `split` natively; `regexp_extract`, `regexp_extract_all`, `regexp_instr` via fallthrough | All regexp expressions (`rlike`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`, `regexp_replace`, `split`) |
@@ -65,9 +65,9 @@ The **Rust engine** is faster but cannot match Java regex semantics for every pa
 choice is itself the opt-in, setting `spark.comet.exec.regexp.engine=rust` declares acceptance of those
 differences without a separate per-expression flag.
 
-The **Java engine** is the default. It is gated behind `spark.comet.exec.scalaUDF.codegen.enabled`
-(also default `true`) so the codegen dispatcher can be disabled globally without changing the regex
-engine selector.
+The **Java engine** is the default but the underlying codegen dispatcher is experimental and gated behind
+`spark.comet.exec.scalaUDF.codegen.enabled=true`; the behavior, configuration, and supported expressions
+may change in future releases.
 
 ## Why the engines differ
 
@@ -129,7 +129,7 @@ shape and want to avoid the JNI overhead of the Java engine, switching to the Ru
 `allowIncompatible=true` is generally safe.
 
 For anything that uses backreferences, lookaround, or relies on Java's specific Unicode or line-handling
-defaults, use the Java engine.
+defaults, use the experimental Java engine.
 
 [`java.util.regex`]: https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
 [`regex`]: https://docs.rs/regex/latest/regex/

docs/source/user-guide/latest/iceberg.md

@@ -157,12 +157,12 @@ Iceberg ships several `ScalaUDF`s that surface in user queries and maintenance a
   (`INT_ORDERED_BYTES`, `LONG_ORDERED_BYTES`, ..., `INTERLEAVE_BYTES`) over the sort key columns
   during compaction.
 
-[Scala UDF and Java UDF Support](scala_java_udfs.md)
-(`spark.comet.exec.scalaUDF.codegen.enabled=true`, the default) routes these UDFs through native
-execution so the project, exchange, and sort operators around them stay on the Comet path
-end-to-end. Disabling the flag causes the enclosing operator to fall back to Spark, which forces
-a columnar-to-row roundtrip and demotes the surrounding shuffle from `CometExchange` to
-`CometColumnarExchange`.
+By default these UDFs cause the enclosing operator to fall back to Spark, which forces a
+columnar-to-row roundtrip and demotes the surrounding shuffle from `CometExchange` to
+`CometColumnarExchange`. Enabling the experimental
+[Scala UDF and Java UDF Support](scala_java_udfs.md) feature
+(`spark.comet.exec.scalaUDF.codegen.enabled=true`) routes these UDFs through native execution so
+the project, exchange, and sort operators around them stay on the Comet path end-to-end.
 
 ### Task input metrics
 

docs/source/user-guide/latest/scala_java_udfs.md

@@ -23,11 +23,13 @@ Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](htt
 
 This page covers Spark's `ScalaUDF` (Scala `udf(...)`, `spark.udf.register(...)` over Scala or Java functional interfaces, and SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`). Other UDF kinds (Python / Pandas, Hive, aggregate) are out of scope and continue to fall back to Spark.
 
+This feature is experimental and disabled by default.
+
 ## Configuration
 
 | Key                                         | Default | Description                                                                                                        |
 | ------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------ |
-| `spark.comet.exec.scalaUDF.codegen.enabled` | `true`  | When `true`, eligible `ScalaUDF`s run on the Comet path. When `false`, the enclosing operator falls back to Spark. |
+| `spark.comet.exec.scalaUDF.codegen.enabled` | `false` | When `true`, eligible `ScalaUDF`s run on the Comet path. When `false`, the enclosing operator falls back to Spark. |
 
 ## Supported
 

spark/src/main/scala/org/apache/comet/CometConf.scala

@@ -365,14 +365,14 @@ object CometConf extends ShimCometConf {
   val COMET_SCALA_UDF_CODEGEN_ENABLED: ConfigEntry[Boolean] =
     conf("spark.comet.exec.scalaUDF.codegen.enabled")
       .category(CATEGORY_EXEC)
-      .doc("Whether to route Spark `ScalaUDF` expressions through Comet's Arrow-direct codegen " +
-        "dispatcher. When enabled (the default), a supported ScalaUDF is compiled into a " +
-        "per-batch kernel that reads and writes Arrow vectors directly from native execution. " +
-        "When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing " +
-        "operator. The same dispatcher backs `spark.comet.exec.regexp.engine=java` so the " +
-        "regex family routes through it as well.")
+      .doc("Experimental. Whether to route Spark `ScalaUDF` expressions through Comet's " +
+        "Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into " +
+        "a per-batch kernel that reads and writes Arrow vectors directly from native " +
+        "execution. When disabled, plans containing a ScalaUDF fall back to Spark for the " +
+        "enclosing operator. The same dispatcher backs `spark.comet.exec.regexp.engine=java` " +
+        "so the regex family routes through it as well.")
       .booleanConf
-      .createWithDefault(true)
+      .createWithDefault(false)
 
   val REGEXP_ENGINE_RUST = "rust"
   val REGEXP_ENGINE_JAVA = "java"
@@ -384,8 +384,8 @@ object CometConf extends ShimCometConf {
         "Selects the engine used to evaluate Spark regular-expression expressions. " +
           s"`$REGEXP_ENGINE_JAVA` (default) routes through the Arrow-direct codegen dispatcher " +
           "so Spark's own `doGenCode` (backed by `java.util.regex.Pattern`) runs inside the " +
-          "Comet pipeline; this falls back to Spark when " +
-          s"${COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false. `$REGEXP_ENGINE_RUST` runs the " +
+          s"Comet pipeline; this requires ${COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true and " +
+          s"falls back to Spark otherwise. `$REGEXP_ENGINE_RUST` runs the " +
           "native DataFusion regexp engine when an implementation exists; setting this is " +
           "itself the opt-in for the semantic differences between Java and Rust regex. " +
           "Expressions without a native Rust implementation (`regexp_extract`, " +

spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala

@@ -43,8 +43,8 @@ import org.apache.comet.udf.codegen.CometScalaUDFCodegen
  *   - Python / Pandas UDFs.
  *   - Hive `GenericUDF` / `SimpleUDF`.
  *
- * Gated by [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]] (default `true`). When disabled, plans
- * containing a `ScalaUDF` fall back to Spark for the enclosing operator.
+ * Gated by [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]]. When disabled, plans containing a
+ * `ScalaUDF` fall back to Spark for the enclosing operator.
  *
  * [[emitJvmCodegenDispatch]] exposes the same closure-serialize + dispatcher-proto path to other
  * serdes that want to keep a built-in Spark expression inside the Comet pipeline when no native

spark/src/main/scala/org/apache/comet/serde/datetime.scala

@@ -653,8 +653,9 @@ object CometDateFormat extends CometExpressionSerde[DateFormatClass] {
     "Format strings in a curated allow-list run natively via DataFusion's `to_char` for UTC " +
       "sessions. Other format strings (including non-literal formats), as well as non-UTC " +
       "sessions, route through Spark's own `DateFormatClass.doGenCode` via the Arrow-direct " +
-      "codegen dispatcher (default). Set `spark.comet.exec.scalaUDF.codegen.enabled=false` to " +
-      "fall back to Spark for those cases instead.")
+      "codegen dispatcher when `spark.comet.exec.scalaUDF.codegen.enabled=true`. When the " +
+      "codegen dispatcher is disabled (default) the operator falls back to Spark in those " +
+      "cases.")
 
   override def convert(
       expr: DateFormatClass,

spark/src/main/scala/org/apache/comet/serde/strings.scala

@@ -63,10 +63,10 @@ private object RegexpRoute {
   /**
    * Pick a route given the user's config and whether a native Rust implementation exists for the
    * expression. `engine=java` (default) routes through the codegen dispatcher when
-   * [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]] is true (also the default); otherwise Spark
-   * fallback. `engine=rust` runs native if available, otherwise transparently falls back to the
-   * JVM codegen dispatcher when it is enabled, and only declines (Spark fallback) when neither
-   * the native path nor the dispatcher can serve the expression.
+   * [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]] is true; otherwise Spark fallback.
+   * `engine=rust` runs native if available, otherwise transparently falls back to the JVM codegen
+   * dispatcher when it is enabled, and only declines (Spark fallback) when neither the native
+   * path nor the dispatcher can serve the expression.
    */
   def choose(exprName: String, hasNative: Boolean): RegexpRoute = {
     val engine = CometConf.COMET_REGEXP_ENGINE.get()
@@ -91,7 +91,8 @@ private object RegexpRoute {
         } else {
           Fallback(
             s"$exprName requires ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true when " +
-              s"${CometConf.COMET_REGEXP_ENGINE.key}=${CometConf.REGEXP_ENGINE_JAVA}.")
+              s"${CometConf.COMET_REGEXP_ENGINE.key}=${CometConf.REGEXP_ENGINE_JAVA}. " +
+              "The codegen dispatcher is experimental and disabled by default.")
         }
 
       case other => Fallback(s"Unknown ${CometConf.COMET_REGEXP_ENGINE.key}=$other")

spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala

@@ -99,11 +99,6 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
   /**
    * Pin the sentinel-query convention for fixtures that route an `expect_error` through the
    * codegen dispatcher. See [[ExpectError]] for the failure mode this guards against.
-   *
-   * Fires for fixtures that explicitly opt into the codegen dispatcher via `--CONFIG`. The
-   * dispatcher now defaults to enabled, but most expression fixtures use their own native paths
-   * and do not exercise the dispatcher, so we still key off the explicit opt-in rather than
-   * forcing every fixture in the repo to add a sentinel.
    */
   private def requireSentinelForCodegenExpectError(
       relativePath: String,