feat: under engine=rust, fall through to JVM dispatcher for unimplemented regex

Previously engine=rust returned Spark fallback for regexp_extract,
regexp_extract_all, and regexp_instr because they have no native Rust
path. With the codegen dispatcher now enabled by default, prefer the
JVM dispatcher over Spark in that case so users still get Comet
acceleration with full Spark semantics. Only decline to native and
dispatcher are both unavailable.

Also document the per-expression
spark.comet.expression.<ClassName>.enabled disable knobs in the regex
compatibility guide, and add a regression test that exercises the new
rust→JVM fallthrough.

Author: andygrove

docs/source/user-guide/latest/compatibility/regex.md

@@ -30,19 +30,36 @@ expression inside Comet's Arrow-direct codegen dispatcher (the same dispatcher u
 - `rust` — run the Rust engine when an expression has a native implementation. Setting this is itself
   the opt-in for the semantic differences between Java and Rust regex (no separate `allowIncompatible`
   flag needed). Expressions without a native Rust implementation (`regexp_extract`,
-  `regexp_extract_all`, `regexp_instr`) fall back to Spark.
+  `regexp_extract_all`, `regexp_instr`) fall through to the Java engine so users still get Comet
+  acceleration with full Spark semantics.
 
 With pure defaults (`engine=java`, `scalaUDF.codegen.enabled=true`), all regex expressions run on
 the Comet path with full Spark compatibility.
 
+## Disabling Comet for individual regex expressions
+
+Each regex expression has a per-class `spark.comet.expression.<ClassName>.enabled` flag (default
+`true`) that disables Comet's serde for that expression and forces a Spark fallback. This is
+useful for narrowing a regression or comparing performance on a single operator without changing
+the engine selector:
+
+| Expression          | Config                                                   |
+| ------------------- | -------------------------------------------------------- |
+| `rlike`             | `spark.comet.expression.RLike.enabled=false`             |
+| `regexp_extract`    | `spark.comet.expression.RegExpExtract.enabled=false`     |
+| `regexp_extract_all`| `spark.comet.expression.RegExpExtractAll.enabled=false`  |
+| `regexp_instr`      | `spark.comet.expression.RegExpInStr.enabled=false`       |
+| `regexp_replace`    | `spark.comet.expression.RegExpReplace.enabled=false`     |
+| `split`             | `spark.comet.expression.StringSplit.enabled=false`       |
+
 ## Choosing an engine
 
-|                      | Rust engine                             | Java engine (default)                                                                                               |
-| -------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
-| **Compatibility**    | Differs from Java regex (see below)     | 100% compatible with Spark                                                                                          |
-| **Feature coverage** | `rlike`, `regexp_replace`, `split` only | All regexp expressions (`rlike`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`, `regexp_replace`, `split`) |
-| **Performance**      | Fully native, no JNI overhead           | One JNI round-trip per batch (Arrow vectors stay columnar)                                                          |
-| **Pattern support**  | Linear-time subset only                 | All Java regex features (backreferences, lookaround, etc.)                                                          |
+|                      | Rust engine                                                                                                       | Java engine (default)                                                                                               |
+| -------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
+| **Compatibility**    | Differs from Java regex (see below)                                                                               | 100% compatible with Spark                                                                                          |
+| **Feature coverage** | `rlike`, `regexp_replace`, `split` natively; `regexp_extract`, `regexp_extract_all`, `regexp_instr` via fallthrough | All regexp expressions (`rlike`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`, `regexp_replace`, `split`) |
+| **Performance**      | Fully native, no JNI overhead                                                                                     | One JNI round-trip per batch (Arrow vectors stay columnar)                                                          |
+| **Pattern support**  | Linear-time subset only                                                                                           | All Java regex features (backreferences, lookaround, etc.)                                                          |
 
 The **Rust engine** is faster but cannot match Java regex semantics for every pattern. Because the engine
 choice is itself the opt-in, setting `spark.comet.exec.regexp.engine=rust` declares acceptance of those

spark/src/main/scala/org/apache/comet/CometConf.scala

@@ -388,9 +388,10 @@ object CometConf extends ShimCometConf {
           s"${COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false. `$REGEXP_ENGINE_RUST` runs the " +
           "native DataFusion regexp engine when an implementation exists; setting this is " +
           "itself the opt-in for the semantic differences between Java and Rust regex. " +
-          "Affected expressions: rlike, regexp_extract, regexp_extract_all, regexp_replace, " +
-          "regexp_instr, and split (the extract/instr family has no native Rust path; they " +
-          s"fall back to Spark under `$REGEXP_ENGINE_RUST`).")
+          "Expressions without a native Rust implementation (`regexp_extract`, " +
+          "`regexp_extract_all`, `regexp_instr`) fall through to the JVM codegen dispatcher " +
+          s"under `$REGEXP_ENGINE_RUST` so users still get Comet acceleration with full " +
+          "Spark semantics.")
       .stringConf
       .transform(_.toLowerCase(Locale.ROOT))
       .checkValues(Set(REGEXP_ENGINE_RUST, REGEXP_ENGINE_JAVA))

spark/src/main/scala/org/apache/comet/serde/strings.scala

@@ -64,7 +64,9 @@ private object RegexpRoute {
    * Pick a route given the user's config and whether a native Rust implementation exists for the
    * expression. `engine=java` (default) routes through the codegen dispatcher when
    * [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]] is true (also the default); otherwise Spark
-   * fallback. `engine=rust` runs native if available; else Spark fallback.
+   * fallback. `engine=rust` runs native if available, otherwise transparently falls back to the
+   * JVM codegen dispatcher when it is enabled, and only declines (Spark fallback) when neither
+   * the native path nor the dispatcher can serve the expression.
    */
   def choose(exprName: String, hasNative: Boolean): RegexpRoute = {
     val engine = CometConf.COMET_REGEXP_ENGINE.get()
@@ -74,12 +76,13 @@ private object RegexpRoute {
       case CometConf.REGEXP_ENGINE_RUST =>
         if (hasNative) {
           Native
+        } else if (codegenEnabled) {
+          JvmCodegen
         } else {
           Fallback(
-            s"$exprName has no native Rust implementation. Set " +
-              s"${CometConf.COMET_REGEXP_ENGINE.key}=${CometConf.REGEXP_ENGINE_JAVA} with " +
-              s"${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true to route through the " +
-              "codegen dispatcher.")
+            s"$exprName has no native Rust implementation and " +
+              s"${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false, so neither the rust " +
+              "engine nor the JVM codegen dispatcher can serve this expression.")
         }
 
       case CometConf.REGEXP_ENGINE_JAVA =>

spark/src/test/scala/org/apache/comet/CometRegExpJvmSuite.scala

@@ -390,4 +390,18 @@ class CometRegExpJvmSuite extends CometTestBase with AdaptiveSparkPlanHelper {
           |""".stripMargin))
     }
   }
+
+  test("engine=rust falls through to JVM dispatcher for expressions with no native path") {
+    withSQLConf(CometConf.COMET_REGEXP_ENGINE.key -> CometConf.REGEXP_ENGINE_RUST) {
+      withSubjects("abc123def", "no match", null, "xyz789") {
+        // regexp_extract / regexp_extract_all / regexp_instr have no native rust path; under
+        // engine=rust they should still run on Comet via the JVM codegen dispatcher rather than
+        // falling back to Spark.
+        checkSparkAnswerAndOperator(
+          sql("SELECT s, regexp_extract(s, '([a-z]+)(\\\\d+)', 2) FROM t"))
+        checkSparkAnswerAndOperator(sql("SELECT s, regexp_extract_all(s, '\\\\d+', 0) FROM t"))
+        checkSparkAnswerAndOperator(sql("SELECT s, regexp_instr(s, '\\\\d+', 0) FROM t"))
+      }
+    }
+  }
 }