fix: implement Spark-compatible null handling for arrays_overlap

Replace DataFusion's array_has_any (which treats NULL == NULL) with a
custom implementation that follows Spark's three-valued logic:
- true when arrays share a common non-null element
- null when no common non-null elements but either array has nulls
- false when no common elements and no nulls

Closes #3645

Author: andygrove

docs/source/user-guide/latest/compatibility.md

@@ -58,6 +58,18 @@ Expressions that are not 100% Spark-compatible will fall back to Spark by defaul
 `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name. See
 the [Comet Supported Expressions Guide](expressions.md) for more information on this configuration setting.
 
+## Array Functions
+
+### ArraysOverlap
+
+Comet's `arrays_overlap` implementation follows Spark's null handling semantics: when no common non-null elements
+exist but either array contains null elements, the result is `null` rather than `false`. This matches Spark's
+three-valued logic where `arrays_overlap(array(1, null), array(null, 2))` returns `null`.
+
+Comet currently uses `ScalarValue`-based comparison for complex element types (structs, nested arrays), which may
+have subtle differences from Spark's equality semantics for these types. Primitive and string element types use
+native comparisons that match Spark.
+
 ## Regular Expressions
 
 Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's

docs/source/user-guide/latest/expressions.md

@@ -245,7 +245,7 @@ Comet supports using the following aggregate functions within window contexts wi
 | ArrayRemove    | Yes               |                                                                                                                                                                                           |
 | ArrayRepeat    | No                |                                                                                                                                                                                           |
 | ArrayUnion     | No                | Behaves differently than spark. Comet sorts the input arrays before performing the union, while Spark preserves the order of the first array and appends unique elements from the second. |
-| ArraysOverlap  | No                |                                                                                                                                                                                           |
+| ArraysOverlap  | No                | See [ArraysOverlap](compatibility.md#arraysoverlap) in the compatibility guide.                                                                                                           |
 | CreateArray    | Yes               |                                                                                                                                                                                           |
 | ElementAt      | Yes               | Input must be an array. Map inputs are not supported.                                                                                                                                     |
 | Flatten        | Yes               |                                                                                                                                                                                           |