docs: Rewrite supported expressions page to show complete overview of what is and is not supported by Comet (#4550)

Author: andygrove

docs/source/contributor-guide/adding_a_new_expression.md

@@ -27,6 +27,10 @@ Before you start, have a look through [these slides](https://docs.google.com/pre
 
 You may have a specific expression in mind that you'd like to add, but if not, you can review the [expression coverage document](spark_expressions_support.md) to see which expressions are not yet supported.
 
+When you add or change an expression, update **both** the coverage checklist
+(`spark_expressions_support.md`) and the user-facing status in
+[Supported Spark Expressions](../user-guide/latest/expressions.md).
+
 ## Implementing the Expression
 
 Once you have the expression you'd like to add, you should take inventory of the following:

docs/source/contributor-guide/spark_expressions_support.md

@@ -106,9 +106,9 @@
 - [ ] percentile_approx
 - [ ] percentile_cont
 - [ ] percentile_disc
-- [ ] regr_avgx
-- [ ] regr_avgy
-- [ ] regr_count
+- [x] regr_avgx
+- [x] regr_avgy
+- [x] regr_count
 - [ ] regr_intercept
 - [ ] regr_r2
 - [ ] regr_slope
@@ -243,7 +243,7 @@
   - Spark 4.1.1 (audited 2026-05-27): `inputTypes` tightened to `Seq(ArrayType, IntegralType)` (analysis-time only); runtime unchanged.
 - [ ] sequence
 - [ ] shuffle
-- [ ] slice
+- [x] slice
 - [x] sort_array
   - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
   - Spark 3.5.8 (audited 2026-05-27): baseline. `SortArray(base, ascendingOrder) extends BinaryExpression with ArraySortLike`; the second arg must be a `Literal(_: Boolean, BooleanType)`. Comet `CometSortArray` flags `Incompatible` under strict floating-point and falls back for nested arrays whose innermost element is `Struct` or `Null`.
@@ -267,7 +267,7 @@
   - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
   - Spark 4.0.1 (audited 2026-05-27): `>>` parses to `ShiftRight`. Comet `CometShiftRight` mirrors the same operand-cast logic as `CometShiftLeft`.
   - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [ ] `>>>`
+- [x] `>>>`
 - [x] `^`
   - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
   - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseXor(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseXor` to the proto's `bitwise_xor` binary expression.
@@ -311,8 +311,9 @@
 
 ### collection_funcs
 
-- [ ] array_size
-- [ ] cardinality
+- [x] array_size
+  - Native via `size`; returns -1 instead of NULL for NULL input (https://github.com/apache/datafusion-comet/issues/4560).
+- [x] cardinality
 - [x] concat
   - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
   - Spark 3.5.8 (audited 2026-05-27): baseline. `Concat(children) extends ComplexTypeMergingExpression with QueryErrorsBase`; `allowedTypes = Seq(StringType, BinaryType, ArrayType)`; result type is the merged child type. Empty children is allowed and returns the empty string of the result type.
@@ -355,7 +356,7 @@
   - Spark 3.5.8 (audited 2026-05-27): `NullIf(left, right, replacement) extends RuntimeReplaceable with InheritAnalysisRules`; the analyzer rewrites to `If(EqualTo(left, right), Literal(null, left.dataType), left)`. Comet handles via `CometIf` plus `CometEqualTo`.
   - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
   - Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
-- [ ] nullifzero
+- [x] nullifzero
 - [x] nvl
   - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
   - Spark 3.5.8 (audited 2026-05-27): `Nvl(left, right, replacement) extends RuntimeReplaceable`; analyzer rewrites to `Coalesce(Seq(left, right))`. Comet handles via `CometCoalesce`.
@@ -371,7 +372,7 @@
   - Spark 3.5.8 (audited 2026-05-27): the `CASE WHEN ... THEN ...` SQL form lowers to `CaseWhen(branches: Seq[(Expression, Expression)], elseValue: Option[Expression])`. Spark evaluates left-to-right with short-circuit; result type is the merged branch type. Comet routes via `CometCaseWhen` to the native `CaseWhen` proto.
   - Spark 4.0.1 (audited 2026-05-27): adds the `withNewAlwaysEvaluatedInputs` optimizer hook; semantics unchanged.
   - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [ ] zeroifnull
+- [x] zeroifnull
 
 ### conversion_funcs
 
@@ -486,7 +487,8 @@
   - Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
 - [x] trunc
 - [ ] try_make_interval
-- [ ] try_make_timestamp
+- [x] try_make_timestamp
+  - Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).
 - [ ] try_to_date
 - [ ] try_to_time
 - [ ] try_to_timestamp
@@ -505,13 +507,13 @@
 
 - [x] explode
   - Handled at the operator level as a `GenerateExec` (`CometExplodeExec`), not via the expression serde maps, so it is not auto-detected by the function-registry checkbox logic. Compatible for array inputs; map inputs fall back ([#2837](https://github.com/apache/datafusion-comet/issues/2837)).
-- [ ] explode_outer
+- [x] explode_outer
   - Same `CometExplodeExec` path as `explode`, but the `outer=true` case is `Incompatible` (empty arrays are not preserved as null outputs) and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
 - [ ] inline
 - [ ] inline_outer
 - [x] posexplode
   - Handled at the operator level as a `GenerateExec` (`CometExplodeExec`), like `explode`. Compatible for array inputs; map inputs fall back ([#2837](https://github.com/apache/datafusion-comet/issues/2837)).
-- [ ] posexplode_outer
+- [x] posexplode_outer
   - Same `CometExplodeExec` path as `posexplode`, but the `outer=true` case is `Incompatible` and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
 - [ ] stack
 
@@ -556,7 +558,8 @@
 
 ### json_funcs
 
-- [ ] from_json
+- [x] from_json
+  - Partial native support, marked `Incompatible` (requires explicit schema).
 - [x] get_json_object
   - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
   - Spark 3.5.8 (audited 2026-05-27): baseline. `BinaryExpression with ExpectsInputTypes with CodegenFallback`; `inputTypes = Seq(StringType, StringType) -> StringType`. Eval is inline and uses Jackson with `RawStyle` output. Foldable paths are parsed once. Returns NULL for invalid JSON, missing paths, or `JsonProcessingException`.
@@ -567,7 +570,8 @@
 - [ ] json_object_keys
 - [ ] json_tuple
 - [ ] schema_of_json
-- [ ] to_json
+- [x] to_json
+  - Partial native support; options and map/array inputs fall back.
 
 ### lambda_funcs
 
@@ -629,7 +633,7 @@
   - Spark 3.5.8 (audited 2026-05-27): baseline. `StringToMap(text, pairDelim, keyValueDelim) extends TernaryExpression`; splits `text` on `pairDelim`, then each pair on `keyValueDelim` (default `","` and `":"`). Uses `ArrayBasedMapBuilder` for duplicate-key handling. Wired as `CometScalarFunction("str_to_map")`.
   - Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeNonCSAICollation`; uses `CollationAwareUTF8String.splitSQL` with a `collationId`. Runtime unchanged for `UTF8_BINARY`.
   - Spark 4.1.1 (audited 2026-05-27): adds the `legacySplitTruncate` flag (driven by `spark.sql.legacy.truncateForEmptyRegexSplit`) to both `splitSQL` calls (https://github.com/apache/datafusion-comet/issues/4477). The Comet native impl does not honour this flag; behaviour matches the non-legacy default.
-- [ ] try_element_at
+- [x] try_element_at
 
 ### math_funcs
 
@@ -681,7 +685,8 @@
   - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.toDegrees, "DEGREES")` unchanged across versions.
 - [x] div
   - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `IntegralDivide(left, right, evalMode)`. Non-decimal operands are cast to `DecimalType(19, 0)`; result is recomputed per `IntegralDivide.resultDecimalType`, wrapped in `CheckOverflow`, then cast to `Long`. ANSI overflow for `Long.MinValue div -1` and decimal-overflow ANSI cases are covered by existing tests.
-- [ ] e
+- [x] e
+  - Foldable; rewritten to a literal by ConstantFolding (like `pi`).
 - [x] exp
   - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(StrictMath.exp, "EXP")` unchanged. ULP-level differences vs DataFusion `exp` are possible but unflagged.
 - [x] expm1
@@ -729,7 +734,7 @@
   - See `misc_funcs / rand`.
 - [x] randn
   - See `misc_funcs / randn`.
-- [ ] random
+- [x] random
 - [ ] randstr
 - [x] rint
   - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.rint, "ROUND")` with `funcName = "rint"`. Passthrough to DataFusion `rint` (round-half-to-even).
@@ -764,7 +769,7 @@
   - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to `Subtract(.., EvalMode.TRY)`. Integer path uses `checked_sub`; decimal uses `WideDecimalBinaryExpr` as needed.
 - [x] unhex
   - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Unhex(child, failOnError)`. Spark 4.x widens input to `StringTypeWithCollation` and wraps the inner call in try/catch; Comet `CometUnhex` forwards `failOnError` to native `spark_unhex` but does not gate on collation.
-- [ ] uniform
+- [x] uniform
 - [x] width_bucket
   - Spark 3.5.8 (audited 2026-05-27): introduced; not available in 3.4.3.
   - Spark 4.0.1, 4.1.1 (audited 2026-05-27): same semantics; `NullIntolerant` -> `nullIntolerant: Boolean` refactor.
@@ -782,11 +787,15 @@
 - [ ] bitmap_construct_agg
 - [ ] bitmap_count
 - [ ] bitmap_or_agg
-- [ ] current_catalog
-- [ ] current_database
-- [ ] current_schema
-- [ ] current_user
-- [ ] equal_null
+- [x] current_catalog
+  - Resolved to a literal by the analyzer (`ReplaceCurrentLike`).
+- [x] current_database
+  - Resolved to a literal by the analyzer (`ReplaceCurrentLike`).
+- [x] current_schema
+  - Alias of `current_database`; resolved to a literal by the analyzer.
+- [x] current_user
+  - Resolved to a literal by the analyzer; same as `user`.
+- [x] equal_null
 - [ ] from_avro
 - [ ] from_protobuf
 - [ ] hll_sketch_estimate
@@ -819,7 +828,8 @@
 - [ ] schema_of_avro
 - [ ] schema_of_variant
 - [ ] schema_of_variant_agg
-- [ ] session_user
+- [x] session_user
+  - Alias of `current_user`; resolved to a literal by the analyzer.
 - [x] spark_partition_id
   - Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `SparkPartitionID() extends LeafExpression with Nondeterministic`; returns the integer index of the partition being processed. Comet emits an empty `SparkPartitionId` proto.
   - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
@@ -841,7 +851,8 @@
 - [ ] try_parse_json
 - [ ] try_reflect
 - [ ] try_variant_get
-- [ ] typeof
+- [x] typeof
+  - Foldable; resolved to a literal before Comet sees the plan.
 - [x] user
   - Spark 3.4.3 (audited 2026-05-27): `CurrentUser() extends LeafExpression with Unevaluable`; the analyzer's `ResolveCurrentLike` rule replaces it with a `StringType` literal of the current user name before Comet sees the plan. No Comet serde needed; the literal flows through `CometLiteral`.
   - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
@@ -940,8 +951,8 @@
   - Spark 3.5.8 (audited 2026-05-27): baseline. `Or(left, right) extends BinaryOperator with Predicate`; short-circuit left-to-right. Comet routes via `CometOr` to the proto's `or` binary expression.
   - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
   - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [ ] regexp
-- [ ] regexp_like
+- [x] regexp
+- [x] regexp_like
 - [x] rlike
   - See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR #4461). Uses the Rust `regex` crate, which differs from Java's `Pattern` engine; requires `spark.comet.expression.regexp.allowIncompatible=true`.
 
@@ -956,7 +967,7 @@
 - [x] character_length
 - [x] chr
 - [ ] collate
-- [ ] collation
+- [x] collation
 - [x] concat_ws
 - [x] contains
 - [x] decode
@@ -978,7 +989,7 @@
 - [x] lower
 - [x] lpad
 - [x] ltrim
-- [ ] luhn_check
+- [x] luhn_check
 - [ ] make_valid_utf8
 - [ ] mask
 - [x] octet_length
@@ -1006,7 +1017,7 @@
 - [x] substr
 - [x] substring
 - [x] substring_index
-- [ ] to_binary
+- [x] to_binary
 - [ ] to_char
 - [ ] to_number
 - [ ] to_varchar
@@ -1052,8 +1063,10 @@
 
 - [ ] cume_dist
 - [ ] dense_rank
-- [ ] lag
-- [ ] lead
+- [x] lag
+  - Supported via `CometWindowExec` (operator level), not the expression serde.
+- [x] lead
+  - Supported via `CometWindowExec` (operator level), not the expression serde.
 - [ ] nth_value
 - [ ] ntile
 - [ ] percent_rank