Skip to content

json_funcs.md

Latest commit

 

History

History
46 lines (33 loc) · 3.01 KB

json_funcs.md

File metadata and controls

46 lines (33 loc) · 3.01 KB

json_funcs Expression Audits

Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide Spark Expression Support for current support status.

from_json

  • Partial native support, marked Incompatible (requires explicit schema).

get_json_object

  • Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
  • Spark 3.5.8 (audited 2026-05-27): baseline. BinaryExpression with ExpectsInputTypes with CodegenFallback; inputTypes = Seq(StringType, StringType) -> StringType. Eval is inline and uses Jackson with RawStyle output. Foldable paths are parsed once. Returns NULL for invalid JSON, missing paths, or JsonProcessingException.
  • Spark 4.0.1 (audited 2026-05-27): the eval is extracted into a GetJsonObjectEvaluator helper (no behaviour change). The trait set now mixes in DefaultStringProducingExpression, and inputTypes is widened to StringTypeWithCollation(supportsTrimCollation = true) for both arguments.
  • Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
  • Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet's native parser (built on serde_json) rejects both, so those inputs require spark.comet.expression.GetJsonObject.allowIncompatible=true and may still produce different results. Non-default Spark 4.0 string collations are not propagated (apache#2190).

json_array_length

  • LengthOfJsonArray: UnaryExpression with ExpectsInputTypes with CodegenFallback; inputTypes = Seq(StringType) -> IntegerType. Returns NULL for NULL input, invalid JSON, or non-array JSON; otherwise the number of top-level array elements.
  • Runs through the codegen dispatcher by default for byte-exact Spark compatibility.
  • Known incompatibility: the native path (built on serde_json) requires strict JSON, so single-quoted JSON, unescaped control characters, and trailing content require spark.comet.expression.LengthOfJsonArray.allowIncompatible=true and may still produce different results.

to_json

  • Partial native support; options and map/array inputs fall back.