Skip to content

Commit ca1d470

Browse files
committed
Address review: doc note placement, map/struct nested coverage, keep UDF assert
- De-indent the Hive-serde TIME note to align with other datetime types. - Cover nested TIME in map and struct (not just array) in the directory-write test. - Keep assert/contains for the wrapped Hive UDF error; the inner UNSUPPORTED_DATATYPE has no cause attached on _LEGACY_ERROR_TEMP_3084 (tracked by SPARK-57750). Co-authored-by: Isaac
1 parent 6e89017 commit ca1d470

3 files changed

Lines changed: 30 additions & 21 deletions

File tree

docs/sql-ref-datatypes.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ Spark SQL and DataFrames support the following data types:
4848
time-zone.
4949
- `TimeType(precision)`: Represents values comprising values of fields hour, minute and second with the number of decimal digits `precision` following the decimal point in the seconds field, without a time-zone.
5050
The range of values is from `00:00:00` to `23:59:59` for min precision `0`, and to `23:59:59.999999999` for max precision `9`. The default precision is `6`.
51-
- Note: Apache Hive has no TIME type, so `TimeType` is not supported in Hive SerDe interop. Storing it in a Hive SerDe table (including `INSERT OVERWRITE DIRECTORY ... STORED AS`) or passing it to a Hive UDF/UDAF/UDTF raises an error rather than silently converting the value.
51+
- Note: Apache Hive has no TIME type, so `TimeType` is not supported in Hive SerDe interop. Storing it in a Hive SerDe table (including `INSERT OVERWRITE DIRECTORY ... STORED AS`) or passing it to a Hive UDF/UDAF/UDTF raises an error rather than silently converting the value.
5252
- `TimestampType`: Timestamp with local time zone(TIMESTAMP_LTZ). It represents values comprising values of fields year, month, day,
5353
hour, minute, and second, with the session local time-zone. The timestamp value represents an
5454
absolute point in time.

sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -710,25 +710,32 @@ class InsertSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
710710

711711
test("SPARK-57556: nested TIME type is unsupported when writing to a Hive serde directory") {
712712
// Exercises HiveFileFormat.supportDataType's recursion into nested types: a TIME nested inside
713-
// an array must also be rejected, with the full (array) column type reported.
714-
withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
715-
withTempDir { dir =>
716-
// InsertIntoHiveDirCommand wraps the failure in a SparkException, so assert on the cause.
717-
val e = intercept[SparkException] {
718-
sql(
719-
s"""
720-
|INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
721-
|STORED AS PARQUET
722-
|SELECT array(TIME'12:01:02') AS c
723-
""".stripMargin)
713+
// an array, map or struct must also be rejected, with the full column type reported.
714+
Seq(
715+
"array(TIME'12:01:02')" -> ArrayType(TimeType(), containsNull = false),
716+
"map('k', TIME'12:01:02')" -> MapType(StringType, TimeType(), valueContainsNull = false),
717+
"named_struct('f', TIME'12:01:02')" ->
718+
new StructType().add("f", TimeType(), nullable = false)
719+
).foreach { case (selectExpr, columnType) =>
720+
withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
721+
withTempDir { dir =>
722+
// InsertIntoHiveDirCommand wraps the failure in a SparkException, so assert on the cause.
723+
val e = intercept[SparkException] {
724+
sql(
725+
s"""
726+
|INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
727+
|STORED AS PARQUET
728+
|SELECT $selectExpr AS c
729+
""".stripMargin)
730+
}
731+
checkError(
732+
exception = e.getCause.asInstanceOf[AnalysisException],
733+
condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
734+
parameters = Map(
735+
"columnName" -> "`c`",
736+
"columnType" -> s"\"${columnType.sql}\"",
737+
"format" -> "Hive"))
724738
}
725-
checkError(
726-
exception = e.getCause.asInstanceOf[AnalysisException],
727-
condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
728-
parameters = Map(
729-
"columnName" -> "`c`",
730-
"columnType" -> s"\"${ArrayType(TimeType()).sql}\"",
731-
"format" -> "Hive"))
732739
}
733740
}
734741
}

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -411,8 +411,10 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton {
411411
test("SPARK-57556: TIME type is unsupported as a Hive UDF argument") {
412412
withUserDefinedFunction("testGenericUDFHash" -> true) {
413413
sql(s"CREATE TEMPORARY FUNCTION testGenericUDFHash AS '${classOf[GenericUDFHash].getName}'")
414-
// The Hive UDF resolver wraps the underlying failure, but the message must still clearly
415-
// identify the unsupported TIME type rather than surfacing a MatchError/internal error.
414+
// The Hive UDF resolver wraps the underlying failure in `_LEGACY_ERROR_TEMP_3084` without
415+
// attaching it as a cause (see SPARK-57750), so we can only assert on the wrapped message.
416+
// It must still clearly identify the unsupported TIME type rather than surfacing a
417+
// MatchError/internal error.
416418
val e = intercept[AnalysisException] {
417419
sql("SELECT testGenericUDFHash(TIME'12:01:02')").collect()
418420
}

0 commit comments

Comments
 (0)