Skip to content

Commit 5d491f6

Browse files
committed
revert [SPARK-43752] and add test
### What changes were proposed in this pull request? This PR reverts the code changes from #55127 (commit eca57ea) and moves the test to `InsertIntoTests` where it properly runs across V2 INSERT SQL test suites. Specifically: - **Removes** the `V2WriteCommand` case added to `Analyzer.ResolveReferences` and `ResolveColumnDefaultInCommandInputQuery` - **Removes** the test from `ResolveDefaultColumnsSuite` (wrong location for a V2 end-to-end test) - **Adds** the test to `InsertIntoSQLOnlyTests` in `InsertIntoTests.scala`, which runs in suites with `includeSQLOnlyTests = true`: `DataSourceV2SQLSuite`, `DataSourceV2SQLSessionCatalogSuite`, and `V1WriteFallbackSessionCatalogSuite`. This is correct since `DEFAULT` is SQL-only syntax. ### Why are the changes needed? The code added by #55127 is dead code. For `INSERT INTO v2_table VALUES (1, DEFAULT)`, the resolution flow is: 1. The parser produces `InsertIntoStatement` for **both** V1 and V2 tables. 2. In the Resolution batch, `ResolveReferences` dispatches `InsertIntoStatement` to `resolveColumnDefaultInCommandInputQuery`, which resolves DEFAULT using the table schema. For V2 tables, the `DataSourceV2Relation` schema includes column default metadata (via `CatalogV2Util.encodeDefaultValue`), so DEFAULT is resolved correctly at this stage. 3. `ResolveInsertInto` only converts `InsertIntoStatement` to a `V2WriteCommand` (`AppendData`/`OverwriteByExpression`/`OverwritePartitionsDynamic`) **after** the query is fully resolved — by which point DEFAULT is already gone. No code path (SQL or DataFrame API) creates a `V2WriteCommand` with unresolved DEFAULT attributes. The `PlanResolutionSuite` test "INSERT INTO table with default column value" already verifies this resolution at the plan level. The test is still valuable and passes without the code changes — this PR moves it to the proper location. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Moved the existing test to `InsertIntoSQLOnlyTests` in `InsertIntoTests.scala`. The test verifies: - `INSERT INTO` with partial DEFAULT values - `INSERT INTO` with all DEFAULT values - `INSERT OVERWRITE` with DEFAULT values ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code 4.6 Closes #55348 from cloud-fan/SPARK-43752-revert. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 6d2840a commit 5d491f6

4 files changed

Lines changed: 17 additions & 36 deletions

File tree

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1418,12 +1418,6 @@ class Analyzer(
14181418
// to resolve column "DEFAULT" in the child plans so that they must be unresolved.
14191419
case s: SetVariable => resolveColumnDefaultInCommandInputQuery(s)
14201420

1421-
// SPARK-43752: resolve column "DEFAULT" in V2 write commands before the
1422-
// query is fully resolved, matching the InsertIntoStatement behavior above.
1423-
case v2: V2WriteCommand
1424-
if v2.table.resolved && v2.query.containsPattern(UNRESOLVED_ATTRIBUTE) =>
1425-
resolveColumnDefaultInCommandInputQuery(v2)
1426-
14271421
// Skip FetchCursor - let ResolveFetchCursor handle variable resolution
14281422
// This prevents ResolveReferences from trying to resolve target variables as columns
14291423
case s: SingleStatement if s.parsedPlan.isInstanceOf[FetchCursor] => s

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveColumnDefaultInCommandInputQuery.scala

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -89,12 +89,6 @@ class ResolveColumnDefaultInCommandInputQuery(val catalogManager: CatalogManager
8989
// defined for the n-th variable of the SET.
9090
s.withNewChildren(Seq(resolveColumnDefault(s.sourceQuery, expectedQuerySchema)))
9191

92-
// SPARK-43752: resolve column "DEFAULT" in V2 write commands.
93-
case v2: V2WriteCommand if conf.enableDefaultColumns && v2.table.resolved &&
94-
v2.query.containsPattern(UNRESOLVED_ATTRIBUTE) =>
95-
val expectedQuerySchema = v2.table.schema
96-
v2.withNewQuery(resolveColumnDefault(v2.query, expectedQuerySchema))
97-
9892
case _ => plan
9993
}
10094

sql/core/src/test/scala/org/apache/spark/sql/ResolveDefaultColumnsSuite.scala

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -308,28 +308,4 @@ class ResolveDefaultColumnsSuite extends QueryTest with SharedSparkSession {
308308
checkAnswer(sql(s"SELECT * FROM $tableName"), Seq(Row(0, user)))
309309
}
310310
}
311-
312-
test("SPARK-43752: column DEFAULT in V2 write commands") {
313-
withSQLConf(
314-
"spark.sql.catalog.testcat" ->
315-
classOf[connector.catalog.InMemoryTableCatalog].getName) {
316-
sql("CREATE TABLE testcat.t(c1 INT DEFAULT 42, c2 STRING DEFAULT 'hello')")
317-
// INSERT INTO (AppendData) with DEFAULT
318-
sql("INSERT INTO testcat.t VALUES (1, DEFAULT)")
319-
checkAnswer(
320-
sql("SELECT * FROM testcat.t"),
321-
Row(1, "hello"))
322-
// INSERT INTO with all DEFAULT
323-
sql("INSERT INTO testcat.t VALUES (DEFAULT, DEFAULT)")
324-
checkAnswer(
325-
sql("SELECT * FROM testcat.t ORDER BY c1"),
326-
Seq(Row(1, "hello"), Row(42, "hello")))
327-
// INSERT OVERWRITE (OverwriteByExpression) with DEFAULT
328-
sql("INSERT OVERWRITE testcat.t VALUES (100, DEFAULT)")
329-
checkAnswer(
330-
sql("SELECT * FROM testcat.t"),
331-
Row(100, "hello"))
332-
sql("DROP TABLE testcat.t")
333-
}
334-
}
335311
}

sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -505,6 +505,23 @@ trait InsertIntoSQLOnlyTests
505505
verifyTable(t1, Seq("a" -> "b").toDF("id", "data"))
506506
}
507507
}
508+
509+
test("InsertInto: column DEFAULT values") {
510+
val t1 = s"${catalogAndNamespace}tbl"
511+
withTable(t1) {
512+
sql(s"CREATE TABLE $t1 (c1 INT DEFAULT 42, c2 STRING DEFAULT 'hello') USING $v2Format")
513+
sql(s"INSERT INTO $t1 VALUES (1, DEFAULT)")
514+
checkAnswer(sql(s"SELECT * FROM $t1"), Row(1, "hello"))
515+
516+
sql(s"INSERT INTO $t1 VALUES (DEFAULT, DEFAULT)")
517+
checkAnswer(
518+
sql(s"SELECT * FROM $t1 ORDER BY c1"),
519+
Seq(Row(1, "hello"), Row(42, "hello")))
520+
521+
sql(s"INSERT OVERWRITE $t1 VALUES (100, DEFAULT)")
522+
checkAnswer(sql(s"SELECT * FROM $t1"), Row(100, "hello"))
523+
}
524+
}
508525
}
509526
}
510527

0 commit comments

Comments
 (0)