Skip to content

Commit ef439bb

Browse files
committed
docs: explain classic.SparkSession downcast in BoundedDatasetFactory
Per /ultrareview feedback: the one-line comment didn't make clear why the cast is safe. Expand it to note that SparkSession.builder() always returns a classic.SparkSession at runtime, which is why the downcast avoids reflection.
1 parent 0d2854a commit ef439bb

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

runners/spark/4/src/main/java/org/apache/beam/runners/spark/structuredstreaming/io/BoundedDatasetFactory.java

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,10 @@ public static <T> Dataset<WindowedValue<T>> createDatasetFromRows(
8686
Params<T> params = new Params<>(encoder, options, session.sparkContext().defaultParallelism());
8787
BeamTable<T> table = new BeamTable<>(source, params);
8888
LogicalPlan logicalPlan = DataSourceV2Relation.create(table, Option.empty(), Option.empty());
89-
// In Spark 4.0+, Dataset$ moved to org.apache.spark.sql.classic; cast session accordingly.
89+
// In Spark 4.0+, Dataset$ moved to org.apache.spark.sql.classic and its ofRows() now
90+
// takes the classic SparkSession subclass. The runtime instance returned by
91+
// SparkSession.builder() is always a classic.SparkSession, so the downcast is safe and
92+
// avoids reflection.
9093
return (Dataset<WindowedValue<T>>)
9194
Dataset$.MODULE$
9295
.ofRows((org.apache.spark.sql.classic.SparkSession) session, logicalPlan)

0 commit comments

Comments
 (0)