Skip to content

Commit 627bb85

Browse files
committed
docs: document schema evolution behavior for native_datafusion scan
1 parent e9ea8e3 commit 627bb85

1 file changed

Lines changed: 4 additions & 0 deletions

File tree

docs/source/contributor-guide/parquet_scans.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,10 @@ cause Comet to fall back to Spark.
6262
- No support for `input_file_name()`, `input_file_block_start()`, or `input_file_block_length()` SQL functions.
6363
The `native_datafusion` scan does not use Spark's `FileScanRDD`, so these functions cannot populate their values.
6464
- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to `true`
65+
- Schema evolution (type promotion such as int→long, float→double) is always enabled. DataFusion's native Parquet
66+
reader handles type promotion automatically, regardless of the `spark.comet.schemaEvolution.enabled` setting.
67+
This means queries that would fail in Spark or other Comet scan implementations when schema evolution is disabled
68+
will succeed with `native_datafusion`.
6569

6670
The `native_iceberg_compat` scan has the following additional limitation that may produce incorrect results
6771
without falling back to Spark:

0 commit comments

Comments
 (0)