You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/parquet_scans.md
+1-2Lines changed: 1 addition & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,8 +41,7 @@ implementation:
41
41
- Improves performance
42
42
43
43
> **Note on mutable buffers:** Both `native_comet` and `native_iceberg_compat` use reusable mutable buffers
44
-
> when transferring data from JVM to native code via Arrow FFI. The `native_iceberg_compat` implementation uses DataFusion's native Parquet reader for data columns, bypassing Comet's mutable buffer infrastructure entirely. However, partition columns still use `ConstantColumnReader`, which relies on Comet's mutable buffers that are reused across batches. This means native operators that
45
-
> buffer data (such as `SortExec` or `ShuffleWriterExec`) must perform deep copies to avoid data corruption.
44
+
> when transferring data from JVM to native code via Arrow FFI. The `native_iceberg_compat` implementation uses DataFusion's native Parquet reader for data columns, bypassing Comet's mutable buffer infrastructure entirely. However, partition columns still use `ConstantColumnReader`, which relies on Comet's mutable buffers that are reused across batches. This means native operators that buffer data (such as `SortExec` or `ShuffleWriterExec`) must perform deep copies to avoid data corruption.
46
45
> See the [FFI documentation](ffi.md) for details on the `arrow_ffi_safe` flag and ownership semantics.
47
46
48
47
The `native_datafusion` and `native_iceberg_compat` scans share the following limitations:
0 commit comments