docs: introduce Arrow-native terminology in README and user guide

Comet's documentation conflates several distinct ideas under the word
'native': implementation language (Rust vs JVM), pipeline membership
(handled by Comet vs falls back to Spark), and data format (Arrow vs
Spark rows). The vocabulary clean-up in #4419 splits these out, and
this PR rolls in only the README and user-guide prose, with no code
or operator renames.

- 'Arrow-native' is now the term for the data-format property that
  unifies the pipeline.
- 'Comet pipeline' replaces 'the native Comet path' / 'accelerated by
  Comet' for membership.
- 'Rust-implemented' / 'native Rust' is used for the implementation-
  language axis. Bare 'native execution' / 'runs natively' / 'the
  native path' as vague adjectives are removed.

The biggest single rewrite is in user-guide/latest/understanding-comet-plans.md,
where the 'three kinds of nodes' framing becomes four (Arrow-native
Rust operators, Arrow-native JVM expressions, Arrow-native JVM
plumbing, Spark fallback).

Contributor-guide files, plugin-overview prose, and the about/ pages
are deliberately out of scope and will follow in a separate PR.

Author: andygrove

README.md

@@ -35,10 +35,12 @@ under the License.
 
 <img src="docs/source/_static/images/DataFusionComet-Logo-Light.png" width="512" alt="logo"/>
 
-Apache DataFusion Comet is a high-performance accelerator for Apache Spark, built on top of the powerful
-[Apache DataFusion] query engine. Comet is designed to significantly enhance the
-performance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the
-Spark ecosystem without requiring any code changes.
+Apache DataFusion Comet is a high-performance accelerator for Apache Spark. Comet keeps Spark queries
+**Arrow-native end-to-end**: operators, expressions, shuffle, and broadcast all stay in Apache Arrow
+columnar format, avoiding the per-row overhead of Spark's row-based engine. Within the Arrow-native
+pipeline, operators and expressions execute as Rust code (via the [Apache DataFusion] query engine)
+or as JVM code that operates directly on Arrow batches. Comet integrates with the Spark ecosystem
+without requiring any code changes.
 
 **Comet provides a ~2x speedup for TPC-DS @ SF 1000 (1TB), resulting in ~50% cost savings.**
 
@@ -58,17 +60,22 @@ See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contribut
 
 ## What Comet Accelerates
 
-Comet replaces Spark operators and expressions with native Rust implementations that run on Apache DataFusion.
-It uses Apache Arrow for zero-copy data transfer between the JVM and native code.
+Comet replaces Spark operators and expressions with implementations that consume and produce Apache Arrow
+batches. Most run as native Rust code on top of Apache DataFusion; some run as JVM code over Arrow batches.
+Either way the work stays in the Comet pipeline without falling back to Spark's row-based engine.
 
 - **Parquet scans**: native Parquet reader integrated with Spark's query planner
 - **Apache Iceberg**: accelerated Parquet scans when reading Iceberg tables from Spark
   (see the [Iceberg guide](https://datafusion.apache.org/comet/user-guide/iceberg.html))
-- **Shuffle**: native columnar shuffle with support for hash and range partitioning
+- **Shuffle**: Arrow-IPC columnar shuffle with support for hash and range partitioning, in a native Rust
+  implementation paired with a JVM fallback for unsupported partition key types
 - **Expressions**: hundreds of supported Spark expressions across math, string, datetime, array,
   map, JSON, hash, and predicate categories
 - **Aggregations**: hash aggregate with support for `FILTER (WHERE ...)` clauses
 - **Joins**: hash join, sort-merge join, and broadcast join
+- **Scala/Java UDFs**: experimental support for keeping Scala/Java scalar UDFs in the Comet pipeline
+  via Spark's whole-stage codegen (see the
+  [Scala UDF guide](https://datafusion.apache.org/comet/user-guide/scala_java_udfs.html))
 
 For the authoritative lists, see the [supported expressions](https://datafusion.apache.org/comet/user-guide/expressions.html)
 and [supported operators](https://datafusion.apache.org/comet/user-guide/operators.html) pages.

docs/source/user-guide/latest/compatibility/expressions/cast.md

@@ -24,9 +24,9 @@ Cast operations in Comet fall into three levels of support:
 - **C (Compatible)**: The results match Apache Spark
 - **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
   will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-  `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
-  recommended for production use.
-- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
+  `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run inside the Comet pipeline,
+  but this is not recommended for production use.
+- **U (Unsupported)**: Comet does not provide an implementation of this cast expression and the query stage will fall back to
   Spark.
 - **N/A**: Spark does not support this cast.
 
@@ -83,9 +83,9 @@ is the same regardless of the session timezone setting.
 
 In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric
 types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles
-this by short-circuiting to a null literal during query planning, so no native execution
-is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before
-execution reaches Comet).
+this by short-circuiting to a null literal during query planning, so the cast is removed
+before reaching the runtime. In ANSI and Try modes, Spark rejects these casts at analysis
+time (before execution reaches Comet).
 
 ## String to Timestamp
 
@@ -139,7 +139,7 @@ The result is always a wall-clock timestamp with no timezone conversion or DST a
 Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
 `spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
 disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.
+its cast implementation on unexpected inputs.
 
 When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
 Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific

docs/source/user-guide/latest/compatibility/scans.md

@@ -63,8 +63,8 @@ The following limitation may produce incorrect results without falling back to S
 The following limitation raises an error at scan time rather than falling back to Spark:
 
 - Invalid UTF-8 bytes in `STRING` columns. Spark permits arbitrary byte sequences in a `STRING`
-  column (for example from `CAST(X'C1' AS STRING)`), but Comet's native execution path is built on
-  Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose `STRING` column contains
+  column (for example from `CAST(X'C1' AS STRING)`), but Comet's pipeline is built on Arrow,
+  whose string type is strictly UTF-8. Reading a Parquet file whose `STRING` column contains
   non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`. Disable Comet for the
   query, or cast the column to `BINARY` before persisting, if you need to preserve non-UTF-8 bytes.
   See [#4121](https://github.com/apache/datafusion-comet/issues/4121).

docs/source/user-guide/latest/datasources.md

@@ -23,9 +23,10 @@
 
 ### Parquet
 
-Parquet scans are performed natively by Comet if all data types in the schema are supported. When the scan
-falls back to Spark, enabling `spark.comet.convert.parquet.enabled` will immediately convert the data into
-Arrow format, allowing native execution to happen after that, but the process may not be efficient.
+Parquet scans are read directly into Arrow by Comet's Rust scan when all data types in the schema are supported.
+When the scan falls back to Spark, enabling `spark.comet.convert.parquet.enabled` will immediately convert the
+data into Arrow format so the rest of the plan can stay in the Comet pipeline, though the conversion itself
+may not be efficient.
 
 ### Apache Iceberg
 
@@ -35,17 +36,17 @@ Comet accelerates Iceberg scans of Parquet files. See the [Iceberg Guide] for mo
 
 ### CSV
 
-Comet provides experimental native CSV scan support. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV files
-are read natively for improved performance. This feature is experimental and performance benefits are
-workload-dependent.
+Comet provides an experimental Rust-implemented CSV scan. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV
+files are read directly into Arrow for improved performance. This feature is experimental and performance benefits
+are workload-dependent.
 
 Alternatively, when `spark.comet.convert.csv.enabled` is enabled, data from Spark's CSV reader is immediately
-converted into Arrow format, allowing native execution to happen after that.
+converted into Arrow format so the rest of the plan can stay in the Comet pipeline.
 
 ### JSON
 
-Comet does not provide native JSON scan, but when `spark.comet.convert.json.enabled` is enabled, data is immediately
-converted into Arrow format, allowing native execution to happen after that.
+Comet does not provide a Rust-implemented JSON scan, but when `spark.comet.convert.json.enabled` is enabled,
+data is immediately converted into Arrow format so the rest of the plan can stay in the Comet pipeline.
 
 ## Data Catalogs
 

docs/source/user-guide/latest/iceberg.md

@@ -22,7 +22,7 @@
 ## Native Reader
 
 Comet's native Iceberg reader relies on reflection to extract `FileScanTask`s from Iceberg, which are
-then serialized to Comet's native execution engine (see
+then serialized to DataFusion for execution (see
 [PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
 
 The example below uses Spark's package downloader to retrieve Comet $COMET_VERSION and Iceberg
@@ -157,12 +157,12 @@ Iceberg ships several `ScalaUDF`s that surface in user queries and maintenance a
   (`INT_ORDERED_BYTES`, `LONG_ORDERED_BYTES`, ..., `INTERLEAVE_BYTES`) over the sort key columns
   during compaction.
 
-By default these UDFs cause the enclosing operator to fall back to Spark, which forces a
-columnar-to-row roundtrip and demotes the surrounding shuffle from `CometExchange` to
+By default these UDFs cause the enclosing operator to fall back to Spark, which forces an
+Arrow-to-row roundtrip and demotes the surrounding shuffle from `CometExchange` to
 `CometColumnarExchange`. Enabling the experimental
 [Scala UDF and Java UDF Support](scala_java_udfs.md) feature
-(`spark.comet.exec.scalaUDF.codegen.enabled=true`) routes these UDFs through native execution so
-the project, exchange, and sort operators around them stay on the Comet path end-to-end.
+(`spark.comet.exec.scalaUDF.codegen.enabled=true`) keeps these UDFs in the Comet pipeline so
+the project, exchange, and sort operators around them stay accelerated end-to-end.
 
 ### Task input metrics
 

docs/source/user-guide/latest/scala_java_udfs.md

@@ -19,24 +19,24 @@
 
 # Scala UDF and Java UDF Support
 
-Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html) on the native Comet path. The presence of a UDF does not force the enclosing operator off the native path; surrounding native operators stay native.
+Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html) inside the Comet pipeline. The UDF body is JVM bytecode produced by Spark's whole-stage codegen, but it runs over Arrow batches alongside the surrounding Arrow-native operators rather than triggering a fallback to row-based Spark execution. The presence of a UDF does not force the enclosing operator out of the pipeline; surrounding Comet operators continue to run as usual.
 
 This page covers Spark's `ScalaUDF` (Scala `udf(...)`, `spark.udf.register(...)` over Scala or Java functional interfaces, and SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`). Other UDF kinds (Python / Pandas, Hive, aggregate) are out of scope and continue to fall back to Spark.
 
 This feature is experimental and disabled by default.
 
 ## Configuration
 
-| Key                                         | Default | Description                                                                                                        |
-| ------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------ |
-| `spark.comet.exec.scalaUDF.codegen.enabled` | `false` | When `true`, eligible `ScalaUDF`s run on the Comet path. When `false`, the enclosing operator falls back to Spark. |
+| Key                                         | Default | Description                                                                                                                |
+| ------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
+| `spark.comet.exec.scalaUDF.codegen.enabled` | `false` | When `true`, eligible `ScalaUDF`s run inside the Comet pipeline. When `false`, the enclosing operator falls back to Spark. |
 
 ## Supported
 
 - User functions registered via `udf(...)`, `spark.udf.register(...)` (Scala or Java functional interfaces), or SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`.
 - Scalar input/output types: `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Decimal`, `String`, `Binary`, `Date`, `Timestamp`, `TimestampNTZ`.
 - Complex input/output types with arbitrary nesting: `ArrayType`, `StructType`, `MapType`.
-- Composition with other Catalyst expressions inside the argument tree (e.g. `myUdf(upper(s))` runs as one native unit).
+- Composition with other Catalyst expressions inside the argument tree (e.g. `myUdf(upper(s))` runs as a single compiled kernel without leaving the Comet pipeline).
 - Higher-order functions (`transform`, `filter`, `exists`, `aggregate`, `zip_with`, `map_filter`, `map_zip_with`, etc.) inside the argument tree.
 
 ## Not supported
@@ -45,7 +45,7 @@ This feature is experimental and disabled by default.
 - Table UDFs and generators.
 - Python `@udf` and Pandas `@pandas_udf`.
 - Hive `GenericUDF` and `SimpleUDF`.
-- `CalendarIntervalType`, `NullType`, and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; for native execution, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
+- `CalendarIntervalType`, `NullType`, and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep the work in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
 - Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
 
 When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.

docs/source/user-guide/latest/tuning.md

@@ -61,8 +61,8 @@ The valid pool types are:
 - `fair_unified` (default when `spark.memory.offHeap.enabled=true` is set)
 - `greedy_unified`
 
-Both pool types are shared across all native execution contexts within the same Spark task. When
-Comet executes a shuffle, it runs two native execution contexts concurrently (e.g. one for
+Both pool types are shared across all DataFusion execution contexts within the same Spark task. When
+Comet executes a shuffle, it runs two DataFusion execution contexts concurrently (e.g. one for
 pre-shuffle operators and one for the shuffle writer). The shared pool ensures that the combined
 memory usage stays within the per-task limit.