docs: update Spark version support and add version compatibility page (#4138)

Author: andygrove

README.md

@@ -81,7 +81,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl
 
 ## Getting Started
 
-Comet supports Apache Spark 3.4 and 3.5, and provides experimental support for Spark 4.0. See the
+Comet supports Apache Spark 3.4, 3.5, and 4.0, and provides experimental support for Spark 4.1 and 4.2. See the
 [installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed
 version, Java, and Scala compatibility matrix.
 

docs/source/user-guide/latest/compatibility/index.md

@@ -28,6 +28,7 @@ This guide documents areas where Comet's behavior is known to differ from Spark.
 - **Regular expressions**: differences between the Rust regexp crate and Java's regex engine.
 - **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning.
 - **Expressions**: per-expression compatibility notes, including cast.
+- **Spark versions**: version-specific known issues and limitations.
 
 ```{toctree}
 :maxdepth: 1
@@ -37,4 +38,5 @@ floating-point
 regex
 operators
 expressions/index
+spark-versions
 ```

docs/source/user-guide/latest/compatibility/spark-versions.md

@@ -0,0 +1,61 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Spark Version Compatibility
+
+This page documents known issues and limitations specific to each supported Apache Spark version.
+
+For general compatibility information that applies across all Spark versions, see the other pages in this
+compatibility guide.
+
+## Spark 3.4
+
+Spark 3.4.3 is supported with Java 11/17 and Scala 2.12/2.13.
+
+## Spark 3.5
+
+Spark 3.5.8 is supported with Java 11/17 and Scala 2.12/2.13.
+
+## Spark 4.0
+
+Spark 4.0.2 is supported with Java 17 and Scala 2.13.
+
+### Known Limitations
+
+- **Collation support** ([#1947](https://github.com/apache/datafusion-comet/issues/1947),
+  [#4051](https://github.com/apache/datafusion-comet/issues/4051)): Spark 4.0 introduced collation
+  support. Non-default collated strings are not yet supported by Comet and will fall back to Spark.
+
+## Spark 4.1 (Experimental)
+
+Spark 4.1.1 is provided as experimental support with Java 17 and Scala 2.13.
+
+```{warning}
+Spark 4.1 support is experimental and intended for development and testing only. It should not be used
+in production.
+```
+
+## Spark 4.2 (Experimental)
+
+Spark 4.2.0-preview4 is provided as experimental support with Java 17 and Scala 2.13.
+
+```{warning}
+Spark 4.2 support is experimental and targets a preview release of Spark. It is intended for early
+evaluation only and should not be used in production.
+```

docs/source/user-guide/latest/installation.md

@@ -42,15 +42,17 @@ Other versions may work well enough for development and evaluation purposes.
 | 3.5.6         | 11/17        | 2.12/2.13     | Yes               | No                    |
 | 3.5.7         | 11/17        | 2.12/2.13     | Yes               | Yes                   |
 | 3.5.8         | 11/17        | 2.12/2.13     | Yes               | Yes                   |
+| 4.0.2         | 17           | 2.13          | Yes               | Yes                   |
 
 Note that we do not test the full matrix of supported Java and Scala versions in CI for every Spark version.
 
 Experimental support is provided for the following versions of Apache Spark and is intended for development/testing
 use only and should not be used in production yet.
 
-| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |
-| ------------- | ------------ | ------------- | ----------------- | --------------------- |
-| 4.0.2         | 17           | 2.13          | Yes               | Yes                   |
+| Spark Version  | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |
+| -------------- | ------------ | ------------- | ----------------- | --------------------- |
+| 4.1.1          | 17           | 2.13          | Compile only      | No                    |
+| 4.2.0-preview4 | 17           | 2.13          | Compile only      | No                    |
 
 Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by
 Cloud Service Providers.
@@ -81,7 +83,9 @@ Here are the direct links for downloading the Comet $COMET_VERSION jar file.
 - [Comet plugin for Spark 3.4 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.13/$COMET_VERSION/comet-spark-spark3.4_2.13-$COMET_VERSION.jar)
 - [Comet plugin for Spark 3.5 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/$COMET_VERSION/comet-spark-spark3.5_2.12-$COMET_VERSION.jar)
 - [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar)
-- [Comet plugin for Spark 4.0 / Scala 2.13 (Experimental)](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)
+- [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)
+- [Comet plugin for Spark 4.1 / Scala 2.13 (Experimental)](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.1_2.13/$COMET_VERSION/comet-spark-spark4.1_2.13-$COMET_VERSION.jar)
+- [Comet plugin for Spark 4.2 / Scala 2.13 (Experimental)](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.2_2.13/$COMET_VERSION/comet-spark-spark4.2_2.13-$COMET_VERSION.jar)
 <!-- ENDIF -->
 
 ## Building from source