feat(phase-7): Spark 3.4/4.0 Delta version matrix + TPC-DS docs

Adds delta-spark test dependencies for all three Spark profiles:
  - spark-3.4: delta-spark 2.4.0 (Delta 2.x, no DV support)
  - spark-3.5: delta-spark 3.3.2 (Delta 3.x, full DV/column-mapping)
  - spark-4.0: delta-spark 4.0.0 (Delta 4.x, experimental)

Each profile includes the failureaccess companion dep and
Spark/Hadoop exclusions matching the spark-3.5 pattern.

CI workflow updated to test all three Spark versions in matrix.

Also adds TPC-DS plan stability fixture generation docs to the
contributor guide (procedure for adding q*.native_delta_compat
golden files).

Documentation updated: Spark 3.4/4.0 no longer listed as unsupported,
now listed with their Delta version constraints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: schenksj

.github/workflows/delta_spark_test.yml

@@ -103,7 +103,10 @@ jobs:
       matrix:
         os: [ubuntu-24.04]
         java-version: [17]
-        spark-version: [{short: '3.5', full: '3.5.8'}]
+        spark-version:
+          - {short: '3.4', full: '3.4.3'}
+          - {short: '3.5', full: '3.5.8'}
+          - {short: '4.0', full: '4.0.1'}
       fail-fast: false
     name: delta-native/${{ matrix.os }}/spark-${{ matrix.spark-version.full }}/java-${{ matrix.java-version }}
     runs-on: ${{ matrix.os }}

docs/source/contributor-guide/delta-spark-tests.md

@@ -102,3 +102,24 @@ python benchmarks/tpc/tpcbench.py \
 
 Engine configs are in `benchmarks/tpc/engines/comet-delta.toml` and
 `benchmarks/tpc/engines/comet-delta-hashjoin.toml`.
+
+## TPC-DS Plan Stability Fixtures
+
+To generate the `q*.native_delta_compat/` plan stability golden files:
+
+1. Add `CometConf.SCAN_NATIVE_DELTA_COMPAT` to the `scanImpls` list in
+   `CometPlanStabilitySuite.scala` (line 66).
+
+2. Ensure TPC-DS data exists as Delta tables (see `create-delta-tables.py`
+   in the Benchmarks section above).
+
+3. Generate golden files:
+
+```bash
+SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl spark \
+    -Dsuites=org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite \
+    test -Pspark-3.5 -Dmaven.gitcommitid.skip
+```
+
+4. Commit the generated `q*.native_delta_compat/extended.txt` files under
+   `spark/src/test/resources/tpcds-plan-stability/approved-plans-*`.

docs/source/user-guide/latest/delta.md

@@ -126,4 +126,5 @@ The following scenarios will fall back to Spark's native Delta reader:
 - Tables with `rowTracking` enabled
 - Change Data Feed (`readChangeFeed`) queries
 - The `_metadata.row_index` virtual column
-- Spark 3.4 and 4.0 (currently tested on Spark 3.5 only)
+- Spark 3.4 uses Delta 2.4.x (DVs not supported in Delta 2.x; simpler feature set)
+- Spark 4.0 uses Delta 4.0.x (experimental)

spark/pom.xml

@@ -218,6 +218,29 @@ under the License.
           <version>9.4.53.v20231009</version>
           <scope>test</scope>
         </dependency>
+        <!-- Delta 2.4.x targets Spark 3.4.x -->
+        <dependency>
+          <groupId>io.delta</groupId>
+          <artifactId>delta-spark_${scala.binary.version}</artifactId>
+          <version>2.4.0</version>
+          <scope>test</scope>
+          <exclusions>
+            <exclusion>
+              <groupId>org.apache.spark</groupId>
+              <artifactId>*</artifactId>
+            </exclusion>
+            <exclusion>
+              <groupId>org.apache.hadoop</groupId>
+              <artifactId>*</artifactId>
+            </exclusion>
+          </exclusions>
+        </dependency>
+        <dependency>
+          <groupId>com.google.guava</groupId>
+          <artifactId>failureaccess</artifactId>
+          <version>1.0.2</version>
+          <scope>test</scope>
+        </dependency>
       </dependencies>
     </profile>
 
@@ -299,6 +322,29 @@ under the License.
           <version>11.0.24</version>
           <scope>test</scope>
         </dependency>
+        <!-- Delta 4.0.x targets Spark 4.0.x -->
+        <dependency>
+          <groupId>io.delta</groupId>
+          <artifactId>delta-spark_${scala.binary.version}</artifactId>
+          <version>4.0.0</version>
+          <scope>test</scope>
+          <exclusions>
+            <exclusion>
+              <groupId>org.apache.spark</groupId>
+              <artifactId>*</artifactId>
+            </exclusion>
+            <exclusion>
+              <groupId>org.apache.hadoop</groupId>
+              <artifactId>*</artifactId>
+            </exclusion>
+          </exclusions>
+        </dependency>
+        <dependency>
+          <groupId>com.google.guava</groupId>
+          <artifactId>failureaccess</artifactId>
+          <version>1.0.2</version>
+          <scope>test</scope>
+        </dependency>
       </dependencies>
     </profile>
     <profile>