Fix CI for Spark 4.0: use Python 3.9 and version-marker pyspark dep

PySpark 4.0 requires Python >=3.9. Update the CI matrix to carry a
PYTHON_VERSION per entry (3.8 for Spark 3.x, 3.9 for Spark 4.x) and
use it in the setup-python step.

Split the pyspark optional dep in pyproject.toml into two
version-marker entries so poetry can resolve correctly on both
Python 3.8 (pyspark <4.0) and Python 3.9+ (pyspark <5.0).

Affirm-Skill: acli-jira
Affirm-Skill: att-deploy-check
Affirm-Skill: att-test
Affirm-Skill: buildkite-debug
Affirm-Skill: capture-context
Affirm-Skill: cmt-writer
Affirm-Skill: commit-and-push
Affirm-Skill: create-adr
Affirm-Skill: create-cmt-ticket
Affirm-Skill: create-tech-spec
Affirm-Skill: export-conversation-record
Affirm-Skill: gather-best-practices
Affirm-Skill: mcp-debug
Affirm-Skill: multi-thor-provisioner
Affirm-Skill: resolve-pr-comments
Affirm-Skill: save-output
Affirm-Skill: security-scanner-for-skill-md
Affirm-Skill: skill-evaluator
Affirm-Skill: thor-control
Affirm-Skill: tickets-from-plan
Affirm-Skill: write-implementation-plan

Author: m-aciek

.github/workflows/base.yml

@@ -15,22 +15,27 @@ jobs:
         include:
           - PYSPARK_VERSION: "3.1.3"
             JAVA_VERSION: "11"
+            PYTHON_VERSION: "3.8"
           - PYSPARK_VERSION: "3.2"
             JAVA_VERSION: "11"
+            PYTHON_VERSION: "3.8"
           - PYSPARK_VERSION: "3.3"
             JAVA_VERSION: "11"
+            PYTHON_VERSION: "3.8"
           - PYSPARK_VERSION: "3.5"
             JAVA_VERSION: "11"
+            PYTHON_VERSION: "3.8"
           - PYSPARK_VERSION: "4.0.0"
             JAVA_VERSION: "17"
+            PYTHON_VERSION: "3.9"
 
     steps:
       - uses: actions/checkout@v3
 
       - uses: actions/setup-python@v2
-        name: Install Python 3.8
+        name: Install Python ${{matrix.PYTHON_VERSION}}
         with:
-          python-version: 3.8
+          python-version: ${{matrix.PYTHON_VERSION}}
 
       - uses: actions/setup-java@v1
         name: Setup Java ${{matrix.JAVA_VERSION}}

pyproject.toml

@@ -31,7 +31,10 @@ classifiers = [
 python = ">=3.8,<4"
 numpy = ">=1.14.1"
 pandas = ">=0.23.0"
-pyspark = { version = ">=2.4.7,<5.0.0", optional = true }
+pyspark = [
+    { version = ">=2.4.7,<4.0.0", optional = true, python = ">=3.8,<3.9" },
+    { version = ">=2.4.7,<5.0.0", optional = true, python = ">=3.9" },
+]
 
 [tool.poetry.dev-dependencies]
 pytest = "^6.2.4"