[SPARK-56688][INFRA][PYTHON] Reorganize pyspark tests to empty a CI slot

### What changes were proposed in this pull request?

We emptied a CI slot for future usage.

After some optimization work, we have a CI slot `pyspark-core, pyspark-errors, pyspark-streaming, pyspark-logger` that normally just takes 30 minutes. It's a waste of slot because we can only have 20 concurrent jobs. This PR splits the workload into other slots

* `pyspark-core, pyspark-errors, pyspark-logger` -> `pyspark-sql, pyspark-resource, pyspark-testing`
* `pyspark-streaming` -> `pyspark-structured-streaming, pyspark-structured-streaming-connect`
* pip test, which used to follow `pyspark-logger` -> follow `pyspark-pipelines` now

We should still be able to keep all the CI slots below 90 minutes most of the time (120 is the limit). And we can have a new slot.

### Why are the changes needed?

The reason we want a new slot, is to have a CI test against old client (4.0 client for example) to check backward compatibility issues. We have broken the client multiple times this year and we have 0 tests to gate it.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

It's a CI change only.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #55638 from gaogaotiantian/rearrange-pyspark-tests.

Authored-by: Tian Gao <gaogaotiantian@hotmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit cff6c0d)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

Author: gaogaotiantian

.github/workflows/build_and_test.yml

@@ -555,13 +555,11 @@ jobs:
           - ${{ inputs.java }}
         modules:
           - >-
-            pyspark-sql, pyspark-resource, pyspark-testing
-          - >-
-            pyspark-core, pyspark-errors, pyspark-streaming, pyspark-logger
+            pyspark-sql, pyspark-resource, pyspark-testing, pyspark-core, pyspark-errors, pyspark-logger
           - >-
             pyspark-mllib, pyspark-ml, pyspark-ml-connect, pyspark-pipelines
           - >-
-            pyspark-structured-streaming, pyspark-structured-streaming-connect
+            pyspark-streaming, pyspark-structured-streaming, pyspark-structured-streaming-connect
           - >-
             pyspark-connect
           - >-
@@ -578,10 +576,9 @@ jobs:
           # Always run if pyspark == 'true', even infra-image is skip (such as non-master job)
           # In practice, the build will run in individual PR, but not against the individual commit
           # in Apache Spark repository.
-          - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-sql, pyspark-resource, pyspark-testing' }}
-          - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-core, pyspark-errors, pyspark-streaming, pyspark-logger' }}
+          - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-sql, pyspark-resource, pyspark-testing, pyspark-core, pyspark-errors, pyspark-logger' }}
           - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-mllib, pyspark-ml, pyspark-ml-connect' }}
-          - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-structured-streaming, pyspark-structured-streaming-connect' }}
+          - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-streaming, pyspark-structured-streaming, pyspark-structured-streaming-connect' }}
           - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark != 'true' && 'pyspark-connect' }}
           # pyspark-install is very slow so we only run it when it's changed or explicity requested
           - modules: ${{ fromJson(needs.precondition.outputs.required).pyspark-install != 'true' && 'pyspark-install' }}
@@ -665,7 +662,7 @@ jobs:
       env: ${{ fromJSON(inputs.envs) }}
       shell: 'script -q -e -c "bash {0}"'
       run: |
-        if [[ "$MODULES_TO_TEST" == *"pyspark-errors"* ]]; then
+        if [[ "$MODULES_TO_TEST" == *"pyspark-pipelines"* ]]; then
           export SKIP_PACKAGING=false
           echo "Python Packaging Tests Enabled!"
         fi