Skip to content

Commit 383fd94

Browse files
thinkallCopilot
thinkall
and
Copilot
committed
ci: split tests into spark/notspark variants and aggregate coverage
Mirror the structure of the internal Azure DevOps pipeline (.pipelines/build.yml in the internal fork) in the public GitHub Actions workflow: * New matrix axis 'test-type' with two values, 'notspark' and 'spark', each with its own --ignore list and pytest -m filter: - notspark: --ignore=test/autogen --ignore=test/spark -m 'not spark' - spark: --ignore=test/autogen --ignore=test/spark/test_internal_mlflow.py -m 'spark' This keeps test/spark/test_internal_mlflow.py (which depends on the internal MLflow tracking server and unconditionally imports pyspark at module level) from breaking collection in either variant. * The 'spark' variant only runs where pyspark is installed by the workflow today: ubuntu-latest with Python 3.11 / 3.12 / 3.13. It is excluded for windows-latest and for Python 3.10. * All Linux test jobs now run under 'coverage run' (not just the 3.11 job). Each Linux job combines its parallel-mode coverage shards into one .coverage file and uploads it as a uniquely named artifact. A new 'coverage' aggregator job downloads every per-job artifact, runs 'coverage combine' across them, generates a single coverage.xml and uploads that one combined report to Codecov. This replaces the previous per-3.11-job Codecov upload. * The 'Save dependencies' step is now gated to a single matrix entry (ubuntu / 3.11 / notspark) on push to main so that parallel jobs do not race on the unit-tests-installed-dependencies branch. Coverage is intentionally not collected on Windows runners to avoid Linux/Windows path mismatches when combining .coverage data files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 35e9f75 commit 383fd94

1 file changed

Lines changed: 104 additions & 15 deletions

File tree

‎.github/workflows/python-package.yml‎

Lines changed: 104 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,14 @@ jobs:
4141
matrix:
4242
os: [ubuntu-latest, windows-latest]
4343
python-version: ["3.10", "3.11", "3.12", "3.13"]
44+
test-type: [notspark, spark]
45+
exclude:
46+
# OSS CI does not install pyspark on Windows runners.
47+
- os: windows-latest
48+
test-type: spark
49+
# OSS CI does not install pyspark on Python 3.10.
50+
- python-version: "3.10"
51+
test-type: spark
4452
steps:
4553
- uses: actions/checkout@v4
4654
- name: Set up Python ${{ matrix.python-version }}
@@ -64,6 +72,7 @@ jobs:
6472
pip install -e .
6573
python -c "import flaml"
6674
pip install -e .[test]
75+
pip install coverage
6776
- name: On Ubuntu with pyspark, pin pandas<3 (pyspark doesn't support pandas 3.0 yet)
6877
if: matrix.os == 'ubuntu-latest' && matrix.python-version != '3.10'
6978
run: |
@@ -106,26 +115,59 @@ jobs:
106115
- name: Clear pip cache
107116
run: |
108117
pip cache purge
109-
- name: Test with pytest
118+
# Tests are split into a "notspark" variant (everything except Spark
119+
# tests, with `-m "not spark"`) and a "spark" variant (only Spark
120+
# tests, with `-m "spark"`), mirroring the internal Azure DevOps
121+
# pipeline at .pipelines/build.yml in the internal fork.
122+
- name: Test with pytest (notspark)
123+
if: matrix.test-type == 'notspark'
110124
timeout-minutes: 120
111-
if: matrix.python-version != '3.11'
125+
shell: bash
112126
run: |
113-
pytest test/ --ignore=test/autogen --ignore=test/spark/test_internal_mlflow.py --reruns 2 --reruns-delay 10
114-
- name: Coverage
127+
coverage run -m pytest test/ \
128+
--ignore=test/autogen \
129+
--ignore=test/spark \
130+
--reruns 2 --reruns-delay 10 \
131+
-m "not spark"
132+
- name: Test with pytest (spark)
133+
if: matrix.test-type == 'spark'
115134
timeout-minutes: 120
116-
if: matrix.python-version == '3.11'
135+
shell: bash
117136
run: |
118-
pip install coverage
119-
coverage run -a -m pytest test --ignore=test/autogen --ignore=test/spark/test_internal_mlflow.py --reruns 2 --reruns-delay 10
120-
coverage xml
121-
- name: Upload coverage to Codecov
122-
if: matrix.python-version == '3.11'
123-
uses: codecov/codecov-action@v3
137+
coverage run -m pytest test/ \
138+
--ignore=test/autogen \
139+
--ignore=test/spark/test_internal_mlflow.py \
140+
--reruns 2 --reruns-delay 10 \
141+
-m "spark"
142+
# .coveragerc enables `parallel = true`, so each subprocess writes its
143+
# own .coverage.<host>.<pid>.<rand> shard. Combine them into a single
144+
# per-job .coverage file so one artifact = one job's coverage.
145+
- name: Combine per-job coverage data
146+
if: always() && matrix.os == 'ubuntu-latest'
147+
shell: bash
148+
run: |
149+
set -e
150+
shopt -s nullglob
151+
shards=( .coverage .coverage.* )
152+
if [ ${#shards[@]} -eq 0 ]; then
153+
echo "No coverage shards produced; skipping combine."
154+
exit 0
155+
fi
156+
coverage combine || true
157+
ls -la .coverage* || true
158+
- name: Upload coverage data artifact
159+
if: always() && matrix.os == 'ubuntu-latest' && hashFiles('.coverage') != ''
160+
uses: actions/upload-artifact@v4
124161
with:
125-
file: ./coverage.xml
126-
flags: unittests
162+
name: coverage-data-${{ matrix.os }}-py${{ matrix.python-version }}-${{ matrix.test-type }}
163+
path: .coverage
164+
if-no-files-found: ignore
165+
include-hidden-files: true
166+
retention-days: 1
127167
- name: Save dependencies
128-
if: github.ref == 'refs/heads/main'
168+
# Run for a single matrix entry on push to main to avoid concurrent
169+
# writes to the unit-tests-installed-dependencies branch.
170+
if: github.ref == 'refs/heads/main' && matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11' && matrix.test-type == 'notspark'
129171
shell: bash
130172
run: |
131173
git config --global user.name 'github-actions[bot]'
@@ -139,7 +181,54 @@ jobs:
139181
pip freeze > installed_all_dependencies_${{ matrix.python-version }}_${{ matrix.os }}.txt
140182
python test/check_dependency.py > installed_first_tier_dependencies_${{ matrix.python-version }}_${{ matrix.os }}.txt
141183
git add installed_*dependencies*.txt
142-
mv coverage.xml ./coverage_${{ matrix.python-version }}_${{ matrix.os }}.xml || true
184+
coverage xml -o ./coverage_${{ matrix.python-version }}_${{ matrix.os }}.xml || true
143185
git add -f ./coverage_${{ matrix.python-version }}_${{ matrix.os }}.xml || true
144186
git commit -m "Update installed dependencies for Python ${{ matrix.python-version }} on ${{ matrix.os }}" || exit 0
145187
git push origin "$BRANCH" --force
188+
189+
# Combine coverage data files from every test-matrix job into a single
190+
# XML report and upload it to Codecov as one merged result. Coverage is
191+
# only collected on Linux runners (see the build job) to avoid path
192+
# mismatches between Linux and Windows in the combined data file.
193+
coverage:
194+
name: Combine coverage and upload to Codecov
195+
if: always()
196+
needs: build
197+
runs-on: ubuntu-latest
198+
steps:
199+
- uses: actions/checkout@v4
200+
- uses: actions/setup-python@v5
201+
with:
202+
python-version: "3.11"
203+
- name: Install coverage
204+
run: pip install coverage
205+
- name: Download all coverage data artifacts
206+
uses: actions/download-artifact@v4
207+
with:
208+
path: coverage_artifacts
209+
pattern: coverage-data-*
210+
- name: Combine coverage data and produce XML
211+
shell: bash
212+
run: |
213+
set -euo pipefail
214+
shopt -s nullglob
215+
mkdir -p combined
216+
i=0
217+
for f in coverage_artifacts/*/.coverage; do
218+
i=$((i + 1))
219+
cp "$f" "combined/.coverage.$i"
220+
done
221+
if [ "$i" -eq 0 ]; then
222+
echo "No coverage data files were uploaded by build jobs."
223+
exit 0
224+
fi
225+
echo "Combining $i coverage data files..."
226+
coverage combine combined/.coverage.*
227+
coverage xml -i -o coverage.xml
228+
coverage report -m || true
229+
- name: Upload combined coverage to Codecov
230+
if: hashFiles('coverage.xml') != ''
231+
uses: codecov/codecov-action@v3
232+
with:
233+
file: ./coverage.xml
234+
flags: unittests

0 commit comments

Comments
 (0)