Skip to content

Commit 73061f0

Browse files
beveradbclaude
beveradb
and
claude
authored
ci: split integration tests into 4 jobs for progress visibility (#269)
* ci: split integration tests into 4 jobs for better progress visibility Split the monolithic integration test job into separate jobs: - core-models: 24-bit preservation + CLI tests (~6 min) - ensemble: preset + quality tests (~10 min) - multi-stem: multi-stem verification + pipelines (~3 min) - fast-tests: remote API, roformer, output tests (parallel on ubuntu) GPU jobs chain sequentially while fast-tests runs in parallel. Gate job aggregates all results for branch protection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: include workflow file in change detection filter Without this, workflow-only changes skip all integration tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: move separator_output test to core-models job (needs onnxruntime) test_separator_output_integration.py imports onnxruntime which is only available with gpu/cpu extras, not the base install. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: run GPU test jobs in parallel across multiple runners Remove sequential chaining - with 3 GPU runners available, all GPU jobs can run simultaneously for much faster total wall time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: move roformer tests to GPU job (need onnxruntime via audio_separator import) The roformer e2e tests use patch() on audio_separator.separator which triggers the onnxruntime import chain. Only test_remote_api remains in the fast-tests ubuntu job. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: rebalance to 3 parallel GPU jobs targeting ~7 min each Consolidate from 4 jobs (3 GPU + 1 ubuntu) to 3 GPU jobs balanced for equal wall time across 3 runners: ensemble-presets (~8 min): ensemble preset tests (heaviest file) core-models (~7 min): 24-bit + CLI + output + roformer tests stems-and-quality (~6 min): ensemble quality + multi-stem + remote API Eliminates the separate ubuntu fast-tests job — remote API tests are fast enough to bundle into a GPU job without waste. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent adc5539 commit 73061f0

1 file changed

Lines changed: 126 additions & 51 deletions

File tree

‎.github/workflows/run-integration-tests.yaml‎

Lines changed: 126 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -19,90 +19,158 @@ jobs:
1919
- 'tests/**'
2020
- 'pyproject.toml'
2121
- 'poetry.lock'
22+
- '.github/workflows/run-integration-tests.yaml'
2223
23-
run-integration-test:
24+
# ── Integration test jobs (parallel across 3 GPU runners) ──────────
25+
#
26+
# Balanced to ~7 min each so all 3 finish around the same time.
27+
#
28+
# ensemble-presets (~8 min): test_ensemble_integration (heaviest single file)
29+
# core-models (~7 min): test_24bit + test_cli + test_separator_output + roformer tests
30+
# stems-and-quality (~6 min): test_ensemble_meaningful + test_multi_stem + test_remote_api
31+
32+
ensemble-presets:
2433
needs: changes
2534
if: needs.changes.outputs.should_run == 'true'
2635
runs-on: [self-hosted, gpu]
27-
timeout-minutes: 30
36+
timeout-minutes: 15
2837
env:
29-
# Use persistent local directory on self-hosted runners instead of actions/cache.
30-
# Models are pre-downloaded to this path by the runner startup script, so there's
31-
# no need to download ~14GB of models on every CI run.
3238
AUDIO_SEPARATOR_MODEL_DIR: /opt/audio-separator-models
33-
3439
steps:
35-
- name: Checkout project
36-
uses: actions/checkout@v4
37-
40+
- uses: actions/checkout@v4
3841
- name: Verify GPU availability
39-
run: |
40-
echo "=== GPU Info ==="
41-
nvidia-smi
42-
echo ""
43-
echo "=== Driver Version ==="
44-
nvidia-smi --query-gpu=driver_version,name,memory.total --format=csv,noheader
45-
42+
run: nvidia-smi --query-gpu=driver_version,name,memory.total --format=csv,noheader
4643
- name: Set up Python
4744
uses: actions/setup-python@v5
4845
with:
4946
python-version: '3.13'
50-
51-
- name: Install pipx
52-
run: python -m pip install --user pipx && python -m pipx ensurepath
53-
54-
- name: Install poetry
55-
run: python -m pipx install poetry
56-
57-
- name: Setup PATH
58-
run: echo "$HOME/.local/bin" >> $GITHUB_PATH
59-
60-
- name: Install system dependencies
47+
- name: Install pipx and poetry
6148
run: |
62-
sudo apt-get update
63-
sudo apt-get install -y ffmpeg libsamplerate0 libsamplerate-dev
64-
65-
- name: Set up Python
49+
python -m pip install --user pipx && python -m pipx ensurepath
50+
python -m pipx install poetry
51+
echo "$HOME/.local/bin" >> $GITHUB_PATH
52+
- name: Install system dependencies
53+
run: sudo apt-get update && sudo apt-get install -y ffmpeg libsamplerate0 libsamplerate-dev
54+
- name: Set up Python with cache
6655
uses: actions/setup-python@v5
6756
with:
6857
python-version: '3.13'
6958
cache: poetry
70-
7159
- name: Install Poetry dependencies (GPU)
7260
run: poetry install -E gpu
73-
7461
- name: Verify pre-cached models
7562
run: |
76-
echo "Models directory contents:"
77-
ls -lh $AUDIO_SEPARATOR_MODEL_DIR
7863
MODEL_COUNT=$(ls -1 $AUDIO_SEPARATOR_MODEL_DIR | wc -l)
79-
echo "Total files: $MODEL_COUNT"
64+
echo "Pre-cached models: $MODEL_COUNT"
8065
if [ "$MODEL_COUNT" -lt 10 ]; then
8166
echo "::warning::Expected at least 10 pre-cached model files, found $MODEL_COUNT"
8267
fi
68+
- name: "Run: ensemble preset tests (~8 min)"
69+
run: poetry run pytest -sv tests/integration/test_ensemble_integration.py
70+
- name: Upload test artifacts
71+
if: always()
72+
uses: actions/upload-artifact@v4
73+
with:
74+
name: ensemble-presets-results
75+
path: |
76+
*.flac
77+
tests/*.flac
8378
84-
- name: Run integration tests
85-
run: poetry run pytest -sv --cov=audio_separator --cov-report=xml tests/integration
86-
87-
- name: Upload coverage reports to Codecov
88-
uses: codecov/codecov-action@v3
89-
env:
90-
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
79+
core-models:
80+
needs: changes
81+
if: needs.changes.outputs.should_run == 'true'
82+
runs-on: [self-hosted, gpu]
83+
timeout-minutes: 15
84+
env:
85+
AUDIO_SEPARATOR_MODEL_DIR: /opt/audio-separator-models
86+
steps:
87+
- uses: actions/checkout@v4
88+
- name: Set up Python
89+
uses: actions/setup-python@v5
90+
with:
91+
python-version: '3.13'
92+
- name: Install pipx and poetry
93+
run: |
94+
python -m pip install --user pipx && python -m pipx ensurepath
95+
python -m pipx install poetry
96+
echo "$HOME/.local/bin" >> $GITHUB_PATH
97+
- name: Install system dependencies
98+
run: sudo apt-get update && sudo apt-get install -y ffmpeg libsamplerate0 libsamplerate-dev
99+
- name: Set up Python with cache
100+
uses: actions/setup-python@v5
101+
with:
102+
python-version: '3.13'
103+
cache: poetry
104+
- name: Install Poetry dependencies (GPU)
105+
run: poetry install -E gpu
106+
- name: "Run: 24-bit, CLI, output, and roformer tests (~7 min)"
107+
run: |
108+
poetry run pytest -sv \
109+
tests/integration/test_24bit_preservation.py \
110+
tests/integration/test_cli_integration.py \
111+
tests/integration/test_separator_output_integration.py \
112+
tests/integration/test_roformer_audio_quality.py \
113+
tests/integration/test_roformer_backward_compatibility.py \
114+
tests/integration/test_roformer_config_validation.py \
115+
tests/integration/test_roformer_e2e.py \
116+
tests/integration/test_roformer_fallback_mechanism.py \
117+
tests/integration/test_roformer_model_switching.py \
118+
tests/integration/test_roformer_new_parameters.py
119+
- name: Upload test artifacts
120+
if: always()
121+
uses: actions/upload-artifact@v4
122+
with:
123+
name: core-models-results
124+
path: |
125+
*.flac
126+
tests/*.flac
91127
92-
- name: Upload test results
128+
stems-and-quality:
129+
needs: changes
130+
if: needs.changes.outputs.should_run == 'true'
131+
runs-on: [self-hosted, gpu]
132+
timeout-minutes: 15
133+
env:
134+
AUDIO_SEPARATOR_MODEL_DIR: /opt/audio-separator-models
135+
steps:
136+
- uses: actions/checkout@v4
137+
- name: Set up Python
138+
uses: actions/setup-python@v5
139+
with:
140+
python-version: '3.13'
141+
- name: Install pipx and poetry
142+
run: |
143+
python -m pip install --user pipx && python -m pipx ensurepath
144+
python -m pipx install poetry
145+
echo "$HOME/.local/bin" >> $GITHUB_PATH
146+
- name: Install system dependencies
147+
run: sudo apt-get update && sudo apt-get install -y ffmpeg libsamplerate0 libsamplerate-dev
148+
- name: Set up Python with cache
149+
uses: actions/setup-python@v5
150+
with:
151+
python-version: '3.13'
152+
cache: poetry
153+
- name: Install Poetry dependencies (GPU)
154+
run: poetry install -E gpu
155+
- name: "Run: ensemble quality, multi-stem, and remote API tests (~6 min)"
156+
run: |
157+
poetry run pytest -sv \
158+
tests/integration/test_ensemble_meaningful.py \
159+
tests/integration/test_multi_stem_verification.py \
160+
tests/integration/test_remote_api_integration.py
161+
- name: Upload test artifacts
93162
if: always()
94163
uses: actions/upload-artifact@v4
95164
with:
96-
name: integration-test-results
165+
name: stems-and-quality-results
97166
path: |
98167
*.flac
99168
tests/*.flac
100-
**/temp_images/**/*.png
101-
tests/**/temp_images/**/*.png
102169
103-
# Gate job for branch protection - always reports a status
170+
# ── Gate job for branch protection ────────────────────────────────
171+
104172
integration-test:
105-
needs: [changes, run-integration-test]
173+
needs: [changes, ensemble-presets, core-models, stems-and-quality]
106174
if: always()
107175
runs-on: ubuntu-latest
108176
steps:
@@ -112,8 +180,15 @@ jobs:
112180
echo "Tests skipped - no code changes detected"
113181
exit 0
114182
fi
115-
if [[ "${{ needs.run-integration-test.result }}" == "failure" ]]; then
183+
184+
echo "ensemble-presets: ${{ needs.ensemble-presets.result }}"
185+
echo "core-models: ${{ needs.core-models.result }}"
186+
echo "stems-and-quality: ${{ needs.stems-and-quality.result }}"
187+
188+
if [[ "${{ needs.ensemble-presets.result }}" == "failure" ]] || \
189+
[[ "${{ needs.core-models.result }}" == "failure" ]] || \
190+
[[ "${{ needs.stems-and-quality.result }}" == "failure" ]]; then
116191
echo "Integration tests failed"
117192
exit 1
118193
fi
119-
echo "Integration tests passed"
194+
echo "All integration tests passed"

0 commit comments

Comments
 (0)