Skip to content

Commit c404061

Browse files
ko3n1gclaude
ko3n1g
and
claude
committed
ci(speech): split L0_Unit_Tests_GPU_ASR into 5 parallel buckets
The monolithic job ran 40+ min total. Each bucket targets ≤10 min, distributed by observed wall-clock time from run 25112807335. Bucket mapping (approx times): 1 (~6.6m): confidence/ + fast decoding tests 2 (~9.8m): streaming/rnnt decoding + inference/ + k2/ + mixins/ 3 (~8.6m): numba/ + hybrid/interctc/local_attn models 4 (~10.6m): test_asr_multitask_model_bpe.py (single file, irreducible) 5 (~4.9m): rnnt encoder models + remaining small tests + utils/ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: oliver könig <okoenig@nvidia.com>
1 parent 0558934 commit c404061

6 files changed

Lines changed: 129 additions & 3 deletions

File tree

‎.github/workflows/cicd-main-speech.yml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,21 @@ jobs:
4646
fail-fast: false
4747
matrix:
4848
include:
49-
- script: L0_Unit_Tests_GPU_ASR
49+
- script: L0_Unit_Tests_GPU_ASR_1
5050
runner: ${{ inputs.runner }}
51-
timeout: 60
51+
timeout: 15
52+
- script: L0_Unit_Tests_GPU_ASR_2
53+
runner: ${{ inputs.runner }}
54+
timeout: 15
55+
- script: L0_Unit_Tests_GPU_ASR_3
56+
runner: ${{ inputs.runner }}
57+
timeout: 15
58+
- script: L0_Unit_Tests_GPU_ASR_4
59+
runner: ${{ inputs.runner }}
60+
timeout: 15
61+
- script: L0_Unit_Tests_GPU_ASR_5
62+
runner: ${{ inputs.runner }}
63+
timeout: 15
5264
- script: L0_Unit_Tests_CPU_ASR
5365
runner: ${{ inputs.runner }}
5466
cpu-only: true

‎tests/functional_tests/L0_Unit_Tests_GPU_ASR_1.sh‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Copyright (c) 2020-2025, NVIDIA CORPORATION.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
NEMO_NUMBA_MINVER=0.53 CUDA_VISIBLE_DEVICES=0 coverage run -a --data-file=/workspace/.coverage --source=/workspace/ -m pytest \
15+
tests/collections/asr/confidence \
16+
tests/collections/asr/decoding/test_batched_beam_decoding.py \
17+
tests/collections/asr/decoding/test_batched_beam_hyps.py \
18+
tests/collections/asr/decoding/test_batched_hyps_and_alignments.py \
19+
tests/collections/asr/decoding/test_biasing_multi_model.py \
20+
tests/collections/asr/decoding/test_ctc_decoding.py \
21+
tests/collections/asr/decoding/test_cuda_graph_rnnt_greedy_decoding.py \
22+
tests/collections/asr/decoding/test_multi_task_decoding.py \
23+
-m "not pleasefixme" --with_downloads

‎tests/functional_tests/L0_Unit_Tests_GPU_ASR_2.sh‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Copyright (c) 2020-2025, NVIDIA CORPORATION.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
NEMO_NUMBA_MINVER=0.53 CUDA_VISIBLE_DEVICES=0 coverage run -a --data-file=/workspace/.coverage --source=/workspace/ -m pytest \
15+
tests/collections/asr/decoding/test_multi_task_streaming_decoding.py \
16+
tests/collections/asr/decoding/test_rnnt_alignments.py \
17+
tests/collections/asr/decoding/test_rnnt_decoding.py \
18+
tests/collections/asr/decoding/test_streaming_buffer.py \
19+
tests/collections/asr/decoding/test_streaming_decoding.py \
20+
tests/collections/asr/inference \
21+
tests/collections/asr/k2 \
22+
tests/collections/asr/mixins \
23+
-m "not pleasefixme" --with_downloads

‎tests/functional_tests/L0_Unit_Tests_GPU_ASR_3.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Copyright (c) 2020-2025, NVIDIA CORPORATION.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
NEMO_NUMBA_MINVER=0.53 CUDA_VISIBLE_DEVICES=0 coverage run -a --data-file=/workspace/.coverage --source=/workspace/ -m pytest \
15+
tests/collections/asr/numba \
16+
tests/collections/asr/test_asr_classification_model.py \
17+
tests/collections/asr/test_asr_context_biasing.py \
18+
tests/collections/asr/test_asr_ctc_encoder_model_bpe.py \
19+
tests/collections/asr/test_asr_ctcencdec_model.py \
20+
tests/collections/asr/test_asr_datasets.py \
21+
tests/collections/asr/test_asr_eou.py \
22+
tests/collections/asr/test_asr_exportables.py \
23+
tests/collections/asr/test_asr_filterbankfeatures_seq_len.py \
24+
tests/collections/asr/test_asr_hybrid_rnnt_ctc_model_bpe.py \
25+
tests/collections/asr/test_asr_hybrid_rnnt_ctc_model_bpe_prompt.py \
26+
tests/collections/asr/test_asr_hybrid_rnnt_ctc_model_char.py \
27+
tests/collections/asr/test_asr_interctc_models.py \
28+
tests/collections/asr/test_asr_lhotse_dataset.py \
29+
tests/collections/asr/test_asr_lhotse_speaker_dataset.py \
30+
tests/collections/asr/test_asr_local_attn.py \
31+
tests/collections/asr/test_asr_metrics.py \
32+
tests/collections/asr/test_asr_modules.py \
33+
tests/collections/asr/test_asr_multitalker_models.py \
34+
-m "not pleasefixme" --with_downloads

tests/functional_tests/L0_Unit_Tests_GPU_ASR.sh renamed to tests/functional_tests/L0_Unit_Tests_GPU_ASR_4.sh

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,6 @@
1111
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
14-
python -c "from nemo.collections.asr.models import ASRModel" && NEMO_NUMBA_MINVER=0.53 CUDA_VISIBLE_DEVICES=0 coverage run -a --data-file=/workspace/.coverage --source=/workspace/ -m pytest tests/collections/asr -m "not pleasefixme" --with_downloads
14+
NEMO_NUMBA_MINVER=0.53 CUDA_VISIBLE_DEVICES=0 coverage run -a --data-file=/workspace/.coverage --source=/workspace/ -m pytest \
15+
tests/collections/asr/test_asr_multitask_model_bpe.py \
16+
-m "not pleasefixme" --with_downloads

‎tests/functional_tests/L0_Unit_Tests_GPU_ASR_5.sh‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Copyright (c) 2020-2025, NVIDIA CORPORATION.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
NEMO_NUMBA_MINVER=0.53 CUDA_VISIBLE_DEVICES=0 coverage run -a --data-file=/workspace/.coverage --source=/workspace/ -m pytest \
15+
tests/collections/asr/test_asr_parts_submodules_batchnorm.py \
16+
tests/collections/asr/test_asr_regression_model.py \
17+
tests/collections/asr/test_asr_rnnt_encdec_model.py \
18+
tests/collections/asr/test_asr_rnnt_encoder_model_bpe.py \
19+
tests/collections/asr/test_asr_samplers.py \
20+
tests/collections/asr/test_asr_subsampling.py \
21+
tests/collections/asr/test_boosting_tree.py \
22+
tests/collections/asr/test_conformer_encoder.py \
23+
tests/collections/asr/test_custom_tokenizer.py \
24+
tests/collections/asr/test_jasper_block.py \
25+
tests/collections/asr/test_label_datasets.py \
26+
tests/collections/asr/test_ngram_lm.py \
27+
tests/collections/asr/test_padding_and_batch_size_invariance.py \
28+
tests/collections/asr/test_preprocessing_segment.py \
29+
tests/collections/asr/test_ssl_models.py \
30+
tests/collections/asr/test_text_to_text_dataset.py \
31+
tests/collections/asr/utils \
32+
-m "not pleasefixme" --with_downloads

0 commit comments

Comments
 (0)