Skip to content

Commit 0f30308

Browse files
Copilotfunctionstackx
Copilot
and
functionstackx
committed
Refactor pip install to use GitHub Actions steps with Python caching
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
1 parent c24a0a4 commit 0f30308

9 files changed

Lines changed: 153 additions & 30 deletions

‎.github/workflows/collect-results.yml‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,15 @@ jobs:
1919
token: ${{ secrets.REPO_PAT }}
2020
fetch-depth: 0
2121

22+
- name: Set up Python
23+
uses: actions/setup-python@v6
24+
with:
25+
python-version: '3.12'
26+
cache: 'pip'
27+
28+
- name: Install dependencies
29+
run: pip install matplotlib
30+
2231
- name: Download JSON artifacts
2332
uses: actions/download-artifact@v6
2433
with:
@@ -38,9 +47,7 @@ jobs:
3847
path: agg_${{ inputs.exp-name || 'all' }}.json
3948

4049
- name: Plot performance
41-
run: |
42-
pip install -q matplotlib
43-
python3 utils/plot_perf.py results/ ${{ inputs.exp-name || 'all' }}
50+
run: python3 utils/plot_perf.py results/ ${{ inputs.exp-name || 'all' }}
4451
- name: Upload performance graphs
4552
uses: actions/upload-artifact@v5
4653
with:

‎.github/workflows/e2e-tests.yml‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,17 @@ jobs:
1818
- name: Checkout code
1919
uses: actions/checkout@v5
2020

21+
- name: Set up Python
22+
uses: actions/setup-python@v6
23+
with:
24+
python-version: '3.12'
25+
cache: 'pip'
26+
27+
- name: Install dependencies
28+
run: pip install pydantic pyyaml
29+
2130
- id: get-jobs
2231
run: |
23-
pip install pydantic
2432
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py ${{ inputs.generate-cli-command }})
2533
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
2634
@@ -70,15 +78,21 @@ jobs:
7078
token: ${{ secrets.REPO_PAT }}
7179
fetch-depth: 0
7280

81+
- name: Set up Python
82+
uses: actions/setup-python@v6
83+
with:
84+
python-version: '3.12'
85+
cache: 'pip'
86+
87+
- name: Install dependencies
88+
run: pip install PyGithub
89+
7390
- name: Download results artifacts
7491
uses: actions/download-artifact@v6
7592
with:
7693
path: ${{ env.RESULTS_DIR }}
7794
pattern: results_*
7895

79-
- name: Install python dependencies
80-
run: pip install PyGithub
81-
8296
- name: Calculate success rate
8397
run: python3 utils/calc_success_rate.py $STATS_FILENAME
8498

‎.github/workflows/full-sweep-1k1k-scheduler.yml‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,17 @@ jobs:
1414
- name: Checkout code
1515
uses: actions/checkout@v5
1616

17+
- name: Set up Python
18+
uses: actions/setup-python@v6
19+
with:
20+
python-version: '3.12'
21+
cache: 'pip'
22+
23+
- name: Install dependencies
24+
run: pip install pydantic pyyaml
25+
1726
- id: get-dsr1-configs
1827
run: |
19-
pip install pydantic
2028
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py full-sweep --config-files ${GITHUB_WORKSPACE}/.github/configs/nvidia-master.yaml ${GITHUB_WORKSPACE}/.github/configs/amd-master.yaml --seq-lens 1k1k --model-prefix dsr1)
2129
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
2230
@@ -28,9 +36,17 @@ jobs:
2836
- name: Checkout code
2937
uses: actions/checkout@v5
3038

39+
- name: Set up Python
40+
uses: actions/setup-python@v6
41+
with:
42+
python-version: '3.12'
43+
cache: 'pip'
44+
45+
- name: Install dependencies
46+
run: pip install pydantic pyyaml
47+
3148
- id: get-gptoss-configs
3249
run: |
33-
pip install pydantic
3450
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py full-sweep --config-files ${GITHUB_WORKSPACE}/.github/configs/nvidia-master.yaml ${GITHUB_WORKSPACE}/.github/configs/amd-master.yaml --seq-lens 1k1k --model-prefix gptoss)
3551
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
3652
@@ -159,15 +175,21 @@ jobs:
159175
token: ${{ secrets.REPO_PAT }}
160176
fetch-depth: 0
161177

178+
- name: Set up Python
179+
uses: actions/setup-python@v6
180+
with:
181+
python-version: '3.12'
182+
cache: 'pip'
183+
184+
- name: Install dependencies
185+
run: pip install PyGithub
186+
162187
- name: Download results artifacts
163188
uses: actions/download-artifact@v6
164189
with:
165190
path: ${{ env.RESULTS_DIR }}
166191
pattern: results_*
167192

168-
- name: Install python dependencies
169-
run: pip install PyGithub
170-
171193
- name: Calculate success rate
172194
run: python3 utils/calc_success_rate.py $STATS_FILENAME
173195

‎.github/workflows/full-sweep-1k8k-scheduler.yml‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,17 @@ jobs:
1414
- name: Checkout code
1515
uses: actions/checkout@v5
1616

17+
- name: Set up Python
18+
uses: actions/setup-python@v6
19+
with:
20+
python-version: '3.12'
21+
cache: 'pip'
22+
23+
- name: Install dependencies
24+
run: pip install pydantic pyyaml
25+
1726
- id: get-dsr1-configs
1827
run: |
19-
pip install pydantic
2028
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py full-sweep --config-files ${GITHUB_WORKSPACE}/.github/configs/nvidia-master.yaml ${GITHUB_WORKSPACE}/.github/configs/amd-master.yaml --seq-lens 1k8k --model-prefix dsr1)
2129
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
2230
@@ -28,9 +36,17 @@ jobs:
2836
- name: Checkout code
2937
uses: actions/checkout@v5
3038

39+
- name: Set up Python
40+
uses: actions/setup-python@v6
41+
with:
42+
python-version: '3.12'
43+
cache: 'pip'
44+
45+
- name: Install dependencies
46+
run: pip install pydantic pyyaml
47+
3148
- id: get-gptoss-configs
3249
run: |
33-
pip install pydantic
3450
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py full-sweep --config-files ${GITHUB_WORKSPACE}/.github/configs/nvidia-master.yaml ${GITHUB_WORKSPACE}/.github/configs/amd-master.yaml --seq-lens 1k8k --model-prefix gptoss)
3551
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
3652
@@ -114,15 +130,21 @@ jobs:
114130
token: ${{ secrets.REPO_PAT }}
115131
fetch-depth: 0
116132

133+
- name: Set up Python
134+
uses: actions/setup-python@v6
135+
with:
136+
python-version: '3.12'
137+
cache: 'pip'
138+
139+
- name: Install dependencies
140+
run: pip install PyGithub
141+
117142
- name: Download results artifacts
118143
uses: actions/download-artifact@v6
119144
with:
120145
path: ${{ env.RESULTS_DIR }}
121146
pattern: results_*
122147

123-
- name: Install python dependencies
124-
run: pip install PyGithub
125-
126148
- name: Calculate success rate
127149
run: python3 utils/calc_success_rate.py $STATS_FILENAME
128150

‎.github/workflows/full-sweep-8k1k-scheduler.yml‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,17 @@ jobs:
1414
- name: Checkout code
1515
uses: actions/checkout@v5
1616

17+
- name: Set up Python
18+
uses: actions/setup-python@v6
19+
with:
20+
python-version: '3.12'
21+
cache: 'pip'
22+
23+
- name: Install dependencies
24+
run: pip install pydantic pyyaml
25+
1726
- id: get-dsr1-configs
1827
run: |
19-
pip install pydantic
2028
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py full-sweep --config-files ${GITHUB_WORKSPACE}/.github/configs/nvidia-master.yaml ${GITHUB_WORKSPACE}/.github/configs/amd-master.yaml --seq-lens 8k1k --model-prefix dsr1)
2129
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
2230
@@ -28,9 +36,17 @@ jobs:
2836
- name: Checkout code
2937
uses: actions/checkout@v5
3038

39+
- name: Set up Python
40+
uses: actions/setup-python@v6
41+
with:
42+
python-version: '3.12'
43+
cache: 'pip'
44+
45+
- name: Install dependencies
46+
run: pip install pydantic pyyaml
47+
3148
- id: get-gptoss-configs
3249
run: |
33-
pip install pydantic
3450
CONFIG_JSON=$(python3 ${GITHUB_WORKSPACE}/utils/matrix-logic/generate_sweep_configs.py full-sweep --config-files ${GITHUB_WORKSPACE}/.github/configs/nvidia-master.yaml ${GITHUB_WORKSPACE}/.github/configs/amd-master.yaml --seq-lens 8k1k --model-prefix gptoss)
3551
echo "search-space-config=$CONFIG_JSON" >> $GITHUB_OUTPUT
3652
@@ -159,15 +175,21 @@ jobs:
159175
token: ${{ secrets.REPO_PAT }}
160176
fetch-depth: 0
161177

178+
- name: Set up Python
179+
uses: actions/setup-python@v6
180+
with:
181+
python-version: '3.12'
182+
cache: 'pip'
183+
184+
- name: Install dependencies
185+
run: pip install PyGithub
186+
162187
- name: Download results artifacts
163188
uses: actions/download-artifact@v6
164189
with:
165190
path: ${{ env.RESULTS_DIR }}
166191
pattern: results_*
167192

168-
- name: Install python dependencies
169-
run: pip install PyGithub
170-
171193
- name: Calculate success rate
172194
run: python3 utils/calc_success_rate.py $STATS_FILENAME
173195

‎.github/workflows/full-sweep-test.yml‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -48,11 +48,19 @@ jobs:
4848
- name: Checkout code
4949
uses: actions/checkout@v5
5050

51+
- name: Set up Python
52+
uses: actions/setup-python@v6
53+
with:
54+
python-version: '3.12'
55+
cache: 'pip'
56+
57+
- name: Install dependencies
58+
run: pip install pydantic pyyaml
59+
5160
# This looks complicated, but it is just calling generate_sweep_configs.py conditioned on
5261
# discrete inputs (i.e., run_1k1k, run_h100, etc.) to split the test sweep into discrete jobs
5362
- id: generate-configs
5463
run: |
55-
pip install pydantic
5664
5765
set -x
5866
# Build runner type filters based on inputs
@@ -427,15 +435,21 @@ jobs:
427435
token: ${{ secrets.REPO_PAT }}
428436
fetch-depth: 0
429437

438+
- name: Set up Python
439+
uses: actions/setup-python@v6
440+
with:
441+
python-version: '3.12'
442+
cache: 'pip'
443+
444+
- name: Install dependencies
445+
run: pip install PyGithub
446+
430447
- name: Download results artifacts
431448
uses: actions/download-artifact@v6
432449
with:
433450
path: ${{ env.RESULTS_DIR }}
434451
pattern: results_*
435452

436-
- name: Install python dependencies
437-
run: pip install PyGithub
438-
439453
- name: Calculate success rate
440454
run: python3 utils/calc_success_rate.py $STATS_FILENAME
441455

‎.github/workflows/label-validation.yml‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,15 @@ jobs:
1818
- name: Checkout code
1919
uses: actions/checkout@v4
2020

21+
- name: Set up Python
22+
uses: actions/setup-python@v6
23+
with:
24+
python-version: '3.12'
25+
cache: 'pip'
26+
27+
- name: Install dependencies
28+
run: pip install pydantic pyyaml
29+
2130
- id: get-jobs
2231
shell: python
2332
run: |
@@ -49,7 +58,6 @@ jobs:
4958
# Generate configs for standard labels
5059
all_configs = []
5160
if matching:
52-
subprocess.run(['pip', 'install', 'pydantic'], check=True)
5361
5462
for label in matching:
5563
result = subprocess.run([
@@ -125,15 +133,21 @@ jobs:
125133
token: ${{ secrets.REPO_PAT }}
126134
fetch-depth: 0
127135

136+
- name: Set up Python
137+
uses: actions/setup-python@v6
138+
with:
139+
python-version: '3.12'
140+
cache: 'pip'
141+
142+
- name: Install dependencies
143+
run: pip install PyGithub
144+
128145
- name: Download results artifacts
129146
uses: actions/download-artifact@v4
130147
with:
131148
path: ${{ env.RESULTS_DIR }}
132149
pattern: results_*
133150

134-
- name: Install python dependencies
135-
run: pip install PyGithub
136-
137151
- name: Calculate success rate
138152
run: python3 utils/calc_success_rate.py $STATS_FILENAME
139153

‎.github/workflows/test-matrix-logic.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ jobs:
2020
uses: actions/setup-python@v6
2121
with:
2222
python-version: '3.12'
23+
cache: 'pip'
2324

2425
- name: Install dependencies
2526
run: |

‎requirements-workflows.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Requirements for GitHub Actions workflows
2+
# Used by various workflow files to generate configs and process results
3+
pydantic>=2.0.0
4+
pyyaml>=6.0.0
5+
PyGithub>=2.0.0
6+
matplotlib>=3.0.0
7+
pytest>=7.0.0

0 commit comments

Comments
 (0)