Skip to content

Commit b0778e0

Browse files
sd-dbsd-db
authored
chore: add dependency cache for fork PR CI support (#1386)
### Description Fork PRs cannot authenticate to JFrog (no OIDC token available). This adds a cache-based dependency strategy so fork PRs get full CI feedback. - Add warmDepsCache.yml: trusted workflow that downloads all deps via JFrog and saves to GitHub Actions cache (triggers on push to main, daily schedule, and manual dispatch with optional PR number) - Add setup-python-deps composite action: restores cached deps and enables offline mode (UV_OFFLINE + PIP_NO_INDEX) - Update main.yml to use setup-python-deps instead of setup-jfrog-pypi, remove id-token:write permission (no longer needed) ### Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [ ] This PR includes tests, or tests are not required/relevant for this PR - NA - [ ] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section. - NA
1 parent 2f11abb commit b0778e0

File tree

3 files changed

+194
-13
lines changed

3 files changed

+194
-13
lines changed

‎.github/actions/setup-python-deps/action.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: "Setup Python Dependencies"
2+
description: |
3+
Restores pre-cached Python dependencies and enables offline mode.
4+
Outputs cache-hit so callers can fall back to setup-jfrog-pypi on miss.
5+
6+
outputs:
7+
cache-hit:
8+
description: "Whether the dependency cache was restored and offline mode enabled"
9+
value: ${{ steps.uv-cache.outputs.cache-matched-key != '' }}
10+
11+
runs:
12+
using: "composite"
13+
steps:
14+
- name: Restore uv and pip cache
15+
id: uv-cache
16+
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
17+
with:
18+
path: |
19+
~/.cache/uv
20+
~/.cache/pip
21+
key: python-deps-${{ hashFiles('uv.lock', 'pyproject.toml') }}-latest
22+
restore-keys: python-deps-${{ hashFiles('uv.lock', 'pyproject.toml') }}-
23+
24+
- name: Restore pre-commit cache
25+
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
26+
with:
27+
path: ~/.cache/pre-commit
28+
key: pre-commit-deps-${{ hashFiles('.pre-commit-config.yaml') }}-latest
29+
restore-keys: pre-commit-deps-${{ hashFiles('.pre-commit-config.yaml') }}-
30+
31+
- name: Enable offline mode
32+
if: steps.uv-cache.outputs.cache-matched-key != ''
33+
shell: bash
34+
run: |
35+
echo "UV_OFFLINE=true" >> "$GITHUB_ENV"
36+
echo "PIP_NO_INDEX=1" >> "$GITHUB_ENV"
37+
mkdir -p ~/.config/pip
38+
printf '[global]\nno-index = true\n' > ~/.config/pip/pip.conf

‎.github/workflows/main.yml‎

Lines changed: 22 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
# **what?**
22
# Runs code quality checks, unit tests, and verifies python build on
3-
# all code commited to the repository. This workflow should not
4-
# require any secrets since it runs for PRs from forked repos.
5-
# By default, secrets are not passed to workflows running from
6-
# a forked repo.
3+
# all code commited to the repository. Dependencies are served from a
4+
# pre-populated cache (see warmDepsCache.yml) when available, with a
5+
# JFrog OIDC fallback when the cache is cold.
76

87
# **why?**
98
# Ensure code for dbt meets a certain quality standard.
@@ -54,16 +53,17 @@ jobs:
5453
env:
5554
UV_FROZEN: "1"
5655

57-
strategy:
58-
fail-fast: false
59-
6056
steps:
6157
- name: Check out the repository
6258
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
6359

64-
- name: Setup JFrog PyPI Proxy
65-
uses: ./.github/actions/setup-jfrog-pypi
60+
- name: Setup Python Dependencies
61+
id: deps
62+
uses: ./.github/actions/setup-python-deps
6663

64+
- name: Setup JFrog PyPI Proxy (fallback)
65+
if: steps.deps.outputs.cache-hit != 'true'
66+
uses: ./.github/actions/setup-jfrog-pypi
6767

6868
- name: Set up Python
6969
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
@@ -109,9 +109,13 @@ jobs:
109109
- name: Check out the repository
110110
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
111111

112-
- name: Setup JFrog PyPI Proxy
113-
uses: ./.github/actions/setup-jfrog-pypi
112+
- name: Setup Python Dependencies
113+
id: deps
114+
uses: ./.github/actions/setup-python-deps
114115

116+
- name: Setup JFrog PyPI Proxy (fallback)
117+
if: steps.deps.outputs.cache-hit != 'true'
118+
uses: ./.github/actions/setup-jfrog-pypi
115119

116120
- name: Set up Python ${{ matrix.python-version }}
117121
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
@@ -129,6 +133,7 @@ jobs:
129133

130134
# Only run coverage comment once (not for all python versions)
131135
- name: Coverage Comment
136+
id: coverage_comment
132137
if: matrix.python-version == '3.12' && github.event_name == 'pull_request'
133138
uses: py-cov-action/python-coverage-comment-action@7188638f871f721a365d644f505d1ff3df20d683 # v3
134139
with:
@@ -154,9 +159,13 @@ jobs:
154159
- name: Check out the repository
155160
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
156161

157-
- name: Setup JFrog PyPI Proxy
158-
uses: ./.github/actions/setup-jfrog-pypi
162+
- name: Setup Python Dependencies
163+
id: deps
164+
uses: ./.github/actions/setup-python-deps
159165

166+
- name: Setup JFrog PyPI Proxy (fallback)
167+
if: steps.deps.outputs.cache-hit != 'true'
168+
uses: ./.github/actions/setup-jfrog-pypi
160169

161170
- name: Set up Python
162171
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5

‎.github/workflows/warmDepsCache.yml‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Warm Python Dependency Cache
2+
#
3+
# Pre-downloads all Python dependencies via JFrog Artifactory and saves them
4+
# to the GitHub Actions cache. PR workflows (including fork PRs, which cannot
5+
# authenticate to JFrog) restore this cache and build fully offline.
6+
#
7+
# Triggers:
8+
# - push to main when dependency files change (keeps cache fresh)
9+
# - daily schedule (prevents 7-day GitHub Actions cache eviction)
10+
# - manual dispatch (with optional PR number to warm cache for a fork's deps)
11+
12+
name: Warm Python Dependency Cache
13+
14+
on:
15+
push:
16+
branches: [main]
17+
paths:
18+
- "uv.lock"
19+
- "pyproject.toml"
20+
- ".pre-commit-config.yaml"
21+
schedule:
22+
- cron: "0 6 * * *" # Daily at 06:00 UTC
23+
workflow_dispatch:
24+
inputs:
25+
pr_number:
26+
description: "PR number to warm cache for (reads lockfiles from the PR branch). Leave empty to warm from main."
27+
required: false
28+
type: string
29+
30+
permissions:
31+
id-token: write
32+
contents: read
33+
pull-requests: read
34+
35+
jobs:
36+
warm-cache:
37+
runs-on:
38+
group: databricks-protected-runner-group
39+
labels: linux-ubuntu-latest
40+
41+
env:
42+
UV_FROZEN: "1"
43+
44+
steps:
45+
- name: Checkout main branch
46+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
47+
48+
- name: Overlay PR dependency files
49+
if: inputs.pr_number != ''
50+
shell: bash
51+
run: |
52+
set -euo pipefail
53+
54+
PR_DATA=$(curl -sLS \
55+
-H "Accept: application/vnd.github+json" \
56+
-H "Authorization: Bearer ${{ github.token }}" \
57+
"https://api.github.com/repos/${{ github.repository }}/pulls/${{ inputs.pr_number }}")
58+
59+
FORK_REPO=$(echo "$PR_DATA" | jq -r '.head.repo.full_name')
60+
FORK_REF=$(echo "$PR_DATA" | jq -r '.head.ref')
61+
62+
echo "Warming cache for PR #${{ inputs.pr_number }} from ${FORK_REPO}@${FORK_REF}"
63+
64+
# Fetch only lockfiles from the fork — .github/actions/ always
65+
# comes from main to prevent code injection from forks.
66+
git remote add fork "https://github.com/${FORK_REPO}.git"
67+
git fetch --depth=1 fork "${FORK_REF}"
68+
git checkout FETCH_HEAD -- uv.lock pyproject.toml .pre-commit-config.yaml
69+
70+
- name: Setup JFrog PyPI Proxy
71+
uses: ./.github/actions/setup-jfrog-pypi
72+
73+
- name: Set up Python
74+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
75+
with:
76+
python-version: "3.10"
77+
78+
- name: Install uv
79+
uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4
80+
81+
- name: Install Hatch
82+
uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc # install
83+
84+
- name: Install Python versions for test matrix
85+
run: uv python install 3.10 3.11 3.12 3.13
86+
87+
- name: Create all hatch environments (populates uv cache)
88+
run: |
89+
set -euo pipefail
90+
hatch env create default
91+
hatch env create test.py3.10
92+
hatch env create test.py3.11
93+
hatch env create test.py3.12
94+
hatch env create test.py3.13
95+
hatch env create verify
96+
97+
- name: Warm pre-commit cache
98+
run: pre-commit install-hooks
99+
100+
- name: Build and warm pip cache for verify environment
101+
run: |
102+
set -euo pipefail
103+
hatch -v build
104+
# Run verify to populate ~/.cache/pip/ with runtime transitive deps.
105+
# Allow failure — we only care about populating the cache.
106+
hatch run verify:check-all || true
107+
108+
- name: Generate cache key
109+
id: cache-key
110+
shell: bash
111+
run: |
112+
# Hash first so consumers can prefix-match on the hash alone.
113+
# Timestamp suffix ensures each run creates a new immutable entry;
114+
# consumers pick the latest timestamp for a given hash.
115+
TIMESTAMP=$(date -u +%Y%m%d%H%M%S)
116+
LOCK_HASH="${{ hashFiles('uv.lock', 'pyproject.toml') }}"
117+
echo "python-deps-key=python-deps-${LOCK_HASH}-${TIMESTAMP}" >> "$GITHUB_OUTPUT"
118+
119+
PRECOMMIT_HASH="${{ hashFiles('.pre-commit-config.yaml') }}"
120+
echo "pre-commit-key=pre-commit-deps-${PRECOMMIT_HASH}-${TIMESTAMP}" >> "$GITHUB_OUTPUT"
121+
122+
- name: Save uv and pip cache
123+
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
124+
with:
125+
path: |
126+
~/.cache/uv
127+
~/.cache/pip
128+
key: ${{ steps.cache-key.outputs.python-deps-key }}
129+
130+
- name: Save pre-commit cache
131+
uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
132+
with:
133+
path: ~/.cache/pre-commit
134+
key: ${{ steps.cache-key.outputs.pre-commit-key }}

0 commit comments

Comments
 (0)