Skip to content

Commit 0082e4c

Browse files
asnareasnare
authored
Modernize project setup, migrate from hatch to uv and pin dependencies (#2374)
## Changes This PR updates this project to use uv instead of Hatch. ### Relevant implementation details - All `Makefile` targets have been updated to use `uv` instead of `hatch`. - The CI/CD workflows have been updated to work with `uv` instead of `hatch`. - Locking information for all third-party dependencies has been added via `uv.lock` and `.build-constraints.txt`. - Workflow permissions are now scoped more narrowly. - Contribution documentation has been fixed. - The release workflow has been removed to prevent confusion. - The docusaurus `Makefile` targets have been refactored, and npm/yarn configured to not run any of the scripts associated with packages; we do not need these. Auto-update of the `yarn.lock` should no longer occur. ### Caveats/things to watch out for when reviewing: Aside from the GHA-related changes, some of the project setup can be checked manually while CI/CD isn't available: ```shell make clean dev fmt test labs test ``` The documentation setup needs further work, especially if a mirror should be used for packages instead of direct access to the npm/yarn package registries. This will be addressed in a subsequent PR. ### Tests - manually tested
1 parent 4cc88c7 commit 0082e4c

23 files changed

Lines changed: 4422 additions & 2282 deletions

‎.build-constraints.txt‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
hatchling==1.29.0 \
2+
--hash=sha256:50af9343281f34785fab12da82e445ed987a6efb34fd8c2fc0f6e6630dbcc1b0 \
3+
--hash=sha256:793c31816d952cee405b83488ce001c719f325d9cda69f1fc4cd750527640ea6
4+
packaging==26.0 \
5+
--hash=sha256:00243ae351a257117b6a241061796684b084ed1c516a08c48a3f7e147a9d80b4 \
6+
--hash=sha256:b36f1fef9334a5588b4166f8bcd26a14e521f2b55e6b9de3aaa80d3ff7a37529
7+
# via hatchling
8+
pathspec==1.0.4 \
9+
--hash=sha256:0210e2ae8a21a9137c0d470578cb0e595af87edaa6ebf12ff176f14a02e0e645 \
10+
--hash=sha256:fb6ae2fd4e7c921a165808a552060e722767cfa526f99ca5156ed2ce45a5c723
11+
# via hatchling
12+
pluggy==1.6.0 \
13+
--hash=sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3 \
14+
--hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746
15+
# via hatchling
16+
tomli==2.4.1 ; python_full_version < '3.11' \
17+
--hash=sha256:01f520d4f53ef97964a240a035ec2a869fe1a37dde002b57ebc4417a27ccd853 \
18+
--hash=sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe \
19+
--hash=sha256:136443dbd7e1dee43c68ac2694fde36b2849865fa258d39bf822c10e8068eac5 \
20+
--hash=sha256:1d8591993e228b0c930c4bb0db464bdad97b3289fb981255d6c9a41aedc84b2d \
21+
--hash=sha256:2190f2e9dd7508d2a90ded5ed369255980a1bcdd58e52f7fe24b8162bf9fedbd \
22+
--hash=sha256:2c1c351919aca02858f740c6d33adea0c5deea37f9ecca1cc1ef9e884a619d26 \
23+
--hash=sha256:36d2bd2ad5fb9eaddba5226aa02c8ec3fa4f192631e347b3ed28186d43be6b54 \
24+
--hash=sha256:3d48a93ee1c9b79c04bb38772ee1b64dcf18ff43085896ea460ca8dec96f35f6 \
25+
--hash=sha256:47149d5bd38761ac8be13a84864bf0b7b70bc051806bc3669ab1cbc56216b23c \
26+
--hash=sha256:4ab97e64ccda8756376892c53a72bd1f964e519c77236368527f758fbc36a53a \
27+
--hash=sha256:4b605484e43cdc43f0954ddae319fb75f04cc10dd80d830540060ee7cd0243cd \
28+
--hash=sha256:504aa796fe0569bb43171066009ead363de03675276d2d121ac1a4572397870f \
29+
--hash=sha256:51529d40e3ca50046d7606fa99ce3956a617f9b36380da3b7f0dd3dd28e68cb5 \
30+
--hash=sha256:52c8ef851d9a240f11a88c003eacb03c31fc1c9c4ec64a99a0f922b93874fda9 \
31+
--hash=sha256:559db847dc486944896521f68d8190be1c9e719fced785720d2216fe7022b662 \
32+
--hash=sha256:5a881ab208c0baf688221f8cecc5401bd291d67e38a1ac884d6736cbcd8247e9 \
33+
--hash=sha256:5cb41aa38891e073ee49d55fbc7839cfdb2bc0e600add13874d048c94aadddd1 \
34+
--hash=sha256:5e262d41726bc187e69af7825504c933b6794dc3fbd5945e41a79bb14c31f585 \
35+
--hash=sha256:5ee18d9ebdb417e384b58fe414e8d6af9f4e7a0ae761519fb50f721de398dd4e \
36+
--hash=sha256:7008df2e7655c495dd12d2a4ad038ff878d4ca4b81fccaf82b714e07eae4402c \
37+
--hash=sha256:734e20b57ba95624ecf1841e72b53f6e186355e216e5412de414e3c51e5e3c41 \
38+
--hash=sha256:7c7e1a961a0b2f2472c1ac5b69affa0ae1132c39adcb67aba98568702b9cc23f \
39+
--hash=sha256:7f86fd587c4ed9dd76f318225e7d9b29cfc5a9d43de44e5754db8d1128487085 \
40+
--hash=sha256:7f94b27a62cfad8496c8d2513e1a222dd446f095fca8987fceef261225538a15 \
41+
--hash=sha256:88dceee75c2c63af144e456745e10101eb67361050196b0b6af5d717254dddf7 \
42+
--hash=sha256:8a650c2dbafa08d42e51ba0b62740dae4ecb9338eefa093aa5c78ceb546fcd5c \
43+
--hash=sha256:8d65a2fbf9d2f8352685bc1364177ee3923d6baf5e7f43ea4959d7d8bc326a36 \
44+
--hash=sha256:96481a5786729fd470164b47cdb3e0e58062a496f455ee41b4403be77cb5a076 \
45+
--hash=sha256:a120733b01c45e9a0c34aeef92bf0cf1d56cfe81ed9d47d562f9ed591a9828ac \
46+
--hash=sha256:b1d22e6e9387bf4739fbe23bfa80e93f6b0373a7f1b96c6227c32bef95a4d7a8 \
47+
--hash=sha256:b8c198f8c1805dc42708689ed6864951fd2494f924149d3e4bce7710f8eb5232 \
48+
--hash=sha256:c2541745709bad0264b7d4705ad453b76ccd191e64aa6f0fc66b69a293a45ece \
49+
--hash=sha256:c742f741d58a28940ce01d58f0ab2ea3ced8b12402f162f4d534dfe18ba1cd6a \
50+
--hash=sha256:c7f2c7f2b9ca6bdeef8f0fa897f8e05085923eb091721675170254cbc5b02897 \
51+
--hash=sha256:d312ef37c91508b0ab2cee7da26ec0b3ed2f03ce12bd87a588d771ae15dcf82d \
52+
--hash=sha256:d4d8fe59808a54658fcc0160ecfb1b30f9089906c50b23bcb4c69eddc19ec2b4 \
53+
--hash=sha256:da25dc3563bff5965356133435b757a795a17b17d01dbc0f42fb32447ddfd917 \
54+
--hash=sha256:eab21f45c7f66c13f2a9e0e1535309cee140182a9cdae1e041d02e47291e8396 \
55+
--hash=sha256:eb0dc4e38e6a1fd579e5d50369aa2e10acfc9cace504579b2faabb478e76941a \
56+
--hash=sha256:ec9bfaf3ad2df51ace80688143a6a4ebc09a248f6ff781a9945e51937008fcbc \
57+
--hash=sha256:ede3e6487c5ef5d28634ba3f31f989030ad6af71edfb0055cbbd14189ff240ba \
58+
--hash=sha256:f3c6818a1a86dd6dca7ddcaaf76947d5ba31aecc28cb1b67009a5877c9a64f3f \
59+
--hash=sha256:f758f1b9299d059cc3f6546ae2af89670cb1c4d48ea29c3cacc4fe7de3058257 \
60+
--hash=sha256:f8f0fc26ec2cc2b965b7a3b87cd19c5c6b8c5e5f436b984e85f486d652285c30 \
61+
--hash=sha256:fd0409a3653af6c147209d267a0e4243f0ae46b011aa978b1080359fddc9b6cf \
62+
--hash=sha256:ff18e6a727ee0ab0388507b89d1bc6a22b138d1e2fa56d1ad494586d61d2eae9 \
63+
--hash=sha256:ff2983983d34813c1aeb0fa89091e76c3a22889ee83ab27c5eeb45100560c049
64+
# via hatchling
65+
trove-classifiers==2026.1.14.14 \
66+
--hash=sha256:00492545a1402b09d4858605ba190ea33243d361e2b01c9c296ce06b5c3325f3 \
67+
--hash=sha256:1f9553927f18d0513d8e5ff80ab8980b8202ce37ecae0e3274ed2ef11880e74d
68+
# via hatchling

‎.codegen.json‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,12 @@
33
"src/databricks/labs/lakebridge/__about__.py": "__version__ = \"$VERSION\""
44
},
55
"toolchain": {
6-
"required": ["hatch"],
7-
"pre_setup": ["hatch env create"],
6+
"required": ["make", "uv"],
7+
"pre_setup": ["make dev"],
88
"prepend_path": ".venv/bin",
9-
"acceptance_path": "tests/integration"
9+
"acceptance_path": "tests/integration",
10+
"test": [
11+
"make test"
12+
]
1013
}
1114
}

‎.github/actions/jfrog-auth/action.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: 'Authenticate for JFrog'
2+
description: 'Authenticate with JFrog using OIDC based on the GitHub repository.'
3+
# TODO: Factor out into an external action, once releases are allowed.
4+
outputs:
5+
jfrog-access-token:
6+
description: "Access token for JFrog"
7+
value: "${{ steps.jfrog-auth.outputs.jfrog-access-token }}"
8+
runs:
9+
using: "composite"
10+
steps:
11+
- id: jfrog-auth
12+
name: Authenticate against JFrog
13+
shell: bash
14+
run: |
15+
"${GITHUB_ACTION_PATH}/jfrog-auth" "${ACTIONS_ID_TOKEN_REQUEST_URL}" "${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
16+
17+
- id: detect-cmds
18+
name: Detecting python package/project managers.
19+
shell: bash
20+
run: |
21+
for cmd in pip3 uv
22+
do
23+
command -v "${cmd}" > /dev/null && found=true || found=false
24+
printf '::debug::%s\n' "Found ${cmd}: ${found}"
25+
printf '%s=%s\n' "command_${cmd}" "${found}" >> "${GITHUB_OUTPUT}"
26+
done
27+
28+
- name: Configure pip for JFrog
29+
if: "${{ steps.detect-cmds.outputs.command_pip3 == 'true' }}"
30+
shell: bash
31+
env:
32+
JFROG_ACCESS_TOKEN: "${{ steps.jfrog-auth.outputs.jfrog-access-token }}"
33+
run: |
34+
umask 077
35+
cat > "$RUNNER_TEMP/.pip.conf" <<EOF
36+
[global]
37+
index-url = https://gha-service-account:${JFROG_ACCESS_TOKEN}@databricks.jfrog.io/artifactory/api/pypi/db-pypi/simple
38+
EOF
39+
printf '%s=%s\n' 'PIP_CONFIG_FILE' "${RUNNER_TEMP}/.pip.conf" >> "${GITHUB_ENV}"
40+
41+
- name: Configure uv for JFrog
42+
if: "${{ steps.detect-cmds.outputs.command_uv == 'true' }}"
43+
shell: bash
44+
env:
45+
JFROG_ACCESS_TOKEN: "${{ steps.jfrog-auth.outputs.jfrog-access-token }}"
46+
UV_INDEX_URL: 'https://databricks.jfrog.io/artifactory/api/pypi/db-pypi/simple'
47+
run: |
48+
uv auth login "${UV_INDEX_URL}" --username gha-service-account --password "${JFROG_ACCESS_TOKEN}"
49+
printf "%s=%s\n" 'UV_INDEX_URL' "${UV_INDEX_URL}" >> "${GITHUB_ENV}"
50+
printf "%s=%s\n" 'UV_FROZEN' '1' >> "${GITHUB_ENV}"

‎.github/actions/jfrog-auth/jfrog-auth‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#!/bin/sh
2+
#
3+
# Obtain a JFrog access token, assuming GitHub OIDC.
4+
#
5+
set -eu
6+
7+
_request_url="$1"
8+
_request_token="$2"
9+
10+
#
11+
# Step 1: Obtain the OIDC identifier token from GitHub.
12+
#
13+
printf '::debug::%s\n' "Fetching OIDC identifier token from GitHub..."
14+
_id_token="$(curl -sLS \
15+
-H 'User-Agent: actions/oidc-client' \
16+
-H "Authorization: Bearer ${_request_token}" \
17+
"${_request_url}&audience=jfrog-github" |
18+
jq -r .value)"
19+
printf '::add-mask::%s\n' "${_id_token}"
20+
21+
#
22+
# Step 2: Exchange it for the JFrog access token.
23+
#
24+
printf '::debug::%s\n' "Exchanging OIDC identifier token for JFrog access token..."
25+
_access_token=$(curl -sLS \
26+
--json "{\"grant_type\": \"urn:ietf:params:oauth:grant-type:token-exchange\", \"subject_token_type\":\"urn:ietf:params:oauth:token-type:id_token\", \"subject_token\": \"${_id_token}\", \"provider_name\": \"github-actions\"}" \
27+
"https://databricks.jfrog.io/access/api/v1/oidc/token" |
28+
jq -r .access_token)
29+
printf '::add-mask::%s\n' "${_access_token}"
30+
31+
if [ -z "${_access_token}" ] || [ "${_access_token}" = 'null' ]
32+
then
33+
printf '::error::%s\n' "Could not fetch JFrog access token."
34+
exit 1
35+
fi
36+
37+
printf '%s=%s\n' 'jfrog-access-token' "${_access_token}" >> "${GITHUB_OUTPUT}"

‎.github/dependabot.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,16 @@
11
version: 2
22
updates:
3-
- package-ecosystem: "pip"
3+
- package-ecosystem: "uv"
44
directory: "/"
5+
cooldown:
6+
default-days: 7
7+
exclude:
8+
- "databricks*"
59
schedule:
610
interval: "daily"
7-
- package-ecosystem: "github-actions"
8-
directory: "/"
11+
- package-ecosystem: "npm"
12+
directory: "/docs/lakebridge"
13+
cooldown:
14+
default-days: 7
915
schedule:
1016
interval: "daily"

‎.github/scripts/setup_local_oracle.sh‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,16 @@
1-
#!/usr/bin/env bash
1+
#!/bin/bash
2+
# TODO: Replace this script with a GHA service container on the workflow.
23
set -Eeuo pipefail
34

45
# Config
56
ORACLE_CONTAINER="${ORACLE_CONTAINER:-oracle-free}"
6-
ORACLE_IMAGE="${ORACLE_IMAGE:-container-registry.oracle.com/database/free:latest-lite}"
7+
ORACLE_IMAGE="${ORACLE_IMAGE:-container-registry.oracle.com/database/free@sha256:481dbb4a1ea7cac6aadd354ff42b48fb7e4df955725158f237ad58c8fca1f458}" # latest-lite
78
ORACLE_PORT="${ORACLE_PORT:-1521}"
89
ORACLE_PWD="${ORACLE_PWD:?export ORACLE_PWD for SYS}"
910
ORACLE_SID="${ORACLE_SID:-FREEPDB1}"
1011

1112
# Dependencies
12-
command -v docker >/dev/null || { echo "Docker not installed" >&2; exit 2; }
13+
command -v docker >/dev/null || { printf "Docker not installed\n" >&2; exit 2; }
1314

1415
# Image present?
1516
docker image inspect "${ORACLE_IMAGE}" >/dev/null 2>&1 || docker pull "${ORACLE_IMAGE}"
@@ -24,19 +25,19 @@ else
2425
-p "${ORACLE_PORT}:1521" \
2526
-e ORACLE_PWD="${ORACLE_PWD}" \
2627
-d "${ORACLE_IMAGE}" >/dev/null
27-
echo "Starting Oracle container..."
28+
printf "Starting Oracle container...\n"
2829
fi
2930

30-
echo "Waiting up to 5 minutes for Oracle to be healthy..."
31+
printf "Waiting up to 5 minutes for Oracle to be healthy...\n"
3132
MAX_WAIT=300; WAIT_INTERVAL=5; waited=0
3233
while :; do
3334
state="$(docker inspect -f '{{.State.Health.Status}}' "${ORACLE_CONTAINER}" 2>/dev/null || true)"
34-
echo "health=${state:-unknown} waited=${waited}s"
35+
printf "health=%s waited=%ss\n" "${state:-unknown}" "${waited}"
3536
[[ "$state" == "healthy" ]] && break
36-
(( waited >= MAX_WAIT )) && { echo "ERROR: Oracle not healthy in 300s" >&2; exit 1; }
37+
(( waited >= MAX_WAIT )) && { printf "ERROR: Oracle not healthy in 300s\n" >&2; exit 1; }
3738
sleep "$WAIT_INTERVAL"; waited=$((waited + WAIT_INTERVAL))
3839
done
39-
echo "Oracle is fully started."
40+
printf "Oracle is fully started.\n"
4041

4142
# SQL bootstrap as SYSDBA, target SYSTEM schema
4243
docker exec -i "${ORACLE_CONTAINER}" bash -lc \
@@ -92,4 +93,4 @@ WHEN NOT MATCHED THEN INSERT (ID, DESCR, YEAR, DATEE)
9293
COMMIT;
9394
SQL
9495

95-
echo "Oracle DDL/DML completed."
96+
printf "Oracle DDL/DML completed.\n"

‎.github/scripts/setup_mssql_odbc.sh‎

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,41 @@
1-
#!/usr/bin/env bash
2-
3-
set -xve
4-
#Repurposed from https://github.com/Yarden-zamir/install-mssql-odbc
1+
#!/bin/bash
2+
#
3+
# TODO: Replace with a job container that has msodbcsql18 pre-installed.
4+
#
5+
# Install the Microsoft ODBC driver for SQL Server.
6+
#
7+
# The FILE_MANIFEST from Microsoft is verified against a pinned checksum
8+
# before it is trusted to verify the package repository configuration.
9+
#
10+
# Repurposed from https://github.com/Yarden-zamir/install-mssql-odbc
11+
#
12+
set -eu
513

614
VERSION_ID="$(awk -F= '$1=="VERSION_ID"{gsub(/"/, "", $2); print $2}' /etc/os-release)"
715
PKG_NAME="packages-microsoft-prod.deb"
816

17+
# Pinned FILE_MANIFEST checksums per Ubuntu version.
18+
case "${VERSION_ID}" in
19+
22.04) manifest_sha256="3fc171cfaeb40e8c2103267b28277c43de02269bd35ac7cb897b2bbd9a40f256" ;;
20+
24.04) manifest_sha256="52830b24a2c53300daa3991cb91997288e7440c3c66782ac003e4cf6c0271d04" ;;
21+
*) printf "Unsupported Ubuntu version: %s\n" "${VERSION_ID}" >&2; exit 1 ;;
22+
esac
23+
924
FILE_MANIFEST_URL="https://packages.microsoft.com/config/ubuntu/${VERSION_ID}/FILE_MANIFEST"
1025
FILE_MANIFEST="hashes.txt"
1126

12-
curl -sSL -o "${FILE_MANIFEST}" "${FILE_MANIFEST_URL}"
27+
curl -fsSL -o "${FILE_MANIFEST}" "${FILE_MANIFEST_URL}"
28+
printf '%s %s\n' "${manifest_sha256}" "${FILE_MANIFEST}" | sha256sum -c >/dev/null
1329

14-
expected_hash=$(grep "${PKG_NAME}" "${FILE_MANIFEST}" | cut -d',' -f2)
30+
expected_hash="$(awk -F, -v pkg="${PKG_NAME}" '$1==pkg {print $2}' "${FILE_MANIFEST}")"
1531

16-
curl -sSL -O "https://packages.microsoft.com/config/ubuntu/${VERSION_ID}/${PKG_NAME}"
32+
curl -fsSL -O "https://packages.microsoft.com/config/ubuntu/${VERSION_ID}/${PKG_NAME}"
1733
printf "%s *packages-microsoft-prod.deb\n" "${expected_hash}" | sha256sum -c -
1834

1935
# Install the Microsoft package repository configuration file using dpkg.
20-
# The `--force-confold` option ensures that existing configuration files are not overwritten
21-
# The `DEBIAN_FRONTEND=noninteractive` environment variable suppresses interactive prompts
36+
# The --force-confold option ensures that existing configuration files are not overwritten.
37+
# The DEBIAN_FRONTEND=noninteractive environment variable suppresses interactive prompts.
2238
sudo DEBIAN_FRONTEND=noninteractive dpkg --force-confold -i packages-microsoft-prod.deb
23-
#rm packages-microsoft-prod.deb
2439

2540
sudo apt-get update
2641
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18

0 commit comments

Comments
 (0)