Skip to content

Commit 09233b5

Browse files
JohTJohT
authored
Merge pull request #584 from JohT/feature/migrate-to-uv-as-primary-package-manager
Migrate to uv as primary package manager for Python
2 parents 451da9e + e13c00e commit 09233b5

25 files changed

Lines changed: 1944 additions & 262 deletions

‎.github/instructions/python-dependencies.instructions.md‎

Lines changed: 15 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,43 @@
11
---
2-
applyTo: "requirements.txt,conda-environment.yml"
2+
applyTo: "pyproject.toml,conda-environment.yml"
33
---
44
# Python Dependency Conventions
55

6-
Applies to [requirements.txt](../../requirements.txt), [conda-environment.yml](../../conda-environment.yml).
6+
[pyproject.toml](../../pyproject.toml), [conda-environment.yml](../../conda-environment.yml).
77

88
## Sync Rule
99

10-
Both files must stay **in sync**: same packages, same pinned versions. Package in one must be in other.
10+
`pyproject.toml` canonical source. `conda-environment.yml` stays synced: same packages, same pinned versions.
1111

12-
Exceptions (document inline with comment if they differ):
13-
- `conda-environment.yml` may split packages (e.g. `plotly[kaleido]` in pip vs. `plotly` + `python-kaleido` in conda)
14-
- `conda-environment.yml` may use `pip:` block for packages not on conda-forge
12+
Exceptions (inline comment if differ):
13+
- `conda-environment.yml` splits packages (e.g. `plotly[kaleido]` in pyproject.toml vs. `plotly` + `python-kaleido` in conda)
14+
- `conda-environment.yml` uses `pip:` block for packages not on conda-forge
1515

1616
## Versioning
1717

1818
- Pin all versions — no ranges, no `>=`, no unpinned entries
19-
- `requirements.txt`: `==` (e.g. `pandas==2.2.3`)
20-
- `conda-environment.yml`: `=` (e.g. `pandas=2.2.3`)
19+
- `pyproject.toml`: `==` (e.g. `"pandas==2.3.3"`)
20+
- `conda-environment.yml`: `=` (e.g. `pandas=2.3.3`)
2121

22-
## Adding or Updating
22+
## Updating
2323

24-
- **Ask user before introducing new dependencies**
24+
- **Ask user before adding dependencies**
2525
- Update both files when changing version
2626

2727
## Verification
2828

29-
**`requirements.txt`** — venv dry-run from [.github/workflows/internal-check-python-venv-support.yml](../../.github/workflows/internal-check-python-venv-support.yml):
29+
**`pyproject.toml`** — uv sync from [.github/workflows/internal-check-python-uv-support.yml](../../.github/workflows/internal-check-python-uv-support.yml):
3030

3131
```shell
32-
python -m venv .venv
33-
source ./scripts/activatePythonEnvironment.sh
34-
.venv/bin/pip install --dry-run --quiet --requirement requirements.txt
35-
! .venv/bin/pip install --dry-run --requirement requirements.txt 2>/dev/null | grep -q "Would install"
32+
uv sync --frozen
33+
uv lock --check
34+
uv run python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"
3635
```
3736

3837
**`conda-environment.yml`** — via [scripts/activateCondaEnvironment.sh](../../scripts/activateCondaEnvironment.sh):
3938

4039
```shell
41-
source ./scripts/activateCondaEnvironment.sh
40+
PYTHON_PACKAGE_MANAGER=conda source ./scripts/activateCondaEnvironment.sh
4241
```
4342

4443
Fallback (conda directly):

‎.github/instructions/renovate.instructions.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
applyTo: "renovate.json"
3+
---
4+
# Renovate Config Conventions
5+
6+
## Valid JSON
7+
8+
File must be valid JSON. No trailing commas. Validate with:
9+
10+
```shell
11+
npx --yes --package renovate -- renovate-config-validator
12+
```
13+
14+
## $schema
15+
16+
Top-level field must have schema reference:
17+
18+
```json
19+
{
20+
"$schema": "https://docs.renovatebot.com/renovate-schema.json"
21+
}
22+
```
23+
24+
## customManagers
25+
26+
Each entry requires:
27+
28+
- `"customType": "regex"` — mandatory for regex managers
29+
- `"description"` — short human-readable string. State what updated, where.
30+
31+
Example:
32+
33+
```json
34+
{
35+
"description": "Update NEO4J_VERSION constant in shell scripts",
36+
"customType": "regex",
37+
"fileMatch": ["^scripts/[^/]*\\.sh$"],
38+
"matchStrings": ["NEO4J_VERSION:-\\\"?(?<currentValue>.*?)\\\""]
39+
}
40+
```
41+
42+
## packageRules
43+
44+
Each entry requires:
45+
46+
- `"description"` — short string. Clarify rule purpose.
47+
48+
Example:
49+
50+
```json
51+
{
52+
"description": "Keep Python ML libraries in sync",
53+
"matchDatasources": ["pypi"],
54+
"matchPackageNames": ["scikit-**", "umap-learn"],
55+
"groupName": "python-machine-learning-libs"
56+
}
57+
```
58+
59+
## Escaping
60+
61+
Use `\\` in `matchStrings` + `fileMatch`. Single `\` → invalid JSON.
62+
63+
- `.` → `\\.`
64+
- `"` → `\\\"`

‎.github/prompts/plan-uvAsPrimaryPackageManager.prompt.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Plan: uv as Primary Package Manager
2+
3+
uv becomes the primary (default) Python package manager using `pyproject.toml` + `uv.lock` as the canonical dependency source. Conda remains a supported optional path. venv is removed entirely. All wiring — scripts, workflows, Renovate, docs, and instruction files — is updated accordingly.
4+
5+
---
6+
7+
## Phase 1 — Canonical Dependency Source
8+
1. **Create `pyproject.toml`**: `[project]` section with `requires-python = ">=3.12"`, `dependencies` pinned with `==` (exact versions from `requirements.txt`); `[tool.uv]` for uv settings
9+
2. **Delete `requirements.txt`** — Renovate's `uv` manager takes over version tracking
10+
3. **Fix drift in `conda-environment.yml`**: `pandas=2.2.3` → `2.3.3`; `neo4j==5.28.2` → `6.1.0` in its pip block
11+
4. **Update header comment** in `conda-environment.yml`: "sync to `pyproject.toml`" (not requirements.txt)
12+
13+
**Verification**: `uv sync` completes cleanly; `uv lock --check` passes.
14+
15+
## Phase 2 — Activation Scripts *(parallel with Phase 3)*
16+
5. **Create `scripts/activateUvEnvironment.sh`**: skips early if `PYTHON_PACKAGE_MANAGER != 'uv'`; hard-fails if `uv` not found (with install URL); runs `uv sync --frozen`; activates `.venv/bin/activate` (Unix) or `.venv/Scripts/activate` (Windows Git Bash) — cross-platform via `operatingSystemFunctions.sh`; shellcheck-clean, follows [shell-scripts.instructions.md](.github/instructions/shell-scripts.instructions.md)
17+
6. **Delete `scripts/activatePythonEnvironment.sh`**
18+
7. **Modify `scripts/activateCondaEnvironment.sh`**: replace `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV == 'true'` skip-guard → `PYTHON_PACKAGE_MANAGER != 'conda'`; leave deprecated var in comment
19+
20+
## Phase 3 — Environment Variables *(parallel with Phase 2)*
21+
8. **Add `PYTHON_PACKAGE_MANAGER`** env var (default: `uv`, valid: `uv` | `conda`) to both new activation scripts
22+
9. **Deprecate `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV`** — mark in all referencing scripts
23+
10. **Remove `PYTHON_ENVIRONMENT_FILE`** — referenced only by the deleted `activatePythonEnvironment.sh`
24+
25+
## Phase 4 — Pipeline Script Wiring *(depends on 2 + 3)*
26+
11. **Modify `scripts/reports/compilations/PythonReports.sh`**: replace `source activatePythonEnvironment.sh` → `source activateUvEnvironment.sh`; keep `activateCondaEnvironment.sh` call; update comment
27+
12. **Modify `scripts/checkCompatibility.sh`**: replace `oneOf "conda" "venv"` → `oneOf "uv" "conda"`; replace `checkOptionalCommand "virtualenv"` → `checkOptionalCommand "uv"` (with astral.sh URL); remove `python`/`pip` as required commands (uv manages its own Python)
28+
29+
## Phase 5 — Workflows *(depends on 2 + 3 + 4)*
30+
13. **Rename** `internal-check-python-venv-support.yml` → `internal-check-python-uv-support.yml`: install uv via `astral-sh/setup-uv`; `uv sync --frozen`; verify package imports; trigger paths: `pyproject.toml`, `uv.lock`, `scripts/activateUvEnvironment.sh`
31+
14. **Create** `.github/workflows/internal-check-conda-support.yml`: setup-miniconda + `conda-environment.yml`; verify activation + imports; trigger paths: `conda-environment.yml`, `scripts/activateCondaEnvironment.sh`
32+
15. **Modify `public-analyze-code-graph.yml`** (public API — no breaking changes):
33+
- Add `python-package-manager` input: `type: string`, `default: 'uv'`, `required: false`, self-contained description for external users
34+
- Update `use-venv_virtual_python_environment` description: "Deprecated. Use `python-package-manager` instead."
35+
- Add uv setup step (conditional `inputs.python-package-manager != 'conda'`) using `astral-sh/setup-uv`
36+
- Keep conda setup step (conditional `inputs.python-package-manager == 'conda'`)
37+
- Pass `PYTHON_PACKAGE_MANAGER: ${{ inputs.python-package-manager }}` to the analysis env
38+
- ⚠️ **Behavior change**: callers relying on conda being the effective default must now explicitly pass `python-package-manager: 'conda'`
39+
16. **Modify `internal-java-code-analysis.yml`**: remove `!requirements.txt` override from `paths-ignore` (2 occurrences); no other changes — inherits uv default
40+
17. **Modify `internal-typescript-code-analysis.yml`**: same `!requirements.txt` removal
41+
42+
## Phase 6 — Renovate *(parallel with Phase 5)*
43+
18. **Modify `renovate.json`**: add `"uv"` to `matchManagers` in the "Only use stable versions" packageRule; remove `"pip_requirements"` from the same list; Renovate's `uv` manager auto-discovers `pyproject.toml` + `uv.lock`
44+
- 🔘 **Decision point**: enable the currently-disabled conda customManagers? (change `fileMatch` from `conda-environment-temporarily-disabled.yml` → `conda-environment.yml`)
45+
46+
## Phase 7 — Documentation *(depends on all above)*
47+
19. **`README.md`**: update Python prerequisites (uv primary, conda optional)
48+
20. **`COMMANDS.md`**: add uv setup commands; keep conda manual setup section
49+
21. **`ENVIRONMENT_VARIABLES.md`**: add `PYTHON_PACKAGE_MANAGER`, deprecate `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV`, remove `PYTHON_ENVIRONMENT_FILE`
50+
22. **Regenerate `SCRIPTS.md`** (run generation script after all script changes)
51+
23. **`AGENTS.md`, `CLAUDE.md`, `.github/copilot-instructions.md`**: update instruction table row for `python-dependencies.instructions.md` (applyTo now covers `pyproject.toml`)
52+
53+
## Phase 8 — Instruction Files *(depends on all above)*
54+
24. **Modify `.github/instructions/python-dependencies.instructions.md`**: update `applyTo` to `pyproject.toml,conda-environment.yml`; update sync rule (`pyproject.toml` is canonical); update verification to `uv sync --check` / `uv lock --check`
55+
56+
---
57+
58+
## Summary Table
59+
60+
| File | Change |
61+
|------|--------|
62+
| `pyproject.toml` | **NEW** — canonical dep source |
63+
| `requirements.txt` | **DELETED** |
64+
| `conda-environment.yml` | drift fix + header comment update |
65+
| `scripts/activateUvEnvironment.sh` | **NEW** |
66+
| `scripts/activatePythonEnvironment.sh` | **DELETED** |
67+
| `scripts/activateCondaEnvironment.sh` | gate on `PYTHON_PACKAGE_MANAGER` |
68+
| `scripts/reports/compilations/PythonReports.sh` | swap activation script |
69+
| `scripts/checkCompatibility.sh` | uv/conda check, remove pip/python |
70+
| `.github/workflows/internal-check-python-venv-support.yml` | **RENAMED** → uv-support.yml |
71+
| `.github/workflows/internal-check-conda-support.yml` | **NEW** |
72+
| `.github/workflows/public-analyze-code-graph.yml` | add `python-package-manager` param |
73+
| `.github/workflows/internal-java-code-analysis.yml` | remove `!requirements.txt` exception |
74+
| `.github/workflows/internal-typescript-code-analysis.yml` | remove `!requirements.txt` exception |
75+
| `renovate.json` | add `uv` manager, remove `pip_requirements` |
76+
| `ENVIRONMENT_VARIABLES.md` | new var, deprecated var, removed var |
77+
| `SCRIPTS.md` | regenerated |
78+
| `README.md`, `COMMANDS.md` | uv setup docs |
79+
| `AGENTS.md`, `CLAUDE.md`, `.github/copilot-instructions.md` | instruction table update |
80+
| `.github/instructions/python-dependencies.instructions.md` | applyTo + sync rule + verification |
81+
82+
## Verification Checklist
83+
1. `uv sync --frozen` completes cleanly from project root
84+
2. `uv lock --check` — lockfile is consistent with `pyproject.toml`
85+
3. `uv run python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"` — all imports succeed
86+
4. `shellcheck scripts/activateUvEnvironment.sh scripts/activateCondaEnvironment.sh` — no warnings
87+
5. `npx --yes markdown-link-check --quiet --progress --config=markdown-lint-check-config.json README.md COMMANDS.md ENVIRONMENT_VARIABLES.md` — no broken links
88+
6. CI: `internal-check-python-uv-support.yml` passes
89+
7. CI: `internal-check-conda-support.yml` passes
90+
8. CI: `public-analyze-code-graph.yml` smoke-tested with default (uv) and with `python-package-manager: conda`
91+
92+
## Renovate Decision
93+
- Should Renovate's conda customManagers be re-enabled now that `conda-environment.yml` is officially the conda canonical file? Currently disabled on purpose. Enabling makes Renovate auto-update conda packages but the custom matchers only cover conda packages (not the nested pip block). Answer: Yes, but it is fine when conda versions diverge from pyproject.toml since conda is now explicitly optional and secondary. Therefore, enable it and change it so that it only matches `conda-environment.yml` and only takes conda package updates into account (ignore pip block).
94+
95+
## Key Decisions Made
96+
- **Canonical source**: `pyproject.toml` + `uv.lock`
97+
- **`requirements.txt`**: deleted entirely; Renovate `uv` manager handles updates
98+
- **`conda-environment.yml`**: kept; synced to `pyproject.toml` (not deleted)
99+
- **Drift fix** (in this PR): `pandas=2.3.3`, `neo4j==6.1.0` (use requirements.txt versions as truth)
100+
- **Script**: new `activateUvEnvironment.sh`; delete `activatePythonEnvironment.sh`
101+
- **Env var**: new `PYTHON_PACKAGE_MANAGER` (default: `uv`); `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV` deprecated
102+
- **Public workflow**: add `python-package-manager` string param (default `'uv'`); keep old param deprecated-but-present
103+
- **Behavior change flag**: callers relying on old conda default (VENV=false) now get uv. Must set `python-package-manager: 'conda'` explicitly.
104+
- **CI**: uv = primary (existing internal workflows), conda = new dedicated check workflow
105+
- **Renovate**: enable `uv` manager; remove/disable `pip_requirements` since no more requirements.txt
106+
- **Windows Git Bash / WSL**: supported in `activateUvEnvironment.sh`
107+
- **Jupyter**: `ipykernel==7.2.0` already in both files — no changes needed

‎.github/workflows/internal-check-conda-support.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Check Python Conda environment
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
# Only watch changes related to the Conda environment
8+
paths:
9+
- 'conda-environment.yml'
10+
- 'scripts/activateCondaEnvironment.sh'
11+
- '.github/workflows/internal-check-conda-support.yml' # or when this file changed
12+
13+
jobs:
14+
check-conda-environment:
15+
runs-on: ubuntu-22.04
16+
strategy:
17+
matrix:
18+
include:
19+
- python: 3.12
20+
miniforge: 24.9.0-0
21+
22+
steps:
23+
- name: Checkout GIT Repository
24+
uses: actions/checkout@v6
25+
26+
- name: (Python Setup) Use version ${{ matrix.python }} with Conda package manager Miniforge
27+
uses: conda-incubator/setup-miniconda@v3
28+
with:
29+
python-version: ${{ matrix.python }}
30+
miniforge-version: ${{ matrix.miniforge }}
31+
activate-environment: codegraph
32+
environment-file: ./conda-environment.yml
33+
auto-activate-base: false
34+
show-channel-urls: true
35+
36+
- name: (Python Setup) Conda environment info
37+
shell: bash -el {0}
38+
run: |
39+
conda info
40+
conda list
41+
42+
- name: Activate Conda environment and verify required packages are importable
43+
env:
44+
PYTHON_PACKAGE_MANAGER: "conda"
45+
PREPARE_CONDA_ENVIRONMENT: "false" # Already prepared above
46+
shell: bash -el {0}
47+
run: |
48+
source ./scripts/activateCondaEnvironment.sh
49+
python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"

‎.github/workflows/internal-check-python-uv-support.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Check Python uv environment
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
# Only watch changes related to Python uv environment
8+
paths:
9+
- 'pyproject.toml'
10+
- 'uv.lock'
11+
- 'scripts/activateUvEnvironment.sh'
12+
- '.github/workflows/internal-check-python-uv-support.yml' # or when this file changed
13+
14+
jobs:
15+
check-python-uv-environment:
16+
runs-on: ubuntu-22.04
17+
strategy:
18+
matrix:
19+
include:
20+
- python: 3.12
21+
22+
steps:
23+
- name: Checkout GIT Repository
24+
uses: actions/checkout@v6
25+
26+
- name: (uv Setup) Install uv
27+
uses: astral-sh/setup-uv@v6
28+
with:
29+
python-version: ${{ matrix.python }}
30+
31+
- name: (uv Setup) Sync dependencies from lockfile
32+
run: uv sync --frozen
33+
34+
- name: Verify activation script and required packages
35+
run: |
36+
source ./scripts/activateUvEnvironment.sh
37+
python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"
38+
echo "✓ Activation script and imports successful"

‎.github/workflows/internal-check-python-venv-support.yml‎

Lines changed: 0 additions & 48 deletions
This file was deleted.

‎.github/workflows/internal-java-code-analysis.yml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@ on:
88
paths-ignore:
99
- '**/*.md'
1010
- '**/*.txt'
11-
- '!requirements.txt'
1211
- '**/*.css'
1312
- '**/*.html'
1413
- '**/*.js'
@@ -27,7 +26,6 @@ on:
2726
paths-ignore:
2827
- '**/*.md'
2928
- '**/*.txt'
30-
- '!requirements.txt'
3129
- '**/*.css'
3230
- '**/*.html'
3331
- '**/*.js'

0 commit comments

Comments
 (0)