Migrate to uv as primary package manager for Python

.github/instructions/python-dependencies.instructions.md

@@ -1,44 +1,43 @@
 ---
-applyTo: "requirements.txt,conda-environment.yml"
+applyTo: "pyproject.toml,conda-environment.yml"
 ---
 # Python Dependency Conventions
 
-Applies to [requirements.txt](../../requirements.txt), [conda-environment.yml](../../conda-environment.yml).
+[pyproject.toml](../../pyproject.toml), [conda-environment.yml](../../conda-environment.yml).
 
 ## Sync Rule
 
-Both files must stay **in sync**: same packages, same pinned versions. Package in one must be in other.
+`pyproject.toml` canonical source. `conda-environment.yml` stays synced: same packages, same pinned versions.
 
-Exceptions (document inline with comment if they differ):
-- `conda-environment.yml` may split packages (e.g. `plotly[kaleido]` in pip vs. `plotly` + `python-kaleido` in conda)
-- `conda-environment.yml` may use `pip:` block for packages not on conda-forge
+Exceptions (inline comment if differ):
+- `conda-environment.yml` splits packages (e.g. `plotly[kaleido]` in pyproject.toml vs. `plotly` + `python-kaleido` in conda)
+- `conda-environment.yml` uses `pip:` block for packages not on conda-forge
 
 ## Versioning
 
 - Pin all versions — no ranges, no `>=`, no unpinned entries
-- `requirements.txt`: `==` (e.g. `pandas==2.2.3`)
-- `conda-environment.yml`: `=` (e.g. `pandas=2.2.3`)
+- `pyproject.toml`: `==` (e.g. `"pandas==2.3.3"`)
+- `conda-environment.yml`: `=` (e.g. `pandas=2.3.3`)
 
-## Adding or Updating
+## Updating
 
-- **Ask user before introducing new dependencies**
+- **Ask user before adding dependencies**
 - Update both files when changing version
 
 ## Verification
 
-**`requirements.txt`** — venv dry-run from [.github/workflows/internal-check-python-venv-support.yml](../../.github/workflows/internal-check-python-venv-support.yml):
+**`pyproject.toml`** — uv sync from [.github/workflows/internal-check-python-uv-support.yml](../../.github/workflows/internal-check-python-uv-support.yml):
 
 ```shell
-python -m venv .venv
-source ./scripts/activatePythonEnvironment.sh
-.venv/bin/pip install --dry-run --quiet --requirement requirements.txt
-! .venv/bin/pip install --dry-run --requirement requirements.txt 2>/dev/null | grep -q "Would install"
+uv sync --frozen
+uv lock --check
+uv run python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"
 ```
 
 **`conda-environment.yml`** — via [scripts/activateCondaEnvironment.sh](../../scripts/activateCondaEnvironment.sh):
 
 ```shell
-source ./scripts/activateCondaEnvironment.sh
+PYTHON_PACKAGE_MANAGER=conda source ./scripts/activateCondaEnvironment.sh
 ```
 
 Fallback (conda directly):

.github/instructions/renovate.instructions.md

@@ -0,0 +1,64 @@
+---
+applyTo: "renovate.json"
+---
+# Renovate Config Conventions
+
+## Valid JSON
+
+File must be valid JSON. No trailing commas. Validate with:
+
+```shell
+npx --yes --package renovate -- renovate-config-validator
+```
+
+## $schema
+
+Top-level field must have schema reference:
+
+```json
+{
+  "$schema": "https://docs.renovatebot.com/renovate-schema.json"
+}
+```
+
+## customManagers
+
+Each entry requires:
+
+- `"customType": "regex"` — mandatory for regex managers
+- `"description"` — short human-readable string. State what updated, where.
+
+Example:
+
+```json
+{
+  "description": "Update NEO4J_VERSION constant in shell scripts",
+  "customType": "regex",
+  "fileMatch": ["^scripts/[^/]*\\.sh$"],
+  "matchStrings": ["NEO4J_VERSION:-\\\"?(?<currentValue>.*?)\\\""]
+}
+```
+
+## packageRules
+
+Each entry requires:
+
+- `"description"` — short string. Clarify rule purpose.
+
+Example:
+
+```json
+{
+  "description": "Keep Python ML libraries in sync",
+  "matchDatasources": ["pypi"],
+  "matchPackageNames": ["scikit-**", "umap-learn"],
+  "groupName": "python-machine-learning-libs"
+}
+```
+
+## Escaping
+
+Use `\\` in `matchStrings` + `fileMatch`. Single `\` → invalid JSON.
+
+- `.` → `\\.`
+- `"` → `\\\"`

.github/prompts/plan-uvAsPrimaryPackageManager.prompt.md

@@ -0,0 +1,107 @@
+# Plan: uv as Primary Package Manager
+
+uv becomes the primary (default) Python package manager using `pyproject.toml` + `uv.lock` as the canonical dependency source. Conda remains a supported optional path. venv is removed entirely. All wiring — scripts, workflows, Renovate, docs, and instruction files — is updated accordingly.
+
+---
+
+## Phase 1 — Canonical Dependency Source
+1. **Create `pyproject.toml`**: `[project]` section with `requires-python = ">=3.12"`, `dependencies` pinned with `==` (exact versions from `requirements.txt`); `[tool.uv]` for uv settings
+2. **Delete `requirements.txt`** — Renovate's `uv` manager takes over version tracking
+3. **Fix drift in `conda-environment.yml`**: `pandas=2.2.3` → `2.3.3`; `neo4j==5.28.2` → `6.1.0` in its pip block
+4. **Update header comment** in `conda-environment.yml`: "sync to `pyproject.toml`" (not requirements.txt)
+
+**Verification**: `uv sync` completes cleanly; `uv lock --check` passes.
+
+## Phase 2 — Activation Scripts *(parallel with Phase 3)*
+5. **Create `scripts/activateUvEnvironment.sh`**: skips early if `PYTHON_PACKAGE_MANAGER != 'uv'`; hard-fails if `uv` not found (with install URL); runs `uv sync --frozen`; activates `.venv/bin/activate` (Unix) or `.venv/Scripts/activate` (Windows Git Bash) — cross-platform via `operatingSystemFunctions.sh`; shellcheck-clean, follows [shell-scripts.instructions.md](.github/instructions/shell-scripts.instructions.md)
+6. **Delete `scripts/activatePythonEnvironment.sh`**
+7. **Modify `scripts/activateCondaEnvironment.sh`**: replace `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV == 'true'` skip-guard → `PYTHON_PACKAGE_MANAGER != 'conda'`; leave deprecated var in comment
+
+## Phase 3 — Environment Variables *(parallel with Phase 2)*
+8. **Add `PYTHON_PACKAGE_MANAGER`** env var (default: `uv`, valid: `uv` | `conda`) to both new activation scripts
+9. **Deprecate `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV`** — mark in all referencing scripts
+10. **Remove `PYTHON_ENVIRONMENT_FILE`** — referenced only by the deleted `activatePythonEnvironment.sh`
+
+## Phase 4 — Pipeline Script Wiring *(depends on 2 + 3)*
+11. **Modify `scripts/reports/compilations/PythonReports.sh`**: replace `source activatePythonEnvironment.sh` → `source activateUvEnvironment.sh`; keep `activateCondaEnvironment.sh` call; update comment
+12. **Modify `scripts/checkCompatibility.sh`**: replace `oneOf "conda" "venv"` → `oneOf "uv" "conda"`; replace `checkOptionalCommand "virtualenv"` → `checkOptionalCommand "uv"` (with astral.sh URL); remove `python`/`pip` as required commands (uv manages its own Python)
+
+## Phase 5 — Workflows *(depends on 2 + 3 + 4)*
+13. **Rename** `internal-check-python-venv-support.yml` → `internal-check-python-uv-support.yml`: install uv via `astral-sh/setup-uv`; `uv sync --frozen`; verify package imports; trigger paths: `pyproject.toml`, `uv.lock`, `scripts/activateUvEnvironment.sh`
+14. **Create** `.github/workflows/internal-check-conda-support.yml`: setup-miniconda + `conda-environment.yml`; verify activation + imports; trigger paths: `conda-environment.yml`, `scripts/activateCondaEnvironment.sh`
+15. **Modify `public-analyze-code-graph.yml`** (public API — no breaking changes):
+    - Add `python-package-manager` input: `type: string`, `default: 'uv'`, `required: false`, self-contained description for external users
+    - Update `use-venv_virtual_python_environment` description: "Deprecated. Use `python-package-manager` instead."
+    - Add uv setup step (conditional `inputs.python-package-manager != 'conda'`) using `astral-sh/setup-uv`
+    - Keep conda setup step (conditional `inputs.python-package-manager == 'conda'`)
+    - Pass `PYTHON_PACKAGE_MANAGER: ${{ inputs.python-package-manager }}` to the analysis env
+    - ⚠️ **Behavior change**: callers relying on conda being the effective default must now explicitly pass `python-package-manager: 'conda'`
+16. **Modify `internal-java-code-analysis.yml`**: remove `!requirements.txt` override from `paths-ignore` (2 occurrences); no other changes — inherits uv default
+17. **Modify `internal-typescript-code-analysis.yml`**: same `!requirements.txt` removal
+
+## Phase 6 — Renovate *(parallel with Phase 5)*
+18. **Modify `renovate.json`**: add `"uv"` to `matchManagers` in the "Only use stable versions" packageRule; remove `"pip_requirements"` from the same list; Renovate's `uv` manager auto-discovers `pyproject.toml` + `uv.lock`
+    - 🔘 **Decision point**: enable the currently-disabled conda customManagers? (change `fileMatch` from `conda-environment-temporarily-disabled.yml` → `conda-environment.yml`)
+
+## Phase 7 — Documentation *(depends on all above)*
+19. **`README.md`**: update Python prerequisites (uv primary, conda optional)
+20. **`COMMANDS.md`**: add uv setup commands; keep conda manual setup section
+21. **`ENVIRONMENT_VARIABLES.md`**: add `PYTHON_PACKAGE_MANAGER`, deprecate `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV`, remove `PYTHON_ENVIRONMENT_FILE`
+22. **Regenerate `SCRIPTS.md`** (run generation script after all script changes)
+23. **`AGENTS.md`, `CLAUDE.md`, `.github/copilot-instructions.md`**: update instruction table row for `python-dependencies.instructions.md` (applyTo now covers `pyproject.toml`)
+
+## Phase 8 — Instruction Files *(depends on all above)*
+24. **Modify `.github/instructions/python-dependencies.instructions.md`**: update `applyTo` to `pyproject.toml,conda-environment.yml`; update sync rule (`pyproject.toml` is canonical); update verification to `uv sync --check` / `uv lock --check`
+
+---
+
+## Summary Table
+
+| File | Change |
+|------|--------|
+| `pyproject.toml` | **NEW** — canonical dep source |
+| `requirements.txt` | **DELETED** |
+| `conda-environment.yml` | drift fix + header comment update |
+| `scripts/activateUvEnvironment.sh` | **NEW** |
+| `scripts/activatePythonEnvironment.sh` | **DELETED** |
+| `scripts/activateCondaEnvironment.sh` | gate on `PYTHON_PACKAGE_MANAGER` |
+| `scripts/reports/compilations/PythonReports.sh` | swap activation script |
+| `scripts/checkCompatibility.sh` | uv/conda check, remove pip/python |
+| `.github/workflows/internal-check-python-venv-support.yml` | **RENAMED** → uv-support.yml |
+| `.github/workflows/internal-check-conda-support.yml` | **NEW** |
+| `.github/workflows/public-analyze-code-graph.yml` | add `python-package-manager` param |
+| `.github/workflows/internal-java-code-analysis.yml` | remove `!requirements.txt` exception |
+| `.github/workflows/internal-typescript-code-analysis.yml` | remove `!requirements.txt` exception |
+| `renovate.json` | add `uv` manager, remove `pip_requirements` |
+| `ENVIRONMENT_VARIABLES.md` | new var, deprecated var, removed var |
+| `SCRIPTS.md` | regenerated |
+| `README.md`, `COMMANDS.md` | uv setup docs |
+| `AGENTS.md`, `CLAUDE.md`, `.github/copilot-instructions.md` | instruction table update |
+| `.github/instructions/python-dependencies.instructions.md` | applyTo + sync rule + verification |
+
+## Verification Checklist
+1. `uv sync --frozen` completes cleanly from project root
+2. `uv lock --check` — lockfile is consistent with `pyproject.toml`
+3. `uv run python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"` — all imports succeed
+4. `shellcheck scripts/activateUvEnvironment.sh scripts/activateCondaEnvironment.sh` — no warnings
+5. `npx --yes markdown-link-check --quiet --progress --config=markdown-lint-check-config.json README.md COMMANDS.md ENVIRONMENT_VARIABLES.md` — no broken links
+6. CI: `internal-check-python-uv-support.yml` passes
+7. CI: `internal-check-conda-support.yml` passes
+8. CI: `public-analyze-code-graph.yml` smoke-tested with default (uv) and with `python-package-manager: conda`
+
+## Renovate Decision
+- Should Renovate's conda customManagers be re-enabled now that `conda-environment.yml` is officially the conda canonical file? Currently disabled on purpose. Enabling makes Renovate auto-update conda packages but the custom matchers only cover conda packages (not the nested pip block). Answer: Yes, but it is fine when conda versions diverge from pyproject.toml since conda is now explicitly optional and secondary. Therefore, enable it and change it so that it only matches `conda-environment.yml` and only takes conda package updates into account (ignore pip block).
+
+## Key Decisions Made
+- **Canonical source**: `pyproject.toml` + `uv.lock`
+- **`requirements.txt`**: deleted entirely; Renovate `uv` manager handles updates
+- **`conda-environment.yml`**: kept; synced to `pyproject.toml` (not deleted)
+- **Drift fix** (in this PR): `pandas=2.3.3`, `neo4j==6.1.0` (use requirements.txt versions as truth)
+- **Script**: new `activateUvEnvironment.sh`; delete `activatePythonEnvironment.sh`
+- **Env var**: new `PYTHON_PACKAGE_MANAGER` (default: `uv`); `USE_VIRTUAL_PYTHON_ENVIRONMENT_VENV` deprecated
+- **Public workflow**: add `python-package-manager` string param (default `'uv'`); keep old param deprecated-but-present
+  - **Behavior change flag**: callers relying on old conda default (VENV=false) now get uv. Must set `python-package-manager: 'conda'` explicitly.
+- **CI**: uv = primary (existing internal workflows), conda = new dedicated check workflow
+- **Renovate**: enable `uv` manager; remove/disable `pip_requirements` since no more requirements.txt
+- **Windows Git Bash / WSL**: supported in `activateUvEnvironment.sh`
+- **Jupyter**: `ipykernel==7.2.0` already in both files — no changes needed

.github/workflows/internal-check-conda-support.yml

@@ -0,0 +1,49 @@
+name: Check Python Conda environment
+
+on:
+  pull_request:
+    branches:
+      - main
+    # Only watch changes related to the Conda environment
+    paths:
+      - 'conda-environment.yml'
+      - 'scripts/activateCondaEnvironment.sh'
+      - '.github/workflows/internal-check-conda-support.yml' # or when this file changed
+
+jobs:
+  check-conda-environment:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        include:
+        - python: 3.12
+          miniforge: 24.9.0-0
+
+    steps:
+    - name: Checkout GIT Repository
+      uses: actions/checkout@v6
+
+    - name: (Python Setup) Use version ${{ matrix.python }} with Conda package manager Miniforge
+      uses: conda-incubator/setup-miniconda@v3
+      with:
+        python-version: ${{ matrix.python }}
+        miniforge-version: ${{ matrix.miniforge }}
+        activate-environment: codegraph
+        environment-file: ./conda-environment.yml
+        auto-activate-base: false
+        show-channel-urls: true
+
+    - name: (Python Setup) Conda environment info
+      shell: bash -el {0}
+      run: |
+        conda info
+        conda list
+
+    - name: Activate Conda environment and verify required packages are importable
+      env:
+        PYTHON_PACKAGE_MANAGER: "conda"
+        PREPARE_CONDA_ENVIRONMENT: "false" # Already prepared above
+      shell: bash -el {0}
+      run: |
+        source ./scripts/activateCondaEnvironment.sh
+        python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"

.github/workflows/internal-check-python-uv-support.yml

@@ -0,0 +1,38 @@
+name: Check Python uv environment
+
+on:
+  pull_request:
+    branches:
+      - main
+    # Only watch changes related to Python uv environment
+    paths:
+      - 'pyproject.toml'
+      - 'uv.lock'
+      - 'scripts/activateUvEnvironment.sh'
+      - '.github/workflows/internal-check-python-uv-support.yml' # or when this file changed
+
+jobs:
+  check-python-uv-environment:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        include:
+        - python: 3.12
+
+    steps:
+    - name: Checkout GIT Repository
+      uses: actions/checkout@v6
+
+    - name: (uv Setup) Install uv
+      uses: astral-sh/setup-uv@v6
+      with:
+        python-version: ${{ matrix.python }}
+
+    - name: (uv Setup) Sync dependencies from lockfile
+      run: uv sync --frozen
+
+    - name: Verify activation script and required packages
+      run: |
+        source ./scripts/activateUvEnvironment.sh
+        python -c "import ipykernel, pandas, sklearn, umap, shap, neo4j, plotly, optuna"
+        echo "✓ Activation script and imports successful"

.github/workflows/internal-java-code-analysis.yml

@@ -8,7 +8,6 @@ on:
     paths-ignore: 
       - '**/*.md'
       - '**/*.txt'
-      - '!requirements.txt'
       - '**/*.css'
       - '**/*.html'
       - '**/*.js'
@@ -27,7 +26,6 @@ on:
     paths-ignore: 
       - '**/*.md'
       - '**/*.txt'
-      - '!requirements.txt'
       - '**/*.css'
       - '**/*.html'
       - '**/*.js'