databrickslabs
diff --git a/‎.cursor/agents/docker.md‎
Lines changed: 95 additions & 2 deletions b/‎.cursor/agents/docker.md‎
Lines changed: 95 additions & 2 deletions
diff --git a/‎.cursor/commands/gbx-ci-act.md‎
Lines changed: 62 additions & 0 deletions b/‎.cursor/commands/gbx-ci-act.md‎
Lines changed: 62 additions & 0 deletions
diff --git a/‎.cursor/commands/gbx-ci-act.sh‎
Lines changed: 52 additions & 0 deletions b/‎.cursor/commands/gbx-ci-act.sh‎
Lines changed: 52 additions & 0 deletions
diff --git a/‎.cursor/commands/gbx-lint-python.sh‎
Lines changed: 3 additions & 1 deletion b/‎.cursor/commands/gbx-lint-python.sh‎
Lines changed: 3 additions & 1 deletion
diff --git a/‎.cursor/commands/gbx-test-notebooks.sh‎
Lines changed: 5 additions & 2 deletions b/‎.cursor/commands/gbx-test-notebooks.sh‎
Lines changed: 5 additions & 2 deletions
diff --git a/‎.cursor/commands/gbx-test-python-docs.sh‎
Lines changed: 1 addition & 1 deletion b/‎.cursor/commands/gbx-test-python-docs.sh‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎.cursor/commands/gbx-test-sql-docs.sh‎
Lines changed: 1 addition & 1 deletion b/‎.cursor/commands/gbx-test-sql-docs.sh‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎.cursor/commands/gbx-versions-audit.md‎
Lines changed: 68 additions & 0 deletions b/‎.cursor/commands/gbx-versions-audit.md‎
Lines changed: 68 additions & 0 deletions
@@ -470,6 +470,90 @@ gbx:docker:rebuild --start
 - **Mounted to**: `/root/geobrix/scripts/docker/m2/`
 - **Purpose**: Persist Maven dependencies between container restarts
 
+## Registry proxies (optional)
+
+The dev container is built to route through whatever registry URLs the host
+environment supplies — useful if you sit behind a network that blocks public
+PyPI / Maven Central, or if you want a single team-wide pin set:
+
+| Tool | Source of URL | Configured by |
+|---|---|---|
+| pip | host env `PIP_INDEX_URL` forwarded as `--build-arg` | `Dockerfile` writes `/etc/pip.conf` + sets `PIP_INDEX_URL` env only when set; `build_smart.sh` auto-forwards |
+| Maven | `scripts/docker/m2/settings.xml` `<mirror>` block (gitignored, host-local) | `docker_maven_setup.sh` |
+
+Export `PIP_INDEX_URL` before building (or `build_smart.sh` picks it up
+automatically). Leave it unset and the build uses public PyPI.
+
+If a `pip install` step in the Dockerfile fails with `Connection refused` or
+`Could not find a version` listing only old releases, your proxy is either
+unreachable or has an embargo on recently-published versions — fall back to
+the prior stable release of the offending package.
+
+## Local GitHub Actions dry-runs with `act`
+
+Separate from the dev container (`geobrix-dev`), there's a second Docker image
+purpose-built for **local CI validation** — `geobrix-ci-runner:local`. It's
+shaped like a GitHub-hosted runner (`catthehacker/ubuntu:runner-24.04`,
+digest-pinned). pip/Maven/npm registry URLs are build-arg injected from the
+host env (`PIP_INDEX_URL`, `MAVEN_MIRROR_URL`, `NPM_REGISTRY_URL`) — set them
+to a private proxy if your network requires it; leave them unset to use
+public registries.
+
+### When to use
+
+- After editing any `.github/workflows/*.yml` or `.github/actions/*/action.yml`
+- Before push, to catch typos, action SHA pin breakage, step ordering issues
+- To debug a CI failure that doesn't reproduce locally
+
+### Quickstart
+
+```bash
+brew install act              # one-time
+gbx:ci:act -l                  # list jobs across workflows
+gbx:ci:act -W .github/workflows/build_main.yml -j build   # run one job
+gbx:ci:act push                # simulate a push event
+```
+
+First run builds the runner image (~5 min, cached after).
+
+### How real `.github/` stays untouched
+
+`act` parses workflow + composite-action YAML on the host filesystem *before*
+any container starts, so we need the overlay on disk — not just inside the
+container. `scripts/ci-local/run-act.sh` regenerates a mirror at
+`.cache/act-workspace/` (gitignored) on every run:
+
+- `.github/` is freshly copied (~100 KB, ~50 ms) with the jfrog-auth stub
+  overlaid on top.
+- Every other top-level entry (`pom.xml`, `src/`, `scripts/`, `.git`, …) is
+  symlinked back to the real project, so workflow content is identical.
+- `act --bind` runs from inside the mirror; `actions/checkout` becomes a
+  no-op (uses the bind-mounted workspace as-is).
+
+The real `.github/` tree on disk is never modified — only the mirror's copy
+is. JFrog OIDC can't run locally (no real GitHub OIDC issuer); pip / Maven /
+npm fall back to whatever proxies were build-arg-injected into the runner
+image (public registries by default).
+
+### Files
+
+| Path | Purpose |
+|---|---|
+| `scripts/ci-local/Dockerfile.gha-runner` | Runner image build |
+| `scripts/ci-local/{pip.conf,maven-settings.xml,npmrc}` | Proxy configs baked into the image |
+| `scripts/ci-local/jfrog-auth-stub/action.yml` | No-op overlay |
+| `scripts/ci-local/run-act.sh` | act invocation with overlay mount |
+| `scripts/ci-local/README.md` | Detailed mechanics + caveats |
+| `.cursor/commands/gbx-ci-act.{sh,md}` | Cursor command wrapper |
+
+### Caveats
+
+- **JFrog OIDC**: mocked locally (stub action). Real OIDC exchange runs only in CI.
+- **`runs-on: larger-runners`**: treated as a label alias; you don't actually get a "larger" machine — just whatever Docker resources are available.
+- **Real GitHub event payloads**: `act` mocks `head_sha`, `head_ref`, etc.
+- **Secrets**: only `GITHUB_TOKEN` is provided (auto-mocked); workflows fall back via the `REPO_ACCESS_TOKEN || GITHUB_TOKEN` pattern. `CODECOV_TOKEN` is missing but the upload step has `fail_ci_if_error: false`.
+- **Org-level runner-group policy**: not simulated (which is fine — local runs use a local Docker container regardless).
+
 ### Settings File
 - **Location**: `scripts/docker/m2/settings.xml`
 - **Key settings**:
@@ -483,9 +567,18 @@ gbx:docker:rebuild --start
 ## Environment Variables
 
 ### Key Variables in Container
+Pinned in `scripts/docker/Dockerfile` (DBR 17.3 LTS aligned). Run
+`gbx:versions:audit` to see all of them. The most load-bearing:
+
 ```bash
-SPARK_VERSION=3.5.3         # Spark version
-GDAL_VERSION=3.10.0         # GDAL version
+SPARK_VERSION=4.0.0         # Spark version (DBR 17.3 LTS)
+NUMPY_VERSION=2.1.3         # NumPy 2.x (DBR 17.3 LTS)
+PANDAS_VERSION=2.2.3        # pandas (DBR 17.3 LTS)
+PIP_VERSION=25.0.1          # pip (DBR 17.3 LTS)
+SETUPTOOLS_VERSION=74.0.0   # setuptools (DBR 17.3 LTS)
+WHEEL_VERSION=0.45.1        # wheel (DBR 17.3 LTS)
+# GDAL is NOT in DBR; built from ubuntugis PPA.
+# Python bindings auto-detect via `gdal-config --version` (currently 3.11.4).
 JUPYTER_PLATFORM_DIRS=1     # Suppress Jupyter warnings
 ```
 
 
@@ -0,0 +1,62 @@
+# Run GitHub Actions locally with act
+
+Validate `.github/workflows/*.yml` changes before pushing by running them locally via [act](https://github.com/nektos/act). Catches ~80% of CI bugs (typos, missing env, action SHA pin issues, step ordering, composite-action structure errors) without a push-and-iterate loop.
+
+## Usage
+
+```bash
+bash .cursor/commands/gbx-ci-act.sh [act-arguments...]
+```
+
+## Examples
+
+```bash
+# List jobs across all workflows
+gbx:ci:act -l
+
+# Run one job from one workflow
+gbx:ci:act -W .github/workflows/build_main.yml -j build
+
+# Simulate a push event (runs all workflows triggered on push)
+gbx:ci:act push
+
+# Simulate a PR event
+gbx:ci:act pull_request
+
+# Pass any other arg through to act
+gbx:ci:act --help
+```
+
+## First-time setup
+
+```bash
+brew install act
+```
+
+The first invocation builds `geobrix-ci-runner:local` (~5 min). Subsequent runs reuse the image.
+
+## What's wired
+
+- **Runner image**: `catthehacker/ubuntu:runner-24.04` (digest-pinned). pip/Maven/npm registry URLs are build-arg injected from the host env (`PIP_INDEX_URL`, `MAVEN_MIRROR_URL`, `NPM_REGISTRY_URL`); set them to a private proxy if your network requires it, otherwise leave unset to use public registries.
+- **Platform map**: `ubuntu-latest`, `ubuntu-24.04`, `ubuntu-22.04`, `larger`, `linux-ubuntu-latest` → `geobrix-ci-runner:local`
+- **JFrog auth stub**: Real `.github/actions/jfrog-auth/action.yml` is bind-mounted over inside the act container with a no-op stub. Real `.github/` on disk is never modified.
+
+## Coverage
+
+| Catches | Doesn't catch |
+|---|---|
+| YAML syntax errors | JFrog OIDC token exchange (mocked) |
+| Action SHA pin issues | `larger-runners` actually being larger |
+| Step ordering | Real GitHub event payloads (head_sha, head_ref) |
+| Composite action structure | Real secrets (CODECOV_TOKEN, REPO_ACCESS_TOKEN) |
+| pip/Maven/npm install correctness | Org-level runner-group access policy |
+| Matrix expansion | Protected env / branch protection gating |
+| Conditional `if:` evaluation | |
+
+For the gaps, push to a draft PR and let real CI exercise them.
+
+## Notes
+
+- The runner image is roughly 2 GB; cached locally after first build
+- `act` reuses the image across runs; rebuild via `docker rmi geobrix-ci-runner:local && gbx:ci:act -l`
+- Full mechanics in `scripts/ci-local/README.md`; runbook context in `.cursor/agents/docker.md`
@@ -0,0 +1,52 @@
+#!/bin/bash
+# gbx:ci:act - Run GitHub Actions workflows locally via `act`, against a
+# project-shaped runner image. The real .github/ tree is NEVER modified —
+# the jfrog-auth composite is bind-mounted with a no-op stub at runtime.
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+source "$SCRIPT_DIR/common.sh"
+
+show_help() {
+    show_banner "Local CI dry-run with act"
+    echo -e "${CYAN}Usage:${NC}"
+    echo -e "  ${GREEN}gbx:ci:act${NC} ${YELLOW}[act-arguments...]${NC}"
+    echo ""
+    echo -e "${CYAN}Common invocations:${NC}"
+    echo -e "  ${GREEN}gbx:ci:act -l${NC}                                    List jobs"
+    echo -e "  ${GREEN}gbx:ci:act -W .github/workflows/build_main.yml -j build${NC}    Run a specific job"
+    echo -e "  ${GREEN}gbx:ci:act push${NC}                                  Simulate a push event"
+    echo -e "  ${GREEN}gbx:ci:act pull_request${NC}                          Simulate a PR event"
+    echo -e "  ${GREEN}gbx:ci:act --help${NC}                                Pass-through to act --help"
+    echo ""
+    echo -e "${CYAN}First run (one-time, ~5 min):${NC}"
+    echo -e "  Builds ${YELLOW}geobrix-ci-runner:local${NC} Docker image."
+    echo -e "  Requires act: ${YELLOW}brew install act${NC}"
+    echo ""
+    echo -e "${CYAN}What's pre-baked into the runner image:${NC}"
+    echo -e "  • pip / Maven / npm registry URLs are build-arg injected from your env"
+    echo -e "    (${YELLOW}PIP_INDEX_URL${NC} / ${YELLOW}MAVEN_MIRROR_URL${NC} / ${YELLOW}NPM_REGISTRY_URL${NC});"
+    echo -e "    set them to a private proxy if your network requires it, or leave unset"
+    echo -e "    to default to public registries."
+    echo ""
+    echo -e "${CYAN}Caveats:${NC}"
+    echo -e "  • JFrog OIDC is mocked (no real token); pip/maven/npm flow via the registry baked in at build time"
+    echo -e "  • ${YELLOW}runs-on: larger-runners${NC} is treated as a label only — no actual"
+    echo -e "    larger machine; uses your local Docker resources"
+    echo -e "  • Real .github/ files are never modified — see scripts/ci-local/README.md"
+    echo ""
+    echo -e "${CYAN}Iteration loop:${NC}"
+    echo -e "  1. Edit a workflow in .github/workflows/"
+    echo -e "  2. ${GREEN}gbx:ci:act -W <workflow> -j <job>${NC}"
+    echo -e "  3. When clean, push and let real CI exercise OIDC + larger runners"
+    echo ""
+}
+
+if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
+    show_help
+    exit 0
+fi
+
+show_banner "Local CI dry-run with act"
+exec bash "$PROJECT_ROOT/scripts/ci-local/run-act.sh" "$@"
@@ -79,7 +79,9 @@ run_fix_host() {
         if [ ! -d "$venv_dir" ] || [ ! -x "$venv_dir/bin/isort" ] || [ ! -x "$venv_dir/bin/black" ] || [ ! -x "$venv_dir/bin/flake8" ]; then
             echo -e "${CYAN}Creating venv at ${YELLOW}$venv_dir${NC} and installing dev deps..."
             python3 -m venv "$venv_dir" || { echo -e "${RED}Failed to create venv.${NC}"; exit 1; }
-            "$venv_dir/bin/pip" install -q --upgrade pip
+            # Pin bootstrap to DBR 17.3 LTS — keep in sync with .github/actions/{scala,python}_build/action.yml,
+            # scripts/docker/Dockerfile, scripts/geobrix-gdal-init.sh.
+            "$venv_dir/bin/pip" install -q --upgrade pip==25.0.1 setuptools==74.0.0 wheel==0.45.1
             (cd "$PY_DIR" && "$venv_dir/bin/pip" install -q -e ".[dev]") || { echo -e "${RED}Failed to install python/geobrix[dev].${NC}"; exit 1; }
             echo -e "${GREEN}Venv ready.${NC}"
         fi
 
@@ -145,8 +145,11 @@ ${INCLUDE_INTEGRATION:+export GBX_NOTEBOOK_INCLUDE_INTEGRATION=1}
 ${ALLOW_ABSOLUTE_READS:+export GBX_NOTEBOOK_ALLOW_ABSOLUTE_READS=1}
 ${ALLOW_ABSOLUTE_WRITES:+export GBX_NOTEBOOK_ALLOW_ABSOLUTE_WRITES=1}
 cd /root/geobrix
-pip install -e /root/geobrix/python/geobrix --break-system-packages -q 2>/dev/null || true
-pip install nbformat nbconvert --break-system-packages -q 2>/dev/null || true
+# Install the local geobrix package (no deps — runtime deps are already in
+# the hash-pinned dev container lockfile). nbformat / nbconvert are also
+# pre-installed there; no ad-hoc pip install needed (would defeat the
+# supply-chain hardening).
+pip install --no-deps -e /root/geobrix/python/geobrix --break-system-packages -q 2>/dev/null || true
 python3 /root/geobrix/notebooks/tests/run_notebooks_cell_by_cell.py ${PATH_ARG:+"$PATH_ARG"}
 "
 
 
@@ -192,7 +192,7 @@ if [ \"$SKIP_BUILD\" != 'true' ]; then
     cd /root/geobrix/python/geobrix && python3 -m build && cd /root/geobrix
     echo ''
     echo 'Installing Python package (editable)...'
-    pip install -e /root/geobrix/python/geobrix --break-system-packages -q
+    pip install --no-deps -e /root/geobrix/python/geobrix --break-system-packages -q
     echo ''
 fi
 
 
@@ -118,7 +118,7 @@ if [ \"$SKIP_BUILD\" != 'true' ]; then
     echo '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
     mvn package -DskipTests -q
     cd /root/geobrix/python/geobrix && python3 -m build && cd /root/geobrix
-    pip install -e /root/geobrix/python/geobrix --break-system-packages -q
+    pip install --no-deps -e /root/geobrix/python/geobrix --break-system-packages -q
     echo ''
 fi
 echo '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
 
@@ -0,0 +1,68 @@
+# Audit pinned Python tooling versions
+
+Inventories every place a pinned package version lives across CI, the dev container Dockerfile, the Databricks cluster init script, and the lint venv setup. Use it to **detect drift** between locations, and as the entry point for **bumping pins to a new DBR LTS**.
+
+## Usage
+
+```bash
+bash .cursor/commands/gbx-versions-audit.sh [OPTIONS]
+```
+
+## Options
+
+- `--package <name>` — Show only one package (e.g. `pip`, `numpy`, `pyspark`).
+- `--log <path>` — Write output to log file.
+- `--help` — Display help.
+
+## Examples
+
+```bash
+# Full audit (all tracked packages, all locations)
+gbx:versions:audit
+
+# Just numpy
+gbx:versions:audit --package numpy
+
+# Just pyspark — useful when bumping Spark
+gbx:versions:audit --package pyspark
+```
+
+## What's tracked
+
+Source of truth (priority order):
+
+1. **DBR LTS release notes** — for any package shipped with the runtime:
+   - Core build/runtime: `pip`, `setuptools`, `wheel`, `cython`, `numpy`, `pandas`, `pyspark`/`spark`, `requests`, `python`.
+   - Jupyter / notebook stack (dev container): `ipython`, `ipykernel`, `ipywidgets`, `jupyterlab`, `jupyter_server`, `jupyter_core`, `jupyter_client`, `tornado`, `nbformat`, `nbconvert`.
+2. **Current PyPI stable** — for tools NOT in DBR: `build`, `pytest-cov`, `geopandas` (pinned to be compatible with DBR's pandas 2.2.x), `keplergl` (last stable; requires `ipywidgets >= 7`).
+3. **CI matrix** — for tools intentionally divergent from DBR: `pytest 8.4.2` (kept ahead of DBR 8.3.5 because tests don't run in DBR).
+4. **System apt + ubuntugis PPA** — for GDAL native + Python bindings; auto-detected from `gdal-config --version`.
+5. **Intentional DBR divergence** — `pyzmq 25.1.1` (Dockerfile only). DBR has 26.2.0 but our Jupyter ZMQ workaround requires the older pin. Re-evaluate on next DBR LTS bump.
+
+## Files audited
+
+- `.github/actions/scala_build/action.yml`
+- `.github/actions/python_build/action.yml`
+- `.github/workflows/build_*.yml` and `codecov-*.yml` (matrix values)
+- `scripts/docker/Dockerfile`
+- `scripts/geobrix-gdal-init.sh`
+- `.cursor/commands/gbx-lint-python.sh`
+- `python/geobrix/pyproject.toml` (dev/test extras — currently unpinned)
+- `notebooks/tests/requirements.txt` (lower bounds — not pinned)
+
+If a new file pins versions, add it to the `FILES` array in `gbx-versions-audit.sh`.
+
+## Workflow: bumping to a new DBR LTS
+
+1. Get the DBR LTS release-notes URL (e.g. `https://docs.databricks.com/aws/en/release-notes/runtime/18.3lts`).
+2. Run `gbx:versions:audit` and capture the output.
+3. Ask Claude:
+   > "Bump pinned versions to DBR 18.3 LTS from `<URL>`. Here's the current audit: `<paste>`. Favor DBR versions where listed; pin packages not in DBR (`build`, `pytest-cov`) to current PyPI stable; keep `pytest` at the current CI matrix value (we don't run tests in DBR)."
+4. Re-run `gbx:versions:audit` to verify everything moved together.
+5. Commit: `chore(deps): align pins to DBR 18.3 LTS`.
+
+## Notes
+
+- The audit is **read-only** — never modifies files. All edits happen via Claude after you review the audit.
+- Pyproject.toml dev/test extras (`isort`, `black`, `flake8`, `build`, `pytest`, etc.) are intentionally unpinned to allow `pip install -e .[dev]` to resolve cleanly. The hard pins live in CI/Docker/init script — those are the trust boundaries.
+- The audit prints a "Files audited" section at the end with `✓`/`✗` markers — a `✗` means a tracked file is missing (e.g. renamed/deleted). Update `FILES` in the script to fix.