fix(commands): address code-review findings in gbx:test:* --host mode

Follow-up to the --host commit; fixes six review findings, all in this
session's new host-mode code.

- PYSPARK_PYTHON not reaching Spark workers (gbx-test-python.sh): the
  `VAR=x eval "..."` command-prefix form does NOT export to the python
  grandchild, so workers fell back to system python3 (No module named
  pandas). New shared helper activate_host_python_env exports
  PYSPARK_PYTHON/PYSPARK_DRIVER_PYTHON and unsets PROJ_DATA/PROJ_LIB;
  all five host branches call it (kills the copy-pasted block too).
- bench tests ran nowhere (coverage gap): the hand-maintained LIGHT_DIRS
  list omitted `bench`, so the heavy leg's conftest collect_ignore skipped
  it and the light leg didn't list it. Now the light-dir set is sourced
  from python/geobrix/test/conftest.py _LIGHT_TEST_DIRS (single source of
  truth) via host_light_test_dirs; the heavy leg runs plain `pytest test/`
  and lets that conftest's dependency-aware collect_ignore exclude the
  light dirs (no hand-rolled --ignore list to drift).
- PROJ_DATA= empty-string vs unset: now unset (via the helper).
- Unquoted --ignore/paths through eval: pytest is now invoked with a real
  args array (marker_args + path args), not an eval-interpolated string,
  so a repo path with spaces can't word-split.
- Docs drift: five command .md Host-mode sections still described the
  abandoned single .venv-host / requirements-dev-container.txt model;
  updated to the two-venv (.venv-host-ci / .venv-host-pyrx) model.
- Notebooks on host: the runner's nested `python -m venv` isolation fails
  on arca (no ensurepip). Host branch now sets GBX_NOTEBOOK_ISOLATED_ENV=0
  and installs nbformat/nbconvert into .venv-host-pyrx, running notebooks
  directly in it.

Re-verified in a clean shell (no pre-exported PYSPARK_PYTHON): unit heavy
146 passed, light leg green with bench now executing, function-info clean.
Remaining failures are all pre-existing and unrelated to --host: bench
fnspec references RST_FromFile.scala (absent on this branch), doc-test #51
/#52, /Volumes-hardcoded reads, and the gdal_examples import collision.

Co-authored-by: Isaac

Author: sllynn

scripts/commands/common.sh

@@ -267,6 +267,40 @@ ensure_host_test_venv() {
     echo "${venv_dir}/bin"
 }
 
+# Prepare the process environment so a host --host pytest run drives Spark + rasterio correctly.
+# Call with the venv bin dir (from ensure_host_test_venv) BEFORE launching pytest. Two exports the
+# forked Spark JVM's Python workers and rasterio need:
+#   - PYSPARK_PYTHON / PYSPARK_DRIVER_PYTHON = the venv interpreter, so Spark workers use the venv
+#     (pandas/pyarrow for Arrow UDFs live there, not in system python3). These MUST be real exports
+#     in the current shell — a `VAR=x eval "..."` command-prefix does NOT propagate to the python
+#     grandchild, so workers would silently fall back to system python3 (ModuleNotFound: pandas).
+#   - unset PROJ_DATA / PROJ_LIB so the venv rasterio uses its own bundled proj.db (layout >=6);
+#     the arca env points these at the older $HOME GDAL proj.db (layout 3), which rasterio rejects
+#     (CRSError "... another PROJ installation"). unset (not empty-string) is required — an empty
+#     PROJ_DATA is a search path of "", not a fallback to bundled data. The JVM GDAL sets PROJ_LIB
+#     internally via SetConfigOption (the /usr/share/proj bridge), so the heavy tier is unaffected.
+# Usage:  activate_host_python_env "$VENV_BIN"
+activate_host_python_env() {
+    local venv_bin="$1"
+    export PYSPARK_PYTHON="${venv_bin}/python"
+    export PYSPARK_DRIVER_PYTHON="${venv_bin}/python"
+    unset PROJ_DATA PROJ_LIB
+}
+
+# The light-tier test dirs (single source of truth: python/geobrix/test/conftest.py _LIGHT_TEST_DIRS).
+# Their modules import light-only deps (rasterio/shapely/pandas/h3/…) at collection time, so they run
+# only in the pyrx venv; the ci venv's conftest collect_ignore skips them (rasterio absent). Echoed
+# space-separated. Falls back to a hardcoded list only if the conftest can't be parsed.
+host_light_test_dirs() {
+    local conftest="${PROJECT_ROOT}/python/geobrix/test/conftest.py"
+    local dirs
+    dirs="$(awk '/_LIGHT_TEST_DIRS *= *\[/{f=1;next} f&&/\]/{f=0} f{gsub(/[",[:space:]]/,""); if($0!="") print}' "$conftest" 2>/dev/null | tr '\n' ' ')"
+    if [ -z "${dirs// /}" ]; then
+        dirs="bench ds pyrx pyvx pygx pmtiles_light stac vizx sample"
+    fi
+    echo "$dirs"
+}
+
 # Run a command inside the isolated pyrx venv (host, no Docker).
 # Usage: run_in_pyrx_venv "<command string>"
 # Requires gbx:venv:sync to have been run (venv at $PROJECT_ROOT/.venv-pyrx).
@@ -291,4 +325,5 @@ export RED GREEN YELLOW BLUE CYAN NC DOCKER_MAVEN_ENV
 export -f check_docker resolve_log_path setup_log_file show_banner show_separator \
           print_report_link open_report generate_timestamp warn_if_jar_stale \
           print_banner print_separator setup_log run_in_pyrx_venv validate_set \
-          require_host_gdal_env ensure_host_test_venv 2>/dev/null || true
+          require_host_gdal_env ensure_host_test_venv activate_host_python_env \
+          host_light_test_dirs 2>/dev/null || true

scripts/commands/gbx-test-docs.md

@@ -34,7 +34,7 @@ bash scripts/commands/gbx-test-docs.sh [OPTIONS]
 
 ## Host mode (arca, no Docker)
 
-With `--host` the orchestrator runs on the host instead of `docker exec geobrix-dev`, forwarding `--host` to each child suite (python-docs, sql-docs, scala-docs). Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The venv-based child suites build `.venv-host` from `python/geobrix/requirements-dev-container.txt` on first run (the exact CI pins, minus native-source-only `pdal` which arca can't build); `--rebuild-venv` is forwarded to them. The scala-docs child runs `mvn` directly and needs only the sourced GDAL env. See the `geobrix-arca` plugin for the full setup.
+With `--host` the orchestrator runs on the host instead of `docker exec geobrix-dev`, forwarding `--host` to each child suite (python-docs, sql-docs, scala-docs). Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The venv-based child suites build a host test venv from the exact CI-pinned locks via `uv` on first run — `.venv-host-pyrx` from `python/geobrix/requirements-pyrx-ci.txt` (light-tier deps); neither CI lock contains `pdal` (source-only, unbuildable on arca); `--rebuild-venv` is forwarded to them. The scala-docs child runs `mvn` directly and needs only the sourced GDAL env. See the `geobrix-arca` plugin for the full setup.
 
 **Sample data (default):** The command sets `GBX_SAMPLE_DATA_ROOT=/Volumes/main/default/test-data` inside the container so doc tests use the minimal bundle (host path `sample-data/Volumes/main/default/test-data`). This is required for running docs unit tests on remote/CI. Use `--no-sample-data-root` to leave it unset (e.g. to use a full bundle or your own env).
 

scripts/commands/gbx-test-function-info.md

@@ -11,14 +11,14 @@ bash scripts/commands/gbx-test-function-info.sh [OPTIONS]
 ## Options
 
 - `--skip-generate` - Skip the generator; run only pytest in `docs/tests-function-info/`
-- `--host` - Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses a `.venv-host` from the pinned lock and runs against a host-built JAR. See "Host mode" below.
+- `--host` - Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses `.venv-host-pyrx` from the pinned CI lock and runs against a host-built JAR. See "Host mode" below.
 - `--rebuild-venv` - (with `--host`) force-rebuild the host test venv.
 - `--log <path>` - Write output to log file
 - `--help` - Display help
 
 ## Host mode (arca, no Docker)
 
-With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The pytest registers functions via the built JAR; no sample data is needed. The first run builds `.venv-host` from `python/geobrix/requirements-dev-container.txt` (the exact CI pins, minus native-source-only `pdal` which arca can't build). See the `geobrix-arca` plugin for the full setup.
+With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The pytest registers functions via the built JAR; no sample data is needed. The first run builds a host test venv from the exact CI-pinned lock via `uv` — `.venv-host-pyrx` from `python/geobrix/requirements-pyrx-ci.txt` (the light-tier deps: rasterio/pandas/h3/vizx). Neither CI lock contains `pdal` (source-only, unbuildable on arca and not needed here). See the `geobrix-arca` plugin for the full setup.
 
 ## Default behavior (inside Docker)
 

scripts/commands/gbx-test-function-info.sh

@@ -71,14 +71,8 @@ if [ "$USE_HOST" = true ]; then
     # --- Host (arca) path: no Docker. The pytest registers functions via the built JAR. ---
     require_host_gdal_env || exit 1
     VENV_BIN=$(ensure_host_test_venv pyrx) || exit 1
-    # Spark Python workers must use the venv interpreter (pandas/pyarrow for Arrow UDFs live there, not in system python3).
-    export PYSPARK_PYTHON="$VENV_BIN/python"
-    export PYSPARK_DRIVER_PYTHON="$VENV_BIN/python"
-    # The venv rasterio bundles its own libproj/proj.db (layout >=6); the arca env points PROJ_DATA/PROJ_LIB
-    # at the older $HOME GDAL proj.db (layout 3), which rasterio refuses. Unset them for the Python side so
-    # rasterio uses its bundled data. The JVM GDAL sets PROJ_LIB internally via SetConfigOption (the
-    # /usr/share/proj bridge), so this does not affect the heavy tier.
-    unset PROJ_DATA PROJ_LIB
+    # Wire Spark workers to the venv python and unset PROJ_DATA/PROJ_LIB (see common.sh).
+    activate_host_python_env "$VENV_BIN"
     warn_if_jar_stale "$PROJECT_ROOT"
     unset JAVA_TOOL_OPTIONS
     export JUPYTER_PLATFORM_DIRS=1

scripts/commands/gbx-test-notebooks.md

@@ -14,7 +14,7 @@ bash scripts/commands/gbx-test-notebooks.sh [OPTIONS]
 
 **Common**
 
-- `--host` – Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses a `.venv-host` from the pinned lock and runs against a host-built JAR. See "Host mode" below.
+- `--host` – Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses `.venv-host-pyrx` from the pinned CI lock and runs against a host-built JAR. See "Host mode" below.
 - `--rebuild-venv` – (with `--host`) force-rebuild the host test venv.
 - `--log <path>` – Write output to log (filename → `test-logs/<name>`).
 - `--path <path>` – Limit scope: subdir (e.g. `sample-data`, `fixtures`), a single `.ipynb`, or a test file (e.g. `test_notebook_via_script.py`). With a `.py` path, runs **pytest** for that file instead of the cell-by-cell runner.
@@ -23,7 +23,9 @@ bash scripts/commands/gbx-test-notebooks.sh [OPTIONS]
 
 ## Host mode (arca, no Docker)
 
-With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The notebook runner executes from the host venv against the built JAR. The first run builds `.venv-host` from `python/geobrix/requirements-dev-container.txt` (the exact CI pins, minus native-source-only `pdal` which arca can't build). See the `geobrix-arca` plugin for the full setup.
+With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The notebook runner executes from the host venv against the built JAR. The first run builds a host test venv from the exact CI-pinned lock via `uv` — `.venv-host-pyrx` from `python/geobrix/requirements-pyrx-ci.txt` (the light-tier deps: rasterio/pandas/h3/vizx), plus `nbformat`/`nbconvert` (needed by the runner, installed on demand). Neither CI lock contains `pdal` (source-only, unbuildable on arca and not needed here). Unlike the container, the host path runs the notebooks directly in `.venv-host-pyrx` (`GBX_NOTEBOOK_ISOLATED_ENV=0`) rather than a nested `python -m venv` (which fails on arca — no `ensurepip`/`python3-venv`).
+
+**Note (bare host):** notebooks that write sample bundles to the literal `/Volumes` path (the data-download notebook) fail on a bare host with `Permission denied: '/Volumes'` — those need a real UC Volumes mount (`sudo ln -sfn "$PWD/sample-data/Volumes" /Volumes`) and, for Sentinel-2 fixtures, `pystac-client`/`planetary-computer`. The runner itself works on host; these are data/mount limitations. See the `geobrix-arca` plugin for the full setup.
 
 **Read/write path behavior (absolute vs relative)**
 

scripts/commands/gbx-test-notebooks.sh

@@ -133,17 +133,22 @@ if [ "$USE_HOST" = true ]; then
     # --- Host (arca) path: no Docker; run the notebook runner from .venv-host ---
     require_host_gdal_env || exit 1
     VENV_BIN=$(ensure_host_test_venv pyrx) || exit 1
-    # Spark Python workers must use the venv interpreter (pandas/pyarrow for Arrow UDFs live there, not in system python3).
-    export PYSPARK_PYTHON="$VENV_BIN/python"
-    export PYSPARK_DRIVER_PYTHON="$VENV_BIN/python"
-    # The venv rasterio bundles its own libproj/proj.db (layout >=6); the arca env points PROJ_DATA/PROJ_LIB
-    # at the older $HOME GDAL proj.db (layout 3), which rasterio refuses. Unset them for the Python side so
-    # rasterio uses its bundled data. The JVM GDAL sets PROJ_LIB internally via SetConfigOption (the
-    # /usr/share/proj bridge), so this does not affect the heavy tier.
-    unset PROJ_DATA PROJ_LIB
+    # The notebook runner needs nbformat/nbconvert (not in the pyrx lock — the runner normally installs
+    # them into its own nested venv). We run directly in .venv-host-pyrx (see below), so ensure they're
+    # present here. Idempotent; uv no-ops when already installed.
+    if ! "$VENV_BIN/python" -c "import nbformat, nbconvert" >/dev/null 2>&1; then
+        echo -e "${CYAN}Installing nbformat/nbconvert into the host venv...${NC}"
+        uv pip install --python "$VENV_BIN/python" -q nbformat nbconvert \
+            || { echo -e "${RED}❌ failed to install nbformat/nbconvert${NC}"; exit 1; }
+    fi
+    # Wire Spark workers to the venv python and unset PROJ_DATA/PROJ_LIB (see common.sh).
+    activate_host_python_env "$VENV_BIN"
     unset JAVA_TOOL_OPTIONS
     export JUPYTER_PLATFORM_DIRS=1
-    export GBX_NOTEBOOK_ISOLATED_ENV=1
+    # Do NOT force the runner's nested-venv isolation on host: .venv-host-pyrx (plus nbformat/nbconvert
+    # installed just above) is already the isolated env, and the runner's `python -m venv` nested-venv
+    # path fails on arca (no ensurepip / python3-venv). Running directly in .venv-host-pyrx is the isolation.
+    export GBX_NOTEBOOK_ISOLATED_ENV=0
     [ -n "$NOTEBOOK_VERBOSITY" ] && export GBX_NOTEBOOK_VERBOSITY="$NOTEBOOK_VERBOSITY"
     [ "$INCLUDE_INTEGRATION" = true ] && export GBX_NOTEBOOK_INCLUDE_INTEGRATION=1
     [ "$ALLOW_ABSOLUTE_READS" = true ] && export GBX_NOTEBOOK_ALLOW_ABSOLUTE_READS=1

scripts/commands/gbx-test-python-docs.md

@@ -58,7 +58,7 @@ bash scripts/commands/gbx-test-python-docs.sh [OPTIONS]
 
 **Other**
 
-- `--host` – Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses a `.venv-host` from the pinned lock and a host-built JAR. See "Host mode" below.
+- `--host` – Run on the host (arca), not the Docker container. Requires `source ~/.local/geobrix-gdal-env.sh` first (provisioned by the `geobrix-arca` plugin); builds/reuses `.venv-host-pyrx` from the pinned CI lock and a host-built JAR. See "Host mode" below.
 - `--rebuild-venv` – (with `--host`) force-rebuild the host test venv.
 - `--log <path>` – Log file (filename → `test-logs/<name>`). Prefer timestamped names for tracking.
 - `--markers <markers>` – Pytest markers (e.g. `"not slow"`).
@@ -69,7 +69,7 @@ bash scripts/commands/gbx-test-python-docs.sh [OPTIONS]
 
 ## Host mode (arca, no Docker)
 
-With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The first run builds `.venv-host` from `python/geobrix/requirements-dev-container.txt` (the exact CI pins, minus native-source-only `pdal` which arca can't build). Sample data reads from the on-disk `sample-data/…/test-data` mirror via `GBX_SAMPLE_DATA_ROOT`. See the `geobrix-arca` plugin for the full setup.
+With `--host` the command runs directly on the host instead of `docker exec geobrix-dev`. Prerequisites: `source ~/.local/geobrix-gdal-env.sh` (native GDAL + Java 17 + PYTHONPATH) and `uv` on PATH, with `PIP_INDEX_URL` pointing at the internal pip proxy. The first run builds `.venv-host-pyrx` from `python/geobrix/requirements-pyrx-ci.txt` (the exact light-tier CI pins: rasterio/pandas/h3/vizx) via `uv`; neither CI lock contains `pdal` (source-only, unbuildable on arca and not needed here). Sample data reads from the on-disk `sample-data/…/test-data` mirror via `GBX_SAMPLE_DATA_ROOT`. See the `geobrix-arca` plugin for the full setup.
 
 ## Examples
 

scripts/commands/gbx-test-python-docs.sh

@@ -142,8 +142,10 @@ if [ "$USE_HOST" = true ]; then
     require_host_gdal_env || exit 1
     local_base="$PROJECT_ROOT/docs/tests/python"
     TEST_PATH="${local_base}/${REL_PATH}"
-    IGNORE_API_ARG=""
-    [ "$SKIP_SQL_TESTS" = true ] && IGNORE_API_ARG="--ignore=${local_base}/api"
+    # Real args array (space-safe), not an eval-interpolated string, so a repo path with spaces
+    # doesn't word-split the --ignore value.
+    ignore_args=()
+    [ "$SKIP_SQL_TESTS" = true ] && ignore_args=(--ignore="${local_base}/api")
 
     echo -e "${CYAN}🎯 Test path: ${YELLOW}$TEST_PATH${NC}  ${CYAN}(host)${NC}"
     [ "$SKIP_SQL_TESTS" = true ] && echo -e "${CYAN}🚫 Excluding api/ (use gbx:test:sql-docs or --suite api)${NC}"
@@ -152,14 +154,8 @@ if [ "$USE_HOST" = true ]; then
     echo ""
 
     VENV_BIN=$(ensure_host_test_venv pyrx) || exit 1
-    # Spark Python workers must use the venv interpreter (pandas/pyarrow for Arrow UDFs live there, not in system python3).
-    export PYSPARK_PYTHON="$VENV_BIN/python"
-    export PYSPARK_DRIVER_PYTHON="$VENV_BIN/python"
-    # The venv rasterio bundles its own libproj/proj.db (layout >=6); the arca env points PROJ_DATA/PROJ_LIB
-    # at the older $HOME GDAL proj.db (layout 3), which rasterio refuses. Unset them for the Python side so
-    # rasterio uses its bundled data. The JVM GDAL sets PROJ_LIB internally via SetConfigOption (the
-    # /usr/share/proj bridge), so this does not affect the heavy tier.
-    unset PROJ_DATA PROJ_LIB
+    # Wire Spark workers to the venv python and unset PROJ_DATA/PROJ_LIB (see common.sh).
+    activate_host_python_env "$VENV_BIN"
 
     if [ "$SKIP_BUILD" != true ]; then
         show_separator
@@ -175,7 +171,8 @@ if [ "$USE_HOST" = true ]; then
     show_separator
     echo -e "${CYAN}Running Python documentation tests (host)...${NC}"
     show_separator
-    eval "\"$VENV_BIN/python\" -m pytest \"$TEST_PATH\" -v $MARKERS --tb=short --color=yes $IGNORE_API_ARG"
+    marker_args=(); [ -n "$MARKERS" ] && eval "marker_args=($MARKERS)"
+    "$VENV_BIN/python" -m pytest "$TEST_PATH" -v --tb=short --color=yes "${marker_args[@]}" "${ignore_args[@]}"
     EXIT_CODE=$?
 else
     # --- Docker path (unchanged) ---