Merge pull request #17 from nmfs-opensci/copilot/remove-env-yml-files

Reorganize environment files into /environment directory

Author: eeholmes

.github/scripts/filter_and_validate_packages.py

@@ -26,7 +26,8 @@ def parse_env_files(repo_root: Path) -> Dict[str, Set[str]]:
     Returns:
         Dictionary mapping filename to set of package names
     """
-    env_files = sorted(repo_root.glob("env-*.yml"))
+    env_dir = repo_root / "environment"
+    env_files = sorted(env_dir.glob("env-*.yml"))
     packages_by_file = {}
 
     for env_file in env_files:

.github/workflows/build-and-push.yml

@@ -9,8 +9,7 @@ on:
       - 'action.yml'
       - '.github/workflows/build-and-push.yml'
       - 'Dockerfile'
-      - 'env-*.yml'
-      - 'environment.yml'
+      - 'environment/env-*.yml'
       - 'install.R'
       - 'apt.txt'
       - 'Desktop/**'

Dockerfile

@@ -13,15 +13,15 @@ USER root
 # Install Zotero; must be run before apt since apt install zotero requires this script be run first
 RUN wget -qO- https://raw.githubusercontent.com/retorquere/zotero-deb/master/install.sh | bash
 
-# Copy the repo files into /tmp (env-*.yml, install.R, etc)
+# Copy the repo files into /tmp (environment/*.yml, install.R, etc)
 COPY . /tmp/
 
 # Update conda env sequentially to prevent hairy solve
-RUN /pyrocket_scripts/install-conda-packages.sh /tmp/env-core1.yml || (echo "install env-core1 failed" && exit 1)
-RUN /pyrocket_scripts/install-conda-packages.sh /tmp/env-core2.yml || (echo "install env-core2 failed" && exit 1)
-RUN /pyrocket_scripts/install-conda-packages.sh /tmp/env-geo.yml || (echo "install env-geo failed" && exit 1)
-RUN /pyrocket_scripts/install-conda-packages.sh /tmp/env-viz.yml || (echo "install env-viz failed" && exit 1)
-RUN /pyrocket_scripts/install-conda-packages.sh /tmp/env-qgis.yml || (echo "install env-qgis failed" && exit 1)
+RUN /pyrocket_scripts/install-conda-packages.sh /tmp/environment/env-core1.yml || (echo "install env-core1 failed" && exit 1)
+RUN /pyrocket_scripts/install-conda-packages.sh /tmp/environment/env-core2.yml || (echo "install env-core2 failed" && exit 1)
+RUN /pyrocket_scripts/install-conda-packages.sh /tmp/environment/env-geo.yml || (echo "install env-geo failed" && exit 1)
+RUN /pyrocket_scripts/install-conda-packages.sh /tmp/environment/env-viz.yml || (echo "install env-viz failed" && exit 1)
+RUN /pyrocket_scripts/install-conda-packages.sh /tmp/environment/env-qgis.yml || (echo "install env-qgis failed" && exit 1)
 RUN /pyrocket_scripts/install-apt-packages.sh /tmp/apt.txt || (echo "install-apt-packages.sh failed" && exit 1)
 RUN /pyrocket_scripts/install-desktop.sh /tmp/Desktop|| (echo "setup-desktop.sh failed" && exit 1)
 

NOTES.md

@@ -2,7 +2,7 @@
 
 ## 2026-02-08
 
-* In January 2026, the solver stopped solving. It was also getting long, over an hour. I broke the environment.yml into 5 smaller files and add these sequentially to the conda notebook env. This seemed to fix the impossible to solve environment.
+* In January 2026, the solver stopped solving. It was also getting long, over an hour. I broke the main environment file into 5 smaller files (environment/env-*.yml) and add these sequentially to the conda notebook env. This seemed to fix the impossible to solve environment.
 * Removed some unneeded packages
 * The R package installs were messed up. First install_geospatial.sh needs to be run first. Then other packages.
     - When I fixed the problem of installs going to /home (and then getting wiped out), I caused installs via install-r-packages.sh to go to source rather than binary.

README.md

@@ -16,7 +16,7 @@ TeXLive and Quarto are installed along with MyST and JupyterBook.
 
 Python 3.11 is installed with a conda environment called notebook that is activated on opening the container. R 4.5.X is installed and operates separate from the conda notebook environment (conda is not on the PATH when using R). R can be used from RStudio or JupyterLab and the same R environment is used.
 
-* [Python packages (source env files)](env-core1.yml)
+* [Python packages (source env files)](environment/env-core1.yml)
 * [R packages (source install file)](install.R)
 * [Pinned Python packages](packages-python-pinned.yaml) - Auto-generated list of all Python packages with pinned versions
 * [Pinned R packages](packages-r-pinned.R) - Auto-generated list of all R packages with pinned versions
@@ -29,7 +29,7 @@ Python 3.11 is installed with a conda environment called notebook that is activa
 
 ## Customizing py-rocket-geospatial-2
 
-* edit the Python packages here `env-*.yml`
+* edit the Python packages here `environment/env-*.yml`
 * edit the R packages here `install.R`
 * update the QGIS, CoastWatch Utilities, and Panoply installs here `Dockerfile`
 * update the systems installs here `apt.txt`
@@ -39,16 +39,16 @@ If the changes are core functionality, not scientific, put in an [issue in py-ro
 ### Package Pinning and Validation
 
 The repository automatically maintains pinned package versions with validation:
-- `packages-python-pinned.yaml` - Contains Python packages from env-*.yml files with exact versions (not all 900+ conda packages)
+- `packages-python-pinned.yaml` - Contains Python packages from environment/env-*.yml files with exact versions (not all 900+ conda packages)
 - `packages-r-pinned.R` - Contains all R packages from the site-library with exact versions
 - `build.log` - Validation report showing if all packages from env files and rocker scripts are present
 
 The [Pin Package Versions workflow](.github/workflows/pin-packages.yml):
 - Runs automatically after each successful build
 - Can be manually triggered from the Actions tab
 - Extracts package versions from the published Docker image
-- **Filters Python packages** to only include those specified in env-*.yml files
-- **Validates Python packages** that all packages from env-*.yml files are present in the container
+- **Filters Python packages** to only include those specified in environment/env-*.yml files
+- **Validates Python packages** that all packages from environment/env-*.yml files are present in the container
 - **Validates R packages** that all packages from install.R, /rocker_scripts/install_geospatial.sh, and /rocker_scripts/install_tidyverse.sh are present in the container
 - Creates a PR with:
   - Updated pinned package files
@@ -73,7 +73,7 @@ FROM ghcr.io/nmfs-opensci/container-images/py-rocket-geospatial-2:2026.02.08
 USER root
 
 COPY . /tmp/
-RUN /pyrocket_scripts/install-conda-packages.sh /tmp/environment.yml || echo "install-conda-packages.sh failed" || true
+RUN /pyrocket_scripts/install-conda-packages.sh /tmp/your-environment.yml || echo "install-conda-packages.sh failed" || true
 RUN /pyrocket_scripts/install-r-packages.sh /tmp/install.R || echo "install-r-package.sh failed" || true
 RUN rm -rf /tmp/*