WIP: Test lowest versions of all required and optional dependencies #3639

weiji14 · 2024-11-21T02:08:57Z

Using --resolution lowest-direct for now, because --resolution lowest grabs many other transitive dependencies that may be hard to install (e.g. missing wheels for newer Python versions, so requires compilation from source). We could switch to --resolution lowest once the Python ecosystem does lower bound pins a bit more thoroughly (which may take years).

weiji14 · 2024-11-21T02:03:41Z

Getting this error at https://github.com/GenericMappingTools/pygmt/actions/runs/11945150103/job/33297317606?pr=3639#step:7:35:

Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/runner/work/pygmt/pygmt/pygmt/__init__.py", line 24, in <module> from pygmt import datasets File "/home/runner/work/pygmt/pygmt/pygmt/datasets/__init__.py", line 7, in <module> from pygmt.datasets.earth_age import load_earth_age File "/home/runner/work/pygmt/pygmt/pygmt/datasets/earth_age.py", line 12, in <module> from pygmt.datasets.load_remote_dataset import _load_remote_dataset File "/home/runner/work/pygmt/pygmt/pygmt/datasets/load_remote_dataset.py", line 9, in <module> from pygmt.clib import Session File "/home/runner/work/pygmt/pygmt/pygmt/clib/__init__.py", line 9, in <module> from pygmt.clib.session import Session, __gmt_version__ File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 111, in <module> _libgmt = load_libgmt() File "/home/runner/work/pygmt/pygmt/pygmt/clib/loading.py", line 62, in load_libgmt raise GMTCLibNotFoundError("\n".join(error_msg)) pygmt.exceptions.GMTCLibNotFoundError: Error loading GMT shared library at '/home/runner/micromamba/envs/pygmt/lib/libgmt.so'. /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/runner/micromamba/envs/pygmt/lib/./libgdal.so.34) Error loading GMT shared library at 'libgmt.so'. /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/runner/micromamba/envs/pygmt/lib/python3.10/lib-dynload/../.././libgdal.so.34) make: *** [Makefile:35: _runtest] Error 1

Similar to issues mentioned at GenericMappingTools/try-gmt#50 (comment) and GenericMappingTools/try-gmt#37 (comment). Seems to only be on the Ubuntu/Linux build.

weiji14 · 2026-02-04T02:45:32Z

Oof, it's not liking the GMT from conda-forge now. Traceback from https://github.com/GenericMappingTools/pygmt/actions/runs/21656150792/job/62431201928?pr=3639#step:7:39:

Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/runner/work/pygmt/pygmt/pygmt/__init__.py", line 24, in <module> from pygmt import datasets File "/home/runner/work/pygmt/pygmt/pygmt/datasets/__init__.py", line 28, in <module> from pygmt.datasets.samples import list_sample_data, load_sample_data File "/home/runner/work/pygmt/pygmt/pygmt/datasets/samples.py", line 11, in <module> from pygmt.src import which File "/home/runner/work/pygmt/pygmt/pygmt/src/__init__.py", line 5, in <module> from pygmt.src.basemap import basemap File "/home/runner/work/pygmt/pygmt/pygmt/src/basemap.py", line 9, in <module> from pygmt.clib import Session File "/home/runner/work/pygmt/pygmt/pygmt/clib/__init__.py", line 9, in <module> from pygmt.clib.session import Session, __gmt_version__ File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 116, in <module> _libgmt = load_libgmt() ^^^^^^^^^^^^^ File "/home/runner/work/pygmt/pygmt/pygmt/clib/loading.py", line 62, in load_libgmt raise GMTCLibNotFoundError("\n".join(error_msg)) pygmt.exceptions.GMTCLibNotFoundError: Error loading GMT shared library at '/home/runner/micromamba/envs/pygmt/lib/libgmt.so'. /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by /home/runner/micromamba/envs/pygmt/lib/./libgdal.so.37) Error loading GMT shared library at 'libgmt.so'. libgmt.so: cannot open shared object file: No such file or directory make: *** [Makefile:35: _runtest] Error 1

GMT was installed via micromamba. Not sure if that libstdc++.so.6: version CXXABI_1.3.15' not found` error is from libgdal or elsewhere?

weiji14 · 2024-11-21T02:05:43Z

We discussed making NetCDF an optional dependency before in #429. Maybe we could revisit that discussion?

That's a good point. NetCDF4 was needed by xarray when reading and writing netCDF files. As mentioned in #239 (comment), previously, our load_xxx functions (e.g., load_earth_relief) called xarray.load_dataarray to load local netCDF grids into an xarray.DataArray object, making netCDF a highly "required" dependency.

After we refactor our load_xxx function in #3120, we no longer call xarray.load_dataarray, so netCDF should be no longer highly needed by PyGMT.

I've opened PR #3643 to see how the PyGMT relies on the netCDF package.

Edit: There are only 14 failures in the Python 3.11 + Ubuntu CI job after removing netCDF entirely (see https://github.com/GenericMappingTools/pygmt/actions/runs/11968416461/job/33367210857?pr=3643) and most of them are caused by load_static_earth_relief.

There is another error that popped up on Ubuntu-22.04-ARM64 at https://github.com/GenericMappingTools/pygmt/actions/runs/14457975280/job/40545076044?pr=3639#step:7:1079, related to static_earth_relief too:

==================================== ERRORS ==================================== __________________ ERROR at setup of test_clib_read_data_grid __________________ self = CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/home/runner/.gmt/cache/static_earth_relief.nc', mode='r', kwa...r': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}, manager_id='26bc1ee9-cb26-4f6f-bcea-ac669a2faa62') needs_lock = True def _acquire_with_cache_info(self, needs_lock=True): """Acquire a file, returning the file and whether it was cached.""" with self._optional_lock(needs_lock): try: > file = self._cache[self._key] ../.venv/lib/python3.11/site-packages/xarray/backends/file_manager.py:210: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <xarray.backends.lru_cache.LRUCache object at 0xff802d7a2b80> key = [<class 'netCDF4._netCDF4.Dataset'>, ('/home/runner/.gmt/cache/static_earth_relief.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '26bc1ee9-cb26-4f6f-bcea-ac669a2faa62'] def __getitem__(self, key: K) -> V: # record recent use of the key by moving it to the front of the list with self._lock: > value = self._cache[key] E KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/runner/.gmt/cache/static_earth_relief.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '26bc1ee9-cb26-4f6f-bcea-ac669a2faa62'] ../.venv/lib/python3.11/site-packages/xarray/backends/lru_cache.py:56: KeyError During handling of the above exception, another exception occurred: @pytest.fixture(scope="module", name="expected_xrgrid") def fixture_expected_xrgrid(): """ The expected xr.DataArray object for the static_earth_relief.nc file. """ > return load_dataarray(which("@static_earth_relief.nc")) ../pygmt/tests/test_clib_read_data.py:30: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../pygmt/io.py:44: in load_dataarray with xr.open_dataarray(filename_or_obj, **kwargs) as dataarray: ../.venv/lib/python3.11/site-packages/xarray/backends/api.py:686: in open_dataarray dataset = open_dataset( ../.venv/lib/python3.11/site-packages/xarray/backends/api.py:525: in open_dataset backend_ds = backend.open_dataset( ../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:588: in open_dataset store = NetCDF4DataStore.open( ../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:389: in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) ../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:336: in __init__ self.format = self.ds.data_model ../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:398: in ds return self._acquire() ../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:392: in _acquire with self._manager.acquire_context(needs_lock) as root: ../../../../.local/share/uv/python/cpython-3.11.12-linux-aarch64-gnu/lib/python3.11/contextlib.py:137: in __enter__ return next(self.gen) ../.venv/lib/python3.11/site-packages/xarray/backends/file_manager.py:198: in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) ../.venv/lib/python3.11/site-packages/xarray/backends/file_manager.py:216: in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) src/netCDF4/_netCDF4.pyx:2489: in netCDF4._netCDF4.Dataset.__init__ ??? src/netCDF4/_netCDF4.pyx:2092: in netCDF4._netCDF4._get_vars ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E RuntimeError: only endian='native' allowed for NETCDF3 files, got 'big' (variable 'lon', group '/') src/netCDF4/_netCDF4.pyx:4062: RuntimeError

This seems to be fixed in Unidata/netcdf4-python#1355, released in netcdf4=1.7.2. So we can either set the minimum version to netCDF4>=1.7.2 and/or remove netCDF4 in #3643. Would prefer the latter, because netCDF4=1.7.2 is the latest version released on Oct 2024, so not very old.

Edit: netCDF4 is no longer a required dependency of PyGMT since #3643, so no need to worry about this anymore.

weiji14 · 2024-11-21T02:10:44Z

Setting lower bounds here, otherwise uv will try to install super old versions that may not work anymore, some which actually go back to Python 2! For e.g., it tried to install rioxarray=0.0.1 at https://github.com/GenericMappingTools/pygmt/actions/runs/11945100244/job/33297179010#step:7:45. Happy to reconsider some of these version pins if needed.

Edit see comments at #3639 (comment) and #3639 (comment) below.

weiji14 · 2025-06-09T03:47:29Z

Getting this test failure on windows-2022 at https://github.com/GenericMappingTools/pygmt/actions/runs/15524973929/job/43703740279?pr=3639#step:7:1095:

_______________ [doctest] pygmt.datasets.tile_map.load_tile_map _______________ 118 >>> raster.sizes 119 Frozen({'band': 3, 'y': 256, 'x': 512}) 120 >>> raster.coords # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE 121 Coordinates: 122 * band (band) uint8... 1 2 3 123 * y (y) float64... -7.081e-10 -7.858e+04 ... -1.996e+07 -2.004e+07 124 * x (x) float64... -2.004e+07 -1.996e+07 ... 1.996e+07 2.004e+07 125 spatial_ref int... 0 126 >>> # CRS is set only if rioxarray is available 127 >>> if hasattr(raster, "rio"): Expected: 'EPSG:3857' Got: 'PROJCS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Mercator_1SP"],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +wktext +no_defs"],AUTHORITY["EPSG","3857"]]' D:\a\pygmt\pygmt\pygmt\datasets\tile_map.py:127: DocTestFailure

Similar situation to the one reported at #3575. We could perhaps pin to a rioxarray version that requires rasterio>=1.4.2? E.g. rioxarray=0.19.0 pins to rasterio>=1.4.3 (corteva/rioxarray#850). However, that version was released too recently on 22 Apr 2025 (<2 months).

Suggestions on what to do?

The actual error messages are

ERROR 1: PROJ: proj_create_from_database: C:\Users\runneradmin\micromamba\envs\pygmt/Library/share/proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 3 whereas a number >= 4 is expected. It comes from another PROJ installation. ERROR 1: PROJ: proj_identify: C:\Users\runneradmin\micromamba\envs\pygmt/Library/share/proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 3 whereas a number >= 4 is expected. It comes from another PROJ installation.

Some related discussions are in corteva/rioxarray#447. I'm unsure if pin rioxarray>=0.19.0 can solve the issue, but please give it a try first.

Hmm you're right, got the same ERROR 1: PROJ: ... message even with rioxarray>0.19.0. Probably because we are using uv/PyPI installed packages with incompatible PROJ versions, instead of conda-forge packages where packages are compiled with the same PROJ lib.

The dependency tree GMT->GDAL->PROJ installs PROJ 9.4.0, while uv/pypi installs pyproj 3.7.1 which contains PROJ 9.5.1.

Perhaps we can install pyproj via micromamba instead?

Perhaps we can install pyproj via micromamba instead?

Tried at commit 4540a5d, same error 😢

What happens if setting env PROJ_WHEEL=YES?

xref: https://pyproj4.github.io/pyproj/dev/installation.html#setup-pyproj

PROJ_WHEEL seems to be relevant only when building the pyproj wheel, but since we're pulling a compiled version of pyproj from PyPI/conda-forge, it probably doesn't do anything.

-Original file line number
+Diff line change
@@ -1,8 +1,8 @@
-    # Test PyGMT with GMT legacy versions on Linux/macOS/Windows
+    # Test PyGMT with GMT legacy versions and old Python dependencies on Linux/macOS/Windows
     #
-    # This workflow runs regular PyGMT tests with GMT legacy versions. Due to the minor
-    # baseline image changes between GMT versions, the workflow only runs the tests but
-    # doesn't do image comparisons.
+    # This workflow runs regular PyGMT tests with GMT legacy versions and old versions of
+    # all optional Python dependencies. Due to the minor baseline image changes between GMT
+    # versions, the workflow only runs the tests but doesn't do image comparisons.
     #
     # It is scheduled to run every Tuesday on the main branch.
     #
@@ Expand All / @@ -12,7 +12,7 @@ on: @@
       # push:
       #   branches: [ main ]
       # Uncomment the 'pull_request' line below to trigger the workflow in PR
-      # pull_request:
+      pull_request:
         # types: [ready_for_review]
         # paths:
         #  - 'pygmt/**'
@@ Expand Down Expand Up / @@ -62,23 +62,6 @@ jobs: @@
                 python=3.10
                 gmt=${{ matrix.gmt_version }}
                 ghostscript<10
-                numpy
-                pandas
-                xarray
-                netCDF4
-                packaging
-                contextily
-                geopandas
-                ipython
-                pyarrow
-                rioxarray
-                sphinx-gallery
-                make
-                pip
-                python-build
-                pytest
-                pytest-doctestplus
-                pytest-mpl
           # Download cached remote files (artifacts) from GitHub
           - name: Download remote data from GitHub
@@ Expand All / @@ -95,10 +78,18 @@ jobs: @@
             env:
               GH_TOKEN: ${{ github.token }}
+          # Install uv package manager
+          - name: Install uv
+            uses: astral-sh/setup-uv@v3
           # Install the package that we want to test
           - name: Install the package
-            run: make install
+            run: |
+              uv run --with pip==23 --resolution lowest-direct --all-extras --dev make install
+              uv pip list
           # Run the tests but skip images
           - name: Run tests
-            run: make test_no_images PYTEST_EXTRA="-r P"
+            run: uv run --with pytest==8 --resolution lowest-direct --all-extras --dev make test_no_images PYTEST_EXTRA="-r P"
+            env:
+              GMT_LIBRARY_PATH: $CONDA_PREFIX/lib

-Original file line number
+Diff line change
@@ Expand Up / @@ -33,18 +33,18 @@ dependencies = [ @@
         "numpy>=1.24",
         "pandas>=2.0",
         "xarray>=2023.04",
-        "netCDF4",
+        "netCDF4>=1.7",
         "packaging",
     ]
     dynamic = ["version"]
     [project.optional-dependencies]
     all = [
-        "contextily",
-        "geopandas",
-        "IPython",  # 'ipython' is not the correct module name.
-        "pyarrow",
-        "rioxarray",
+        "contextily>=1",
+        "geopandas>=0.14",
+        "IPython>=8",  # 'ipython' is not the correct module name.
+        "pyarrow>=14",
+        "rioxarray>=0.14",
     ]
     [project.urls]
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Test lowest versions of all required and optional dependencies #3639

Uh oh!

Diff view

Diff view

There are no files selected for viewing

weiji14 Nov 21, 2024

Uh oh!

weiji14 Nov 21, 2024 •

edited

Loading

Uh oh!

weiji14 Feb 4, 2026

Uh oh!

weiji14 Nov 21, 2024

Uh oh!

seisman Nov 22, 2024 •

edited

Loading

Uh oh!

weiji14 Apr 15, 2025 •

edited

Loading

Uh oh!

weiji14 Nov 21, 2024 •

edited

Loading

Uh oh!

weiji14 Jun 9, 2025 •

edited

Loading

Uh oh!

seisman Jun 9, 2025

Uh oh!

weiji14 Jun 9, 2025

Uh oh!

seisman Jun 13, 2025 •

edited

Loading

Uh oh!

weiji14 Jun 13, 2025

Uh oh!

seisman Jun 24, 2025

Uh oh!

weiji14 Jun 24, 2025

Uh oh!

WIP: Test lowest versions of all required and optional dependencies #3639

Are you sure you want to change the base?

Uh oh!

WIP: Test lowest versions of all required and optional dependencies #3639

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Choose a reason for hiding this comment

Uh oh!

weiji14 Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seisman Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14 Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14 Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14 Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seisman Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14 Nov 21, 2024 •

edited

Loading

seisman Nov 22, 2024 •

edited

Loading

weiji14 Apr 15, 2025 •

edited

Loading

weiji14 Nov 21, 2024 •

edited

Loading

weiji14 Jun 9, 2025 •

edited

Loading

seisman Jun 13, 2025 •

edited

Loading