Installation error: ModuleNotFoundError: No module named 'pydantic'

[aloboa]

I'm trying to install on a linux machine (ubuntu 22.04.5, GPU NVIDIA RTX A4000). I do the following with no problems:

$ conda create -n dtree2 python==3.10
$ conda activate dtree2
$ cd dtree2
$ gdal-config --version 
$ conda install gdal pygeos geopandas numpy opencv -c conda-forge
$ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

But then the last step:

$ pip install git+https://github.com/PatBall1/detectree2.git
Collecting git+https://github.com/PatBall1/detectree2.git
  Cloning https://github.com/PatBall1/detectree2.git to /tmp/pip-req-build-konsnq7s
  Running command git clone --filter=blob:none --quiet https://github.com/PatBall1/detectree2.git /tmp/pip-req-build-konsnq7s
  Resolved https://github.com/PatBall1/detectree2.git to commit 61eb66c27684ccd8b747ac6ac047801390c23cbc
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [47 lines of output]
      /home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/_distutils/dist.py:270: UserWarning: Unknown distribution option: 'test_suite'
        warnings.warn(msg)
      Traceback (most recent call last):
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/_pydantic_compat.py", line 3, in <module>
          from pydantic.v1 import (
      ModuleNotFoundError: No module named 'pydantic'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-konsnq7s/setup.py", line 3, in <module>
          setup(
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
          _setup_distribution = dist = klass(attrs)
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/dist.py", line 319, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 290, in __init__
          self.finalize_options()
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/dist.py", line 680, in finalize_options
          for ep in sorted(loaded, key=by_order):
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/site-packages/setuptools/dist.py", line 679, in <lambda>
          loaded = map(lambda e: e.load(), filtered)
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/importlib/metadata/__init__.py", line 162, in load
          module = import_module(match.group('module'))
        File "/home/alobo2/miniforge3/envs/dtree2/lib/python3.10/importlib/__init__.py", line 126, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/__init__.py", line 11, in <module>
          from ._dynamic_plugin import DynamicPlugin
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/_dynamic_plugin.py", line 17, in <module>
          from npe2._pydantic_compat import BaseModel, ValidationError
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/_pydantic_compat.py", line 21, in <module>
          from pydantic import (
      ModuleNotFoundError: No module named 'pydantic'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[PatBall1]

@aloboa did you solve this?

It looks like the error is due to a missing dependency: pydantic. The traceback suggests that the pydantic module is not found.

Suggested Fix:

Try installing pydantic manually before running the install command:

pip install pydantic

Then, attempt to install detectree2 again:

pip install git+https://github.com/PatBall1/detectree2.git

Additional Debugging:

If that doesn’t resolve the issue, try the following steps:

1. Ensure dependencies are installed

   conda install -c conda-forge pydantic

   or

   pip install pydantic

2. Check if pydantic is installed in the correct environment
   Run:

   python -c "import pydantic; print(pydantic.__version__)"

   If this fails, pydantic is missing from your environment.

3. Use a clean environment
   Sometimes conflicting dependencies cause issues. Try creating a new conda environment and reinstalling everything:

   conda create -n dtree2-clean python=3.10
   conda activate dtree2-clean
   conda install gdal pygeos geopandas numpy opencv -c conda-forge
   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
   pip install git+https://github.com/PatBall1/detectree2.git

If the issue persists, let me know the output of:

pip list | grep pydantic

so I can help diagnose further.

[aloboa]

Let's take the clean approach.This is what I have done:

conda create -n dtree2-clean python=3.10
conda activate dtree2-clean
conda install gdal pygeos geopandas numpy opencv -c conda-forge

Then:

python -c "import pydantic; print(pydantic.__version__)"
                                                                  
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pydantic'

Therefore I do:

conda install -c conda-forge pydantic
python -c "import pydantic; print(pydantic.__version__)"
2.10.6

The torch install looks good:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
Looking in indexes: https://download.pytorch.org/whl/cu118
Collecting torch
  Using cached https://download.pytorch.org/whl/cu118/torch-2.6.0%2Bcu118-cp310-cp310-linux_x86_64.whl.metadata (27 kB)
Collecting torchvision
  Using cached https://download.pytorch.org/whl/cu118/torchvision-0.21.0%2Bcu118-cp310-cp310-linux_x86_64.whl.metadata (6.1 kB)
Collecting torchaudio
  Using cached https://download.pytorch.org/whl/cu118/torchaudio-2.6.0%2Bcu118-cp310-cp310-linux_x86_64.whl.metadata (6.6 kB)
Collecting filelock (from torch)
  Using cached https://download.pytorch.org/whl/filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/alobo2/.local/lib/python3.10/site-packages (from torch) (4.12.2)
Requirement already satisfied: networkx in /home/alobo2/.local/lib/python3.10/site-packages (from torch) (3.3)
Requirement already satisfied: jinja2 in /home/alobo2/.local/lib/python3.10/site-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in /home/alobo2/.local/lib/python3.10/site-packages (from torch) (2024.9.0)
Collecting nvidia-cuda-nvrtc-cu11==11.8.89 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cuda_nvrtc_cu11-11.8.89-py3-none-manylinux1_x86_64.whl (23.2 MB)
Collecting nvidia-cuda-runtime-cu11==11.8.89 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cuda_runtime_cu11-11.8.89-py3-none-manylinux1_x86_64.whl (875 kB)
Collecting nvidia-cuda-cupti-cu11==11.8.87 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cuda_cupti_cu11-11.8.87-py3-none-manylinux1_x86_64.whl (13.1 MB)
Collecting nvidia-cudnn-cu11==9.1.0.70 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cudnn_cu11-9.1.0.70-py3-none-manylinux2014_x86_64.whl (663.9 MB)
Collecting nvidia-cublas-cu11==11.11.3.6 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cublas_cu11-11.11.3.6-py3-none-manylinux1_x86_64.whl (417.9 MB)
Collecting nvidia-cufft-cu11==10.9.0.58 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)
Collecting nvidia-curand-cu11==10.3.0.86 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_curand_cu11-10.3.0.86-py3-none-manylinux1_x86_64.whl (58.1 MB)
Collecting nvidia-cusolver-cu11==11.4.1.48 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cusolver_cu11-11.4.1.48-py3-none-manylinux1_x86_64.whl (128.2 MB)
Collecting nvidia-cusparse-cu11==11.7.5.86 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_cusparse_cu11-11.7.5.86-py3-none-manylinux1_x86_64.whl (204.1 MB)
Collecting nvidia-nccl-cu11==2.21.5 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_nccl_cu11-2.21.5-py3-none-manylinux2014_x86_64.whl (147.8 MB)
Collecting nvidia-nvtx-cu11==11.8.86 (from torch)
  Using cached https://download.pytorch.org/whl/cu118/nvidia_nvtx_cu11-11.8.86-py3-none-manylinux1_x86_64.whl (99 kB)
Collecting triton==3.2.0 (from torch)
  Using cached https://download.pytorch.org/whl/triton-3.2.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.4 kB)
Collecting sympy==1.13.1 (from torch)
  Using cached https://download.pytorch.org/whl/sympy-1.13.1-py3-none-any.whl (6.2 MB)
Collecting mpmath<1.4,>=1.1.0 (from sympy==1.13.1->torch)
  Using cached https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl (536 kB)
Requirement already satisfied: numpy in /home/alobo2/.local/lib/python3.10/site-packages (from torchvision) (1.26.4)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /home/alobo2/.local/lib/python3.10/site-packages (from torchvision) (10.4.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/alobo2/.local/lib/python3.10/site-packages (from jinja2->torch) (2.1.5)
Using cached https://download.pytorch.org/whl/cu118/torch-2.6.0%2Bcu118-cp310-cp310-linux_x86_64.whl (848.7 MB)
Using cached https://download.pytorch.org/whl/triton-3.2.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (166.6 MB)
Using cached https://download.pytorch.org/whl/cu118/torchvision-0.21.0%2Bcu118-cp310-cp310-linux_x86_64.whl (6.5 MB)
Using cached https://download.pytorch.org/whl/cu118/torchaudio-2.6.0%2Bcu118-cp310-cp310-linux_x86_64.whl (3.3 MB)
Using cached https://download.pytorch.org/whl/filelock-3.13.1-py3-none-any.whl (11 kB)
Installing collected packages: triton, mpmath, sympy, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, filelock, nvidia-cusolver-cu11, nvidia-cudnn-cu11, torch, torchvision, torchaudio
Successfully installed filelock-3.13.1 mpmath-1.3.0 nvidia-cublas-cu11-11.11.3.6 nvidia-cuda-cupti-cu11-11.8.87 nvidia-cuda-nvrtc-cu11-11.8.89 nvidia-cuda-runtime-cu11-11.8.89 nvidia-cudnn-cu11-9.1.0.70 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.3.0.86 nvidia-cusolver-cu11-11.4.1.48 nvidia-cusparse-cu11-11.7.5.86 nvidia-nccl-cu11-2.21.5 nvidia-nvtx-cu11-11.8.86 sympy-1.13.1 torch-2.6.0+cu118 torchaudio-2.6.0+cu118 torchvision-0.21.0+cu118 triton-3.2.0

But detectree2 install fails:

pip install git+https://github.com/PatBall1/detectree2.git
Collecting git+https://github.com/PatBall1/detectree2.git
  Cloning https://github.com/PatBall1/detectree2.git to /tmp/pip-req-build-k5t43rxw
  Running command git clone --filter=blob:none --quiet https://github.com/PatBall1/detectree2.git /tmp/pip-req-build-k5t43rxw
  Resolved https://github.com/PatBall1/detectree2.git to commit 0694240b83e63afd5c3989cc7f3f8d862621e837
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      /home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/_distutils/dist.py:270: UserWarning: Unknown distribution option: 'test_suite'
        warnings.warn(msg)
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-k5t43rxw/setup.py", line 3, in <module>
          setup(
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
          _setup_distribution = dist = klass(attrs)
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/dist.py", line 319, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 290, in __init__
          self.finalize_options()
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/dist.py", line 680, in finalize_options
          for ep in sorted(loaded, key=by_order):
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/site-packages/setuptools/dist.py", line 679, in <lambda>
          loaded = map(lambda e: e.load(), filtered)
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
          module = import_module(match.group('module'))
        File "/home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/importlib/__init__.py", line 126, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/__init__.py", line 11, in <module>
          from ._dynamic_plugin import DynamicPlugin
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/_dynamic_plugin.py", line 19, in <module>
          from ._plugin_manager import PluginManager
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/_plugin_manager.py", line 32, in <module>
          from ._command_registry import CommandRegistry
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/_command_registry.py", line 9, in <module>
          from .manifest import utils
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/manifest/__init__.py", line 2, in <module>
          from .schema import ENTRY_POINT, PluginManifest
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/manifest/schema.py", line 24, in <module>
          from ._bases import ImportExportModel
        File "/home/alobo2/.local/lib/python3.10/site-packages/npe2/manifest/_bases.py", line 6, in <module>
          import yaml
      ModuleNotFoundError: No module named 'yaml'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[aloboa]

Note I could install with no problems on our cluster just with:

conda create -n dtree2 python==3.10
conda activate dtree2
conda install gdal pygeos geopandas numpy opencv -c conda-forge
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install git+https://github.com/PatBall1/detectree2.git

[ChristopherKotthoff]

Even though you activate the conda environment, it seems like you are still using your system's python environment at times which is evident by the appearance of /home/alobo2/.local/lib/python3.10/... instead of /home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/... in your stack trace.

This can happen if there are user-site specific packages enabled in your conda environment.
python -m site --user-site to see if there are any and if yes, export PYTHONNOUSERSITE=1 to deactivate temporarily.

Remove or deactivate the PYTHONPATH environmental variable, as this can cause python to look at additional directories outside your conda environment when searching for dependencies.

Also, check your python path (after activating your conda environment):
python -c "import sys; print(sys.path)"
If it does not return /home/alobo2/miniforge3/envs/dtree2-clean/lib/python3.10/..., something in your system is finding your system default python before it finds your conda python.

Hope this solves your issue.

[aloboa]

As you anticipated:

alobo2@alobo-ws:~$ conda activate dtree2
(dtree2) alobo2@alobo-ws:~$ python -m site --user-site
/home/alobo2/.local/lib/python3.10/site-packages
(dtree2) alobo2@alobo-ws:~$ export PYTHONNOUSERSITE=1
(dtree2) alobo2@alobo-ws:~$ python -m site --user-site
/home/alobo2/.local/lib/python3.10/site-packages

But

(dtree2) alobo2@alobo-ws:~$ echo $PYTHONPATH 

(dtree2) alobo2@alobo-ws:~$ which python
/home/alobo2/miniforge3/envs/dtree2/bin/python

Also, are you sure this is related to the error that I am reporting?
Note the python issue occurs in other environments that work with no problems, eg

alobo2@alobo-ws:~$ conda activate enmapbox
(enmapbox) alobo2@alobo-ws:~$ python -m site --user-site
/home/alobo2/.local/lib/python3.12/site-packages
(enmapbox) alobo2@alobo-ws:~$ which python
/home/alobo2/miniforge3/envs/enmapbox/bin/python

alobo2@alobo-ws:~$ conda activate sam
(sam) alobo2@alobo-ws:~$ python -m site --user-site
/home/alobo2/.local/lib/python3.11/site-packages
(sam) alobo2@alobo-ws:~$ which python
/home/alobo2/miniforge3/envs/sam/bin/python

Also in our cluster, whera I had no installation problems for dtree2:

[alobo@earth ~]$ conda activate dtree2
(dtree2) [alobo@earth ~]$ python -m site --user-site
/home/alobo/.local/lib/python3.10/site-packages
(dtree2) [alobo@earth ~]$ which python
~/miniconda3/envs/dtree2/bin/python