Poetry debugging with PyCharm not possible?

[volkerjaenisch]

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.
• [x ] I have searched the documentation and believe that my question is not covered.

Issue

Hi!
I am working on a Poetry enhancement to install Anaconda packages with Poetry. The package resolution already works fine. I just transformed the Anaconda packages into dummy Python Wheels reflecting the conda package dependencies. These dummy wheels I deliver via a private PyPIserver. And Poetry can work with them.

volker@runner:~/workspace/PYTHON4/test_poetry$ poetry add numpy
Using version ^1.20.3 for numpy

Updating dependencies
Resolving dependencies... (0.1s)PackageInfo: Invalid constraint (_libgcc_mutex) found in libgcc-ng-9.3.0 dependencies, skipping
PackageInfo: Invalid constraint (_openmp_mutex (>=4.5)) found in libgcc-ng-9.3.0 dependencies, skipping
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 9 installs, 1 update, 0 removals

  • Installing intel-openmp (2021.4.0)
  • Installing libgcc-ng (9.3.0)
  • Installing mkl (2021.4.0)
  • Installing blas (1.0)
  • Installing libstdcxx-ng (9.3.0)
  • Installing mkl-service (2.4.0)
  • Installing mkl-fft (1.3.1)
  • Installing mkl-random (1.2.2)
  • Installing numpy-base (1.20.3)
  • Updating numpy (1.22.1 -> 1.20.3)

As you can see the dummy Wheels as well as the resolution of the deps works quite well.

But now I have to build a CondaInstaller(BaseInstaller) that does the actual symlinking of the conda-packages into the Poetry-venv. Or if this is not so easy I could utilize Conda to do the same thing. This should be only a couple of lines.

But to enhance the Poetry Code I must be able to debug it from within PyCharm.

Running poetry from PyCharm works without problems. I configured the run-config to utilize ~/.local/bin/poetry with cmd line parameters "add numpy". As Interpreter I can choose the venv poetry is installed in or the system python. The working directory is set to the target venv "test_poetry" where the numpy package should be added. Works like charm.

But when I activate the debugging I get the following stacktrace. This is independet of interpreter settings etc.:

  EnvCommandError

  Command ['/home/volker/workspace/venvs/test-poetry-8iaVxEo2-py3.9/bin/python', '-W', 'ignore', '-'] errored with the following return code 2, and output: 
  unknown option --port
  usage: /home/volker/workspace/venvs/test-poetry-8iaVxEo2-py3.9/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
  Try `python -h' for more information.
  input was : import sys
  
  if hasattr(sys, "real_prefix"):
      print(sys.real_prefix)
  elif hasattr(sys, "base_prefix"):
      print(sys.base_prefix)
  else:
      print(sys.prefix)
  

  at ~/workspace/PYTHON4/poetry/poetry/utils/env.py:1183 in _run
      1179│                 output = subprocess.check_output(
      1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1181│                 )
      1182│         except CalledProcessError as e:
    → 1183│             raise EnvCommandError(e, input=input_)
      1184│ 
      1185│         return decode(output)
      1186│ 
      1187│     def execute(self, bin, *args, **kwargs):

Process finished with exit code 1

This behavior is quite easily to reproduce. Just try to debug a typical installation of Poetry with PyCharm.
This looks to me as if Poetry switches the interpreter and this messes up with the pydevd debug framework. I can actually debug poetry for a while, but in

    def create_venv(
        self, io, name=None, executable=None, force=False
    ):  # type: (IO, Optional[str], Optional[str], bool) -> Env
        if self._env is not None and not force:
            return self._env

        cwd = self._poetry.file.parent
        **env = self.get(reload=True)**

the exception is raised.

Any help appreciated
Volker

[volkerjaenisch]

Update!
I debugged this issue further and actualized my issue ticket at PyCharm.

https://youtrack.jetbrains.com/issue/PY-52864

Cheers,
Volker

[volkerjaenisch]

Workaround found!

Unchecking
Utilizing Settings -> Build, Execution, Deployment -> Python Debugger -> "Attach to subprocess automatically while debugging"

Solves the the problem.

This also disables the debugging of subprocesses, so this is not a general solution but only a workaround.

A clean solution would be for Poetry to check if it is run from PyCharm, debugger. Poetry already checks for some execution environments Anaconda, Mingw, etc, why not also checking for PyCharm debugging?
Then Poetry could include the pydevd proxy of PyCharm (extending the cmd) to enable debugging of the subprocesses.

Cheers,

Volker

[finswimmer]

Hello @volkerjaenisch,

thanks a lot for looking into this and finding a workaround 👍 I realized this problem as well but never tried to have a closer look into it.

Then Poetry could include the pydevd proxy of PyCharm (extending the cmd) to enable debugging of the subprocesses.

Do you have any literature about this? I've no idea where to start :(

fin swimmer

[volkerjaenisch]

Dear @finswimmer

Hello @volkerjaenisch,

thanks a lot for looking into this and finding a workaround +1 I realized this problem as well but never tried to have a closer look into it.

You are welcome!

Then Poetry could include the pydevd proxy of PyCharm (extending the cmd) to enable debugging of the subprocesses.

Do you have any literature about this? I've no idea where to start :(

I would like to have either :-) Best chances are to hop on my issue ticket at PyCharm

https://youtrack.jetbrains.com/issue/PY-52864

and ask for advice there. The more people will join the discussion there the more gravity the issue gets.

Cheers,
Volker

[mitchhentges]

Do you have any literature about this? I've no idea where to start :(

For what it's worth, you can try diving into pydevd.py ($HOME/.local/opt/PyCharm/plugins/python/helpers/pydev/pydevd.py on Linux, you'll know where it is when you debug a file in Python by the first line outputted in the console).
It's a 2000+ line Python script, and there's also IDE behaviour tied into it, so it's tricky to poke without causing it to misbehave.

[PhorstenkampFuzzy]

We are trying to create a pull request for another unrelated Issue and would appreatiate the PyCharm debugger too.

[CarlosDLMC]

Hello @volkerjaenisch ! Thanks for your investigation around this issue.
Could you please share with us the run configuration you are using with pycharm so we can run it?
I am working with poetry for a fast-api project, and this would help me a lot.

Thanks!

[volkerjaenisch]

Dear @CarlosDLMC
Just to state it clearly. There is absolutely no issue to debug your python code is you are using Poetry as a package manager.
The issue that I brought up here is only when you are trying to debug poetry itself.
Cheers,
Volker

[neersighted]

Also, the issue is mostly because of Poetry aggressively using subprocesses to introspect the states of remote Python environments (we run python -c with strings a lot), or to install packages (spawning child pips) -- it's more Poetry stressing the limits of PyCharm than anything else.

[jer-tx]

Not sure if this warrants a new issue created or not but I also am unable to attach the pycarm debugger to poetry. I can run poetry from pycharms run config successfully, but using the debugger or trying to otherwise attach the debugger to the spawned process, it errors out. Redacted the below, replaced the name of the project with "myProject"

/Users/jeremy/PycharmProjects/myProject/.venv/bin/python3.9 -X pycache_prefix=/Users/jeremy/Library/Caches/JetBrains/PyCharm2024.2/cpython-cache /Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py --module --multiprocess --qt-support=auto --client 127.0.0.1 --port 55203 --file poetry run myProject jobs run documents.generate_page 
Connected to pydev debugger (build 242.20224.428)
Configuration file exists at /Users/jeremy/Library/Preferences/pypoetry, reusing this directory.

Consider moving TOML configuration files to /Users/jeremy/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.

Command ['/Users/jeremy/PycharmProjects/myProject/.venv/bin/python', '-I', '-W', 'ignore', '-c', 'import sys\n\nif hasattr(sys, "real_prefix"):\n    print(sys.real_prefix)\nelif hasattr(sys, "base_prefix"):\n    print(sys.base_prefix)\nelse:\n    print(sys.prefix)\n'] errored with the following return code 2

Error output:
unknown option --port
usage: /Users/jeremy/PycharmProjects/myProject/.venv/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.