Discrepancy between pip and Poetry file

[seanswyi]

I'm currently trying to install PyTorch and PyTorch-XLA using a Poetry file but am finding out that it doesn't work. However, when I try to install the packages via pip it does. I'm trying to figure out why it doesn't work for my Poetry configuration.

The pip command that I'm running is taken from the official PyTorch-XLA repository:

pip install torch~=2.2.0 torch_xla[tpu]~=2.2.0 -f https://storage.googleapis.com/libtpu-releases/index.html

My pyproject.toml file looks like this:

[tool.poetry]
name = "xla-test"
version = "0.1.0"
description = ""
authors = ["John Doe"]
readme = "README.md"

[[tool.poetry.source]]
name = "libtpu"
url = "https://storage.googleapis.com/libtpu-releases/index.html"
priority = "supplemental"

[tool.poetry.dependencies]
python = "^3.8"
datasets = "^2.18.0"
torch = { url = "https://download.pytorch.org/whl/cu118/torch-2.2.0%2Bcu118-cp38-cp38-linux_x86_64.whl" }
torch_xla = { version = "~2.2.0", extras = ["tpu"], source = "libtpu" }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

When I run poetry install I get the following error message:

Repository "libtpu" does not exist.

When I try to change the priorty of libtpu to being primary rather than supplemental, Poetry complains that it can't find a compatible version of datasets, which I'm assuming means that it's not searching PyPI.

When I try to add PyPI explicitly as a supplemental source, Poetry complains that libtpu doesn't exist.

Is Poetry not appropriate for this, or am I doing something wrong?

Edit

I forgot to mention that you have to run pip install --upgrade pip first, as mentioned here: pytorch/xla#5802

[dimbleby]

does not reproduce:

$ poetry install
Updating dependencies
Resolving dependencies... (2.3s)

Because xla-test depends on torch_xla[tpu] (~2.2.0) which doesn't match any versions, version solving failed.

[seanswyi]

@dimbleby Ah sorry, I forgot to add that there are some problems w.r.t. XLA installation itself mentioned here: pytorch/xla#5802

You have to run pip install --upgrade pip beforehand it seems.

[dimbleby]

poetry is not using pip here, that cannot - and does not - make a difference

[seanswyi]

Is there a way to run that pip command as a part of the Poetry install process?

[dimbleby]

there is no way and there is no point, you are not making sense

your project configuration asks poetry to find torch_xla at https://storage.googleapis.com/libtpu-releases/index.html. It cannot do that, because it is not there.

none of this reproduces "Repository "libtpu" does not exist."