[MNT] Migrate packaging to pyproject.toml

[fkiraly]

Migrates packaging to pyproject.toml.

• move the dependency sets from requirements to pyproject.toml
• move setup.py and setup.cfg to pyproject.toml

Also carries out a minimal improvement of the testing workflow:

• remove outdated python 3.8 and 3.9
• add concurrency handling

Depends on #597 which should be merged first

[phoeenniixx]

Few Observations:

• torch still supports 1.X, but there is nothing about 2.X support. We should see if it can support torch 2.6+ version completely or not.
• The project uses pytorch-lightning, I think we should move it to the newer lightning version
• There are a lot of "core" dependencies. We should look how can we reduce this stack.
  Some dependencies which I think we should check if we could move to extras:
  • omegaconf
  • torchmetrics
  • tensorboard
  • protobuf
  • matplotlib
  • PyYAML
  • matplotlib
  • einops
  • fsspec
  • rich
  • ipywidgets
• We should also create few more dependency categories in pyproject.toml, like docs etc]
• We should also look into GH workflows, it is still using older python versions

[phoeenniixx]

Should it be "https://github.com/pytorch-tabular/pytorch_tabular" now? given the creation of the org?

[fkiraly]

yes, I think so - although the old URL forwards to the new one, so it is not a breaking error. I will change this.

[fkiraly]

Few Observations:

* `torch` still supports 1.X, but there is nothing about 2.X support. We should see if it can support `torch` 2.6+ version completely or not.

* The project uses `pytorch-lightning`, I think we should move it to the newer `lightning` version

* There are a lot of "core" dependencies. We should look how can we reduce this stack.
  Some dependencies which I think we should check if we could move to `extras`:

* We should also create few more dependency categories in `pyproject.toml`, like `docs` etc]

* We should also  look into GH workflows, it is still using older python versions

Fully agreed with all your points - I think they are already covered by this plan here?
#592

Where your statements are of course more granular than the general plan, as to how exactly we isolate, reduce, or update dependencies.

For general plan, I think it is best if you comment on #592 to align on the precise steps?

[fkiraly]

@phoeenniixx, are you able to start my workflows?

[phoeenniixx]

@phoeenniixx, are you able to start my workflows?

I dont think so, Only maintainers can do so. I dont have write access to this repository. Workflows on my PR is running because I am a member of this org.

[fkiraly]

I will check with @manujosephv on discord, I think he said we both have write access, but currently that does not seem to be the case?