[MNT] Update dependency versions and complete migration to pyproject.toml

[phoeenniixx]

This PR updates the versions of outdated dependencies and migrates build system to pyproject.toml.

Few Observations:

• torch still supports 1.X, but there is nothing about 2.X support. We should see if it can support torch 2.6+ version completely or not.
• The project uses pytorch-lightning, I think we should move it to the newer lightning version
• There are a lot of "core" dependencies. We should look how can we reduce this stack.
  Some dependencies which I think we should check if we could move to extras:
  • omegaconf
  • torchmetrics
  • tensorboard
  • protobuf
  • matplotlib
  • PyYAML
  • matplotlib
  • einops
  • fsspec
  • rich
  • ipywidgets
• We should also create few more dependency categories in pyproject.toml, like docs etc]
• We should also look into GH workflows, it is still using older python versions

---

📚 Documentation preview 📚: https://pytorch-tabular--596.org.readthedocs.build/en/596/

[phoeenniixx]

FYI @manujosephv @fkiraly

[phoeenniixx]

Superceded by #594

[fkiraly]

why that, @phoeenniixx? I think you are doing strictly more than I in my PR.
I would say, reopen and we merge step by step

[fkiraly]

did you notice that the release workflow runs on all PR?

I would suggest to merge #597 before continuing, to avoid triggering a release by accident

[fkiraly]

merged with my PR