Add Trackio Integration for ZenML

[ParagEkbote]

Describe changes

I implemented trackio as an experiment tracker for zenml. As compared to the approach taken on #4128, this PR focuses a larger emphasis on the public API. We can log the trial data as hf dataset, hf space and hf bucket. I'd also like to know where to add the tests for this integration and if additional metadata needs to be logged with a custom method, since trackio does not have a metadata arg and can strictly fail to log trials if a custom argument is present with trackio.init.

Could you please review?

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[ParagEkbote]

Added this section in documentation due to a python exception raised by opentelemetry, which prevents a graceful shutdown, let me know if I should remove this or add a smaller note.

[schustmi]

Can you explain a little the issue that you ran into here? What exception did opentelemetry raise, and does this only happen for you when using the trackio experiment tracker?

[ParagEkbote]

The following exception is seen if we set enable_pipeline_logs=True,

<function shutdown at 0x7a6935999580> Traceback (most recent call last): File "/home/codespace/.python/current/lib/python3.12/logging/__init__.py", line 2271, in shutdown h.flush() File "/workspaces/zenml/.venv/lib/python3.12/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 676, in flush thread.start() File "/home/codespace/.python/current/lib/python3.12/threading.py", line 992, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't create new thread at interpreter shutdown

I have tried this with the comet-ml example and the same exception is visible:

Exception ignored in atexit callback: <function shutdown at 0x7f952b966020>
Traceback (most recent call last):
  File "/home/codespace/.python/current/lib/python3.12/logging/__init__.py", line 2271, in shutdown
    h.flush()
  File "/workspaces/zenml/.venv/lib/python3.12/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py", line 676, in flush
    thread.start()
  File "/home/codespace/.python/current/lib/python3.12/threading.py", line 992, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown

System Info:

Python version: 3.12.1
OS: Ubuntu 24.04.4 LTS

[schustmi]

Can you explain a little the issue that you ran into here? What exception did opentelemetry raise, and does this only happen for you when using the trackio experiment tracker?