[AIR] Add Trackio Integration#61632
Conversation
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a new integration for Trackio with Ray, providing both a setup_trackio function for Ray Train and a TrackioLoggerCallback for Ray Tune. The implementation is clean and follows existing patterns for logger integrations in Ray.
I've identified a few areas for improvement:
- The example in the
setup_trackiodocstring can lead to anAttributeErrorand should be corrected. - The error handling in
TrackioLoggerCallbackcan be improved by logging exceptions instead of silently passing, which will make debugging easier for users.
Overall, this is a great addition. My comments are focused on improving robustness and user experience.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
|
Could you please review the changes? cc: @alexeykudinkin |
|
Which maintainer do I have to reach out to get a first review for this integration? A gentle ping to @alexeykudinkin |
Signed-off-by: Mark Towers <mark.m.towers@gmail.com>
pseudo-rnd-thoughts
left a comment
There was a problem hiding this comment.
Thanks for the PR and integration.
I've made a couple of edits based on personal style, reducing the number of new line between code (IMO new line should show a group of code that works together), changed Dict[Trial, object] to Dict[Trial, trackio.Run].
Ray uses Google docstring styling, so I've moved the docsrings to start on the """ rather than next line.
The PR is missing two important things
- Documentation - The other experiment trackers have a page to explain how to use it. For example comet. To do this, you need to create a file in doc/source/tune/examples/tune-trackio.ipynb
- Testing - Comet, WandB, etc have their own testing to ensure that the implementation continues working. Create a couple of tests in
ray/python/air/tests/test_integration_trackio.py
I've also put some comments on the rest of the file if you could answer them
|
If we choose to pin |
|
Update then run
|
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
…ParagEkbote/ray into add-trackio-integration-for-ray
It seems that the trackio package issues seem to be resolved as seen with the following log: But the CI is failing due to a prior plotly pinned version of |
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 43d36a1. Configure here.
Signed-off-by: Mark Towers <mark@anyscale.com>
Signed-off-by: Mark Towers <mark@anyscale.com>

Description
As described in the issue, this PR adds the integration of trackio with ray. Trackio has a drop-in API similar to W&B, the main difference between the wandb and trackio is that trackio does not rely on a global mutable state, each run is explicitly created, and each run has a fixed lifetime. The API docs have also helped to define the scope of the callback. The runs can be pushed to HF Hub as a dataset or as a Space as seen below, you can view them locally.
trackiois limited to>=0.22.0and<= 0.24.0due touvicornRelated issues
Fixes #60708
Additional information
You can the test the same with the following script:
Local usage
HF Space: https://huggingface.co/spaces/AINovice2005/ray-trackio-dashboard
HF Dataset: https://huggingface.co/datasets/AINovice2005/ray-trackio-experiments