Skip to content

[AIR] Add Trackio Integration#61632

Open
ParagEkbote wants to merge 60 commits into
ray-project:masterfrom
ParagEkbote:add-trackio-integration-for-ray
Open

[AIR] Add Trackio Integration#61632
ParagEkbote wants to merge 60 commits into
ray-project:masterfrom
ParagEkbote:add-trackio-integration-for-ray

Conversation

@ParagEkbote

@ParagEkbote ParagEkbote commented Mar 10, 2026

Copy link
Copy Markdown

Description

As described in the issue, this PR adds the integration of trackio with ray. Trackio has a drop-in API similar to W&B, the main difference between the wandb and trackio is that trackio does not rely on a global mutable state, each run is explicitly created, and each run has a fixed lifetime. The API docs have also helped to define the scope of the callback. The runs can be pushed to HF Hub as a dataset or as a Space as seen below, you can view them locally.

trackio is limited to >=0.22.0 and <= 0.24.0 due to uvicorn

Related issues

Fixes #60708

Additional information

You can the test the same with the following script:

import random
import time

import numpy as np
import trackio

import ray
from ray import tune
from ray.air.integrations.trackio import (
    TrackioLoggerCallback,
    setup_trackio,
)
from ray.train import RunConfig, ScalingConfig
from ray.train.torch import TorchTrainer


PROJECT_NAME = "trackio-ray-demo"

HF_DATASET_ID = (
    "AINovice2005/ray-trackio-experiments"  
)
HF_SPACE_ID = (
    "AINovice2005/ray-trackio-dashboard" 
)

NUM_STEPS = 15


def tune_trainable(config):

    for step in range(NUM_STEPS):

        loss = (config["lr"] * 10) / (step + 1) + random.random()
        accuracy = 1 / (loss + 1e-3)

        # Example artifact
        image = np.random.rand(64, 64, 3)

        tune.report(
            {
                "loss": loss,
                "accuracy": accuracy,
                "image_mean": float(image.mean()),
                "step": step,
            }
        )

        time.sleep(0.2)


def run_tune_example():

    tuner = tune.Tuner(
        tune_trainable,
        param_space={
            "lr": tune.grid_search([0.001, 0.01, 0.1]),
        },
        run_config=tune.RunConfig(
            name="trackio-ray-tune-demo",
            callbacks=[
                TrackioLoggerCallback(
                    project=PROJECT_NAME,
                    # Trackio capabilities
                    auto_log_gpu=True,
                    gpu_log_interval=5,
                    dataset_id=HF_DATASET_ID,
                    space_id=HF_SPACE_ID,
                )
            ],
        ),
    )

    results = tuner.fit()

    print("\nTune finished\n")
    print(results)


def train_loop(config):

    run = setup_trackio(
        config=config,
        project=PROJECT_NAME,
        auto_log_gpu=True,
        gpu_log_interval=5,
        dataset_id=HF_DATASET_ID,
        space_id=HF_SPACE_ID,
    )

    for step in range(NUM_STEPS):

        loss = 5 / (step + 1) + random.random()
        throughput = random.uniform(50, 150)

        # Log metrics
        if run:
            run.log(
                {
                    "loss": loss,
                    "throughput": throughput,
                    "step": step,
                },
                step=step,
            )

        # Example artifact
        sample_image = np.random.rand(64, 64, 3)

        if run:
            run.log(
                {
                    "image_mean": float(sample_image.mean()),
                    "image_std": float(sample_image.std()),
                }
            )

        time.sleep(0.2)

    if run:
        run.finish()


def run_train_example():

    trainer = TorchTrainer(
        train_loop_per_worker=train_loop,
        train_loop_config={"lr": 0.01},
        scaling_config=ScalingConfig(num_workers=1),
        run_config=RunConfig(name="trackio-ray-train-demo"),
    )

    trainer.fit()


def launch_dashboard():

    print("\nLaunching Trackio dashboard...\n")

    trackio.show(
        project=PROJECT_NAME,
        open_browser=True,
    )


if __name__ == "__main__":

    ray.init()

    print("\nRunning Ray Tune experiment\n")
    run_tune_example()

    print("\nRunning Ray Train experiment\n")
    run_train_example()

    print("\nOpening dashboard\n")
    print("Run dashboard manually with:")
    print('trackio show --project "trackio-ray-demo"')

    trackio.finish()
    ray.shutdown()

    print("\nDemo completed\n")

Local usage

image

HF Space: https://huggingface.co/spaces/AINovice2005/ray-trackio-dashboard

HF Dataset: https://huggingface.co/datasets/AINovice2005/ray-trackio-experiments

Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
@ParagEkbote ParagEkbote requested a review from a team as a code owner March 10, 2026 18:46
Comment thread python/ray/air/integrations/trackio.py Outdated

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new integration for Trackio with Ray, providing both a setup_trackio function for Ray Train and a TrackioLoggerCallback for Ray Tune. The implementation is clean and follows existing patterns for logger integrations in Ray.

I've identified a few areas for improvement:

  • The example in the setup_trackio docstring can lead to an AttributeError and should be corrected.
  • The error handling in TrackioLoggerCallback can be improved by logging exceptions instead of silently passing, which will make debugging easier for users.

Overall, this is a great addition. My comments are focused on improving robustness and user experience.

Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py Outdated
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread python/ray/air/integrations/trackio.py Outdated
@ray-gardener ray-gardener Bot added tune Tune-related issues train Ray Train Related Issue community-contribution Contributed by the community labels Mar 10, 2026
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py Outdated
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py Outdated
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread python/ray/air/integrations/trackio.py Outdated
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread python/ray/air/integrations/trackio.py
@ParagEkbote

Copy link
Copy Markdown
Author

Could you please review the changes?

cc: @alexeykudinkin

@ParagEkbote

Copy link
Copy Markdown
Author

Which maintainer do I have to reach out to get a first review for this integration?

A gentle ping to @alexeykudinkin

@pseudo-rnd-thoughts pseudo-rnd-thoughts left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and integration.

I've made a couple of edits based on personal style, reducing the number of new line between code (IMO new line should show a group of code that works together), changed Dict[Trial, object] to Dict[Trial, trackio.Run].
Ray uses Google docstring styling, so I've moved the docsrings to start on the """ rather than next line.

The PR is missing two important things

  • Documentation - The other experiment trackers have a page to explain how to use it. For example comet. To do this, you need to create a file in doc/source/tune/examples/tune-trackio.ipynb
  • Testing - Comet, WandB, etc have their own testing to ensure that the implementation continues working. Create a couple of tests in ray/python/air/tests/test_integration_trackio.py

I've also put some comments on the rest of the file if you could answer them

Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py
Comment thread python/ray/air/integrations/trackio.py Outdated
Comment thread python/ray/air/integrations/trackio.py
@ParagEkbote

Copy link
Copy Markdown
Author

If we choose to pin trackio to the latest version (v.0.26.0) are there any deps beside huggingface_hub that would be needed to be upgraded as well, because I'm not certain that the strict version pins present within Ray would make merging of this integration more tasking/time-consuming than it needs to be?

cc: @pseudo-rnd-thoughts

@pseudo-rnd-thoughts

Copy link
Copy Markdown
Member

Update trackio>=0.5.2,<1 to

trackio>=0.10.0,<0.21
plotly>=6.0.0,<7

then run

bazel run //ci/raydepsets:raydepsets -- build --all-configs

@ParagEkbote

Copy link
Copy Markdown
Author

Update trackio>=0.5.2,<1 to

trackio>=0.10.0,<0.21
plotly>=6.0.0,<7

then run

bazel run //ci/raydepsets:raydepsets -- build --all-configs

It seems that the trackio package issues seem to be resolved as seen with the following log:

[2026-06-04T15:26:13Z] #10 339.6   Downloading trackio-0.20.2-py3-none-any.whl.metadata (14 kB)

But the CI is failing due to a prior plotly pinned version of plotly==5.23.0 set at requirements_compiled.txt, which itself is a transitive dependency of ax-platform. How can we deal with this?

[2026-06-04T15:26:13Z] #10 339.9 The conflict is caused by:
[2026-06-04T15:26:13Z] #10 339.9     The user requested plotly<7 and >=6.0.0
[2026-06-04T15:26:13Z] #10 339.9     The user requested (constraint) plotly==5.23.0

Comment thread python/ray/air/integrations/trackio.py
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Comment thread doc/source/tune/examples/tune-trackio.ipynb
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 43d36a1. Configure here.

Comment thread python/ray/air/integrations/trackio.py
Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>
@ParagEkbote ParagEkbote marked this pull request as draft June 19, 2026 05:10
Mark Towers added 2 commits June 30, 2026 14:54
@pseudo-rnd-thoughts pseudo-rnd-thoughts added the go add ONLY when ready to merge, run all tests label Jun 30, 2026
Mark Towers added 2 commits July 1, 2026 09:52

@pseudo-rnd-thoughts pseudo-rnd-thoughts left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pseudo-rnd-thoughts pseudo-rnd-thoughts marked this pull request as ready for review July 1, 2026 09:31
@pseudo-rnd-thoughts pseudo-rnd-thoughts changed the title Add Trackio Integration for Ray [AIR] Add Trackio Integration Jul 1, 2026
@pseudo-rnd-thoughts pseudo-rnd-thoughts requested a review from a team as a code owner July 7, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community go add ONLY when ready to merge, run all tests train Ray Train Related Issue tune Tune-related issues unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Trackio as a New Backend for Experiment Visualization

2 participants