Skip to content

feat(loggers): add SwanLab logger as a wandb alternative (--swanlab_args)#1385

Open
Luodian wants to merge 3 commits into
mainfrom
feat/swanlab-logger
Open

feat(loggers): add SwanLab logger as a wandb alternative (--swanlab_args)#1385
Luodian wants to merge 3 commits into
mainfrom
feat/swanlab-logger

Conversation

@Luodian

@Luodian Luodian commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Add SwanLab logger as a Weights & Biases alternative

Adds an optional SwanLab results logger for evaluation runs, wired symmetrically to the existing wandb logger and driven by a new --swanlab_args flag. SwanLab is a lightweight, self-hostable experiment tracker many users prefer as a W&B alternative.

Usage

python3 -m lmms_eval \
  --model llava --model_args pretrained=... \
  --tasks mme \
  --swanlab_args project=lmms-eval,exp_name=my-run,mode=cloud \
  --swanlab_log_samples

Install the extra: pip install lmms-eval[swanlab] (or pip install swanlab).

Authenticate with the standard SwanLab mechanism — set SWANLAB_API_KEY (and optionally SWANLAB_HOST for a self-hosted instance). The default project is lmms-eval, overridable via the args string (project=, exp_name=, mode=, and any other swanlab.init kwarg).

What it does

  • SwanLabLogger (lmms_eval/loggers/swanlab_logger.py) mirrors WandbLogger's interface: post_init(results), log_eval_result(), log_eval_samples(samples), finish().
  • log_eval_result() logs task configs to the run config and the flattened per-task numeric metrics as SwanLab scalars.
  • log_eval_samples() (enabled with --swanlab_log_samples) logs a per-task sample count plus a capped JSON preview via swanlab.Text.
  • Wiring in lmms_eval/__main__.py is parallel to wandb: init when --swanlab_args is non-empty, log on the main process after each eval (best-effort, wrapped in try/except), finish at the end.

Notes

  • The module imports fine without swanlab installed; swanlab is imported lazily and raises a clear ImportError (pip install swanlab) only if the logger is actually used without it.
  • SwanLab (0.7.x) has no Table/Artifact type, so wandb's results-as-table and results-as-JSON-artifact steps are not replicated — the scalar metrics cover the leaderboard use-case.
  • No changes to the existing wandb code paths.

Validation

  • py_compile on all changed files; module imports cleanly without swanlab installed (lazy import, clear ImportError on use)
  • Functional smoke test with real swanlab 0.7.15 in mode=offline: init → post_init → log_eval_result → log_eval_samples → finish all pass on representative results/samples dicts
  • ruff clean on the new module; black + isort (pre-commit) pass

Review pass (2026-07-06)

Two self-review commits:

  • Best-effort init: SwanLabLogger(...) construction in __main__.py is now wrapped like the log calls — a bad SWANLAB_API_KEY/SWANLAB_HOST logs a warning and the eval continues without swanlab, instead of aborting a run at startup (swanlab.login() fires during init whenever SWANLAB_API_KEY is set, even for offline runs).
  • swanlab >= 0.8 support: on 0.8.x swanlab.get_run() raises when no run is active (0.7.x returned None), which would have sent every fresh run into the failure path. Both behaviors are handled; verified against real swanlab 0.8.4 (fresh construction reaches swanlab.init exactly once) and 0.7.x semantics are preserved.

@Luodian Luodian marked this pull request as ready for review July 6, 2026 13:56
@Luodian Luodian requested review from kcz358 and pufanyi July 6, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants