feat(loggers): add SwanLab logger as a wandb alternative (--swanlab_args)#1385
Open
Luodian wants to merge 3 commits into
Open
feat(loggers): add SwanLab logger as a wandb alternative (--swanlab_args)#1385Luodian wants to merge 3 commits into
Luodian wants to merge 3 commits into
Conversation
pufanyi
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add SwanLab logger as a Weights & Biases alternative
Adds an optional SwanLab results logger for evaluation runs, wired symmetrically to the existing wandb logger and driven by a new
--swanlab_argsflag. SwanLab is a lightweight, self-hostable experiment tracker many users prefer as a W&B alternative.Usage
Install the extra:
pip install lmms-eval[swanlab](orpip install swanlab).Authenticate with the standard SwanLab mechanism — set
SWANLAB_API_KEY(and optionallySWANLAB_HOSTfor a self-hosted instance). The default project islmms-eval, overridable via the args string (project=,exp_name=,mode=, and any otherswanlab.initkwarg).What it does
SwanLabLogger(lmms_eval/loggers/swanlab_logger.py) mirrorsWandbLogger's interface:post_init(results),log_eval_result(),log_eval_samples(samples),finish().log_eval_result()logs task configs to the run config and the flattened per-task numeric metrics as SwanLab scalars.log_eval_samples()(enabled with--swanlab_log_samples) logs a per-task sample count plus a capped JSON preview viaswanlab.Text.lmms_eval/__main__.pyis parallel to wandb: init when--swanlab_argsis non-empty, log on the main process after each eval (best-effort, wrapped in try/except), finish at the end.Notes
swanlabinstalled;swanlabis imported lazily and raises a clearImportError(pip install swanlab) only if the logger is actually used without it.Table/Artifacttype, so wandb's results-as-table and results-as-JSON-artifact steps are not replicated — the scalar metrics cover the leaderboard use-case.Validation
py_compileon all changed files; module imports cleanly without swanlab installed (lazy import, clearImportErroron use)mode=offline:init → post_init → log_eval_result → log_eval_samples → finishall pass on representative results/samples dictsReview pass (2026-07-06)
Two self-review commits:
SwanLabLogger(...)construction in__main__.pyis now wrapped like the log calls — a badSWANLAB_API_KEY/SWANLAB_HOSTlogs a warning and the eval continues without swanlab, instead of aborting a run at startup (swanlab.login()fires during init wheneverSWANLAB_API_KEYis set, even for offline runs).swanlab.get_run()raises when no run is active (0.7.x returnedNone), which would have sent every fresh run into the failure path. Both behaviors are handled; verified against real swanlab 0.8.4 (fresh construction reachesswanlab.initexactly once) and 0.7.x semantics are preserved.