fix: log the convenience epoch metric once per step (#20902)#21796
Open
gaurav0107 wants to merge 1 commit into
Open
fix: log the convenience epoch metric once per step (#20902)#21796gaurav0107 wants to merge 1 commit into
epoch metric once per step (#20902)#21796gaurav0107 wants to merge 1 commit into
Conversation
…0902) At an epoch boundary several metric groups are flushed at the same step (the training-epoch-end flush, the validation-end flush, and the last train-step flush). Each `step=None` flush re-stamped the convenience `epoch` metric via `setdefault`, so loggers that keep per-call history (e.g. MLflow) recorded two `epoch` datapoints per epoch and misaligned epoch-based x-axes. Track the last `(step, epoch)` for which the convenience `epoch` metric was auto-added and skip re-adding it at the same step. The key is `(step, epoch)` (not `step` alone) so an epoch that advances without an optimizer step (e.g. under gradient accumulation) is still recorded. The tracker is reset per run so a fresh fit/validate/test re-emits `epoch`. Step-indexed loggers (TensorBoard, CSV) are unaffected because they merge by step. Update the logger-history assertions in test_all.py, test_logger_connector.py, and the train/eval loop logging tests to expect the single-`epoch`-per-step behavior, and add a focused regression test.
1440fdc to
2a5ff23
Compare
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #21796 +/- ##
=========================================
- Coverage 87% 79% -8%
=========================================
Files 270 267 -3
Lines 24010 23957 -53
=========================================
- Hits 20787 18839 -1948
- Misses 3223 5118 +1895 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #20902
When metrics are logged with
on_epoch=True, they show up as two points per epoch in the MLflow UI wheneverepochis used as the x-axis.Root cause. At an epoch boundary the logger connector flushes several metric groups at the same step — the training-epoch-end flush and the validation-end flush (plus the last train-step flush when
on_stepmetrics are used). Each of thesestep=Noneflushes re-stamps the convenienceepochmetric viascalar_metrics.setdefault("epoch", ...). Loggers that keep per-call history (MLflow logs oneMetricentry perlog_metricscall) therefore record twoepochdatapoints for every real epoch, so theepochseries has twice as many points as the other metric series and the x-axis mapping is off.Fix. Track the last
(step, epoch)for which the convenienceepochmetric was auto-added and skip re-adding it at the same step. The key is(step, epoch)(notstepalone) so an epoch that advances without an optimizer step — e.g. under gradient accumulation — is still recorded. All real metrics and allstepvalues are unchanged; step-indexed loggers (TensorBoard, CSV) are unaffected because they already overwrite / merge by step.This is a minimal, backward-compatible change contained to
_LoggerConnector.log_metrics. Two existing tests encoded the old (duplicated) behavior and are updated to assert the corrected single-epoch-per-step behavior, and a focused regression test is added.Before submitting
bug).test_log_metrics_no_duplicate_epoch_per_stepand updated existing epoch/step assertions.epochis still logged once per step.PR review
Anyone in the community is welcome to review the PR.