Fix progress bar total under gradient accumulation with iterable datasets#21802
Open
Akrao9 wants to merge 1 commit into
Open
Fix progress bar total under gradient accumulation with iterable datasets#21802Akrao9 wants to merge 1 commit into
Akrao9 wants to merge 1 commit into
Conversation
Akrao9
requested review from
ethanwharris,
justusschock and
tchaton
as code owners
…sets PR Lightning-AI#20869 made `total_train_batches` fall back to `max_steps - global_step` for infinite dataloaders so the bar shows a finite total. That quantity is in optimizer steps, but the bar's counter advances per batch: with `accumulate_grad_batches=N` each optimizer step consumes N batches, so the bar reads e.g. 500000/100000 over a full run (5x overshoot at N=5). The tests added in Lightning-AI#20869 left `accumulate_grad_batches` at its default of 1, where steps and batches coincide, which is why the mismatch went unnoticed. Convert the remaining-step count to batch units before taking the min with `num_training_batches`, and parametrize the regression test over `accumulate_grad_batches` (asserting both the bar total and the final counter, tqdm `.n` / rich `.completed`).
Akrao9
force-pushed
the
fix/progress-bar-total-grad-accumulation
branch
from
2c892bd to
7d24ecf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Follow-up to #20869. That PR made
ProgressBar.total_train_batchesfall back tomax_steps - global_stepfor infinite (no-__len__) dataloaders, so the progress bar shows a finite total instead of?.The problem: that quantity is in optimizer steps, but the bar's counter advances per batch. With
accumulate_grad_batches=N, each optimizer step consumes N batches, so over a full run the bar reads e.g.500000/100000(5× overshoot at N=5) and the ETA is wrong. The regression slipped through because the tests added in #20869 leaveaccumulate_grad_batchesat its default of1, where steps and batches coincide.Fix: convert the remaining-step count to batch units (
* accumulate_grad_batches) before themin()withnum_training_batches. Behavior is unchanged whenaccumulate_grad_batches=1and for finite dataloaders (wherenum_training_batcheswins themin).Test: parametrized the existing
test_tqdm_total_steps_with_iterator_no_lengthoveraccumulate_grad_batches=[1, 4], asserting both the bar total and the final counter (tqdm.n/ rich.completed).Follow-up to #20869; fixes the gradient-accumulation case of #20124.
Before submitting