fix: warn when training batches are fewer than accumulate_grad_batches#21811
fix: warn when training batches are fewer than accumulate_grad_batches#21811giacomo-ciro wants to merge 8 commits into
accumulate_grad_batches#21811Conversation
accumulate_grad_batchesaccumulate_grad_batches
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #21811 +/- ##
=========================================
- Coverage 87% 79% -8%
=========================================
Files 270 267 -3
Lines 24067 24010 -57
=========================================
- Hits 20899 18865 -2034
- Misses 3168 5145 +1977 |
|
@deependujha are the failing checks expected or do I need to do something about it? |
|
@deependujha all checks had passed. I merged master again because it was updated in the meantime. |
|
@deependujha shall we merge? |
|
@deependujha I know you have a lot of PRs on your plate! When you have a moment, could you please approve the workflows so we can merge this? The fix is minimal and won't affect any other tests. |
What does this PR do?
When a training epoch contains fewer batches than
accumulate_grad_batches, Lightning still forces an optimizer step on the last batch of the epoch (a deliberate design choice from #2853, meant to flush leftover gradients when the epoch ends mid-accumulation-cycle). If the whole epoch is shorter thanaccumulate_grad_batches, every batch is the "last batch," so this rule fires on every step and gradient accumulation never actually happens as configured, gradients end up accumulated over fewer batches than intended, with no error or log to indicate it.As agreed with @deependujha in the issue discussion, this PR adds a
PossibleUserWarning(instead of a docs-only fix) raised inFitLoop.setup_data()whennum_training_batches < accumulate_grad_batches, following the same pattern as the existinglog_every_n_stepswarning in the same method.Fixes #21808
This PR does not introduce any breaking changes.
Before submitting
accumulate_grad_batchesis silently ignored when epoch length <accumulate_grad_batches, because Lightning always steps on the last batch #21808.test_warning_with_dataloader_shorter_than_accumulate_grad_batches.