Skip to content

fix: warn when training batches are fewer than accumulate_grad_batches#21811

Open
giacomo-ciro wants to merge 8 commits into
Lightning-AI:masterfrom
giacomo-ciro:master
Open

fix: warn when training batches are fewer than accumulate_grad_batches#21811
giacomo-ciro wants to merge 8 commits into
Lightning-AI:masterfrom
giacomo-ciro:master

Conversation

@giacomo-ciro

@giacomo-ciro giacomo-ciro commented Jul 7, 2026

Copy link
Copy Markdown

What does this PR do?

When a training epoch contains fewer batches than accumulate_grad_batches, Lightning still forces an optimizer step on the last batch of the epoch (a deliberate design choice from #2853, meant to flush leftover gradients when the epoch ends mid-accumulation-cycle). If the whole epoch is shorter than accumulate_grad_batches, every batch is the "last batch," so this rule fires on every step and gradient accumulation never actually happens as configured, gradients end up accumulated over fewer batches than intended, with no error or log to indicate it.

As agreed with @deependujha in the issue discussion, this PR adds a PossibleUserWarning (instead of a docs-only fix) raised in FitLoop.setup_data() when num_training_batches < accumulate_grad_batches, following the same pattern as the existing log_every_n_steps warning in the same method.

Fixes #21808

This PR does not introduce any breaking changes.

Before submitting

Comment thread src/lightning/pytorch/CHANGELOG.md Outdated
@deependujha deependujha changed the title feat: warn when training batches are fewer than accumulate_grad_batches fix: warn when training batches are fewer than accumulate_grad_batches Jul 13, 2026
Comment thread src/lightning/pytorch/CHANGELOG.md Outdated
@codecov-commenter

codecov-commenter commented Jul 13, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79%. Comparing base (7b57f9c) to head (7454080).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

❗ There is a different number of reports uploaded between BASE (7b57f9c) and HEAD (7454080). Click for more details.

HEAD has 158 uploads less than BASE
Flag BASE (7b57f9c) HEAD (7454080)
cpu 80 42
python 5 3
lightning_fabric 26 0
pytest 41 0
python3.12 23 12
python3.13 16 9
python3.12.7 18 9
python3.10 6 3
lightning 30 15
python3.11 12 6
Additional details and impacted files
@@            Coverage Diff            @@
##           master   #21811     +/-   ##
=========================================
- Coverage      87%      79%     -8%     
=========================================
  Files         270      267      -3     
  Lines       24067    24010     -57     
=========================================
- Hits        20899    18865   -2034     
- Misses       3168     5145   +1977     

@giacomo-ciro

Copy link
Copy Markdown
Author

@deependujha are the failing checks expected or do I need to do something about it?

@giacomo-ciro

Copy link
Copy Markdown
Author

@deependujha all checks had passed. I merged master again because it was updated in the meantime.

@giacomo-ciro

Copy link
Copy Markdown
Author

@deependujha shall we merge?

@giacomo-ciro

Copy link
Copy Markdown
Author

@deependujha I know you have a lot of PRs on your plate! When you have a moment, could you please approve the workflows so we can merge this? The fix is minimal and won't affect any other tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

accumulate_grad_batches is silently ignored when epoch length < accumulate_grad_batches, because Lightning always steps on the last batch

3 participants