Commit 5178d95
committed
Fix stateful dataloader state not restored on resume after
Accessing trainer.estimated_stepping_batches in configure_optimizers (e.g. for OneCycleLR) runs FitLoop.setup_data() during strategy setup, before the checkpoint's loop state is restored. The second setup_data() call in fit_loop.run() then early-returns because the combined loader already exists, so the stateful dataloader states from the checkpoint are never loaded and resumed training replays already-seen batches. Load the pending dataloader states in the early-return path and recreate the fetcher iterator so the restored state takes effect.
Fixes #20550estimated_stepping_batches
1 parent fe6b1cc commit 5178d95
3 files changed
Lines changed: 20 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
224 | 224 | | |
225 | 225 | | |
226 | 226 | | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
227 | 234 | | |
228 | 235 | | |
229 | 236 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1248 | 1248 | | |
1249 | 1249 | | |
1250 | 1250 | | |
| 1251 | + | |
1251 | 1252 | | |
1252 | | - | |
| 1253 | + | |
1253 | 1254 | | |
1254 | 1255 | | |
1255 | 1256 | | |
| |||
1258 | 1259 | | |
1259 | 1260 | | |
1260 | 1261 | | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
1261 | 1268 | | |
1262 | 1269 | | |
1263 | 1270 | | |
| |||
1276 | 1283 | | |
1277 | 1284 | | |
1278 | 1285 | | |
| 1286 | + | |
1279 | 1287 | | |
1280 | | - | |
| 1288 | + | |
1281 | 1289 | | |
1282 | 1290 | | |
1283 | 1291 | | |
| |||
1291 | 1299 | | |
1292 | 1300 | | |
1293 | 1301 | | |
1294 | | - | |
| 1302 | + | |
1295 | 1303 | | |
1296 | 1304 | | |
0 commit comments