Commit 7c362d7
fix(jax): write the checkpoint pointer beside save_ckpt (#5726)
## Problem
Fixes #5678. JAX training writes checkpoint directories and the stable
`.jax` link relative to `save_ckpt` (which may include a directory), but
always wrote the `checkpoint` pointer file to the current working
directory with a value that still carried the directory prefix, e.g.
`runs/water/model.ckpt.jax`. The freeze entrypoint looks for the pointer
inside the folder it is given and resolves the pointer's value relative
to that folder (`checkpoint_folder / pointer`). So for `save_ckpt =
runs/water/model.ckpt`, the pointer was written to `./checkpoint` (not
`runs/water/checkpoint`) and, even if relocated, its value would have
double-prefixed to `runs/water/runs/water/model.ckpt.jax`. Passing
`runs/water` to freeze or restart-style tooling could not find or
resolve the checkpoint, even though the matching checkpoint directory
and `.jax` link were written there.
## Fix
Write the pointer into `Path(save_ckpt).parent` and store a value
relative to that directory (the basename only). For the default bare
`save_ckpt` (parent is `.`) the pointer stays in the CWD with the same
value, so existing behavior is unchanged; only directory-valued
`save_ckpt` is affected.
## Test
Adds `source/tests/jax/test_checkpoint_pointer.py`, which drives
`_save_checkpoint` with the checkpoint I/O mocked. The directory case
asserts the pointer lands beside the checkpoint (`subdir/checkpoint`)
with a basename value (`model.ckpt.jax`) and not in the CWD — this fails
on master — and a bare-name control asserts the pointer stays in the
CWD. The trainer's pointer writing previously had no coverage; the
existing freeze test hand-wrote a correct pointer and never exercised
the writer.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Fixed checkpoint “pointer” file placement to be written alongside the
configured checkpoint save directory/path rather than always in the
current working directory.
- Updated pointer contents to reference the correct checkpoint name,
improving downstream checkpoint resolution during freeze/restart.
- **Tests**
- Added unit tests validating pointer location and contents for both
nested save paths and bare checkpoint filenames.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>1 parent d884779 commit 7c362d7
2 files changed
Lines changed: 73 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
841 | 841 | | |
842 | 842 | | |
843 | 843 | | |
844 | | - | |
845 | | - | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
846 | 852 | | |
847 | 853 | | |
848 | 854 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
0 commit comments