Skip to content

Commit 1582436

Browse files
committed
Fix on-the-fly model conversion in Pathways by forwarding storage flags
Pass checkpoint_storage_use_ocdbt and checkpoint_storage_use_zarr3 to to_maxtext subprocess in from_pretrained to ensure compatibility with Pathways parent process.
1 parent a8be563 commit 1582436

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

src/maxtext/utils/model_creation_utils.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -790,6 +790,8 @@ def from_pretrained(
790790
"skip_jax_distributed_system=True",
791791
"--lazy_load_tensors=True",
792792
f"--simulated_cpu_devices_count={simulated_cpu_devices_count}",
793+
f"checkpoint_storage_use_ocdbt={config.checkpoint_storage_use_ocdbt}",
794+
f"checkpoint_storage_use_zarr3={config.checkpoint_storage_use_zarr3}",
793795
]
794796

795797
try:

0 commit comments

Comments
 (0)