Skip to content

Commit a4df6ea

Browse files
authored
Fix the incomplete checkpoint error (GoogleCloudPlatform#1073)
This change fixes the `ValueError: Found incomplete checkpoint error` by setting the dependency in the tasks loop.
1 parent cacc4b6 commit a4df6ea

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

dags/multipod/maxtext_sft_trainer.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@
4747
(SetupMode.STABLE, DockerImage.MAXTEXT_TPU_JAX_STABLE_STACK_CANDIDATE),
4848
(SetupMode.NIGHTLY, DockerImage.MAXTEXT_TPU_STABLE_STACK_NIGHTLY_JAX),
4949
]
50-
50+
test = []
5151
for mode, image in docker_images:
5252
command = (
5353
f'export HF_TOKEN={HF_TOKEN}',
@@ -67,3 +67,6 @@
6767
docker_image=image.value,
6868
test_owner=test_owner.SURBHI_J,
6969
).run()
70+
test.append(maxtext_v4_configs_test)
71+
for i in range(len(test) - 1):
72+
test[i] >> test[i + 1]

0 commit comments

Comments
 (0)