fix: extend timeout for jobset_ttr_drain_restart to 90 minutes - #290
Open
lechipai0209 wants to merge 2 commits into
Open
fix: extend timeout for jobset_ttr_drain_restart to 90 minutes#290lechipai0209 wants to merge 2 commits into
lechipai0209 wants to merge 2 commits into
Conversation
lechipai0209
requested review from
alfredyu-cienet,
andrewyct,
severus-ho and
xibinliu
as code owners
July 6, 2026 07:30
alfredyu-cienet
force-pushed
the
dev
branch
4 times, most recently
from
July 14, 2026 03:12
a577785 to
71b8d8a
Compare
alfredyu-cienet
force-pushed
the
dev
branch
3 times, most recently
from
July 21, 2026 01:14
ccd7f71 to
b588728
Compare
lechipai0209
force-pushed
the
tpu-obs/user/kim/fix_jobset_ttr_drain_restart_timeout
branch
from
July 23, 2026 02:23
6bccef5 to
8838f7b
Compare
Collaborator
|
please finalize the PR description, we could at least mentioning "why" extra information for you, PR description is essentially the same as commit message, which is very close to adding comments in code, where "why" is more important then "what" |
| "validate_interruption_count_gke_host_error": DefaultTimeout, | ||
| "validate_interruption_count_gce_defragmentation": DefaultTimeout, | ||
| "validate_interruption_count_gce_other": DefaultTimeout, | ||
| "validate_interruption_count_gce_migrate_on_hwsw_maintenance": DefaultTimeout, |
Collaborator
There was a problem hiding this comment.
I don't think do line break for a config like this make sense,
so use # pylint: disable=line-too-long to disable it, and then enable back at the proper line
lechipai0209
force-pushed
the
tpu-obs/user/kim/fix_jobset_ttr_drain_restart_timeout
branch
from
July 23, 2026 08:19
8838f7b to
1a5e88e
Compare
Author
thanks. I add more information. |
alfredyu-cienet
force-pushed
the
dev
branch
2 times, most recently
from
July 27, 2026 07:46
54f9a78 to
8e4ea82
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Change
Extend jobset_ttr_drain_restart timeout from 30mins to 90 mins.
Reason
In previous codebase, the timeout of this dag is 30 mins, so task
wait_for_metric_up_loadis often irrupted by dag timeout (this usually takes more than 45mins). The followingclean_uptasks wouldn't be executed. The node pool and pods would be left in cluster until next test flow.When the next test flow is starting, it will skip
create_node_poolfor existing node pool, andensure_all_pods_runningwill reuse existing node pool but couldn't match resource with proper label.Extending dag timeout could prevent the task from being stopped by dag timeout, so the clean up tasks can finish in every test flow.
Checklist
Before submitting this PR, please make sure (put X in square brackets):