Fix: Added a new regex to make sure the restoring logs are caught correctly. - #288
Draft
camiloCienet wants to merge 2 commits into
Draft
Fix: Added a new regex to make sure the restoring logs are caught correctly.#288camiloCienet wants to merge 2 commits into
camiloCienet wants to merge 2 commits into
Conversation
camiloCienet
force-pushed
the
orbax/user/camilo/fix-restore-orbax
branch
from
July 3, 2026 10:51
6121e81 to
06b5510
Compare
alfredyu-cienet
force-pushed
the
dev
branch
3 times, most recently
from
July 9, 2026 08:48
d43f051 to
a577785
Compare
| @@ -467,14 +467,16 @@ def validate_restored_correct_checkpoint( | |||
| namespace="default", | |||
| pod_pattern=pod_pattern, | |||
| text_filter=( | |||
Collaborator
There was a problem hiding this comment.
how about this
from enum import Flag, auto
class FilterMode(Flag):
textPayload = auto()
jsonPayload_message = auto()
in which textPayload will be 0001 (binary) and jsonPayload_message will be 0010 (binary)
def list_log_entries(
...,
text_filters: list[str],
filter_mode: FilterMode,
...,
):
...
filters = []
for txt in text_filters:
if FilterMode.textPayload in filter_mode:
filters.append(f"textPayload: {txt}")
if FilterMode.jsonPayload_message in filter_mode:
filters.append(f"jsonPayload.message: {txt}")
text_filter = f"({" OR ".join(filters)})"
and then in this function, we can do somehting like this
reg_save_event = r"'event_type': 'save'"
reg_restor_event = r"'event_type': '(emergency_)?restore'"
reg_restoring = r"'restoring from this run\'s directory step (\d+)'"
list_log_entries(
...,
text_filters=[reg_save_event, reg_restor_event, reg_restoring],
filter_mode=FilterMode.textPayload | FilterMode.jsonPayload_message,
...,
)
...
message = ...
if re.search(reg_save_event, message):
... # do stuff
elif re.search(reg_restor_event, message) or re.search(reg_restoring, message):
... # do stuff
camiloCienet
force-pushed
the
orbax/user/camilo/fix-restore-orbax
branch
2 times, most recently
from
July 15, 2026 09:11
8859365 to
575f45d
Compare
…reclty. Ref: Refactor list_log_entries function to be more flexible.
camiloCienet
force-pushed
the
orbax/user/camilo/fix-restore-orbax
branch
from
July 15, 2026 09:26
575f45d to
350b255
Compare
alfredyu-cienet
force-pushed
the
dev
branch
4 times, most recently
from
July 23, 2026 01:43
b588728 to
0e938b1
Compare
alfredyu-cienet
force-pushed
the
dev
branch
2 times, most recently
from
July 27, 2026 07:46
54f9a78 to
8e4ea82
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactored log validation utilities to support robust, multi-pattern regex filtering across Orbax and Post-Training DAGs, eliminating duplicate code and hardcoded query strings.
Key Changes
Validation Utilities (orbax/util/validation_util.py):
Replaced the singular text_filter string parameter with a text_filters list across all log validation functions.
Added a FilterMode flag enum (textPayload, jsonPayload_message) to strongly type which payload fields are searched.
Updated list_log_entries to dynamically construct OR-joined Google Cloud Logging regex (=~) queries from the provided filters.
Post-Training Facade (post_training/util/validation_util.py):
Re-exported the FilterMode enum to maintain clean, isolated imports for downstream post-training DAGs.
DAG Migrations:
Orbax: Updated 6 test DAGs (e.g., maxtext_emc_restore_local.py, maxtext_mtc_save_gcs.py) to utilize the new text_filters list and FilterMode flags.
Post-Training (maxtext_sft.py, maxtext_rl.py): Migrated to the new interface. Replaced brittle logical AND GCL string queries with robust, order-agnostic regex patterns to guarantee accurate JSON payload evaluations.
Fixes
It fixes b/530444528
Before submitting this PR, please make sure (put X in square brackets):