Skip to content

Fix: Added a new regex to make sure the restoring logs are caught correctly. - #288

Draft
camiloCienet wants to merge 2 commits into
devfrom
orbax/user/camilo/fix-restore-orbax
Draft

Fix: Added a new regex to make sure the restoring logs are caught correctly.#288
camiloCienet wants to merge 2 commits into
devfrom
orbax/user/camilo/fix-restore-orbax

Conversation

@camiloCienet

@camiloCienet camiloCienet commented Jul 3, 2026

Copy link
Copy Markdown

Summary
Refactored log validation utilities to support robust, multi-pattern regex filtering across Orbax and Post-Training DAGs, eliminating duplicate code and hardcoded query strings.

Key Changes
Validation Utilities (orbax/util/validation_util.py):

Replaced the singular text_filter string parameter with a text_filters list across all log validation functions.
Added a FilterMode flag enum (textPayload, jsonPayload_message) to strongly type which payload fields are searched.
Updated list_log_entries to dynamically construct OR-joined Google Cloud Logging regex (=~) queries from the provided filters.
Post-Training Facade (post_training/util/validation_util.py):

Re-exported the FilterMode enum to maintain clean, isolated imports for downstream post-training DAGs.
DAG Migrations:

Orbax: Updated 6 test DAGs (e.g., maxtext_emc_restore_local.py, maxtext_mtc_save_gcs.py) to utilize the new text_filters list and FilterMode flags.
Post-Training (maxtext_sft.py, maxtext_rl.py): Migrated to the new interface. Replaced brittle logical AND GCL string queries with robust, order-agnostic regex patterns to guarantee accurate JSON payload evaluations.

Fixes

It fixes b/530444528

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run one-shot tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@camiloCienet
camiloCienet force-pushed the orbax/user/camilo/fix-restore-orbax branch from 6121e81 to 06b5510 Compare July 3, 2026 10:51
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 3 times, most recently from d43f051 to a577785 Compare July 9, 2026 08:48
Comment thread dags/orbax/util/validation_util.py Outdated
@@ -467,14 +467,16 @@ def validate_restored_correct_checkpoint(
namespace="default",
pod_pattern=pod_pattern,
text_filter=(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about this

from enum import Flag, auto

class FilterMode(Flag):
    textPayload = auto()
    jsonPayload_message = auto()

in which textPayload will be 0001 (binary) and jsonPayload_message will be 0010 (binary)

def list_log_entries(
    ...,
    text_filters: list[str],
    filter_mode: FilterMode,
    ...,
):
  ...

  filters = []
  for txt in text_filters:
    if FilterMode.textPayload in filter_mode:
      filters.append(f"textPayload: {txt}")
    if FilterMode.jsonPayload_message in filter_mode:
      filters.append(f"jsonPayload.message: {txt}")

  text_filter = f"({" OR ".join(filters)})"

and then in this function, we can do somehting like this

reg_save_event = r"'event_type': 'save'"
reg_restor_event = r"'event_type': '(emergency_)?restore'"
reg_restoring = r"'restoring from this run\'s directory step (\d+)'"

list_log_entries(
    ...,
    text_filters=[reg_save_event, reg_restor_event, reg_restoring],
    filter_mode=FilterMode.textPayload | FilterMode.jsonPayload_message,
    ...,
)

...

message = ...
if re.search(reg_save_event, message):
  ... # do stuff
elif re.search(reg_restor_event, message) or re.search(reg_restoring, message):
  ... # do stuff

@camiloCienet
camiloCienet force-pushed the orbax/user/camilo/fix-restore-orbax branch 2 times, most recently from 8859365 to 575f45d Compare July 15, 2026 09:11
…reclty.

Ref: Refactor list_log_entries function to be more flexible.
@camiloCienet
camiloCienet force-pushed the orbax/user/camilo/fix-restore-orbax branch from 575f45d to 350b255 Compare July 15, 2026 09:26
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 4 times, most recently from b588728 to 0e938b1 Compare July 23, 2026 01:43
@alfredyu-cienet
alfredyu-cienet force-pushed the dev branch 2 times, most recently from 54f9a78 to 8e4ea82 Compare July 27, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants