Description

Fix the Errors in b/535891456:

1. Missing RNGs state handling in GPT-3 PaxML checkpoint conversion:
   When executing convert_gpt3_ckpt_from_paxml.py, verification failed with:

AssertionError: ['model']['decoder']['dropout']['rngs']['aqt']['count'].value not found

Cause: Recent NNX updates added dynamic RNG state (such as AQT count) into model variables, which are not present in PaxML checkpoints.

Fix: Bypassed ['rngs'] key paths during state key verification and array loading map, and added support for non-indexed opt_state paths.

2. On-disk NNX checkpoint layout mismatch:
   When restoring the converted checkpoint in train.py, loading failed with:

ValueError: Checkpoint does not match the model:
  - 'decoder/decoder_norm/bias': missing
  ...

Cause: convert_gpt3_ckpt_from_paxml.py was saving the raw TrainStateNNX structure directly instead of calling train_state_nnx.to_checkpoint_dict(converted_state) to format it into the standard Linen-interchangeable checkpoint layout.

Fix: Added converted_state = train_state_nnx.to_checkpoint_dict(converted_state) prior to saving when cfg.pure_nnx is active.

Test

Test with convert_gpt3_ckpt_from_paxml.py and train.py as in the DAG.
Logs

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.