Fix ModelCheckpoint manual-opt pre-step snapshot not applied when save is deferred to validation

[ErenAta16]

Fixes #21842.

What does this PR do?

ModelCheckpoint's manual-optimization support (added in #21239 for #20947) saves the pre-optimization-step model state captured in pl_module.saved_models, so the checkpoint reflects the state that actually produced the monitored metric rather than whatever the weights happen to be after later optimizer.step() calls.

That swap only happened in on_train_batch_end, the immediate-save path. When the monitored metric is validation-only (monitor="val_loss" and similar, the common case for this feature since manual-optimization training loops rarely log training-time metrics they'd want to checkpoint on), the save gets deferred to on_validation_end instead, via _defer_save_until_validation. That deferred path saved with no swap at all, dumping the live, fully-trained, post-optimization weights, silently defeating the entire point of the feature for its most common use case.

This is a regression of #20947, reintroduced by #21239's own fix, since that PR's test only monitors a training-step metric (monitor="loss"), which is always available in on_train_batch_end and therefore never exercises the _defer_save_until_validation branch.

Fix

Extracted the swap-save-restore logic (previously inline in on_train_batch_end) into _save_checkpoint_with_manual_opt_state, called from both on_train_batch_end and on_validation_end's deferred branch, so the two call sites can't diverge again. on_validation_end only takes this path when not pl_module.automatic_optimization, automatic-optimization checkpointing is untouched since the live weights are already the correct state to save there.

Testing

Added test_model_checkpoint_manual_opt_deferred_validation in tests/tests_pytorch/callbacks/test_model_checkpoint_manual_opt.py, right next to the existing test_model_checkpoint_manual_opt. It monitors val_loss (validation-only) instead of loss (training-step), so it exercises the previously-broken deferred path.

Confirmed red/green:

• Against the pre-fix code: fails with AssertionError: Checkpoint saved the live, post-optimization weights instead of a pre-opt snapshot.
• Against the fix: passes.

Ran the full manual-opt and step-interval-val-metric suites plus the broader ModelCheckpoint test files:

tests/tests_pytorch/callbacks/test_model_checkpoint_manual_opt.py: 3 passed
tests/tests_pytorch/callbacks/test_model_checkpoint_step_interval_val_metric.py: 1 passed
tests/tests_pytorch/checkpointing/test_model_checkpoint.py
tests/tests_pytorch/callbacks/test_model_checkpoint_additional_cases.py
tests/tests_pytorch/callbacks/test_model_checkpoint_edge_cases.py: 135 passed, 2 skipped

The remaining 7 failures in the broader run (test_model_checkpoint_path, test_model_checkpoint_save_last[link], test_model_checkpoint_link_checkpoint*, test_checkpoint_repeated_strategy_extended) are pre-existing environment issues in my local setup (missing tensorboard/tensorboardX, and Windows without symlink privileges for the save_last="link" tests), confirmed by reproducing the exact same failures against unpatched master before making any change.