fix(trainer): require complete multi-run checkpoint publication

[peter941221]

Fixes #3037.

Problem

MultiCheckpointManager could publish checkpoints/step_N/STABLE when its matching policy snapshot was unavailable or incomplete: a rank could fail while writing trainer state, a filesystem broadcast could contain partial files without its own STABLE marker, or the copy into checkpoint storage could fail. A stable marker is the resumability contract, so those states could be selected for resume or retention without a restorable inference policy.

Fix

Checkpoint publication now has three consensus-gated phases: every rank writes its trainer state, the master copies a stable broadcast snapshot, then the master writes the checkpoint STABLE marker. Each result is reduced across ranks, so every rank either records the step or leaves it incomplete.

The copy requires broadcasts/step_N/STABLE, rejects partial broadcast directories, and replaces an unpublished partial weight snapshot on a same-step retry. Multi-run checkpoint history now includes only directories with both STABLE and weight/, so legacy corrupt markers cannot block a retry or evict the last resumable checkpoint.

Verification

Ruff check and format verification pass for the changed files. The focused multi-checkpoint test file passes all six cases: missing broadcast, unstable broadcast, a failed remote rank state write, retry after a partial weight copy, a legacy stable marker without weights, and successful publication. The isolated CPU test harness stubs only the unused optional ring_flash_attn import required during model-module collection.

---

Note

Medium Risk
Changes distributed checkpoint publication and resume eligibility for multi-run training; incorrect consensus logic could block saves or leave bad checkpoints, but scope is limited to multi_ckpt and is covered by new unit tests.

Overview
Multi-run checkpoints are only published when trainer state, weight copy, and STABLE all succeed across ranks, so resume/retention no longer treats half-written steps as valid.

MultiCheckpointManager.save is split into three all_reduce(MIN)-gated phases: per-rank trainer writes, master copy from broadcasts/step_N only if that broadcast has its own STABLE (replacing any unpublished partial weight/ on retry), then master mark_stable. The step is added to ckpt_steps only when every rank agrees publication succeeded; otherwise the step stays incomplete and can be retried.

On manager creation, ckpt_steps is rebuilt from stable steps that also have a weight/ directory, ignoring legacy STABLE markers without matching policy snapshots.

Adds six unit tests covering missing/unstable broadcasts, cross-rank save failure, partial weight retry, and successful publication.

^{Reviewed by Cursor Bugbot for commit 92e61ee. Bugbot is set up for automated code reviews on this repo. Configure here.}