three_pool: move checkpoint consolidation off the train loop (async)#540

Closed

ocg-goodfire wants to merge 10 commits into

feature/multipoolfrom

refactor/3pool-async-consolidation

Commits on May 29, 2026

three_pool: move checkpoint consolidation off the train loop (async)

ocg-goodfire
and
claude
committed
three_pool: fix async-consolidation glue — don't resolve a checkpoint pre-consolidation

ocg-goodfire
and
claude
committed
three_pool: consolidate before CUDA/NCCL init (the child job hung otherwise)

ocg-goodfire
and
claude
committed
three_pool: init distributed before consolidation; CPU-pin assembly; file-poll wait

ocg-goodfire
and
claude
committed
three_pool: fix consolidation hang root cause (unfrozen target) + fail-fast + rerun CLI

ocg-goodfire
and
claude
committed
test: 4-GPU single-block 3-pool config for live e2e validation

ocg-goodfire
and
claude
committed
three_pool: make prune race-tolerant + clean stale scratch on already-consolidated

ocg-goodfire
and
claude
committed
test: 4-GPU 2-save config (peak 8 GPU) for live e2e re-validation

ocg-goodfire
and
claude
committed
test: resume YAML from live-produced p-livee2e05 checkpoint

ocg-goodfire
and
claude
committed
three_pool: document consolidation reliability (fail-fast, rerun CLI) + drop throwaway resume YAMLs

ocg-goodfire
and
claude
committed