Fix continual DPO trainer lifecycle, explicit eval/logging, and DeepSpeed guidance

[Copilot]

The continual DPO pipeline was recreating DeepSpeed/Accelerate trainer state per task, while also doing autoregressive generation and reward scoring from the generic logging path. That combination inflated memory, hid real training throughput, and made the ZeRO-3 setup/documentation misleading for both single-GPU and multi-GPU runs.

• Trainer lifecycle

  • Replace the per-task ContinualDPOTrainer construction pattern with a single long-lived trainer/model lifecycle.
  • Remove the shared-Accelerator reuse hack entirely.
  • Add explicit task dataset switching (set_task_datasets(...)) so continual tasks reuse one prepared trainer instead of rebuilding DeepSpeed state in-process.

• Evaluation and logging path

  • Remove reward-model policy evaluation and sampled completion generation from the generic log() path.
  • Make both behaviors explicit task-end operations via config flags:
    • --eval_policy_metrics
    • --log_completions
    • --completion_logging_batches
  • Prevent metadata/custom logging from triggering full policy generation.

• Reward-model memory behavior

  • Keep reward models out of the normal training path.
  • Load reward models only for explicit evaluation/logging, run them in eval/inference mode, and release them after use.
  • Add clearer task-scoped failure context for explicit policy evaluation.

• DPO script/config cleanup

  • Fix duplicate/import-path issues in dpo_continual.py.
  • Fix sweep config argument issues and rebalance sweep defaults toward bounded-memory runs.
  • Add memory-safety warnings around unbounded sequence lengths, disabled checkpointing, and non-PEFT full-parameter runs.

• DeepSpeed config and launch guidance

  • Clarify that the ZeRO-3 config is a template and that num_processes must match the actual GPU count for multi-GPU sharding.
  • Add a separate single-GPU CPU-offload fallback config for memory-constrained runs.
  • Update README examples to distinguish:
    • fast baseline vs memory-efficient runs
    • LoRA vs QLoRA vs full fine-tuning
    • single-GPU vs multi-GPU DeepSpeed usage

• Regression coverage

  • Add focused non-GPU tests to lock in the intended behavior:
    • no shared-accelerator override
    • no generic trainer log() eval hook
    • one-trainer continual lifecycle
    • explicit reward-model loader path

Example of the new continual-task flow:

trainer = ContinualDPOTrainer(...)

for task_index, dataset in enumerate(continual_dataset):
    if task_index > 0:
        trainer.set_task_datasets(
            train_dataset=dataset[script_args.dataset_train_split],
            eval_dataset=dataset.get(script_args.dataset_test_split),
            dataset_name=f'dataset-{task_index}',
        )

    trainer.train()

    if training_args.eval_policy_metrics:
        reward_model = load_reward_model_for_task(...)
        metrics.update(trainer.evaluate_policy(reward_model=reward_model))