feat: always-on UUID log directories with symlink

[samsja]

What

Logs are now always written to output_dir/runs/<uuid>/ with output_dir/logs symlinked to the current run's directory. This isolates logs per run while preserving full backward compatibility — all existing code that reads get_log_dir(output_dir) / "trainer.log" continues to work unchanged through the symlink.

Why

SLURM retrials and re-runs were clobbering logs from previous attempts. With UUID-isolated run directories, historical logs are preserved automatically without any config changes or user opt-in.

How

output_dir/
  logs → runs/a1b2c3d4.../     # symlink (backward compatible)
  runs/
    a1b2c3d4.../               # current run's actual logs
    f7e6d5c4.../               # previous run's logs (preserved)

• setup_log_dir() in pathing.py creates runs/<uuid>/ and atomically symlinks logs to it
• On resume (ckpt.resume_step set), the existing symlinked directory is reused
• On clean_output_dir, the entire output_dir is wiped (existing behavior), so a fresh UUID is generated
• No config changes, no SLURM template changes — everything goes through the symlink transparently

Files changed

• src/prime_rl/utils/pathing.py — new setup_log_dir() function
• src/prime_rl/entrypoints/rl.py — call setup_log_dir() in rl()
• src/prime_rl/entrypoints/sft.py — call setup_log_dir() in sft(), fix inline output_dir / "logs" to use get_log_dir()
• src/prime_rl/entrypoints/inference.py — call setup_log_dir() in inference()
• tests/unit/utils/test_pathing.py — 6 new tests for symlink creation, writability, UUID isolation, and resume reuse

Test plan

uv run python -m pytest tests/unit/utils/test_pathing.py -v
# 13 passed (7 existing + 6 new)

---

Note

Low Risk
Logging layout only; backward-compatible symlink preserves existing log paths with no auth, checkpoint, or training logic changes.

Overview
Training and inference launches now isolate logs per run under output_dir/runs/<uuid>/, while output_dir/logs stays a symlink to the active run so existing get_log_dir() paths keep working.

A new setup_log_dir() in pathing.py creates the UUID directory and atomically repoints the logs symlink; rl, sft, and inference call it after output_dir is created. Resume (resuming=True) reuses the symlink target so continued runs append to the same log tree; each fresh start gets a new UUID and prior runs/* dirs are left intact. SFT local torchrun --log-dir now goes through get_log_dir() instead of a hardcoded output_dir / "logs".

Unit tests cover symlink creation, writes through the link, new UUID per non-resume call, and resume reuse.

^{Reviewed by Cursor Bugbot for commit f0bbeae. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0fbac0b. Configure here.}

[cursor]

Nested inference rotates log symlink

High Severity

Local RL runs call setup_log_dir on the run output_dir, then spawn an inference subprocess whose entrypoint calls setup_log_dir again with default resuming=False. Because inference.toml omits output_dir, the child defaults to outputs, matching the RL default, so the logs symlink is repointed to a new UUID run and component logs land in different runs/ directories.

Additional Locations (1)

• src/prime_rl/entrypoints/rl.py#L73-L78

 

^{Reviewed by Cursor Bugbot for commit 0fbac0b. Configure here.}

[cursor]

Setup fails on legacy logs folder

Medium Severity

setup_log_dir installs the symlink via os.replace onto output_dir/logs. If logs already exists as a regular directory from a prior release or manual layout, replacement raises on Unix instead of migrating or reusing it, so startup fails even when validate_output_dir allows the run.

 

^{Reviewed by Cursor Bugbot for commit 0fbac0b. Configure here.}