WIP: Modal nightly cron — stacked on Greene PR (do not merge)

[eugenevinitsky]

Summary

WIP — runs the same nightly fan-out as the Greene launcher (sibling PR), on Modal with a built-in cron schedule.

Stacked on top of #497 — base is `ev/nightly_runs_greene` because `modal_app.py` reads the yamls added in that PR at runtime. Merge order: #497, then this.

• `scripts/modal/Dockerfile` — CUDA 12.8.1 / Ubuntu 24.04 / Python 3.12 / uv / torch cu128 base. Multi-arch CUDA (sm_75 / sm_80 / sm_89 / sm_90) so one image runs on T4 / A100 / L4 / L40S / H100 / H200 without rebuild.
• `scripts/modal/modal_app.py` — Modal app with:
  • per-seed `train()` on 1× A100-80GB (cpu=24, memory=32GB, timeout=12h, retries=1)
  • `nightly()` cron at 04:00 UTC fanning out to 6 parallel runs (3 seeds × {single, multi})
  • routes to wandb projects `nightly-single` / `nightly-multi`, group = today's UTC date
• `scripts/modal/README.md` — setup / deploy / manual trigger workflow.

Status

Smoke-validated end-to-end on A100-80GB:

• wandb run live: https://wandb.ai/emerge_/nightly-modal/runs/n8a85v5z
• ~200K SPS, full PPO + Drive metric pipeline confirmed (entropy, episode_return, offroad_rate, num_goals_reached, dnf_rate, …)

Why WIP / do not merge

• Hasn't been run with the multi-agent (`nightly_best.yaml`) yaml on Modal yet — needs to fit in the 12h container timeout.
• Cost is non-trivial — 6 × A100-80GB-hours/night ≈ a few hundred / month. Wait for a real multi-agent number before flipping cron on.
• Image-cache caveat: changing the Dockerfile invalidates everything below it. Want to settle the apt list before this lands so subsequent edits don't trigger 5-min apt rebuilds.

Test plan

• Smoke training run on A100-80GB to ≥10M steps (in progress)
• Manual `modal run scripts/modal/modal_app.py::nightly` to verify the 6-job fan-out
• Multi-agent yaml runs to completion in a 12h container
• Verify wandb run finalization (model upload) lands

🤖 Generated with Claude Code