Support local ephemeral nvme disks

[djeebus]

Local benchmarks indicate that this drop base sandbox latency ~20-30ms

Downsides:

• Live migrations will probably get slower for longer.

---

Note

Replaces PD cache disks with configurable local NVMe SSDs for client/build nodes, provisioning them via startup script and wiring new disk-count variables through Terraform.

• Compute templates (client/build):
  • Replace PD cache disks with dynamic local-ssd NVMe scratch disks (375GB) using dynamic "disk" and range(var.*_cluster_cache_disk_count).
  • Set boot disk auto_delete on client; keep root disk sizing via build_cluster_root_disk_size_gb.
• Startup script (scripts/start-client.sh):
  • Partition local NVMe disks and assemble RAID0 (mdadm) when multiple; persist config.
  • Format XFS, add to /etc/fstab, mount at /orchestrator; create sandbox, template, build dirs.
  • Add 100G swap, persist swap and sysctl tuning; remove sudo usage; minor robustness (retry loops).
  • Template var LOCAL_CACHE_DISK_COUNT passed from Terraform.
• Terraform variables/wiring:
  • Add build_cluster_cache_disk_count and client_cluster_cache_disk_count (with validation) in root and module; pass through in main.tf.
  • Remove cache disk size/type vars from nomad-cluster module (PD-based settings).

^{Written by Cursor Bugbot for commit 4e008cd. This will update automatically on new commits. Configure here.}

[ValentaTomas]

Do we want this?

[djeebus]

No reason to keep a boot or cache disk around after we delete the VM instance, right?

[ValentaTomas]

Maybe in some strange debugging circumstances, but agree here.
Does this mean we are accumulating the disk somewhere right now?

[djeebus]

No, turns out auto_delete is set to true by default. I can remove this if we don't want it be clear.

[ValentaTomas]

Is this relevant anymore?

[djeebus]

I wasn't sure if we want to commit to this, or a/b test this in production. I can assume we'll commit and remove the old stuff.

[ValentaTomas]

Will the local cache cleaner in orchestrator work ok with the RAID?

[djeebus]

Will the local cache cleaner in orchestrator work ok with the RAID?

I'll double check, but assuming it's just looking at files and folders, shouldn't be a problem.

[djeebus]

Yup, unix.Statfs works regardless of how the path is mounted.