feat(fleet): macOS VM backend — darwin jobs in disposable guests via arcbox-daemon (RUN-31)#360
Draft
PeronGH wants to merge 107 commits into
Draft
feat(fleet): macOS VM backend — darwin jobs in disposable guests via arcbox-daemon (RUN-31)#360PeronGH wants to merge 107 commits into
PeronGH wants to merge 107 commits into
Conversation
PeronGH
force-pushed
the
feat/run-31-runner-vm-provisioning
branch
from
2d7e51f to
c2a8777
Compare
Contributor
|
The check suite failed on Step 6: Check formatting (
|
Member
Author
Update — changes on this branch since the PR was openedDraft is still stacked on PR #354 (fleet unenroll — merged) and PR #359. Once #359 lands I'll macOS guest support (merged in from
|
PeronGH
force-pushed
the
feat/run-31-runner-vm-provisioning
branch
from
f4b000f to
0dc1471
Compare
PeronGH
force-pushed
the
feat/run-31-runner-vm-provisioning
branch
from
0dc1471 to
3c61b1c
Compare
MacOSBootLoader, MacPlatform, and the MacHardwareModel / MacMachineIdentifier / MacAuxiliaryStorage sub-configurations, plus NSData FFI helpers for the data representation round-trip. Reuses the existing VirtualMachineConfiguration and device builders unchanged. Slice 1 of PLAN.md (macOS guest VMs).
MacOSRestoreImage (latest_supported / load_from_url + requirements) backed by a generic (id, NSError*) completion block, and an examples/macos_validate.rs that builds and validates a macOS VM configuration end to end. Part of slice 2 of PLAN.md; unblocks Gate A on signed Apple Silicon.
fetchLatestSupportedWithCompletionHandler: and loadFileURL:completionHandler: (verified against the framework headers). Gate A now passes on Apple Silicon: fetch latest restore image -> requirements -> build macOS config -> validate.
MacOSInstaller wraps VZMacOSInstaller (initWithVirtualMachine:restoreImageURL:, installWithCompletionHandler: reusing the (NSError*) state block) with progress polling via NSProgress.fractionCompleted. VirtualMachine gains pub(crate) as_ptr and dispatch_sync so the install runs on the VM's queue. examples/macos_install.rs builds a macOS config with a fresh system disk, installs from a local IPSW, then boots to Running (Gate B). Completes slice 2 of PLAN.md.
MacOSRestoreImage::url() returns the IPSW URL (remote for a fetched latest image, file URL for a loaded one), plus examples/macos_latest_url.rs to print it. Used to obtain an IPSW for the install flow.
New restart-scoped vm_mode (auto|enabled|disabled, mirroring docker_mode) and prepare-scoped macos_runner_image settings across the control proto, env config, persisted settings, observable state, validation, and CLI. Groundwork for the macOS VM backend; nothing consumes them yet.
…, ssh exec New vm.rs consuming arcbox-daemon's MacosService over its gRPC socket: startup probe (ImageList answers + runner image installed + leftover fleet-* sweep), per-job Create/Start, DHCP-lease address wait via Inspect, runner exec over SSH with the image-contract admin login (russh — password auth rules out the OpenSSH binary), and Remove(force) as the single teardown/cancellation primitive with a drop-guard panic net. Routing lands in the next commit.
vm_mode-gated startup probe (init_vm, mirroring init_docker), the vm capability replacing host_runner for the native pair when active, run_vm_job with docker-equivalent cancellation discipline plus a 6.5h runtime watchdog, Option<VmRunner> threaded through both supervisors, and a vm-backend GetAgentInfo feature flag.
FleetImageService gains the macOS kind: Prepare relays the daemon's streamed pull progress and promotes the target on completion; empty requests expand to every host-supported kind (Linux everywhere, macOS on macOS hosts). Same promote-when-runtime-absent semantics as the Docker branch. CLI prepare accepts macos-runner-image; docs cover the fleet consumption path.
Provision, ssh-exec /usr/bin/true, observe exit 0, destroy — the manual verification harness for the VM backend. The guest command becomes an internal parameter of start() to make the cycle exercisable without a real runner invocation.
The prior default host fleet.arcbox.dev does not resolve; the public gateway ingress is gateway.fleet.arcbox.dev. Make it the built-in default so enroll/run work without an ARCBOX_FLEET_GATEWAY override.
…are (PLAT-34) The gateway now acks every heartbeat with a no-op Keepalive down the Attach response stream, so a proxy in front of it (Cloudflare) sees server->client traffic and never idle-times-out the stream. Sync the vendored proto and handle the new variant explicitly in dispatch.
…ponse The agent only started heartbeating after receiving the gateway's response headers, and the gateway only acks a heartbeat once it receives one — so neither side ever sent a byte first. The connection sat fully idle until a load balancer in front reset it on its own idle timeout, delivering the response headers bundled with that reset. Start heartbeating before the request is even sent so the agent always speaks first. Confirmed against the live deployment: the previous build failed identically whether Cloudflare was in front of the gateway or not (same load balancer, same ~60s idle timeout, either way); this build holds the stream open indefinitely and a real offer was dispatched, accepted, and ran to completion. spawn_heartbeat now returns an AbortOnDropHandle so its task is cleaned up on every exit path, not just the one this file's original call site happened to reach.
Adds macOS arm64 (signed) and Linux x86_64/arm64 (native musl) build jobs, each packaged into its own arcbox-fleet-agent-<platform> tarball and attached to the GitHub Release alongside the desktop tarball.
Vendored proto synced with platform PR #112: EnrollRequest carries agent_version (dropping protocol_version), EnrollResponse is a oneof (Enrolled | AgentUpdate). AttachRequest carries a first-message Attach { agent_version, capabilities, host_info_json }; AttachResponse gains AttachAccepted, AttachRejected(AgentUpdate), and HeartbeatAck (rename of Keepalive). Heartbeat trims to just telemetry — the only field that actually changes over time; capabilities and host_info are per-process constants and belong on the Attach handshake. connect_and_serve now sends the Attach message as its first outbound (still speaks-first for PLAT-34) with the full declarative payload, waits for AttachAccepted, and only then starts the heartbeat loop. AttachRejected surfaces as an AgentUpdateRequired error that the reconnect loop parks on (analogous to the RUN-40 credential-rejection park). enroll() branches on EnrollResponse.result and surfaces the expected version on rejection instead of a stringly-typed error. Deletes the x-arcbox-protocol-version metadata header and the PROTOCOL_VERSION const on both sides of the wire.
… macOS New `install-service` / `uninstall-service` subcommands render a LaunchAgent plist against the invoking binary path and the agent's data-dir log path, write it to `~/Library/LaunchAgents/`, and drive `launchctl bootstrap` / `bootout`. No sudo, no static plist file to ship, no manual `launchctl` steps for the operator. The plist runs `arcbox-fleet-agent serve` at login (RunAtLoad), respawns only on non-zero exit (KeepAlive.SuccessfulExit=false — a `launchctl bootout` SIGTERM lets the agent drain runners and exit cleanly without getting resurrected), and tees launchd's own stdout/stderr into the same log dir the agent's logger uses so early-startup output before `arcbox_logging::init` still lands next to the real logs. Uninstall is idempotent — a `bootout` with the job not loaded and a missing plist file both count as success, since the desired end state is "not installed." macOS-only for this cut. Linux systemd (user unit) is the natural follow-up; the CLI shape stays uniform (subcommand exists everywhere, bails with a clear message on non-macOS) so docs and completions don't diverge.
PeronGH
force-pushed
the
feat/run-31-runner-vm-provisioning
branch
from
e0682dd to
530b242
Compare
upload/download-artifact v4 strips permissions — every CI artifact arrives 0644, and the tar builder records on-disk modes, so shipped tarballs (desktop and fleet) contained non-executable binaries. Stage binaries through a shared chmod-0755 helper in both packagers; bundle plist/entitlements files stay 0644. Regression test packages a 0644 input and asserts the tar entry mode.
The platform's fleet vocabulary is GOOS-style everywhere — proto Capability.arch, RunnerArch, the dashboard's MachineArch — all say arm64|amd64. The CI slug linux-x86_64 was the only x86_64 in the system and would force a permanent mapping shim once the gateway starts constructing AgentUpdate.tarball_url from (os, arch). Rename to linux-amd64 before the first fleet tarball ships; the Rust build target is unchanged.
The agent's version is a wire-level contract (the gateway pins one expected agent_version at a time), so inheriting the workspace version meant every desktop release invalidated the entire fleet even with zero fleet changes. fleet/ becomes its own release-please component (tags fleet-agent-vX.Y.Z, starting at 0.5.0 to stay disjoint from the desktop 0.4.x line) and is excluded from the root package, so desktop releases no longer churn agent_version and fleet hotfixes no longer ride the desktop release train. The crate is publish = false, so the coordinated crates.io publish is unaffected.
…kflow release-fleet-agent.yml builds on fleet-agent-v* tags: signed darwin-arm64 plus native linux-amd64/arm64 musl builds, published as bare binaries named arcbox-fleet-agent-<os>-<arch> with .sha256 files (make_latest: false keeps the desktop release as the repo's latest). The download URL is now a pure function of (tag, host_os, host_arch) — the exact shape the gateway will emit in AgentUpdate.binary_url. The agent ships unarchived because the artifact is a single file: the checksum covers exactly the bytes the self-updater executes, and the extraction/exec-bit/inner-directory ceremony of a tarball bought nothing. xtask package-fleet-tarball becomes release fleet-asset; the exec-bit regression test moves to package_tarball, where the desktop tarball still needs it. Fleet jobs leave release.yml.
Vendored fleet.proto synced with the platform auto-update branch: AgentUpdate gains binary_url/binary_sha256, HeartbeatAck gains a mid-stream update payload, and Enroll/Attach carry the agent's own host_os/host_arch (distinct from capabilities — a darwin host serving linux jobs still needs the darwin binary). The agent now reports its platform on both handshakes so the gateway can resolve the download for it.
update.rs implements download → sha256-verify → probe → swap → re-exec. The gateway-pinned checksum is the integrity root, and the --version probe of the downloaded binary is the loop-breaker: a mis-registered release can never install, so the agent can never exec itself into an update cycle. The swap is two renames (never a write over the live path), keeping .prev for forensics; exec preserves argv and the PID, so launchd bookkeeping is undisturbed. Self-update only manages binaries it owns: install-service now installs the invoking binary into <data_dir>/bin/arcbox-fleet-agent and points the plist there, so updates never invalidate the service definition. Unmanaged binaries (dev builds) refuse with a typed error and park. An enrollment refused for version mismatch now carries the pushed download: the CLI enroll path self-updates and re-execs into a retry on the new build (the join token is multi-use).
The attach loop now feeds every AgentUpdate — AttachRejected at the handshake or the new HeartbeatAck.update pushed when the pin moves mid-stream — into one path: drain (no new offers; in-flight jobs and unacked verdicts settle while the stream stays live), then swap and re-exec via the update executor. Payloads without a download keep the existing park behavior, as do unmanaged binaries. The update drain is a separate supervisor flag from the operator drain, so a moot update (the pin moved back to this build before the swap) resumes without cancelling a deliberate local drain. The control API reports the new ENROLLMENT_UPDATING as Draining.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #354 and #359
This branch is cut from a merge of both dependency branches; only the last 5 commits are this PR's content. Once #354 and #359 land (repo uses rebase-merge), this will be rebased with
git rebase --onto origin/master 0c0ac40dand marked ready — the diff then collapses to just the fleet VM backend.What
The fleet agent runs each darwin job in a disposable macOS guest, consuming
arcbox-daemonas a plain gRPC client over its existing socket. No new daemon RPCs: the design (settled after discussion, superseding the RUN-31 issue text) is SSH-exec — the runner is baked into the base image, the agent sshes in and runsrun.sh --jitconfig …, the session's exit is job completion, andRemove(force)is teardown and cancellation.vm_mode(auto|enabled|disabled, restart-scoped) +macos_runner_image(target/current) settings across proto/config/state/CLIvm.rs: daemon probe (ImageList + image installed + crashed-agentfleet-*sweep), Create → Start (the daemon's two-guest license cap is the admission signal) → Inspect-poll for the DHCP-lease address → russh password auth (admin/admin, the runner-image contract; guests are host-only reachable via vmnet NAT) → execrun_vm_jobmirrors the Docker backend's cancellation discipline, plus a 6.5 h runtime watchdog;vm-backendGetAgentInfo feature flagFleetImageService.Preparerelays the daemon'sImagePullstream forIMAGE_KIND_MACOS_RUNNER_IMAGEand promotes on completionVerification
tahoe-base@2026.07.02: provision → ssh exec → exit 0 → destroy in 9.7 s, no leftover guests (vm_round_trip_boots_and_execs_over_ssh, ignored; needs a daemon + pulled image)Closes RUN-31.