Merge pull request #28 from AI45Lab/docs/native-snapshot-fork

release: v2.1.0 — native snapshot-fork, warm-pool CoW fill, prune

Author: ZhiXiao-Lin

CHANGELOG.md

@@ -4,6 +4,44 @@ All notable changes to A3S Box will be documented in this file.
 
 ## [Unreleased]
 
+## [2.1.0] — 2026-06-13
+
+### Added
+- **Native snapshot-fork (Copy-on-Write microVM cloning).** A booted template
+  microVM can be snapshotted and many forks restored from it, instead of cold
+  booting each one. The snapshot captures file-backed guest RAM plus KVM vCPU
+  and virtio device state; each fork maps the RAM file `MAP_PRIVATE` so it pays
+  only for the pages it dirties. Driven by `KRUN_SNAPSHOT_MEM_FILE` /
+  `KRUN_SNAPSHOT_SOCK` (capture) and `KRUN_RESTORE_FROM` (restore), or
+  per-VM via `BoxConfig`/`InstanceSpec`. Verified on `/dev/kvm`: a single fork
+  is ~4× faster than a cold boot (~450 ms → ~110 ms), 100 forks complete in
+  under ~1 s (~8 ms amortized per VM, ~13 MB RSS each), and `exec` runs real
+  commands over virtio-fs inside the restored guest.
+- **Warm pool snapshot-fork fill** (`pool start --snapshot-fork`): the pool
+  cold-boots one template, snapshots it, then restores the rest of the pool
+  from that snapshot. Combined with concurrent (JoinSet) fill this cuts
+  fill-to-8 from ~12.4 s to ~1.9 s. Off by default; opt in with the flag.
+- **`prune` command** (`a3s-box prune`, alias `container-prune`): removes every
+  created, stopped, and dead box in one call, mirroring `docker container
+  prune`. Running and paused boxes are never touched. Requires `--force`.
+- **Per-VM snapshot/restore config seam**: `BoxConfig` and `InstanceSpec` carry
+  `snapshot_mem_file`, `snapshot_sock`, and `restore_from`, so snapshot/restore
+  can be requested per box instead of only through process-global env vars
+  (per-VM config takes precedence over the env).
+
+### Fixed
+- **Concurrent box registration is now atomic.** `run` registered boxes by
+  loading the full state, mutating, and saving under lock; concurrent launches
+  could lose updates and the reconcile pass was O(N²). Registration is now an
+  atomic, reconcile-free append, and the later rollback paths un-register
+  correctly. Verified by launching 100 boxes concurrently with zero lost
+  records.
+- **`pool status` no longer errors when no pool daemon is running** — it exits
+  successfully and reports that nothing is running, matching Docker-style UX.
+- **Restore readiness is faster and OCI-free.** A restored fork skips the OCI
+  pull (the template's cached rootfs is reused) and uses a short crash-detection
+  grace (250 ms fixed → 40 ms) tuned for the restore path.
+
 ## [2.0.7] — 2026-06-06
 
 ### Added

README.md

@@ -20,6 +20,7 @@ A3S Box is built toward production use, but it is not a full Docker, containerd,
 | OCI images | Pull, load, save, tag, inspect, history, remove, and local cache resolution are implemented. Push and cosign signing/verification paths exist and require registry access for end-to-end validation. |
 | Dockerfile build | Honest subset. `FROM`, metadata instructions, `COPY`/`ADD`, and shell-form `RUN` are implemented. `RUN` is isolated with Linux `chroot` and requires root-capable Linux; macOS fails by default unless explicitly unsafe host execution is enabled. |
 | Lifecycle and exec | `run`, `create`, `start`, `stop`, `restart`, `rm`, `wait`, foreground/detached runs, non-PTY exec, PTY exec, logs, stats, and inspect are implemented. |
+| Warm pool and snapshot-fork | A warm pool serves pre-booted sandboxes over a socket. Native snapshot-fork (Copy-on-Write microVM cloning) snapshots one booted template and restores many forks from it, each mapping the template RAM `MAP_PRIVATE`. Verified on `/dev/kvm`: ~4× faster than a cold boot per fork, 100 forks in under ~1 s (~8 ms amortized each). Requires `/dev/kvm`; opt in with `pool start --snapshot-fork` or the `KRUN_SNAPSHOT_*` / `KRUN_RESTORE_FROM` env. |
 | Networking | Default TSI networking, TCP `host:guest` publishing, user-defined bridge networks, network inspect/connect/disconnect/rm, and `/etc/hosts` peer discovery are implemented with documented platform boundaries. |
 | Compose | A useful local subset is implemented: image, command, entrypoint, env, env_file, ports, volumes, depends_on, networks, DNS, tmpfs, workdir, hostname, extra_hosts, labels, healthcheck, restart, CPU/memory, capabilities, and privileged mode. |
 | TEE | AMD SEV-SNP-oriented attestation, RA-TLS, sealing, and secret injection flows exist, plus simulation mode for development. Hardware-backed operation depends on SEV-SNP-capable hosts and libkrun support. TDX is not a productized path. |
@@ -48,7 +49,7 @@ The ignored `core_smoke` suite covers the core CLI path on a real MicroVM host:
 - TCP published ports with host loopback HTTP reachability;
 - bridge network endpoint allocation, peer `/etc/hosts`, connect/disconnect, and force removal cleanup;
 - named volumes, `cp`, `diff`, `export`, `commit`, `snapshot`, restart-policy monitor recovery, and Compose health/volume flow;
-- warm pool (`pool start`/`pool run`): pre-warmed sandboxes served over a socket, with backpressure and multi-image lazy pools; `--deferred` runs each command as the box's real main for full box semantics (real exit code + json-file console logs) with no cold boot.
+- warm pool (`pool start`/`pool run`): pre-warmed sandboxes served over a socket, with backpressure and multi-image lazy pools; `--deferred` runs each command as the box's real main for full box semantics (real exit code + json-file console logs) with no cold boot; `--snapshot-fork` fills the pool by Copy-on-Write restore from one booted template instead of cold booting each sandbox.
 
 The most recent local record in this branch: all 14 ignored `core_smoke` tests
 passed on macOS HVF with an offline Alpine OCI archive, and the ignored
@@ -96,11 +97,11 @@ a3s-box rm web
 
 ## Command surface
 
-A3S Box exposes 55 top-level commands. They are Docker-like, not Docker-identical.
+A3S Box exposes 56 top-level commands. They are Docker-like, not Docker-identical.
 
 | Category | Commands |
 | --- | --- |
-| Lifecycle | `run`, `create`, `start`, `stop`, `restart`, `rm`, `kill`, `pause`, `unpause`, `wait`, `rename` |
+| Lifecycle | `run`, `create`, `start`, `stop`, `restart`, `rm`, `kill`, `pause`, `unpause`, `wait`, `rename`, `prune` |
 | Execution | `exec`, `attach`, `top`, `shell` |
 | Images | `pull`, `push`, `build`, `images`, `rmi`, `tag`, `image-inspect`, `history`, `image-prune`, `save`, `load`, `commit` |
 | Filesystem | `cp`, `export`, `diff` |
@@ -193,7 +194,49 @@ a3s-box snapshot create app checkpoint-1
 a3s-box snapshot restore checkpoint-1 --name restored-app
 ```
 
-Snapshots are configuration/filesystem-oriented Box snapshots, not a live RAM checkpoint facility.
+The `snapshot` command produces configuration/filesystem-oriented Box snapshots, not a live RAM checkpoint. The live RAM Copy-on-Write facility is a separate, lower-level mechanism described in [Warm pool and snapshot-fork](#warm-pool-and-snapshot-fork).
+
+## Warm pool and snapshot-fork
+
+A **warm pool** keeps a set of sandboxes pre-booted and serves them over a Unix
+socket, so a request is answered by an already-running microVM instead of a cold
+boot. It supports backpressure, multi-image lazy pools, and a `--deferred` mode
+that runs each request as the box's real main process (real exit code +
+json-file console logs).
+
+```bash
+a3s-box pool start --image alpine:latest --size 8     # pre-warm 8 sandboxes
+a3s-box pool start --image alpine:latest --size 8 --snapshot-fork   # CoW fill
+a3s-box pool run alpine:latest -- echo hi             # served from the pool
+a3s-box pool status
+a3s-box pool stop
+```
+
+**Snapshot-fork** (`--snapshot-fork`, Linux `/dev/kvm` only) is native
+Copy-on-Write microVM cloning. The pool cold-boots one template sandbox,
+snapshots its file-backed guest RAM together with KVM vCPU and virtio device
+state, and then restores the rest of the pool from that snapshot. Each fork maps
+the template RAM `MAP_PRIVATE`, so it pays only for the pages it dirties. On a
+`/dev/kvm` host this is ~4× faster than a cold boot per fork, completes 100
+forks in under ~1 s (~8 ms amortized each, ~13 MB RSS per fork), and `exec`
+runs real commands over virtio-fs inside the restored guest. It is off by
+default.
+
+The same mechanism is available below the pool through environment variables:
+`KRUN_SNAPSHOT_MEM_FILE` and `KRUN_SNAPSHOT_SOCK` capture a snapshot from a
+booted template, and `KRUN_RESTORE_FROM` restores a fork from it. Per-VM
+`BoxConfig`/`InstanceSpec` fields (`snapshot_mem_file`, `snapshot_sock`,
+`restore_from`) take precedence over the env when set.
+
+## Pruning stopped boxes
+
+```bash
+a3s-box prune --force            # remove every created/stopped/dead box
+a3s-box container-prune --force  # alias
+```
+
+`prune` is the box-only counterpart to `system-prune` (which also removes images
+and networks). Running and paused boxes are never touched.
 
 ## Networking
 
@@ -365,6 +408,9 @@ matrix, and CRI smoke procedures.
 | `A3S_BOX_HOST_SMOKE_TIMEOUT_SECS` | Boot timeout override for ignored host smoke tests. |
 | `A3S_BOX_UNSAFE_HOST_RUN` | Opt into unsafe macOS host execution for Dockerfile `RUN` experiments. |
 | `A3S_BOX_BUILDCACHE_MAX_BYTES` | Cap on the total size of cached build layers at `~/.a3s/buildcache` (oldest evicted first). Default: 2 GiB. |
+| `KRUN_SNAPSHOT_MEM_FILE` | Path the booted template writes its file-backed guest RAM to when capturing a snapshot-fork template. |
+| `KRUN_SNAPSHOT_SOCK` | Control socket the template listens on for the `snapshot <path>` command (Linux `/dev/kvm` only). |
+| `KRUN_RESTORE_FROM` | Path to a snapshot the microVM restores from as a Copy-on-Write fork instead of cold booting. |
 | `RUST_LOG` | Rust tracing log level. |
 
 ## License

src/Cargo.lock

@@ -22,7 +22,7 @@ resolver = "2"
 h2 = { path = "third_party/h2" }
 
 [workspace.package]
-version = "2.0.7"
+version = "2.1.0"
 edition = "2021"
 authors = ["A3S Lab Team"]
 license = "MIT"

src/Cargo.toml

@@ -34,6 +34,7 @@ pub(crate) mod network;
 mod pause;
 mod pool;
 mod port;
+mod prune;
 mod ps;
 mod pull;
 mod push;
@@ -175,6 +176,9 @@ pub enum Command {
     Volume(volume::VolumeArgs),
     /// Show disk usage
     Df(df::DfArgs),
+    /// Remove all stopped boxes (Docker `container prune`)
+    #[command(visible_alias = "container-prune")]
+    Prune(prune::PruneArgs),
     /// Remove all unused data (stopped boxes and unused images)
     SystemPrune(system_prune::SystemPruneArgs),
     /// Show version information
@@ -337,6 +341,7 @@ pub async fn dispatch(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         Command::Network(args) => network::execute(args).await,
         Command::Volume(args) => volume::execute(args).await,
         Command::Df(args) => df::execute(args).await,
+        Command::Prune(args) => prune::execute(args).await,
         Command::SystemPrune(args) => system_prune::execute(args).await,
         Command::Version(args) => version::execute(args).await,
         Command::Info(args) => info::execute(args).await,

src/cli/src/commands/mod.rs

@@ -83,6 +83,11 @@ pub struct PoolStartArgs {
     #[arg(long)]
     pub ksm: bool,
 
+    /// Fill the pool by snapshot-fork: boot one template VM, snapshot it, then
+    /// restore every other slot (MAP_PRIVATE CoW) instead of cold-booting each.
+    #[arg(long)]
+    pub snapshot_fork: bool,
+
     /// Output as JSON
     #[arg(long)]
     pub json: bool,
@@ -282,6 +287,8 @@ struct PoolRegistry {
     deferred: bool,
     /// Mark pooled VM memory KSM-mergeable (host page dedup across same-image VMs).
     ksm: bool,
+    /// Fill the pool by snapshot-fork (one template, restore the rest).
+    snapshot_fork: bool,
 }
 
 impl PoolRegistry {
@@ -300,6 +307,7 @@ impl PoolRegistry {
             min_idle: size,
             max_size,
             idle_ttl_secs: self.ttl,
+            snapshot_fork: self.snapshot_fork,
             ..Default::default()
         };
         let box_config = BoxConfig {
@@ -379,6 +387,7 @@ async fn execute_start(args: PoolStartArgs) -> Result<(), Box<dyn std::error::Er
         ttl: args.ttl,
         deferred: args.deferred,
         ksm: args.ksm,
+        snapshot_fork: args.snapshot_fork,
     });
 
     // Pre-warm the default image, if one was given.
@@ -668,12 +677,19 @@ async fn execute_stop(_args: PoolStopArgs) -> Result<(), Box<dyn std::error::Err
 async fn execute_status(args: PoolStatusArgs) -> Result<(), Box<dyn std::error::Error>> {
     use tokio::net::UnixStream;
 
-    let mut stream = UnixStream::connect(&args.socket).await.map_err(|e| {
-        format!(
-            "Failed to connect to pool daemon at {} ({}). Is `a3s-box pool start` running?",
-            args.socket, e
-        )
-    })?;
+    // No daemon running is not an error for a status query — report "nothing" and
+    // succeed, like `ps` with no boxes. (Only a connected daemon that misbehaves is.)
+    let mut stream = match UnixStream::connect(&args.socket).await {
+        Ok(stream) => stream,
+        Err(_) => {
+            if args.json {
+                println!("[]");
+            } else {
+                println!("No pool daemon running (start one with `a3s-box pool start`).");
+            }
+            return Ok(());
+        }
+    };
 
     write_frame(&mut stream, &serde_json::to_vec(&Request::Status)?).await?;
     let resp: StatusResponse = serde_json::from_slice(&read_frame(&mut stream).await?)?;
@@ -918,6 +934,7 @@ mod tests {
             warm: vec![],
             deferred: false,
             ksm: false,
+            snapshot_fork: false,
             json: false,
         };
         let result = execute_start(args).await;
@@ -936,6 +953,7 @@ mod tests {
             warm: vec![],
             deferred: false,
             ksm: false,
+            snapshot_fork: false,
             json: false,
         };
         let result = execute_start(args).await;
@@ -954,15 +972,15 @@ mod tests {
 
     #[cfg(not(windows))]
     #[tokio::test]
-    async fn test_execute_status_no_daemon_errors() {
-        // With no daemon listening, status fails with a connect hint (not a panic).
+    async fn test_execute_status_no_daemon_succeeds_empty() {
+        // With no daemon listening, status reports "nothing running" and SUCCEEDS —
+        // a status query shouldn't fail just because no pool is up (like `ps`).
         let result = execute_status(PoolStatusArgs {
             socket: "/tmp/a3s-box-pool-does-not-exist.sock".to_string(),
             json: false,
         })
         .await;
-        assert!(result.is_err());
-        assert!(result.unwrap_err().to_string().contains("pool daemon"));
+        assert!(result.is_ok());
     }
 
     #[test]