cocoonstack
diff --git a/‎.gitignore‎
Lines changed: 4 additions & 1 deletion b/‎.gitignore‎
Lines changed: 4 additions & 1 deletion
diff --git a/‎KNOWN_ISSUES.md‎
Lines changed: 31 additions & 1 deletion b/‎KNOWN_ISSUES.md‎
Lines changed: 31 additions & 1 deletion
diff --git a/‎README.md‎
Lines changed: 62 additions & 9 deletions b/‎README.md‎
Lines changed: 62 additions & 9 deletions
diff --git a/‎cmd/core/helpers.go‎
Lines changed: 80 additions & 6 deletions b/‎cmd/core/helpers.go‎
Lines changed: 80 additions & 6 deletions
diff --git a/‎cmd/others/handler.go‎
Lines changed: 9 additions & 2 deletions b/‎cmd/others/handler.go‎
Lines changed: 9 additions & 2 deletions
diff --git a/‎cmd/root.go‎
Lines changed: 3 additions & 2 deletions b/‎cmd/root.go‎
Lines changed: 3 additions & 2 deletions
diff --git a/‎cmd/snapshot/handler.go‎
Lines changed: 4 additions & 5 deletions b/‎cmd/snapshot/handler.go‎
Lines changed: 4 additions & 5 deletions
diff --git a/‎cmd/vm/commands.go‎
Lines changed: 2 additions & 1 deletion b/‎cmd/vm/commands.go‎
Lines changed: 2 additions & 1 deletion
@@ -47,4 +47,7 @@ TODO*
 
 tmp/*
 .cache/*
-.tmp/*
+.tmp/*
+
+.claude/*
+cocoon-test
@@ -69,7 +69,9 @@ This applies to **all CNI plugins** where the upstream network provides DHCP (br
 }
 ```
 
-Cocoon detects when CNI returns no IP allocation and automatically configures the guest for DHCP — cloudimg VMs get `DHCP=ipv4` in their Netplan config, and OCI VMs get DHCP systemd-networkd units generated by the initramfs.
+Cocoon detects when CNI returns no IP allocation and automatically configures the guest for DHCP — cloudimg VMs get `DHCP=ipv4` in their Netplan config, and OCI VMs get DHCP systemd-networkd units generated by the initramfs `cocoon-network` script.
+
+Note: the OCI initramfs uses `IP=off` to prevent the initramfs from running its own DHCP client during boot. DHCP is handled entirely by systemd-networkd after switch_root. The `configure_networking` function is only called when a kernel `ip=` parameter is present (static IP from CNI).
 
 ## Windows VM requires Cloud Hypervisor v50.2
 
@@ -125,3 +127,31 @@ The Windows image's `autounattend.xml` includes defensive power-button configura
 ## Installing patched binaries for Windows
 
 See [`os-image/windows/`](os-image/windows/) for download and installation instructions.
+
+
+## Firecracker snapshot portability
+
+Firecracker snapshots store absolute host paths in the vmstate binary (Rust serde format, not patchable). This means:
+
+- **Same-host clone/restore**: works without restrictions
+- **Cross-host export/import**: requires the target host to use **identical `root_dir` and `run_dir`** (default: `/var/lib/cocoon` and `/var/lib/cocoon/run`) and have the **same OCI image pulled**
+- **CPU/memory overrides**: not supported on clone/restore — Firecracker cannot change machine config after snapshot/load; `--cpu` and `--memory` flags are rejected if they differ from the snapshot values
+- **Drive path redirect**: Cocoon uses a temporary symlink to redirect the source COW path to the clone's COW during `snapshot/load`. This requires a COW flock to serialize with concurrent operations
+
+This is a fundamental Firecracker design limitation. Cloud Hypervisor snapshots do not have this restriction because CH stores device config in a patchable JSON format (`config.json`).
+
+**Upstream fix in progress**: Firecracker [PR #5774](https://github.com/firecracker-microvm/firecracker/pull/5774) adds `drive_overrides` to the `PUT /snapshot/load` API, which would eliminate the symlink redirect and make FC snapshots natively portable. Track this PR for future simplification.
+
+## Firecracker virtio-blk serial numbers
+
+Firecracker does not support virtio-blk serial numbers. Cocoon's OCI init script (`overlay.sh`) uses device paths (`/dev/vdX`) instead of serial names to identify disks when booting under Firecracker. OCI images built from `os-image/ubuntu/overlay.sh` (v0.3+) support both formats automatically. Older images must be rebuilt to work with `--fc`.
+
+## Firecracker clone guest MAC address
+
+Firecracker does not support overriding the guest MAC address during snapshot/load. Cloned FC VMs retain the source VM's guest MAC (baked into the vmstate binary). In Cocoon's TC redirect architecture, each VM runs in an isolated network namespace, so MAC identity is not visible to other VMs or the host bridge — **no MAC conflict occurs in practice**.
+
+On CNI plugins with strict per-veth MAC enforcement (Cilium eBPF, Calico eBPF), the guest MAC vs veth MAC mismatch could theoretically cause packet drops. This has not been observed in testing with the standard bridge CNI.
+
+**Upstream status**: FC's `NetworkOverride` struct only has `iface_id` and `host_dev_name` — no `guest_mac` field. Adding it would follow the existing `VsockOverride` pattern. No issue or PR exists yet.
+
+**Workaround**: If MAC matching is required, run `ip link set dev ethX address <new-mac>` inside the guest after clone (the post-clone hints print the expected MAC values).
@@ -1,6 +1,6 @@
 # Cocoon
 
-Lightweight MicroVM engine built on [Cloud Hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor).
+Lightweight MicroVM engine with dual hypervisor backends: [Cloud Hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) (default) and [Firecracker](https://github.com/firecracker-microvm/firecracker).
 
 ## Features
 
@@ -24,7 +24,8 @@ Lightweight MicroVM engine built on [Cloud Hypervisor](https://github.com/cloud-
 - **Docker-like CLI** — `create`, `run`, `start`, `stop`, `list`, `inspect`, `console`, `rm`, `debug`, `clone`, `status`
 - **Structured logging** — configurable log level (`--log-level`), log rotation (max size / age / backups)
 - **Debug command** — `cocoon vm debug` generates a copy-pasteable `cloud-hypervisor` command for manual debugging
-- **Zero-daemon architecture** — one Cloud Hypervisor process per VM, no long-running daemon
+- **Firecracker backend** — `--fc` flag selects Firecracker for OCI images: ~125ms boot, <5 MiB overhead, minimal attack surface (no UEFI, no qcow2, no Windows)
+- **Zero-daemon architecture** — one hypervisor process per VM, no long-running daemon
 - **Garbage collection** — modular lock-safe GC with cross-module snapshot resolution; protects blobs referenced by running VMs and snapshots
 - **Doctor script** — pre-flight environment check and one-command dependency installation
 
@@ -33,8 +34,9 @@ Lightweight MicroVM engine built on [Cloud Hypervisor](https://github.com/cloud-
 - Linux with KVM (x86_64 or aarch64)
 - Root access (sudo)
 - [Cloud Hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) v51.0+ (for Windows VMs, use our [CH fork](https://github.com/cocoonstack/cloud-hypervisor/tree/dev) and [firmware fork](https://github.com/cocoonstack/rust-hypervisor-firmware/tree/dev) for full compatibility — see [KNOWN_ISSUES.md](KNOWN_ISSUES.md))
+- [Firecracker](https://github.com/firecracker-microvm/firecracker) v1.12+ (optional, for `--fc` backend)
 - `qemu-img` (from qemu-utils, for cloud images)
-- UEFI firmware (`CLOUDHV.fd`, for cloud images)
+- UEFI firmware (`CLOUDHV.fd`, for cloud images, not needed with `--fc`)
 - CNI plugins (`bridge`, `host-local`, `loopback`)
 - Go 1.25+ (build only)
 
@@ -85,6 +87,7 @@ cocoon-check --upgrade
 
 The `--upgrade` flag downloads and installs:
 - Cloud Hypervisor + ch-remote (static binaries)
+- Firecracker (static binary)
 - CLOUDHV.fd firmware (rust-hypervisor-firmware)
 - CNI plugins (bridge, host-local, loopback, etc.)
 
@@ -136,7 +139,7 @@ cocoon
 │   ├── rm [flags] VM [VM...]      Delete VM(s) (--force to stop first)
 │   ├── restore [flags] VM SNAP   Restore a running VM to a snapshot
 │   ├── status [VM...]             Watch VM status in real time
-│   └── debug [flags] IMAGE        Generate CH launch command (dry run)
+│   └── debug [flags] IMAGE        Generate hypervisor launch command (dry run)
 ├── snapshot
 │   ├── save [flags] VM            Create a snapshot from a running VM
 │   ├── list (alias: ls)           List all snapshots
@@ -169,6 +172,7 @@ Applies to `cocoon vm create`, `cocoon vm run`, and `cocoon vm debug`:
 
 | Flag        | Default          | Description                                   |
 | ----------- | ---------------- | --------------------------------------------- |
+| `--fc`      | `false`          | Use Firecracker backend (OCI images only)      |
 | `--name`    | `cocoon-<image>` | VM name                                       |
 | `--cpu`     | `2`              | Boot CPUs                                     |
 | `--memory`  | `1G`             | Memory size (e.g., 512M, 2G)                  |
@@ -356,22 +360,71 @@ cocoon image import win11-25h2 windows-11-25h2.qcow2
 
 For more details, see the [Cloud Hypervisor Windows documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/windows.md).
 
+## Firecracker Backend
+
+Cocoon supports [Firecracker](https://github.com/firecracker-microvm/firecracker) as an alternative hypervisor for workloads that prioritize boot speed and resource density.
+
+```bash
+# Run with Firecracker (--fc only needed for create/run/debug)
+cocoon vm run --fc --name fast-vm ghcr.io/cocoonstack/cocoon/ubuntu:24.04
+
+# Other commands auto-detect the backend — no --fc needed
+cocoon vm list              # shows both CH and FC VMs
+cocoon vm console fast-vm
+cocoon vm stop fast-vm
+
+# Clone infers backend from the snapshot
+cocoon snapshot save fast-vm --name my-snap
+cocoon vm clone my-snap --name clone-vm
+```
+
+### Feature Comparison
+
+| Feature | Cloud Hypervisor | Firecracker |
+|---------|:---:|:---:|
+| OCI images (direct boot) | Y | Y |
+| Cloud images (UEFI boot) | Y | N |
+| Windows guests | Y | N |
+| Snapshot / Clone / Restore | Y | Y |
+| CPU/memory override on clone/restore | Y | N |
+| Multi-queue networking | Y | N |
+| Memory balloon | Y | Y |
+| qcow2 storage | Y | N |
+| Interactive console | Y | Y |
+| HugePages | Y | Y |
+| Boot time | ~200-500ms | ~125ms |
+| Memory overhead | ~10-20 MiB/VM | <5 MiB/VM |
+
+### Limitations
+
+- **OCI images only**: `--fc` is mutually exclusive with `--windows` and rejects cloudimg (UEFI boot) images
+- **Raw disks only**: Firecracker uses raw virtio-blk without serial support; disks are referenced by device path (`/dev/vdX`)
+- **Single-queue networking**: `NetworkConfig.NumQueues` is ignored
+- **No CPU/memory override on clone/restore**: Firecracker cannot change machine config after snapshot/load
+- **Snapshot portability requires same directory layout**: FC snapshots store absolute paths in the vmstate binary (not patchable); cross-host export/import requires the target host to use the same `root_dir`/`run_dir` and have the same OCI image pulled
+- **Console via PTY relay**: a background relay process bridges FC's serial (stdin/stdout) to `console.sock`
+
+### OCI Image Compatibility
+
+OCI images must include a `resolve_disk()` init script that supports device paths (e.g., `/dev/vda`) in addition to virtio serial names. Images built from `os-image/ubuntu/overlay.sh` (v0.3+) support both formats automatically.
+
 ## VM Lifecycle
 
 | State      | Description                                              |
 | ---------- | -------------------------------------------------------- |
 | `creating` | DB placeholder written, disks being prepared             |
-| `created`  | Registered, cloud-hypervisor process not yet started     |
-| `running`  | Cloud-hypervisor process alive, guest is up              |
-| `stopped`  | Cloud-hypervisor process exited cleanly                  |
+| `created`  | Registered, hypervisor process not yet started           |
+| `running`  | Hypervisor process alive, guest is up                    |
+| `stopped`  | Hypervisor process exited cleanly                        |
 | `error`    | Start or stop failed                                     |
 
 ### Shutdown Behavior
 
 - **UEFI VMs (cloudimg)**: ACPI power-button → poll for graceful exit → timeout (default 30s, configurable via `stop_timeout_seconds` in config or `--timeout` flag) → SIGTERM → 5s → SIGKILL
 - **Windows VMs**: ACPI power-button works with our [firmware fork](https://github.com/cocoonstack/rust-hypervisor-firmware/tree/dev) (~13.5s shutdown); with upstream firmware, use `ssh shutdown /s /t 0` before stopping, or `--force` to skip the ACPI timeout (see [KNOWN_ISSUES.md](KNOWN_ISSUES.md))
-- **Direct-boot VMs (OCI)**: `vm.shutdown` API → SIGTERM → 5s → SIGKILL (no ACPI support)
-- **Force stop** (`--force`): skip ACPI, immediate `vm.shutdown` → SIGTERM → SIGKILL
+- **Direct-boot VMs (CH, OCI)**: `vm.shutdown` API → SIGTERM → 5s → SIGKILL (no ACPI support)
+- **Firecracker VMs**: `SendCtrlAltDel` → SIGTERM → 5s → SIGKILL
+- **Force stop** (`--force`): skip ACPI, immediate SIGTERM → SIGKILL
 - PID ownership is verified before sending signals to prevent killing unrelated processes
 
 ### Stop Flags
 
@@ -15,6 +15,7 @@ import (
 	"github.com/cocoonstack/cocoon/config"
 	"github.com/cocoonstack/cocoon/hypervisor"
 	"github.com/cocoonstack/cocoon/hypervisor/cloudhypervisor"
+	"github.com/cocoonstack/cocoon/hypervisor/firecracker"
 	imagebackend "github.com/cocoonstack/cocoon/images"
 	"github.com/cocoonstack/cocoon/images/cloudimg"
 	"github.com/cocoonstack/cocoon/images/oci"
@@ -26,6 +27,12 @@ import (
 	"github.com/cocoonstack/cocoon/utils"
 )
 
+// hypervisorConstructors maps backend type to its constructor.
+var hypervisorConstructors = map[config.HypervisorType]func(*config.Config) (hypervisor.Hypervisor, error){
+	config.HypervisorCH:          func(c *config.Config) (hypervisor.Hypervisor, error) { return cloudhypervisor.New(c) },
+	config.HypervisorFirecracker: func(c *config.Config) (hypervisor.Hypervisor, error) { return firecracker.New(c) },
+}
+
 // BaseHandler provides shared config access for all command handlers.
 type BaseHandler struct {
 	ConfProvider func() *config.Config
@@ -71,11 +78,11 @@ func InitBackends(ctx context.Context, conf *config.Config) ([]imagebackend.Imag
 	if err != nil {
 		return nil, nil, err
 	}
-	ch, err := cloudhypervisor.New(conf)
+	hyper, err := InitHypervisor(conf)
 	if err != nil {
-		return nil, nil, fmt.Errorf("init hypervisor: %w", err)
+		return nil, nil, err
 	}
-	return backends, ch, nil
+	return backends, hyper, nil
 }
 
 // InitImageBackends initializes only image backends (no hypervisor needed).
@@ -100,13 +107,80 @@ func InitImageBackendsForPull(ctx context.Context, conf *config.Config) (*oci.OC
 	return ociStore, cloudimgStore, nil
 }
 
-// InitHypervisor initializes only the hypervisor.
+// InitHypervisor initializes the selected hypervisor backend.
 func InitHypervisor(conf *config.Config) (hypervisor.Hypervisor, error) {
-	ch, err := cloudhypervisor.New(conf)
+	ctor, ok := hypervisorConstructors[conf.Hypervisor()]
+	if !ok {
+		return nil, fmt.Errorf("unknown hypervisor type: %s", conf.Hypervisor())
+	}
+	h, err := ctor(conf)
 	if err != nil {
 		return nil, fmt.Errorf("init hypervisor: %w", err)
 	}
-	return ch, nil
+	return h, nil
+}
+
+// InitAllHypervisors initializes all registered backends for GC.
+// Returns error if any backend fails — GC must not proceed without
+// full blob pinning or it risks deleting referenced layers.
+func InitAllHypervisors(conf *config.Config) ([]hypervisor.Hypervisor, error) {
+	result := make([]hypervisor.Hypervisor, 0, len(hypervisorConstructors))
+	for typ, ctor := range hypervisorConstructors {
+		h, err := ctor(conf)
+		if err != nil {
+			return nil, fmt.Errorf("init %s for GC: %w", typ, err)
+		}
+		result = append(result, h)
+	}
+	return result, nil
+}
+
+// FindHypervisor returns the backend that owns the given VM ref.
+// Tries all registered backends; returns ErrNotFound if no backend has it.
+func FindHypervisor(ctx context.Context, conf *config.Config, ref string) (hypervisor.Hypervisor, error) {
+	hypers, err := InitAllHypervisors(conf)
+	if err != nil {
+		return nil, err
+	}
+	for _, h := range hypers {
+		if _, resolveErr := h.Inspect(ctx, ref); resolveErr == nil {
+			return h, nil
+		}
+	}
+	return nil, fmt.Errorf("VM %q: %w", ref, hypervisor.ErrNotFound)
+}
+
+// ListAllVMs returns VMs from all registered backends, merged.
+func ListAllVMs(ctx context.Context, hypers []hypervisor.Hypervisor) ([]*types.VM, error) {
+	var all []*types.VM
+	for _, h := range hypers {
+		vms, listErr := h.List(ctx)
+		if listErr != nil {
+			continue
+		}
+		all = append(all, vms...)
+	}
+	return all, nil
+}
+
+// RouteRefs groups VM refs by their owning backend.
+// Returns a map from hypervisor to refs it owns, or error if any ref is unresolvable.
+func RouteRefs(ctx context.Context, hypers []hypervisor.Hypervisor, refs []string) (map[hypervisor.Hypervisor][]string, error) {
+	result := map[hypervisor.Hypervisor][]string{}
+	for _, ref := range refs {
+		found := false
+		for _, h := range hypers {
+			if _, resolveErr := h.Inspect(ctx, ref); resolveErr == nil {
+				result[h] = append(result[h], ref)
+				found = true
+				break
+			}
+		}
+		if !found {
+			return nil, fmt.Errorf("VM %q: %w", ref, hypervisor.ErrNotFound)
+		}
+	}
+	return result, nil
 }
 
 // InitNetwork creates the CNI network provider.
 
@@ -20,7 +20,7 @@ func (h Handler) GC(cmd *cobra.Command, _ []string) error {
 	if err != nil {
 		return err
 	}
-	backends, hyper, err := cmdcore.InitBackends(ctx, conf)
+	backends, err := cmdcore.InitImageBackends(ctx, conf)
 	if err != nil {
 		return err
 	}
@@ -37,7 +37,14 @@ func (h Handler) GC(cmd *cobra.Command, _ []string) error {
 	for _, b := range backends {
 		b.RegisterGC(o)
 	}
-	hyper.RegisterGC(o)
+	// Register ALL hypervisor backends so GC protects blobs from both CH and FC VMs.
+	hypers, hyperErr := cmdcore.InitAllHypervisors(conf)
+	if hyperErr != nil {
+		return hyperErr
+	}
+	for _, hyper := range hypers {
+		hyper.RegisterGC(o)
+	}
 	netProvider.RegisterGC(o)
 	snapBackend.RegisterGC(o)
 	if err := o.Run(ctx); err != nil {
 
@@ -59,6 +59,7 @@ var (
 		viper.SetDefault("run_dir", "/var/lib/cocoon/run")
 		viper.SetDefault("log_dir", "/var/log/cocoon")
 		viper.SetDefault("ch_binary", "cloud-hypervisor")
+		viper.SetDefault("fc_binary", "firecracker")
 		viper.SetDefault("cni_conf_dir", "/etc/cni/net.d")
 		viper.SetDefault("cni_bin_dir", "/opt/cni/bin")
 		viper.SetDefault("dns", "8.8.8.8,1.1.1.1")
@@ -83,8 +84,8 @@ var (
 )
 
 // Execute is the main entry point called from main.go.
-func Execute() error {
-	ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+func Execute(ctx context.Context) error {
+	ctx, cancel := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
 	defer cancel()
 	return rootCmd.ExecuteContext(ctx)
 }
 
@@ -30,16 +30,15 @@ func (h Handler) Save(cmd *cobra.Command, args []string) error {
 	}
 	logger := log.WithFunc("cmd.snapshot.save")
 
-	hyper, err := cmdcore.InitHypervisor(conf)
+	vmRef := args[0]
+	hyper, err := cmdcore.FindHypervisor(ctx, conf, vmRef)
 	if err != nil {
-		return err
+		return fmt.Errorf("find VM %s: %w", vmRef, err)
 	}
 	snapBackend, err := cmdcore.InitSnapshot(conf)
 	if err != nil {
 		return err
 	}
-
-	vmRef := args[0]
 	name, _ := cmd.Flags().GetString("name")
 	description, _ := cmd.Flags().GetString("description")
 
@@ -95,7 +94,7 @@ func (h Handler) List(cmd *cobra.Command, _ []string) error {
 	vmRef, _ := cmd.Flags().GetString("vm")
 	var filterIDs map[string]struct{}
 	if vmRef != "" {
-		hyper, hyperErr := cmdcore.InitHypervisor(conf)
+		hyper, hyperErr := cmdcore.FindHypervisor(ctx, conf, vmRef)
 		if hyperErr != nil {
 			return hyperErr
 		}
 
@@ -112,7 +112,7 @@ func Command(h Actions) *cobra.Command {
 
 	debugCmd := &cobra.Command{
 		Use:   "debug [flags] IMAGE",
-		Short: "Generate cloud-hypervisor launch command (dry run)",
+		Short: "Generate hypervisor launch command (dry run)",
 		Args:  cobra.ExactArgs(1),
 		RunE:  h.Debug,
 	}
@@ -149,6 +149,7 @@ func Command(h Actions) *cobra.Command {
 }
 
 func addVMFlags(cmd *cobra.Command) {
+	cmd.Flags().Bool("fc", false, "use Firecracker backend instead of Cloud Hypervisor (OCI images only)")
 	cmd.Flags().String("name", "", "VM name")
 	cmd.Flags().Int("cpu", 2, "boot CPUs")                //nolint:mnd
 	cmd.Flags().String("memory", "1G", "memory size")     //nolint:mnd
Original file line number	Diff line number	Diff line change
`@@ -112,7 +112,7 @@ func Command(h Actions) *cobra.Command {`
`112`	`112`
`113`	`113`	`debugCmd := &cobra.Command{`
`114`	`114`	`Use: "debug [flags] IMAGE",`
`115`		`- Short: "Generate cloud-hypervisor launch command (dry run)",`
	`115`	`+ Short: "Generate hypervisor launch command (dry run)",`
`116`	`116`	`Args: cobra.ExactArgs(1),`
`117`	`117`	`RunE: h.Debug,`
`118`	`118`	`}`
`@@ -149,6 +149,7 @@ func Command(h Actions) *cobra.Command {`
`149`	`149`	`}`
`150`	`150`
`151`	`151`	`func addVMFlags(cmd *cobra.Command) {`
	`152`	`+ cmd.Flags().Bool("fc", false, "use Firecracker backend instead of Cloud Hypervisor (OCI images only)")`
`152`	`153`	`cmd.Flags().String("name", "", "VM name")`
`153`	`154`	`cmd.Flags().Int("cpu", 2, "boot CPUs") //nolint:mnd`
`154`	`155`	`cmd.Flags().String("memory", "1G", "memory size") //nolint:mnd`