Skip to content

feat: nested sandbox support (sandbox-in-sandbox)#424

Open
SauravMishraaa wants to merge 2 commits into
CelestoAI:mainfrom
SauravMishraaa:feature/smolvm-in-smolvm
Open

feat: nested sandbox support (sandbox-in-sandbox)#424
SauravMishraaa wants to merge 2 commits into
CelestoAI:mainfrom
SauravMishraaa:feature/smolvm-in-smolvm

Conversation

@SauravMishraaa

@SauravMishraaa SauravMishraaa commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Enables running smolvm sandbox create inside a SmolVM sandbox (nested VM / sandbox-in-sandbox support).

Previously, running smolvm inside a guest failed because:

  • The auto-selected backend was Firecracker, which is not installed in guest images.
  • QEMU attempted to use KVM, which is unavailable inside a VM.
  • vsock was selected even though /dev/vhost-vsock does not exist in guests.
  • Required packages such as pip and qemu-system-* were not included in the base image.

Changes

backends.py

  • Add _running_inside_smolvm_guest() to detect nested execution via the SMOLVM_NESTED environment variable or the /run/smolvm-guest marker file.
  • Automatically select the QEMU backend instead of Firecracker when running inside a guest.

qemu_args.py

  • Force the TCG accelerator when nested (KVM unavailable).
  • Respect the SMOLVM_QEMU_ACCEL environment variable so acceleration can still be overridden explicitly.

select.py

  • Disable vsock when nested (no /dev/vhost-vsock available).
  • Fall back to SSH for the control channel.

facade.py

  • Cap the default memory at 512 MiB when nested so the inner VM fits inside the outer sandbox without requiring --memory.

cloud_init.py

  • Write nested environment variables to /etc/environment.
  • Add pip.conf.
  • Create /run/smolvm-guest during bootcmd.
  • Install the appropriate architecture-specific qemu-system-* package during runcmd.

Dockerfile.base-rootfs

Bake the following packages into the base image:

  • python3-pip
  • python3-venv
  • qemu-utils
  • Architecture-specific qemu-system-*
  • nftables

preset-init.sh

  • Create the /run/smolvm-guest marker at every boot.
  • Generate pip.conf.
  • Write nested environment variables to /etc/environment during PID 1 startup.

User Flow

After rebuilding the base image and publishing a new release:

smolvm sandbox create --name my-computer
smolvm sandbox ssh my-computer

pip install smolvm

smolvm sandbox create --name computer-inside-computer

Notes

  • Nested virtualization uses TCG (software emulation) because KVM passthrough is unavailable inside a guest.
  • The inner VM is fully functional but slower than native virtualization.
  • The inner sandbox defaults to 512 MiB of memory. This can be overridden with --memory if needed.
  • A base image rebuild and a new PyPI release are required for the complete feature to take effect. The Python logic changes are complete and tested.

Related Issues

Closes #328

Testing

Automated

uv run --with pytest python -m pytest \
    tests/test_qemu_args.py \
    tests/test_comm_select.py \
    tests/test_backends.py \
    tests/test_cli.py
  • No regressions
uv run --with ruff ruff check .
  • Clean
uv run --with ruff ruff format --check .

Manual Validation

Validated end-to-end on Apple Silicon (aarch64):

  1. Installed the modified wheel inside a sandbox.

  2. Created the nested marker:

    touch /run/smolvm-guest
  3. Ran:

    smolvm sandbox create --name computer-inside-computer
  4. Verified that the nested VM booted successfully without requiring any additional flags.

Checklist

  • Scope is focused and limited to this change.
  • Tests added/updated for behavior changes.
  • Documentation updated for user-visible changes.
  • No secrets or sensitive data included.

Summary by CodeRabbit

  • New Features

    • Added a new disk mode option for VM start commands, giving users more control over disk lifecycle behavior.
    • Boot timeout can now be set from an environment variable instead of using a fixed default.
  • Bug Fixes

    • Improved backend and communication selection when running inside a nested SmolVM guest.
    • Added nested-guest setup so VM sessions use the right defaults for memory, networking, and package installation.
    • Updated QEMU handling to choose compatible acceleration and system packages automatically.

@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR adds nested-guest boot markers, guest detection, QEMU selection updates, and CLI defaults. It also expands image bootstrapping with pip, networking, and QEMU packages.

Changes

Nested guest startup and runtime selection

Layer / File(s) Summary
Image packages
scripts/ci/Dockerfile.base-rootfs, src/smolvm/images/builder.py
Base images add Python tooling, qemu utilities, nftables, and Alpine SSH adds QEMU image and system packages.
Boot config
scripts/ci/preset-init.sh, src/smolvm/images/cloud_init.py
Preset init creates the nested guest marker, pip configuration, and environment defaults, while cloud-init adds packages, profile files, boot commands, and QEMU setup.
Guest detection
src/smolvm/runtime/backends.py, src/smolvm/comm/select.py, src/smolvm/facade.py
Backend auto-selection, vsock support, and auto-config memory limits now read the nested guest marker or SMOLVM_NESTED to change runtime behavior.
QEMU accel selection
src/smolvm/runtime/qemu_args.py, tests/test_qemu_args.py
QEMU argument construction centralizes accelerator and CPU selection across virt, microvm, and q35, and the tests reset the new env and module state.
CLI options
src/smolvm/cli/commands/options.py, src/smolvm/cli/commands/app.py
Boot timeout now defaults from SMOLVM_BOOT_TIMEOUT, and preset start adds a --disk-mode choice option.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

QEMU, smolvm-core, CI

Poem

A guest within a guest, still sharp and clean,
The marker goes down where it’s never been seen.
QEMU shifts gears, quiet in the dark,
Boots take their orders, precise and stark.
Tiny kingdoms stack, and the code stays true.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 58.82% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and directly reflects the nested sandbox support feature.
Linked Issues check ✅ Passed The changes implement nested sandbox creation inside SmolVM as requested in #328.
Out of Scope Changes check ✅ Passed The touched files all support nested sandbox behavior, with no clear unrelated changes.
Description check ✅ Passed The PR description matches the required template and includes summary, issue link, testing, and checklist details.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SauravMishraaa SauravMishraaa force-pushed the feature/smolvm-in-smolvm branch 2 times, most recently from 286f970 to 330955f Compare June 26, 2026 17:36
@SauravMishraaa SauravMishraaa force-pushed the feature/smolvm-in-smolvm branch from 330955f to f512e3b Compare June 26, 2026 17:44

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/ci/preset-init.sh`:
- Around line 85-90: The nested SmolVM guest marker is only being created in
/run, so it disappears after reboot and breaks the auto-detection used by
src/smolvm/runtime/backends.py. Update the first-boot init logic in the
preset-init.sh flow to either store the marker in a persistent location or
recreate /run/smolvm-guest on every boot, and make sure the runtime detection
path continues to find the marker reliably.

In `@src/smolvm/cli/commands/app.py`:
- Around line 921-927: The new `--disk-mode` option in `start` is not wired
through and `VMConfig` cannot accept the raw "auto" value. Update the `start`
callback signature to include `disk_mode`, pass it through to `_run_start`, and
convert "auto" to a supported `VMConfig` value before constructing the config.
Use the `start`, `_run_start`, and `VMConfig` symbols to locate and fix the flow
so the command starts without rejecting the option.

In `@src/smolvm/cli/commands/options.py`:
- Around line 113-120: The _boot_timeout_default helper accepts non-real or
non-positive values because it only catches ValueError, so update this function
to validate the parsed timeout from SMOLVM_BOOT_TIMEOUT before returning it. In
_boot_timeout_default, after converting the string to float, reject nan, inf,
zero, and negative values by checking finiteness and positivity, and fall back
to 30.0 for invalid inputs.

In `@src/smolvm/images/builder.py`:
- Around line 562-564: The Alpine package list in build_alpine_ssh() contains a
typo in the qemu system package name, causing apk add to fail. Update the
package entry alongside the other install items in builder.py so the qemu system
package is spelled correctly, keeping the existing install flow unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f2d8436d-2aa6-4953-b1cd-3919ea40e8a5

📥 Commits

Reviewing files that changed from the base of the PR and between dc8388a and 330955f.

📒 Files selected for processing (10)
  • scripts/ci/Dockerfile.base-rootfs
  • scripts/ci/preset-init.sh
  • src/smolvm/cli/commands/app.py
  • src/smolvm/cli/commands/options.py
  • src/smolvm/comm/select.py
  • src/smolvm/images/builder.py
  • src/smolvm/images/cloud_init.py
  • src/smolvm/runtime/backends.py
  • src/smolvm/runtime/qemu_args.py
  • tests/test_qemu_args.py

Comment thread scripts/ci/preset-init.sh
Comment thread src/smolvm/cli/commands/app.py Outdated
Comment thread src/smolvm/cli/commands/options.py
Comment thread src/smolvm/images/builder.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
src/smolvm/cli/commands/app.py (1)

921-927: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Resolve auto before it leaves the CLI.

Line 962 still sends raw "auto" downstream, but src/smolvm/types.py only accepts "isolated" or "shared" for disk_mode. So the default path can still go bang unless you map "auto" to a supported value before _run_start / VMConfig sees it.

Also applies to: 962-962

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/smolvm/cli/commands/app.py` around lines 921 - 927, The CLI disk mode
option currently allows "auto", but downstream code only accepts "isolated" or
"shared", so the default path must be normalized before it reaches VMConfig or
_run_start. Update the app command flow in the CLI handler around the disk-mode
option so that any "auto" value is resolved to a supported value before invoking
VMConfig, while keeping the existing "isolated" and "shared" behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@src/smolvm/cli/commands/app.py`:
- Around line 921-927: The CLI disk mode option currently allows "auto", but
downstream code only accepts "isolated" or "shared", so the default path must be
normalized before it reaches VMConfig or _run_start. Update the app command flow
in the CLI handler around the disk-mode option so that any "auto" value is
resolved to a supported value before invoking VMConfig, while keeping the
existing "isolated" and "shared" behavior unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 59a5dd8b-7249-4ef5-b732-9f4c8f7e6d7b

📥 Commits

Reviewing files that changed from the base of the PR and between 330955f and f512e3b.

📒 Files selected for processing (10)
  • scripts/ci/Dockerfile.base-rootfs
  • scripts/ci/preset-init.sh
  • src/smolvm/cli/commands/app.py
  • src/smolvm/cli/commands/options.py
  • src/smolvm/comm/select.py
  • src/smolvm/images/builder.py
  • src/smolvm/images/cloud_init.py
  • src/smolvm/runtime/backends.py
  • src/smolvm/runtime/qemu_args.py
  • tests/test_qemu_args.py
🚧 Files skipped from review as they are similar to previous changes (9)
  • scripts/ci/preset-init.sh
  • src/smolvm/comm/select.py
  • scripts/ci/Dockerfile.base-rootfs
  • src/smolvm/images/builder.py
  • src/smolvm/images/cloud_init.py
  • src/smolvm/runtime/backends.py
  • tests/test_qemu_args.py
  • src/smolvm/cli/commands/options.py
  • src/smolvm/runtime/qemu_args.py

@aniketmaurya

aniketmaurya commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

this PR is different from expected flow #328

Goal

As a user, I should be able to spin up a SmolVM sandbox, SSH into it, install SmolVM inside a sandbox, and run it like that, similar to how you run Docker in Docker, so sandbox inside a sandbox.

Expected flow

smolvm sandbox create --name my-computer
smolvm ssh my-computer


# from inside the my-computer sandbox
>> pip install smolvm
>> smolvm sandbox create --name computer-inside-computer

@SauravMishraaa SauravMishraaa force-pushed the feature/smolvm-in-smolvm branch from f512e3b to 8e91fba Compare June 27, 2026 06:54
@coderabbitai coderabbitai Bot added the CI label Jun 27, 2026
@SauravMishraaa SauravMishraaa force-pushed the feature/smolvm-in-smolvm branch from 8e91fba to fd8e65d Compare June 27, 2026 07:31
@coderabbitai coderabbitai Bot added Firecracker and removed CI labels Jun 27, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/ci/preset-init.sh`:
- Around line 103-110: The /etc/environment presence check is too broad because
the grep only looks for a SMOLVM_NESTED substring, so any comment or partial
match can skip the whole config block. Tighten the check in the preset-init.sh
environment setup block by anchoring the match to the actual key format
(SMOLVM_NESTED=) or, better, upsert each SMOLVM_* and PIP_BREAK_SYSTEM_PACKAGES
entry individually. Keep the existing cat/appending logic only for missing keys
so the script repairs drift instead of assuming the whole block already exists.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: dc6642f6-f7b6-47dd-8f51-bcd4251a36c2

📥 Commits

Reviewing files that changed from the base of the PR and between 8e91fba and fd8e65d.

📒 Files selected for processing (11)
  • scripts/ci/Dockerfile.base-rootfs
  • scripts/ci/preset-init.sh
  • src/smolvm/cli/commands/app.py
  • src/smolvm/cli/commands/options.py
  • src/smolvm/comm/select.py
  • src/smolvm/facade.py
  • src/smolvm/images/builder.py
  • src/smolvm/images/cloud_init.py
  • src/smolvm/runtime/backends.py
  • src/smolvm/runtime/qemu_args.py
  • tests/test_qemu_args.py
🚧 Files skipped from review as they are similar to previous changes (10)
  • tests/test_qemu_args.py
  • src/smolvm/cli/commands/app.py
  • src/smolvm/facade.py
  • src/smolvm/comm/select.py
  • src/smolvm/images/cloud_init.py
  • src/smolvm/cli/commands/options.py
  • scripts/ci/Dockerfile.base-rootfs
  • src/smolvm/runtime/backends.py
  • src/smolvm/images/builder.py
  • src/smolvm/runtime/qemu_args.py

Comment thread scripts/ci/preset-init.sh Outdated
@SauravMishraaa SauravMishraaa force-pushed the feature/smolvm-in-smolvm branch 4 times, most recently from b1ee9f0 to 1ee3f01 Compare June 27, 2026 09:02
@SauravMishraaa

Copy link
Copy Markdown
Contributor Author

@aniketmaurya Can you review this now?

Comment thread src/smolvm/images/builder.py Outdated
Comment thread src/smolvm/runtime/qemu_args.py
Comment thread scripts/ci/Dockerfile.base-rootfs Outdated
Allow running
╭───── VM Created ──────╮
│ Created VM 'sbx-ohm'. │
╰───────────────────────╯
             VM Details
┌─────────┬─────────────────────────┐
│ Name    │ sbx-ohm                 │
│ Status  │ running                 │
│ OS      │ ubuntu                  │
│ Started │ 2026-06-26 17:17:51 UTC │
└─────────┴─────────────────────────┘
Next: smolvm sandbox shell sbx-ohm
SSH:  smolvm sandbox ssh sbx-ohm
Info: smolvm sandbox info sbx-ohm inside a SmolVM sandbox guest.

Changes:
- backends.py: add _running_inside_smolvm_guest() that detects nested
  context via SMOLVM_NESTED env var or /run/smolvm-guest marker file;
  auto-selects QEMU backend when nested instead of Firecracker
- qemu_args.py: force TCG accelerator when running inside a guest
  (no KVM available); reads SMOLVM_QEMU_ACCEL env var first so it
  can still be overridden explicitly
- comm/select.py: disable vsock when nested (no /dev/vhost-vsock in
  guest); falls back to SSH for the control channel
- cloud_init.py: write nested env vars to /etc/environment (not just
  profile.d), add /etc/pip.conf, write /run/smolvm-guest in bootcmd,
  install arch-aware qemu-system package in runcmd
- Dockerfile.base-rootfs: bake python3-pip, python3-venv, qemu-utils,
  qemu-system-arm/x86 (arch-detected), nftables into the base image
- preset-init.sh: write /run/smolvm-guest marker and /etc/pip.conf
  at PID 1 startup on every boot
- tests/test_qemu_args.py: patch _KVM_DEV and _SMOLVM_GUEST_MARKER
  in autouse fixture so Linux host simulation works from macOS runner

After image rebuild and new release, the user flow becomes:
  smolvm sandbox create --name my-computer --memory 2048
  smolvm sandbox ssh my-computer
  pip install smolvm
  smolvm sandbox create --name inner --os ubuntu --memory 512 --boot-timeout 300
@SauravMishraaa SauravMishraaa force-pushed the feature/smolvm-in-smolvm branch from 1ee3f01 to b676f55 Compare June 30, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Run SmolVM inside a SmolVM sandbox

2 participants