Skip to content

feat: use hyperlight-host 0.16.0 from crates.io - #99

Merged
danbugs merged 5 commits into
mainfrom
experiment/use-hyperlight-main
Jun 27, 2026
Merged

feat: use hyperlight-host 0.16.0 from crates.io#99
danbugs merged 5 commits into
mainfrom
experiment/use-hyperlight-main

Conversation

@danbugs

@danbugs danbugs commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Switch hyperlight-host dependency from a git pin on danbugs/hyperlight to hyperlight-host = "0.16.0" from crates.io. This unblocks publishing hyperlight-unikraft to crates.io (closes #4, unblocks #27).

Key changes:

  • OCI snapshot format: to_file/from_file_uncheckedsave/load with OCI image-layout. Snapshot path changes from snapshot.hls (file) to snapshot/ (directory)
  • map_file_cow: 3-arg → 2-arg (label parameter removed upstream)
  • restore: restore_preserving_file_mappings removed; initrd is re-mapped after every restore()
  • Sparsify removed: OCI format manages storage; sparsify_snapshot and related platform-specific code dropped
  • whp-no-surrogate feature removed: now controlled via HYPERLIGHT_MAX_SURROGATES env var at runtime
  • max_surrogates API: configure_surrogates(), InstallOptions.max_surrogates, and Runtime::new() 5th arg for mxc integration
  • Heap size: reduced default from 2560 MiB to 1280 MiB

Test plan

  • cargo check / cargo clippy pass
  • CI passes (runtime tests, benchmarks)
  • mxc compiles with this branch

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linux Benchmarks

Details
Benchmark suite Current: 77c44bd Previous: 897275c Ratio
hello_world (median) 20 ms 20 ms 1
pandas (median) 110 ms 110 ms 1
density (per VM) 8 MB 11 MB 0.73
snapshot (disk) 653 MiB 656 MiB 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows Benchmarks

Details
Benchmark suite Current: 77c44bd Previous: 897275c Ratio
hello_world (median) 307 ms 373 ms 0.82
pandas (median) 957 ms 1092 ms 0.88
density (per VM) 7 MB 656 MB 0.010670731707317074
snapshot (disk) 661 MiB 664 MiB 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@danbugs
danbugs force-pushed the experiment/use-hyperlight-main branch 3 times, most recently from 6b2c81d to b6f7241 Compare June 24, 2026 15:05
@danbugs
danbugs marked this pull request as ready for review June 25, 2026 03:52
Copilot AI review requested due to automatic review settings June 25, 2026 03:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates hyperlight-unikraft’s host tooling to work directly against upstream hyperlight-dev/hyperlight main (pinned to 69c362b6), primarily by migrating snapshot persistence/loading from a single snapshot.hls file to an OCI image-layout directory (snapshot/) and removing custom-branch-only features/APIs.

Changes:

  • Switch hyperlight-host dependency to upstream hyperlight-dev/hyperlight@69c362b6 and drop custom-branch-only features/deps.
  • Migrate snapshot persistence and all references from snapshot.hls to OCI layout in snapshot/ (save/load with OciTag).
  • Update restore behavior to re-map initrd after restore (replacing the old “preserve file mappings” restore path).

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
host/tests/pyhl_runtime.rs Updates test install probing from snapshot.hls to snapshot/.
host/src/pyhl.rs Updates library install/runtime flow to use OCI snapshot directory and upstream snapshot APIs.
host/src/lib.rs Adapts snapshot save/load to upstream OCI format; removes preserving-restore path and snapshot sparsification.
host/src/bin/pyhl.rs Updates CLI setup/run to use snapshot/ directory layout and upstream APIs.
host/src/bin/pydriver_run.rs Updates heap sizing consistent with other host entrypoints.
host/examples/test_cpiovfs.rs Updates snapshot save path to directory form and simplifies output.
host/examples/pyhl_as_library.rs Updates example docs to refer to .pyhl/snapshot/.
host/Cargo.toml Points hyperlight-host to upstream repo/rev; drops no-longer-needed deps/features.
host/Cargo.lock Lockfile refresh consistent with upstream dependency graph changes.
.github/workflows/benchmarks.yml Updates snapshot size reporting to handle snapshot/ directory on Linux/Windows.
Comments suppressed due to low confidence (1)

host/src/bin/pyhl.rs:378

  • The setup early-return condition only checks that snapshot/ is a directory. A partially-written/empty snapshot dir (e.g., interrupted save) would incorrectly be treated as “installed” unless --force, leaving pyhl run broken. Consider checking for an OCI-layout marker (e.g., snapshot/index.json) and, when proceeding with setup, deleting any existing snapshot path (file or dir) before saving a new snapshot to avoid accumulating stale blobs across repeated --force runs.
    if image_installed(&home) && dst_snapshot.is_dir() && !args.force {
        eprintln!(
            "pyhl: image already installed at {} (use --force to overwrite)",
            home.display()
        );

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/benchmarks.yml Outdated
Comment thread host/src/pyhl.rs
Comment thread host/tests/pyhl_runtime.rs Outdated
Comment thread host/src/pyhl.rs
Comment thread host/src/bin/pyhl.rs
Switch from git dep (danbugs/hyperlight) to crates.io hyperlight-host
0.16.0. Key API changes:

- Snapshot format: to_file/from_file_unchecked -> OCI save/load
- snapshot.hls (single file) -> snapshot/ (OCI dir)
- map_file_cow: 3-arg -> 2-arg (label param removed)
- restore_preserving_file_mappings removed; re-map initrd after restore
- Remove sparsify_snapshot (OCI format handles storage)
- Remove whp-no-surrogate feature (now runtime env var)
- Add max_surrogates API to configure_surrogates/InstallOptions/Runtime
- Reduce default heap from 2560 MiB to 1280 MiB

Signed-off-by: danbugs <danilochiarlone@gmail.com>
@danbugs
danbugs force-pushed the experiment/use-hyperlight-main branch from d43b7a7 to eeb2ba4 Compare June 26, 2026 23:57
@danbugs danbugs changed the title experiment: use hyperlight main directly (no custom branch) feat: use hyperlight-host 0.16.0 from crates.io Jun 26, 2026
danbugs added 4 commits June 26, 2026 23:58
Signed-off-by: danbugs <danilochiarlone@gmail.com>
Address Copilot review: check snapshot/index.json instead of just
testing snapshot/ directory existence. Catches interrupted installs
and incomplete OCI layouts with an actionable error message.

Signed-off-by: danbugs <danilochiarlone@gmail.com>
The OCI snapshot format writes guest memory as a dense raw blob.
With CONFIG_PAGING=n in our Unikraft build, the heap pages retain
their boot-time PTEs so the snapshot captures the full 1280 MiB
heap even if most pages are untouched. Post-save sparsification
punches holes in zero-filled 4K regions, reducing disk usage from
~1755 MiB back to ~656 MiB.

Linux uses fallocate(PUNCH_HOLE), Windows uses FSCTL_SET_ZERO_DATA.

Signed-off-by: danbugs <danilochiarlone@gmail.com>
Use GetCompressedFileSizeW to measure actual on-disk allocation
instead of apparent file size, so Windows benchmark reports match
Linux (which already uses stat block count).

Signed-off-by: danbugs <danilochiarlone@gmail.com>
@danbugs
danbugs merged commit 1caba5c into main Jun 27, 2026
87 checks passed
@danbugs
danbugs deleted the experiment/use-hyperlight-main branch June 27, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update Hyperlight dependency to hyperlight-dev/hyperlight

2 participants