Skip to content

Commit 2180119

Browse files
feat: OCI image loader (Hyper.Img.OciLoader) (#24)
1 parent b3c9781 commit 2180119

14 files changed

Lines changed: 668 additions & 7 deletions

File tree

AGENTS.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,16 @@ Coverage is a side effect of good tests, never the target.
132132

133133
## Other conventions
134134

135+
- **Comments earn their place.** No section-divider banners (`# --- foo ---`),
136+
no comment that just restates what the next line does. A comment explains a
137+
non-obvious *why* a reader cannot recover from the code — a workaround, an
138+
invariant, a deliberate trade-off. Prefer self-documenting code: a named
139+
function, a `Unit.*` quantity instead of a bare `1024 * 1024`, a descriptive
140+
variable — over a comment narrating the mechanics. If you reach for a comment
141+
to explain *what*, rename the thing instead.
142+
- Don't hand-roll magic numbers for sizes/durations/bandwidth: use the
143+
`Unit.*` types (`Unit.Information.mib(8)`, not `8 * 1024 * 1024`) and
144+
`use Unit.Operators` for unit-aware arithmetic.
135145
- Add `@spec` to public functions. Dialyzer runs with `:unmatched_returns`,
136146
`:extra_return`, `:missing_return` and will fail the gate on a missing/wrong
137147
spec.

docs/cookbook/intro.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,20 @@ with it.
1717

1818
The absolute best way to get started with `Hyper` is to play with it.
1919

20+
### Requirements
21+
22+
Hyper requires the following software be installed on each node running it:
23+
24+
- [`skopeo`](https://github.com/containers/skopeo)
25+
- [`e2fsprogs`](https://github.com/tytso/e2fsprogs)
26+
27+
Hyper has more runtime dependencies, but they are automatically redistributed
28+
by Hyper.
29+
30+
### Installation
31+
32+
<!-- TODO(markovejnovic): Write this out. -->
33+
2034
### Configuration
2135

2236
Running `Hyper` is involved and requires a large number of pre-requisites. The
@@ -41,3 +55,27 @@ config :hyper,
4155
uid_gid_range: {900_000, 999_999},
4256
layer_dir: "/srv/hyper/layers"
4357
```
58+
59+
<!-- TODO(markovejnovic): Update the config section. -->
60+
61+
### Usage
62+
63+
<!-- TODO(markovejnovic): Write out how to boot hyper etc -->
64+
65+
#### Loading Images
66+
67+
Before an image can be booted, it needs to be loaded into Hyper. Currently, the
68+
only way to load images is through an OCI image, either natively or through the
69+
native interface, or through [gRPC](../grpc.md):
70+
71+
```elixir
72+
{:ok, img_id} = Hyper.Img.OciLoader.load("docker.io/library/alpine:3.19")
73+
```
74+
75+
#### Booting a VM
76+
77+
With the image loaded, and an `img_id` in hand, you can boot it:
78+
79+
```elixir
80+
{:ok, vm} = Hyper.create_vm(%Hyper.Vm.Spec{ img_id: img_id })
81+
```

docs/grpc.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,14 +51,31 @@ from hyper.grpc.v0 import hyper_pb2, hyper_pb2_grpc
5151
client = hyper_pb2_grpc.HyperStub(grpc.aio.insecure_channel("localhost:50051"))
5252
```
5353

54+
### Loading Images
55+
56+
Before you can create a VM you need an image in the cluster. `LoadImage` pulls an
57+
OCI image, builds its rootfs, and records it -- returning the `img_id` you pass to
58+
`CreateVm`. It blocks until the load finishes (this can take minutes), so set a
59+
generous deadline.
60+
61+
```python
62+
loaded = await client.LoadImage(
63+
hyper_pb2.LoadImageRequest(
64+
image_ref="docker.io/library/alpine:3.19",
65+
# label is optional; defaults to image_ref.
66+
)
67+
)
68+
print(loaded.img_id) # pass this to CreateVm
69+
```
70+
5471
### Creating VMs
5572

5673
You can create new VMs with the `CreateVm` RPC.
5774

5875
```python
5976
created = await client.CreateVm(
6077
hyper_pb2.CreateVmRequest(
61-
img_id="img-abc",
78+
img_id=loaded.img_id,
6279
instance_type=hyper_pb2.INSTANCE_TYPE_DECI,
6380
arch=hyper_pb2.ARCHITECTURE_X86_64,
6481
# boot_args is optional; omit it for the default kernel cmdline.

lib/hyper.ex

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,6 @@ defmodule Hyper do
88
@type id :: String.t()
99
end
1010

11-
defmodule Img do
12-
@moduledoc "A content-addressed image: an ordered stack of layers."
13-
@type id :: String.t()
14-
end
15-
1611
@doc """
1712
Create a new virtual machine from an image.
1813

lib/hyper/config.ex

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@ defmodule Hyper.Config do
1919
@losetup_path Application.compile_env(:hyper, :losetup_path, "losetup")
2020
@dmsetup_path Application.compile_env(:hyper, :dmsetup_path, "dmsetup")
2121
@blockdev_path Application.compile_env(:hyper, :blockdev_path, "blockdev")
22+
@skopeo_path Application.compile_env(:hyper, :skopeo_path, "skopeo")
23+
@umoci_path Application.compile_env(:hyper, :umoci_path, nil)
24+
@mke2fs_path Application.compile_env(:hyper, :mke2fs_path, "mke2fs")
2225
@vmlinux Application.compile_env(:hyper, :vmlinux, %{})
2326

2427
@doc """
@@ -62,6 +65,10 @@ defmodule Hyper.Config do
6265
@spec vmlinux_install_dir :: Path.t()
6366
def vmlinux_install_dir, do: Path.join(redist_dir(), "vmlinux")
6467

68+
@doc "Directory where `Hyper.Img.OciLoader.Umoci` installs the default umoci binary."
69+
@spec umoci_install_dir :: Path.t()
70+
def umoci_install_dir, do: Path.join(redist_dir(), "umoci")
71+
6572
@doc """
6673
Path to the directory where all VM chroot's are created (`<work_dir>/jails`).
6774
@@ -114,6 +121,18 @@ defmodule Hyper.Config do
114121
@doc "Path to the blockdev binary."
115122
def blockdev_path, do: @blockdev_path
116123

124+
@doc "Path to the skopeo binary (used by `Hyper.Img.OciLoader` to pull OCI images)."
125+
def skopeo_path, do: @skopeo_path
126+
127+
@doc """
128+
Operator-configured path to the umoci binary, or `nil` (the default) to let
129+
`Hyper.Img.OciLoader.Umoci` download and manage a pinned default.
130+
"""
131+
def umoci_path, do: @umoci_path
132+
133+
@doc "Path to the mke2fs binary (used by `Hyper.Img.OciLoader` to build the ext4 rootfs)."
134+
def mke2fs_path, do: @mke2fs_path
135+
117136
@doc """
118137
Path to the setuid-root device helper (`hyper-suidhelper`). Required: the node
119138
runs unprivileged and routes every `losetup`/`dmsetup`/`blockdev` operation

lib/hyper/grpc/codec.ex

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ defmodule Hyper.Grpc.Codec do
1515
CreateVmResponse,
1616
GetVmResponse,
1717
ListVmsResponse,
18+
LoadImageRequest,
19+
LoadImageResponse,
1820
Vm
1921
}
2022

@@ -72,6 +74,16 @@ defmodule Hyper.Grpc.Codec do
7274
end
7375
end
7476

77+
@spec from_grpc(LoadImageRequest.t()) ::
78+
{:ok, {String.t(), keyword()}} | {:error, :missing_image_ref}
79+
def from_grpc(%LoadImageRequest{image_ref: ref}) when ref in [nil, ""],
80+
do: {:error, :missing_image_ref}
81+
82+
def from_grpc(%LoadImageRequest{image_ref: ref, label: label}) do
83+
opts = if label in [nil, ""], do: [], else: [label: label]
84+
{:ok, {ref, opts}}
85+
end
86+
7587
@doc "Convert a domain result to an outbound response message, or an error to `GRPC.RPCError`."
7688
@spec to_grpc({:created, Hyper.Vm.id(), node()}) :: CreateVmResponse.t()
7789
def to_grpc({:created, vm_id, node}) when is_binary(vm_id),
@@ -85,6 +97,10 @@ defmodule Hyper.Grpc.Codec do
8597
def to_grpc({:vms, vms}),
8698
do: %ListVmsResponse{vms: Enum.map(vms, &vm/1)}
8799

100+
@spec to_grpc({:loaded, Hyper.Img.id()}) :: LoadImageResponse.t()
101+
def to_grpc({:loaded, img_id}) when is_binary(img_id),
102+
do: %LoadImageResponse{img_id: img_id}
103+
88104
@spec to_grpc(:stopped) :: Empty.t()
89105
def to_grpc(:stopped), do: %Empty{}
90106

@@ -124,6 +140,19 @@ defmodule Hyper.Grpc.Codec do
124140
defp rpc_error(reason) when reason in [:no_capacity, :exhausted],
125141
do: GRPC.RPCError.exception(:resource_exhausted, "no capacity")
126142

143+
defp rpc_error(:missing_image_ref),
144+
do: GRPC.RPCError.exception(:invalid_argument, "image_ref is required")
145+
146+
defp rpc_error(:invalid_ref),
147+
do: GRPC.RPCError.exception(:invalid_argument, "image_ref is malformed")
148+
149+
defp rpc_error({:missing_tools, tools}),
150+
do:
151+
GRPC.RPCError.exception(
152+
:failed_precondition,
153+
"node is missing required image tools: #{Enum.join(tools, ", ")}"
154+
)
155+
127156
defp rpc_error(reason),
128157
do: GRPC.RPCError.exception(:internal, "internal error: #{inspect(reason)}")
129158
end

lib/hyper/grpc/server.ex

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,21 @@ defmodule Hyper.Grpc.Server do
1818
GetVmRequest,
1919
GetVmResponse,
2020
ListVmsResponse,
21+
LoadImageRequest,
22+
LoadImageResponse,
2123
StopVmRequest
2224
}
2325

26+
@spec load_image(LoadImageRequest.t(), GRPC.Server.Stream.t()) :: LoadImageResponse.t()
27+
def load_image(%LoadImageRequest{} = req, _stream) do
28+
with {:ok, {ref, opts}} <- Codec.from_grpc(req),
29+
{:ok, img_id} <- Hyper.Img.OciLoader.load(ref, opts) do
30+
Codec.to_grpc({:loaded, img_id})
31+
else
32+
{:error, reason} -> raise Codec.to_grpc({:error, reason})
33+
end
34+
end
35+
2436
@spec create_vm(CreateVmRequest.t(), GRPC.Server.Stream.t()) :: CreateVmResponse.t()
2537
@decorate with_span("Hyper.Grpc.Server.create_vm")
2638
def create_vm(%CreateVmRequest{} = req, _stream) do

lib/hyper/img.ex

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
defmodule Hyper.Img do
2+
@moduledoc """
3+
A content-addressed image: an ordered stack of layers, and the entry point for
4+
putting one into the cluster.
5+
6+
`create/2` ingests a prepared image file -- e.g. the ext4 rootfs produced by
7+
`Hyper.Img.OciLoader` -- into the shared media store and the image database. It
8+
content-addresses the file (sha256 of its bytes = the image id), publishes it
9+
into `Hyper.Config.layer_dir/0` at `layer_<id>.img`, then records it as a
10+
one-layer base image (`blobs` + `images` + `image_layers`). Producers of image
11+
files stay decoupled from the store and DB: they hand a path to `create/2`.
12+
"""
13+
14+
use OpenTelemetryDecorator
15+
16+
require Logger
17+
18+
alias Hyper.Config
19+
alias Hyper.Img.Db.{Blob, Image, ImageLayer, Repo}
20+
21+
@type id :: String.t()
22+
23+
# `Ecto.Multi` is an opaque struct; building it through the pipe trips
24+
# dialyzer's opacity check (a known Ecto false positive), so silence it for the
25+
# one function that assembles a Multi.
26+
@dialyzer {:no_opaque, record: 3}
27+
28+
@doc """
29+
Ingest the image file at `path` into the cluster and return its
30+
content-addressed id.
31+
32+
Content-addresses `path` (sha256 of its bytes = the id), publishes it into the
33+
media store at `layer_<id>.img`, and records it as a one-layer base image. The
34+
file at `path` is consumed -- moved into the store on success, removed on
35+
failure -- so the caller hands off ownership.
36+
37+
`opts[:label]` sets the human-readable `images.label` (defaults to the basename
38+
of `path`).
39+
40+
Idempotent: creating identical bytes again is a no-op that returns the same id.
41+
"""
42+
@spec create(Path.t(), keyword()) :: {:ok, id()} | {:error, term()}
43+
@decorate with_span("Hyper.Img.create", include: [:path, :label])
44+
def create(path, opts \\ []) do
45+
label = Keyword.get(opts, :label, Path.basename(path))
46+
47+
with {:ok, %File.Stat{size: size}} <- File.stat(path),
48+
{:ok, id} <- content_id(path),
49+
{:ok, final, origin} <- publish(path, id),
50+
:ok <- record_or_rollback(id, label, size, final, origin) do
51+
{:ok, id}
52+
else
53+
{:error, _} = err ->
54+
_ = File.rm(path)
55+
err
56+
end
57+
end
58+
59+
# Record the image; if the DB write fails, roll back a file we just created (a
60+
# reused file pre-existed and may back another image, so leave it).
61+
@spec record_or_rollback(id(), String.t(), non_neg_integer(), Path.t(), :created | :reused) ::
62+
:ok | {:error, term()}
63+
defp record_or_rollback(id, label, size, final, origin) do
64+
case record(id, label, size) do
65+
:ok ->
66+
:ok
67+
68+
{:error, _} = err ->
69+
_ = if origin == :created, do: File.rm(final), else: :ok
70+
err
71+
end
72+
end
73+
74+
# Streaming sha256 of `path`, lowercase hex -- the content address.
75+
@spec content_id(Path.t()) :: {:ok, id()} | {:error, term()}
76+
@decorate with_span("Hyper.Img.content_id", include: [:path])
77+
defp content_id(path) do
78+
{:ok, Hyper.Redist.Sha256.file(path)}
79+
rescue
80+
e -> {:error, {:hash_failed, Exception.message(e)}}
81+
end
82+
83+
# Move `src` into the store at its content-addressed path. If the destination
84+
# already exists (identical bytes already published), drop `src` and reuse it.
85+
@spec publish(Path.t(), id()) :: {:ok, Path.t(), :created | :reused} | {:error, term()}
86+
@decorate with_span("Hyper.Img.publish", include: [:id])
87+
defp publish(src, id) do
88+
File.mkdir_p!(Config.layer_dir())
89+
final = final_path(id)
90+
91+
if File.exists?(final) do
92+
Logger.info("image #{id} already present in store; reusing")
93+
_ = File.rm(src)
94+
{:ok, final, :reused}
95+
else
96+
case place(src, final) do
97+
{:ok, ^final} -> {:ok, final, :created}
98+
{:error, _} = err -> err
99+
end
100+
end
101+
end
102+
103+
# An atomic rename when `src` is on the store's filesystem; a copy-then-drop
104+
# across filesystems (rename can't cross a mount).
105+
@spec place(Path.t(), Path.t()) :: {:ok, Path.t()} | {:error, term()}
106+
defp place(src, final) do
107+
case File.rename(src, final) do
108+
:ok ->
109+
{:ok, final}
110+
111+
{:error, :exdev} ->
112+
case File.cp(src, final) do
113+
:ok ->
114+
_ = File.rm(src)
115+
{:ok, final}
116+
117+
{:error, reason} ->
118+
_ = File.rm(final)
119+
{:error, {:publish_failed, reason}}
120+
end
121+
122+
{:error, reason} ->
123+
{:error, {:publish_failed, reason}}
124+
end
125+
end
126+
127+
@spec final_path(id()) :: Path.t()
128+
defp final_path(id), do: Path.join(Config.layer_dir(), "layer_#{id}.img")
129+
130+
# Record the base image: one blob, one image (id == blob id), one layer at
131+
# position 0. All upserts are idempotent so a re-publish of the same bytes is a
132+
# no-op. The blob is inserted before the layer so the FK is satisfied.
133+
@spec record(id(), String.t(), non_neg_integer()) :: :ok | {:error, term()}
134+
defp record(id, label, size) do
135+
multi =
136+
Ecto.Multi.new()
137+
|> Ecto.Multi.insert(
138+
:blob,
139+
Blob.changeset(%Blob{}, %{id: id, kind: :base, size: size}),
140+
on_conflict: :nothing,
141+
conflict_target: :id
142+
)
143+
|> Ecto.Multi.insert(
144+
:image,
145+
Image.changeset(%Image{}, %{id: id, label: label}),
146+
on_conflict: :nothing,
147+
conflict_target: :id
148+
)
149+
|> Ecto.Multi.insert(
150+
:layer,
151+
ImageLayer.changeset(%ImageLayer{}, %{image_id: id, position: 0, blob_id: id}),
152+
on_conflict: :nothing,
153+
conflict_target: [:image_id, :position]
154+
)
155+
156+
case Repo.transaction(multi) do
157+
{:ok, _} -> :ok
158+
{:error, step, reason, _changes} -> {:error, {:record_failed, step, reason}}
159+
end
160+
end
161+
end

0 commit comments

Comments
 (0)