Skip to content

Commit ed6ad5d

Browse files
markovejnovicmarkovejnovic
committed
docs(cookbook): document [tools]/[jails], node tools, user config
1 parent 46dfd96 commit ed6ad5d

3 files changed

Lines changed: 539 additions & 197 deletions

File tree

‎docs/cookbook/config.md‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
# Configuration
2+
3+
Configuring `Hyper` is done through four layers, in priority:
4+
5+
1. Runtime `/etc/hyper/config.exs` is the canonical Elixir way to configure
6+
the system. This allows you to inject arbitrary code to configure `Hyper`.
7+
2. `Hyper` will fall back to reading `/etc/hyper/config.toml` at runtime, on
8+
bootup, on each node.
9+
3. `Hyper` will use its compile-time configuration through `config.ex`.
10+
4. `Hyper` will use defaults.
11+
12+
**Note that not all layers allow all configuration fields to be tweaked.** This
13+
is usually done for security.
14+
15+
## Configuration Files
16+
17+
### `/etc/hyper/config.exs`
18+
19+
The `config.exs` file is exlusively used by the unprivileged `hyper`
20+
application. The purpose of this file is to allow you to load configuration
21+
values at runtime. If you are using a secrets manager, this is the right place
22+
to load the secrets.
23+
24+
### `/etc/hyper/config.toml`
25+
26+
The `/etc/hyper/config.toml` file is used for static configuration. Unlike
27+
`config.exs`, it is used by both `Hyper` and `hyper-suidhelper` which means
28+
that it can impact the behavior of a process running under `root`.
29+
30+
### Compile-Time Config
31+
32+
The compile-time configuration is generally used to fine-tune the performance
33+
of Hyper. You likely do not need to edit most of the configuration fields
34+
exposed by this file for day-to-day usage, but they are available for you to
35+
tweak.
36+
37+
## Configuration Fields
38+
39+
### Tool Configuration
40+
41+
Hyper relies on a large number of external tools, all configured under the
42+
`[tools]` table in `/etc/hyper/config.toml`.
43+
44+
#### Privileged tools (run by the setuid helper)
45+
46+
| Tool | Required | Default | `/etc/hyper/config.toml` |
47+
|------|----------|---------|--------------------------|
48+
| `firecracker` | Yes | - | `tools.firecracker` |
49+
| `jailer` | Yes | - | `tools.jailer` |
50+
| `dmsetup` | No | `/usr/sbin/dmsetup` | `tools.dmsetup` |
51+
| `losetup` | No | `/usr/sbin/losetup` | `tools.losetup` |
52+
| `blockdev` | No | `/usr/sbin/blockdev` | `tools.blockdev` |
53+
54+
> #### Requirements {: .info}
55+
>
56+
> - These paths **can only** be configured through `/etc/hyper/config.toml`.
57+
> Both `Hyper` and `hyper-setuidhelper` rely on these paths being identical.
58+
>
59+
> - The paths **must** be given as absolute paths.
60+
> - The basename **must** match the configuration, eg. `firecracker` must have
61+
> a path `/foo/bar/firecracker`.
62+
> - The tools must be owned by the `root` user.
63+
> - The tools must be exlusively writable by `root`.
64+
65+
#### Node tools (run by the unprivileged node)
66+
67+
| Tool | Required | Default | `/etc/hyper/config.toml` |
68+
|------|----------|---------|--------------------------|
69+
| `skopeo` | No | `skopeo` (on `PATH`) | `tools.skopeo` |
70+
| `mke2fs` | No | `mke2fs` (on `PATH`) | `tools.mke2fs` |
71+
| `umoci` | No | downloaded + cached under `<work_dir>/redist/umoci` | `tools.umoci` |
72+
| `suidhelper` | No | `/usr/local/bin/hyper-suidhelper` | `tools.suidhelper` |
73+
74+
> #### These are not privileged {: .info}
75+
>
76+
> The node runs these directly as the unprivileged hyper user, so — unlike the
77+
> privileged tools above — they carry **no** root-ownership or basename
78+
> requirement. `skopeo`/`mke2fs` default to the bare name resolved on `PATH`;
79+
> leave `umoci` unset to let Hyper download and cache a pinned release. They
80+
> share the one `[tools]` table with the privileged binaries — the helper simply
81+
> ignores the keys it does not own.
82+
83+
## The shared file: `/etc/hyper/config.toml`
84+
85+
> #### Security {: .error}
86+
>
87+
> This file **must** be owned by `root` and be neither group- nor
88+
> world-writable (e.g. mode `0644`). The setuid helper refuses to start
89+
> otherwise — a present-but-untrusted file is treated as operator
90+
> misconfiguration and is fatal (exit `2`), never silently ignored.
91+
92+
### Root keys
93+
94+
| Key | Type | Default | Meaning |
95+
|-----|------|---------|---------|
96+
| `work_dir` | string (absolute path) | `/srv/hyper` | Root of all node-local runtime state. Every other directory is derived from it. Must be an absolute path. Strongly recommended to sit on an NVMe drive. |
97+
98+
The following directories are derived from `work_dir` and are **not**
99+
independently configurable:
100+
101+
| Path | Purpose |
102+
|------|---------|
103+
| `<work_dir>/jails` | Per-VM chroot directories |
104+
| `<work_dir>/socks` | Per-VM control/gRPC sockets |
105+
| `<work_dir>/scratch` | Per-VM copy-on-write writable layers |
106+
| `<work_dir>/layers` | Read-only image layer store |
107+
| `<work_dir>/redist` | Node-downloaded binaries (`vmlinux`, `umoci`) |
108+
109+
### `[jails]` — confinement
110+
111+
| Key | Type | Default | Meaning |
112+
|-----|------|---------|---------|
113+
| `cgroup` | string | `"hyper"` | Parent cgroup under which every VM cgroup is nested (passed to the jailer as `--parent-cgroup`). The operator must create `/sys/fs/cgroup/<name>` and enable subtree control. |
114+
| `uid_gid_range` | `[min, max]` | `[900000, 999999]` | UID/GID band each VM jail is allocated from. `min` must be `>= 1` and `<= max`; `min = 0` is rejected (uid 0 is root, and the jailer skips its privilege drop for uid 0). |
115+
116+
> #### `uid_gid_range` is enforced on both sides {: .warning}
117+
>
118+
> The node only hands out UIDs in this range, and the helper only *accepts*
119+
> UIDs in this range. Because both read the same file, narrowing the band is a
120+
> single edit here — no second place to keep in sync. Nothing else on the host
121+
> may use UIDs/GIDs in this range.
122+
123+
### Complete example
124+
125+
```toml
126+
# Root of all node-local state. Strongly prefer an NVMe-backed mount.
127+
work_dir = "/srv/hyper"
128+
129+
# External binaries. The privileged ones (firecracker..blockdev) must be
130+
# root-owned, not group/world-writable, absolute, and named exactly as their
131+
# key; the node tools (skopeo/mke2fs/umoci/suidhelper) have no such requirement.
132+
[tools]
133+
firecracker = "/opt/firecracker/firecracker" # required; basename must be 'firecracker'
134+
jailer = "/opt/firecracker/jailer" # required; basename must be 'jailer'
135+
# dmsetup = "/usr/sbin/dmsetup" # optional (default shown)
136+
# losetup = "/usr/sbin/losetup" # optional (default shown)
137+
# blockdev = "/usr/sbin/blockdev" # optional (default shown)
138+
# skopeo = "skopeo" # optional node tool (default shown)
139+
# mke2fs = "mke2fs" # optional node tool (default shown)
140+
# umoci = "/usr/bin/umoci" # optional; omit to auto-download
141+
# suidhelper = "/usr/local/bin/hyper-suidhelper" # optional (default shown)
142+
143+
[jails]
144+
cgroup = "hyper" # default
145+
uid_gid_range = [900000, 999999] # default
146+
```
147+
148+
The minimal file is just `work_dir` plus the two required tools — everything
149+
else defaults.
150+
151+
## Node-only configuration (`config :hyper`)
152+
153+
These have no helper counterpart and stay in `config :hyper`. The node's tool
154+
paths (`skopeo`, `mke2fs`, `umoci`, `suidhelper`) used to live here but now read
155+
from the `[tools]` table above — see [Tool Configuration](#tool-configuration).
156+
157+
### Guest kernels
158+
159+
| Key | Where read | Type | Default | Meaning |
160+
|-----|-----------|------|---------|---------|
161+
| `vmlinux` | runtime | `%{arch => path}` | `%{}` | Per-architecture guest kernel images, keyed by `Sys.Arch.t()`. The operator places kernels on the host and points these at them. |
162+
163+
```elixir
164+
config :hyper,
165+
vmlinux: %{x86_64: "/srv/hyper/redist/vmlinux/vmlinux-x86_64"}
166+
```
167+
168+
### Resource budget — `Hyper.Node.Config.Budget`
169+
170+
The per-node resource budget. **Required**: the node refuses to boot if it is
171+
absent. Set it in `config/runtime.exs`. Use the `Unit.*` quantities, never bare
172+
numbers.
173+
174+
| Key | Type | Meaning |
175+
|-----|------|---------|
176+
| `mem_max` | `Unit.Information.t()` | Hard memory cap for this node. |
177+
| `disk_max` | `Unit.Information.t()` | Hard disk cap for this node. |
178+
| `cpu_max_load` | float `0.0..1.0` | CPU-utilization fraction above which the node is considered full. |
179+
| `disk_bw_cap` | `Unit.Bandwidth.t()` | Absolute disk throughput capacity. |
180+
| `disk_bw_max_load` | float `0.0..1.0` | Fraction of `disk_bw_cap` past which disk is saturated. |
181+
| `net_bw_cap` | `Unit.Bandwidth.t()` | Absolute network throughput capacity. |
182+
| `net_bw_max_load` | float `0.0..1.0` | Fraction of `net_bw_cap` past which network is saturated. |
183+
184+
```elixir
185+
config :hyper, Hyper.Node.Config.Budget,
186+
mem_max: Unit.Information.gib(4),
187+
disk_max: Unit.Information.gib(4),
188+
cpu_max_load: 0.8,
189+
disk_bw_cap: Unit.Bandwidth.gibps(1),
190+
disk_bw_max_load: 0.8,
191+
net_bw_cap: Unit.Bandwidth.gibps(1),
192+
net_bw_max_load: 0.8
193+
```
194+
195+
### gRPC server — `Hyper.Grpc.Config`
196+
197+
The public gRPC interface. **Disabled by default.**
198+
199+
| Key | Type | Default | Meaning |
200+
|-----|------|---------|---------|
201+
| `enabled` | boolean | `false` | Whether the server starts. |
202+
| `port` | port number | `50051` | Listen port. |
203+
| `cred` | `GRPC.Credential.t()` \| `nil` | `nil` | TLS credential, or `nil` for plaintext. |
204+
| `adapter_opts` | keyword | `[]` | Forwarded to the server adapter, e.g. `[ip: {0, 0, 0, 0}]`. |
205+
206+
```elixir
207+
config :hyper, Hyper.Grpc.Config,
208+
enabled: true,
209+
port: 50_051,
210+
cred: GRPC.Credential.new(ssl: [certfile: "/path/cert.pem", keyfile: "/path/key.pem"])
211+
```
212+
213+
> #### Co-located nodes {: .info}
214+
>
215+
> Every node binds `:port`. Running multiple nodes on one host requires giving
216+
> each a distinct port. Build the TLS credential where you load your keys
217+
> (e.g. `config/runtime.exs`); Hyper never reads the filesystem on your behalf.
218+
219+
### Layer garbage collector — `Hyper.Img.Db.Gc.Config`
220+
221+
A cluster-wide singleton that prunes unreferenced image layers. Every field has
222+
a default; set only what you change. Durations are `Unit.Time` values, so
223+
overrides belong in `config/runtime.exs`. Set `enabled: false` to never start it.
224+
225+
| Key | Type | Default | Meaning |
226+
|-----|------|---------|---------|
227+
| `enabled` | boolean | `true` | Run the collector at all. |
228+
| `batch_size` | `pos_integer` | `200` | Rows per keyset page (smaller = finer pause granularity). |
229+
| `batch_pause` | `Unit.Time.t()` | `100ms` | Pause between pages within a sweep. |
230+
| `sweep_interval` | `Unit.Time.t()` | `60s` | Rest between completed sweeps. |
231+
| `acquire_interval` | `Unit.Time.t()` | `5s` | How often a standby retries to become the active singleton. |
232+
| `retry` | `Unit.Time.t()` | `60s` | Backoff when the medium or DB is unavailable. |
233+
| `statement_timeout` | `Unit.Time.t()` | `5s` | Cap on each GC DB statement so it can't pin a backend. |
234+
| `grace_period` | `Unit.Time.t()` | `1h` | Never prune a blob younger than this (protects a row whose file is still being published). |
235+
236+
```elixir
237+
config :hyper, Hyper.Img.Db.Gc.Config,
238+
enabled: true,
239+
sweep_interval: Unit.Time.s(30),
240+
grace_period: Unit.Time.s(60 * 60)
241+
```
242+
243+
### Orphaned-resource reaper — `Hyper.Node.Reaper.Config`
244+
245+
A per-node sweeper that reclaims orphaned firecracker cgroups and `hyper-rw-*`
246+
device-mapper volumes left behind by unclean BEAM deaths. Uses a two-strike
247+
grace period (an orphan must be seen on two consecutive ticks before it is
248+
reaped). Set `enabled: false` to never start it.
249+
250+
| Key | Type | Default | Meaning |
251+
|-----|------|---------|---------|
252+
| `enabled` | boolean | `true` | Run the reaper at all. |
253+
| `interval` | `Unit.Time.t()` | `60s` | Rest between reap ticks. |
254+
255+
```elixir
256+
config :hyper, Hyper.Node.Reaper.Config,
257+
enabled: true,
258+
interval: Unit.Time.s(30)
259+
```
260+
261+
### Telemetry (OpenTelemetry)
262+
263+
Tracing is configured in `config/runtime.exs` from environment variables:
264+
265+
| Variable | Effect |
266+
|----------|--------|
267+
| `HONEYCOMB_API_KEY` | Export to `https://api.honeycomb.io` with this key. |
268+
| `OTEL_EXPORTER_OTLP_ENDPOINT` | If `HONEYCOMB_API_KEY` is unset, export to this OTLP/HTTP endpoint (e.g. a local Collector), no auth header. |
269+
270+
If neither is set, tracing is disabled.
271+
272+
### Database and cluster topology
273+
274+
The image-metadata database (`Hyper.Img.Db.Repo`, a standard Ecto/PostgreSQL
275+
repo) and the cluster topology (`:libcluster`) are configured in
276+
`config/config.exs` like any Elixir app. PostgreSQL is a required runtime
277+
dependency — the node will not boot without a reachable instance. See
278+
[Installation](install.md) for connection setup.

0 commit comments

Comments
 (0)