Skip to content

Commit 4516981

Browse files
smoserclaude
smoser
and
claude
committed
docs: add EROFS build and verification walkthrough
Step-by-step guide for producing EROFS images with --format=erofs, inspecting the layer blob without root (fsck.erofs / dump.erofs / fsck.erofs --extract), mounting it (kernel mount or erofsfuse), pulling layer blobs from a registry, and assembling multi-layer images via overlayfs. Includes the current limitations (no +zstd, dm-verity, chunk index) and links from apko_file.md. All commands shown were verified against a real `apko build` of examples/wolfi-base.yaml. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 96c160a commit 4516981

2 files changed

Lines changed: 291 additions & 0 deletions

File tree

‎docs/apko_file.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,3 +268,5 @@ See [layering.md](layering.md) for more information.
268268
`erofs` may also be selected on the command line with `--format=erofs` on `apko build` and `apko publish`. The CLI flag overrides whatever is in the config file.
269269

270270
**Status:** EROFS support is experimental and tracks the spec PR at https://github.com/erofs/erofs-image-spec/pull/1; media types and annotations may change before the spec reaches a stable release. Both single-layer and multi-layer (`layering`) builds are supported. Multi-layer builds emit each non-final layer with `org.erofs.role=overlay-lower` per spec §3.8; the final layer carries no role. `+zstd` compression and dm-verity are not implemented.
271+
272+
See [erofs.md](erofs.md) for a step-by-step guide to building, inspecting, mounting, and pulling EROFS images.

‎docs/erofs.md‎

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
# EROFS Output Format (experimental)
2+
3+
apko can emit image layers as [EROFS](https://erofs.docs.kernel.org/) filesystem images instead of the default gzip-compressed tar.
4+
The format tracks the draft [erofs/erofs-image-spec](https://github.com/erofs/erofs-image-spec) (PR [#1](https://github.com/erofs/erofs-image-spec/pull/1)).
5+
Until the spec reaches a stable release, the media types, annotations, and layer layout used here may change.
6+
7+
## Why EROFS?
8+
9+
- **Mount, don't unpack.** A layer blob is a complete, kernel-mountable read-only filesystem.
10+
You can `mount -t erofs` the layer directly and look at it, without extracting a tarball.
11+
- **Random access.** Container runtimes that consume EROFS images can seek into a layer rather than streaming the whole tar.
12+
- **Designed for sharing.** The spec defines `overlay-lower` and `overlay-data` roles that compose via the kernel's `overlayfs` exactly the way OCI tar layers do.
13+
14+
This document focuses on producing EROFS images and verifying they look legit using widely available tools.
15+
16+
## Prerequisites
17+
18+
To build and inspect EROFS images you need:
19+
20+
- apko built from a revision that contains EROFS support.
21+
- The `erofs-utils` package, which provides `mkfs.erofs`, `fsck.erofs`, and `dump.erofs`.
22+
apko ships a pure-Go writer (no CGO), so `mkfs.erofs` is not required for *producing* images — but `fsck.erofs` and `dump.erofs` are the easiest way to inspect what apko produced.
23+
- To mount an EROFS layer: either the kernel `erofs` module (present in modern Linux distros) plus root for `mount(8)`, or the unprivileged `erofsfuse` binary from `erofs-utils-fuse`.
24+
25+
Install on Wolfi / Chainguard / Alpine:
26+
27+
```sh
28+
sudo apk add erofs-utils # fsck.erofs, dump.erofs, mkfs.erofs
29+
sudo apk add erofs-utils-fuse # erofsfuse (optional, for unprivileged mount)
30+
```
31+
32+
Install on Debian / Ubuntu:
33+
34+
```sh
35+
sudo apt install erofs-utils erofsfuse
36+
```
37+
38+
## Single-layer build
39+
40+
The simplest case: opt into EROFS via the `--format=erofs` flag or `format: erofs` in apko.yaml.
41+
42+
`erofs-demo.yaml`:
43+
44+
```yaml
45+
contents:
46+
keyring:
47+
- https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
48+
repositories:
49+
- https://packages.wolfi.dev/os
50+
packages:
51+
- wolfi-base
52+
53+
cmd: /bin/sh -l
54+
archs:
55+
- host
56+
```
57+
58+
Build into an OCI image layout directory:
59+
60+
```sh
61+
mkdir -p out
62+
apko build erofs-demo.yaml apko-erofs-demo:latest out/ --format=erofs --arch=$(uname -m)
63+
```
64+
65+
The OCI layout under `out/` is a regular OCI image directory — the layer blob just happens to be an EROFS filesystem:
66+
67+
```
68+
out/
69+
├── blobs/sha256/
70+
│ ├── <config-digest> # JSON image config
71+
│ ├── <manifest-digest> # JSON image manifest
72+
│ └── <layer-digest> # raw EROFS filesystem image
73+
├── index.json
74+
└── oci-layout
75+
```
76+
77+
### Verify the manifest references EROFS
78+
79+
```sh
80+
MANIFEST=$(jq -r '.manifests[0].digest | split(":")[1]' out/index.json)
81+
jq . out/blobs/sha256/$MANIFEST
82+
```
83+
84+
Expected (excerpt):
85+
86+
```json
87+
{
88+
"layers": [
89+
{
90+
"mediaType": "application/vnd.erofs",
91+
"size": 16207872,
92+
"digest": "sha256:8a2205cc..."
93+
}
94+
]
95+
}
96+
```
97+
98+
The image config records `erofs` in `os.features` per spec §5.4, signalling to tools that don't implement the spec that they should not attempt to apply the layer as a tar:
99+
100+
```sh
101+
CONFIG=$(jq -r '.config.digest | split(":")[1]' out/blobs/sha256/$MANIFEST)
102+
jq '.["os.features"]' out/blobs/sha256/$CONFIG
103+
# → ["erofs"]
104+
```
105+
106+
## Inspect the layer (no mount required)
107+
108+
The layer blob is a complete EROFS filesystem. You can validate and inspect it without mounting anything.
109+
110+
### Identify the file
111+
112+
```sh
113+
LAYER=$(jq -r '.layers[0].digest | split(":")[1]' out/blobs/sha256/$MANIFEST)
114+
file out/blobs/sha256/$LAYER
115+
# → out/blobs/sha256/...: EROFS filesystem, blocksize=12, exslots=0, ...
116+
```
117+
118+
### Integrity check
119+
120+
```sh
121+
fsck.erofs -d3 out/blobs/sha256/$LAYER
122+
# <I> erofs: No errors found
123+
```
124+
125+
### Dump the superblock
126+
127+
```sh
128+
dump.erofs out/blobs/sha256/$LAYER | head -15
129+
```
130+
131+
This prints the on-disk metadata: block size, inode count, build time, UUID, feature flags.
132+
133+
### Extract without root
134+
135+
`fsck.erofs --extract` reads every inode and writes the resulting tree to a directory.
136+
This is the strongest unprivileged validation you can run: if the image is malformed, extraction fails; if it succeeds, the file tree on disk is exactly what a kernel mount would expose.
137+
138+
```sh
139+
mkdir extracted
140+
fsck.erofs --extract=extracted --xattrs --force out/blobs/sha256/$LAYER
141+
ls extracted/
142+
# bin dev etc home lib ...
143+
cat extracted/etc/os-release
144+
```
145+
146+
## Mount the layer
147+
148+
### Kernel mount (root)
149+
150+
```sh
151+
sudo mkdir -p /mnt/apko-erofs
152+
sudo mount -t erofs -o loop out/blobs/sha256/$LAYER /mnt/apko-erofs
153+
ls /mnt/apko-erofs/
154+
file /mnt/apko-erofs/bin/sh
155+
sudo umount /mnt/apko-erofs
156+
```
157+
158+
The kernel mount is read-only, zero-copy, and exposes xattrs.
159+
If `mount` reports "unknown filesystem type 'erofs'", the kernel module is missing on your system; install it (e.g. `linux-modules-extra-$(uname -r)` on Ubuntu) or use the FUSE path below.
160+
161+
### FUSE mount (unprivileged)
162+
163+
```sh
164+
mkdir -p mnt
165+
erofsfuse out/blobs/sha256/$LAYER mnt/
166+
ls mnt/
167+
fusermount -u mnt/ # or `fusermount3 -u mnt/`
168+
```
169+
170+
`erofsfuse` does not require root, which makes it convenient on dev machines and inside CI containers that lack the kernel module.
171+
172+
## Pulling from a registry
173+
174+
If you push the image with `apko publish` or `crane push`, the registry stores each blob unchanged — including the EROFS layer blob.
175+
Most registry clients can extract layers by digest:
176+
177+
```sh
178+
# Read the manifest and pull layer blobs.
179+
crane manifest registry.example.com/apko-erofs-demo:latest > manifest.json
180+
LAYER_DIGEST=$(jq -r '.layers[0].digest' manifest.json)
181+
crane blob registry.example.com/apko-erofs-demo:latest@$LAYER_DIGEST > layer.erofs
182+
183+
file layer.erofs # EROFS filesystem...
184+
fsck.erofs -d3 layer.erofs # <I> erofs: No errors found
185+
```
186+
187+
Once you have the blob on disk you can inspect or mount it exactly as in the previous sections.
188+
189+
## Multi-layer builds
190+
191+
Combine `format: erofs` with apko's [layering](layering.md) configuration to get one EROFS layer per package group plus a top layer for unowned files.
192+
193+
`erofs-layered.yaml`:
194+
195+
```yaml
196+
contents:
197+
keyring:
198+
- https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
199+
repositories:
200+
- https://packages.wolfi.dev/os
201+
packages:
202+
- wolfi-base
203+
204+
cmd: /bin/sh -l
205+
archs:
206+
- host
207+
208+
layering:
209+
strategy: origin
210+
budget: 4
211+
212+
format: erofs
213+
```
214+
215+
```sh
216+
mkdir -p out-layered
217+
apko build erofs-layered.yaml apko-erofs-layered:latest out-layered/ --arch=$(uname -m)
218+
```
219+
220+
Inspect the manifest:
221+
222+
```sh
223+
MANIFEST=$(jq -r '.manifests[0].digest | split(":")[1]' out-layered/index.json)
224+
jq '.layers[] | {mediaType, role: .annotations["org.erofs.role"]}' out-layered/blobs/sha256/$MANIFEST
225+
```
226+
227+
Expected (last layer carries no role per spec §3.8 rule 1):
228+
229+
```json
230+
{ "mediaType": "application/vnd.erofs", "role": "overlay-lower" }
231+
{ "mediaType": "application/vnd.erofs", "role": "overlay-lower" }
232+
{ "mediaType": "application/vnd.erofs", "role": "overlay-lower" }
233+
{ "mediaType": "application/vnd.erofs", "role": "overlay-lower" }
234+
{ "mediaType": "application/vnd.erofs", "role": null }
235+
```
236+
237+
Each layer is independently mountable as an EROFS filesystem, and each carries its own partial `usr/lib/apk/db/installed` so per-layer scanners (Trivy, Snyk, Grype) can identify the packages it contributes.
238+
239+
### Assemble the full rootfs with overlayfs
240+
241+
The OCI spec composes layers with `overlayfs`-style semantics; for EROFS layers the composition is straightforward.
242+
Mount each layer separately, then stack them with `mount -t overlay`:
243+
244+
```sh
245+
# Pull each layer blob out of the OCI layout.
246+
ROOT=$(pwd)/out-layered/blobs/sha256
247+
mkdir -p mnt/{lower0,lower1,lower2,lower3,top,merged,work,upper}
248+
249+
LAYERS=$(jq -r '.layers[].digest | split(":")[1]' $ROOT/../../blobs/sha256/$MANIFEST)
250+
i=0
251+
for d in $LAYERS; do
252+
sudo mount -t erofs -o loop "$ROOT/$d" "mnt/lower$i" 2>/dev/null || \
253+
erofsfuse "$ROOT/$d" "mnt/lower$i"
254+
i=$((i+1))
255+
done
256+
257+
# In overlayfs, lowerdirs are listed top-down (highest priority first).
258+
# OCI orders layers bottom-up (index 0 is the base), so reverse the order.
259+
sudo mount -t overlay overlay \
260+
-o lowerdir=mnt/lower$((i-1)):mnt/lower$((i-2)):mnt/lower1:mnt/lower0,upperdir=mnt/upper,workdir=mnt/work \
261+
mnt/merged
262+
263+
ls mnt/merged/ # full rootfs
264+
```
265+
266+
Clean up:
267+
268+
```sh
269+
sudo umount mnt/merged
270+
for d in mnt/lower*; do sudo umount "$d" 2>/dev/null || fusermount -u "$d"; done
271+
```
272+
273+
Production runtimes (containerd's erofs snapshotter, podman/CRI-O with the erofs-aware plugin, etc.) automate this assembly; the manual steps above are for verifying that an apko-built EROFS image really does compose into a valid rootfs.
274+
275+
## Current limitations
276+
277+
- **No compression.** apko emits raw `application/vnd.erofs` layers only. The draft spec defines `application/vnd.erofs+zstd` but neither apko's writer nor the underlying go-erofs library writes compressed images yet.
278+
- **No dm-verity.** The spec's verified-mount path (§3.5) is not produced.
279+
- **No chunk index.** Lazy-loading runtimes (per spec §3.4) won't get an index; reads are sequential.
280+
- **No `overlay-data` or `device` roles.** Only `overlay-lower` (and unannotated final) layers are emitted.
281+
- **Spec is draft.** Media-type strings and annotation keys may change before the spec stabilizes. Treat any image built today as experimental.
282+
283+
If you need any of the above, please open an issue.
284+
285+
## See also
286+
287+
- [erofs/erofs-image-spec PR #1](https://github.com/erofs/erofs-image-spec/pull/1) — the layer format spec apko tracks.
288+
- [EROFS kernel documentation](https://erofs.docs.kernel.org/) — on-disk format reference.
289+
- [Layering in apko](layering.md) — how the multi-layer strategy partitions packages into groups.

0 commit comments

Comments
 (0)