`Layer has incorrect checksum` with Ubuntu 26.04 (resolute) base image -- PAX tar format not round-tripped

[jmarrero]

Summary

bootc install --composefs-backend fails with Layer has incorrect checksum when using an image built from ubuntu:resolute (26.04 LTS). The same image built from ubuntu:questing (25.10) works fine.

Root cause

Canonical switched from Docker/debootstrap to umoci (Rockcraft) for building Ubuntu 26.04 container images. umoci produces PAX format tars with sub-second mtime precision in PAX extended headers on every entry:

pax_headers={'mtime': '1776792234.3423805'}

The create_filesystem() function in crates/composefs-oci/src/image.rs reconstructs the tar from the splitstream via cat() and compares its hash to the diff_id. The reconstructed tar doesn't match the original because PAX headers are not preserved in the round-trip:

if config_verity.is_none() {
    let mut layer_stream = repo.open_stream("", Some(layer_verity), Some(TAR_LAYER_CONTENT_TYPE))?;
    let mut context = DigestWrite(Sha256::new());
    layer_stream.cat(repo, &mut context)?;
    let content_hash = crate::sha256_output_to_digest(context.finalize());
    ensure!(content_hash.as_ref() == diff_id, "Layer has incorrect checksum");
}

This code path triggers during generate_boot_image() in boot.rs which passes manifest_verity as None, forcing the untrusted validation path.

Data

	ubuntu:questing (25.10)	ubuntu:resolute (26.04)	fedora-bootc:43
Build tool	Docker/debootstrap	umoci (Rockcraft)	koji/osbuild
TAR format	GNU tar	PAX tar	GNU tar
PAX extended headers	0 / 4,081 entries	6,593 / 6,593 entries	0 / ~40,000 entries
Sub-second mtimes	No	Yes (e.g. 1776792228.527284)	No
Checksum validation	Passes	Fails	Passes

Workaround

Rebuild the base image with podman build --squash-all to flatten PAX layers into a single GNU tar layer:

FROM docker.io/library/ubuntu:resolute

podman build --squash-all -t ubuntu-resolute-squashed:latest .

This strips all PAX headers and produces a composefs-compatible image. See: https://github.com/jmarrero/ubuntu-resolute-squashed

Reproducer

Using https://github.com/bootcrew/mono:

git clone https://github.com/bootcrew/mono.git
cd mono

# Edit ubuntu/Containerfile: change ubuntu:questing to ubuntu:resolute
sed -i 's/ubuntu:questing/ubuntu:resolute/' ubuntu/Containerfile

# Build (requires sudo for rootful podman)
just build ubuntu

# Create disk image -- this fails
just disk-image ubuntu

Error output:

error: Installing to disk: Setting up composefs boot: Creating composefs filesystem
for boot entry discovery: Layer has incorrect checksum

Reverting to ubuntu:questing makes it work:

sed -i 's/ubuntu:resolute/ubuntu:questing/' ubuntu/Containerfile
just build ubuntu
just disk-image ubuntu  # succeeds

Environment

• Host: Fedora 43 (Silverblue), kernel 7.0
• podman 5.7.0
• bootc 1.15.2 (built from source, composefs-rs pinned at rev 54d248f7a7)
• Target: ext4 filesystem

Possible fixes

• generate_boot_image() in boot.rs should propagate the manifest_verity from the pull result instead of passing None, avoiding the checksum validation entirely
• Or: the tar splitstream split_async() / cat() round-trip should preserve PAX extended headers faithfully

Related issues

• composefs fails with Docker v2s2 images bootc-dev/bootc#1703 -- composefs fails with Docker v2s2 images (different error, same area)
• (composefs backend): Linux 7.0.1 fails to boot with error "has no fs-verity digest" bootc-dev/bootc#2174 -- Linux 7.0 fsverity regression (separate boot issue, also affects Ubuntu 26.04)

[jmarrero]

On top of this, I am hitting: bootc-dev/bootc#2174 which I think just means I have to use Ubuntu 25.10 for the example.