add a new UAPI.16 File Manifest spec (WIP) by poettering · Pull Request #213 · uapi-group/specifications

poettering · 2026-04-22T11:05:30Z

No description provided.

poettering · 2026-04-22T11:05:44Z

(mostly posted here to start the discussion)

bluca · 2026-04-22T11:36:50Z

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

Foxboron · 2026-04-22T12:03:41Z

Is the intention of this specc to solve this issue, or is this trying to solve a different problem?

#207

daandemeyer · 2026-04-22T12:07:31Z

@Foxboron Totally different problem, it's designed to replace SHA256SUMS for sysupdate to list remote resources

Foxboron · 2026-04-22T12:14:17Z

@daandemeyer Hmm, should the title be Sysupdate File Manifest spec to nail the usage a bit more down? Else we might end up with multiple "file manifest" specs?

keszybz

I think it'd be nice to use semantic line breaks here. We agreed in general to do this in new documents.

poettering · 2026-04-22T14:07:43Z

@daandemeyer Hmm, should the title be Sysupdate File Manifest spec to nail the usage a bit more down? Else we might end up with multiple "file manifest" specs?

I'd probably call the other spec an "Inode Spec", since that's more what it is. Or "File System Object" or so.

This one here is purely about files.

You are absolutely right, this might be confusing, but it will either way I am sure.

I am against naming this after the software that likely implements this, it's supposed to be a generic spec, independent of any specific implementation.

poettering · 2026-04-22T14:27:16Z

Hmm, we could also consider extending this spec to just cover what is requested in #207 too. I mean, if I grok this right it would just mean adding some more fields to encode UNIX inode properties in full: i.e. inodeType, mode, uid, gid, user, group, major, minor, symlinkTarget and so on. When using this for the download scenario we'd use ignore all these fields I guess, but I see nothing speaking against supporting this too.

poettering · 2026-04-22T14:55:18Z

Posted a new version covering all comments, but not trying to address #207 (would prefer if we did that in a later follow-up PR)

bluca · 2026-04-22T14:57:44Z

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

^^^ ?

poettering · 2026-04-22T15:00:36Z

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

i don't grok this request?

it seems to me that the manifest format here could easily be embedded by mkosi's package manifests if it wants file-level information. But that's something to decide and define in mkosi's format, it's not something we could dictate here?

bluca · 2026-04-22T15:04:32Z

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

i don't grok this request?

it seems to me that the manifest format here could easily be embedded by mkosi's package manifests if it wants file-level information. But that's something to decide and define in mkosi's format, it's not something we could dictate here?

Having a "suggested" key for the object would allow consumers to know what to search for, without having to come up with one for each case. Just like there's a "suggested" filename for the file in the spec?

pothos · 2026-04-22T15:39:37Z

I wouldn't merge this into the mkosi manifest but treat it like the SHA256SUMS file mkosi can generate with --checksum= because this update manifest is supposed to have a specific file name that sysupdate can look for while the mkosi manifest includes the version in its name and thus can't be used as is. So --uapi-file-manifest= would write it and then --sign= would do the gpg signing for this file like it's currently done for SHA256SUMS.

bluca · 2026-04-22T15:42:58Z

Nah it has to be in the same file, at least as an option, as I most definitely do not want to have to deal with having to publish yet another file

AdrianVovk · 2026-04-23T00:40:39Z

+should also be reflected in the POSIX 'w' access mode bit when storing the data in a regular file on disk. If
+unspecified, defaults to false.
+
+`valid*USec` is in µs since UNIX epoch UTC, and shall be an unsigned integer. It defines a validity time


We can expire individual files this way, but what we really aught to be able to do is expire the entire manifest

you could just rewrite each entry, no? doing this a 2nd time on the whole manifest seems redundant to me?

I think the expiration date of an individual file is orthogonal to the expiration date of the manifest.

The manifest should expire to avoid a situation where an attacker serves a stale manifest w/ a stale list of files. Individual files can expire for any additional number of reasons: maybe the file is tied to a TPM secret that self-destructs by a certain deadline. Maybe we expire old versions of the image to make rollbacks harder. Maybe old images expire because we have certificates in there that expire. Really there's any number of reasons to expire an individual file. Point is: you may want to set a different expiration date on the manifest than the individual files

With the scheme you suggest, each time we update the manifest we need to:

Separately keep a database of how the real expiration dates for each file

Compute the expiration date of the manifest

For each file set the expiration date to either the real expiration date or the manifest's expiration date, whichever comes first

So sure it's possible but it's a pain. We'd need separate storage for the real dates, and the manifest becomes less self-descriptive.

AdrianVovk · 2026-04-23T00:50:18Z

+
+A number of future extensions are envisioned:
+
+* Inline cryptographic signatures


We should have this in the manifest from day 1. Part of what we want to do here is get away from GPG, right?

Ack. I'm concerned that inline signatures expose a json parser to untrusted, potentially malicious data. To verify signatures, the signatures themselves need to be skipped so this might require some data mangling. Not sure that can be done safe enough in this context.

Detached signatures aren't too bad though, or? There are alternatives to GPG.

well, in a way https also can provide authentication. but sure we can look into adding this right-away.

Detached signatures aren't too bad though, or? There are alternatives to GPG.

Detached signatures suck because they're separate files but need to be updated atomically as a pair. We've encountered issues where the SHA256SUMS and SHA256SUMS.gpg desync for the client, leading to signature check failures.

The leading hypothesis here is that our CDN is deciding to expire one of the files but not the other one, so there's a period of time where you might get a new manifest but an old signature (or vice versa). To fix this correctly, we'd need to figure out how to tell our CDN explicitly to expire both files together and re-fetch them. In practice, we just set a really short cache time-to-live for those two files and let the signature check fail occasionally for an unlucky client. By the time the client retries the update a while later, the files will probably be sync'd up again

If it's just one file, there's no more race condition to worry about

in a way https also can provide authentication

Not nearly enough for our security model, though! An attacker could totally put sysupdate drop-ins in /run to point it at their own domain, which can use HTTPS without issue, and none of this would be measurable. The manifest must be validated against certificates that are measured into the TPM

AdrianVovk · 2026-04-23T01:11:33Z

Nah it has to be in the same file, at least as an option, as I most definitely do not want to have to deal with having to publish yet another file

Sorry I'm not groking how you can avoid this. mkosi spits out a manifest listing the specific packages it included in a specific image that it built. This spec is about listing all the files in the directory, and in the sysupdate usecase that means across version boundaries. So if anything, the mkosi manifest would have to be embedded in here (describing mkosi's output files) rather than the other way around. But that also doesn't make sense because even if mkosi spits out multiple files you'll get just one manifest, right?

Anyway, this isn't "yet another file" to publish since it'll replace SHA256SUMS

brauner · 2026-04-23T07:10:04Z

+{
+    "mediaType" : "application/vnd.uapi.manifest",
+    "files" : [
+        {
+            "name" : "FooOS.raw",
+            "dataEncoding": "gzip",
+            "encodedDataSize": 5642649603,
+            "dataSize" : 7523532800,
+            "sha256" : "922a9bae0e02b4ffac3e5ed5054230d0689b9c2e25b0178ba82b925f2a0c3e48",
+            "validBeforeUSec" : 1776856773123234
+        },
+        {
+            "name" : "FooOS_esp.raw",
+            "dataFile" : "FooOS.raw",
+            "dataEncoding": "gzip",
+            "encodedDataSize": 5642649603,
+            "dataSize" : 7523532800,
+            "sliceOffset" : 2097152,
+            "sliceSize" : 149175808,
+            "gptLabel" : "EFI System Partition",
+            "gptTypeUuid" : "c12a7328-f81f-11d2-ba4b-00a0c93ec93b",
+            "sha256" : "5dcfd837a4868550cc61c256d9567a974e32a20985afa9e100b8b96755a20cae",
+            "validBeforeUSec" : 1776856773123234
+        },
+        {
+            "name" : "FooOS_root.raw",
+            "dataFile" : "FooOS.raw",
+            "dataEncoding": "gzip",
+            "encodedDataSize": 5642649603,
+            "dataSize" : 7523532800,
+            "sliceOffset" : 351272960,
+            "sliceSize" : 234003200,
+            "sha256" : "00b70a0813e15f309828d3a36156283cba87576c26755e0b2d4cf0951eff8163",
+            "gptLabel" : "FooOS_root",
+            "gptTypeUuid": "4f68bce3-e8cd-4db1-96e7-fbcaf984b709",
+            "readOnly": true,
+            "validBeforeUSec" : 1776856773123234
+        }
+    ]
+}


Let's say someone wants to use this file to describe an ML dataset that has billions of files. Will that scale?

cuI guess because you can have links to other files it wouldn't be a problem.
Curious though: can a manifest point to another manifest?

aplanas · 2026-04-23T09:11:53Z

+`sha256` is the SHA256 hash of the specified slice, formatted in 64 hexadecimal characters. Parsers should
+parse this case-insensitively.
+
+The `gpt*` fields encode fields that we need when placing these resources in a GPT partition table entry. The


I am not sure that the gpt* fields belong to the file object. The gpt data of one object can contradict another one and there are not clear rules about how to resolve them.

Maybe is part of the manifest root? Maybe part of a different UAPI 16 Container Manifest spec?

hmm? not sure i follow? how would they contradict? each other? these are gpt per-partition fields?

bluca · 2026-04-23T12:30:11Z

Nah it has to be in the same file, at least as an option, as I most definitely do not want to have to deal with having to publish yet another file

Sorry I'm not groking how you can avoid this. mkosi spits out a manifest listing the specific packages it included in a specific image that it built. This spec is about listing all the files in the directory, and in the sysupdate usecase that means across version boundaries. So if anything, the mkosi manifest would have to be embedded in here (describing mkosi's output files) rather than the other way around. But that also doesn't make sense because even if mkosi spits out multiple files you'll get just one manifest, right?

The "top-level" build is mkosi, and it's a single mkos build that produces these artifacts. Just like it right now includes extensions DDI metadata in the mkosi manifest, it can also include this metadata.
The other way around would break existing consumers, which would suddenly not find the objects they are looking for anymore. New consumers looking for the new manifest would have no problem though, as they are new.

Anyway, this isn't "yet another file" to publish since it'll replace SHA256SUMS

Well it is, as it's new

poettering · 2026-04-23T13:04:47Z

Having a "suggested" key for the object would allow consumers to know what to search for, without having to come up with one for each case. Just like there's a "suggested" filename for the file in the spec?

I am not convinced that is necessary. For example, let's say we do something to address #207. The way I see this happen would be that for directories they'd contain an additional contents field in the file object that itself is again a manifest object. i.e. you'd nest them nicely. I think it would be really weird to make that field named "Uapi16ManifestFile" just because. It's the contents of the dir, and hence it should be named that way.

poettering · 2026-04-23T13:42:46Z

so i wonder if we should maybe fold the stepping stone and revoked thing into a single "updatePolicy" field or so, giving that a revoked item should never be a stepping stone. And given this state is relevant only when using this manifests for sysupdate-style updaters I think it makes sense to make that clear in the name. Hence:

updatePolicy would be an enum with values good, stepping-stone, revoked:

good → the default for entries that are listed but do not have the field updatePolicy set. An updater should consider this entry as a great, regular update target.
stepping-stone → almost like good, but an updater should never skip over such an entry.
revoked → this is a tainted version, please upgrade asap, and downgrade if there's no upgrade
and then we could define one additional value vanished → this is a pseudo-state for items that where downloaded before but no longer appear in the manifest. In most contexts it should be treated just like revoked.

With this we'd have a single field encoding the whole policy, and it would be self-contained inside the file object (which is a property I like very much)

cyphar · 2026-04-29T04:19:42Z

+        "encodedDataSize" : 2011,
+        "dataSize" : 4711,
+        "dataFile" : "…",
+        "dataUrl" : "http://…",


For those without CDN budgets, it might be nice to allow specifying multiple URLs and the client can pick an arbitrary one?

poettering added do-not-merge The pull request must not be merged new-spec labels Apr 22, 2026

poettering force-pushed the uapi16-file-manifest branch 4 times, most recently from 54f9001 to f4a161c Compare April 22, 2026 11:33

pothos reviewed Apr 22, 2026

View reviewed changes

Comment thread specs/file-manifest.md

pothos reviewed Apr 22, 2026

View reviewed changes

Comment thread specs/file-manifest.md

behrmann reviewed Apr 22, 2026

View reviewed changes

pothos reviewed Apr 22, 2026

View reviewed changes

Comment thread specs/file-manifest.md

keszybz requested changes Apr 22, 2026

View reviewed changes

add a new UAPI.16 File Manifest spec

914c9f6

poettering force-pushed the uapi16-file-manifest branch from f4a161c to 914c9f6 Compare April 22, 2026 14:54

poettering mentioned this pull request Apr 22, 2026

Please develop a filesystem specification (fsspec) #207

Open

AdrianVovk reviewed Apr 23, 2026

View reviewed changes

brauner reviewed Apr 23, 2026

View reviewed changes

Comment thread specs/file-manifest.md

brauner reviewed Apr 23, 2026

View reviewed changes

aplanas reviewed Apr 23, 2026

View reviewed changes

cyphar reviewed Apr 29, 2026

View reviewed changes


		A number of future extensions are envisioned:

		* Inline cryptographic signatures

Conversation

poettering commented Apr 22, 2026

Uh oh!

poettering commented Apr 22, 2026

Uh oh!

bluca commented Apr 22, 2026

Uh oh!

Uh oh!

Uh oh!

Foxboron commented Apr 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daandemeyer commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Foxboron commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

keszybz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

poettering commented Apr 22, 2026

Uh oh!

poettering commented Apr 22, 2026

Uh oh!

poettering commented Apr 22, 2026

Uh oh!

bluca commented Apr 22, 2026

Uh oh!

poettering commented Apr 22, 2026

Uh oh!

bluca commented Apr 22, 2026

Uh oh!

pothos commented Apr 22, 2026

Uh oh!

bluca commented Apr 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdrianVovk commented Apr 23, 2026

Uh oh!

daandemeyer commented Apr 22, 2026 •

edited

Loading

Foxboron commented Apr 22, 2026 •

edited

Loading

poettering Apr 23, 2026 •

edited

Loading

bluca commented Apr 23, 2026 •

edited

Loading