DRAFT: Add design doc for canonical tar#356
Conversation
|
could we split this even further? I like the idea of a canonical tar, and that can be applied to zstd:chunked too, but I don't really see any advantage to split even more from what containers tools are doing, and if possible we should try to fix what is there |
|
I'm interested in the idea of a canonical tar construction for another reason. If you mount a composefs image with --upperdir, it should be possible to "commit" it and create a new OCI image in the composefs repo with a new tarsplit splitstream for the new layer. |
|
Basically, you'd be able to |
Sketch a canonical, byte-reproducible tar serialization for composefs filesystem trees: a deterministic dumpfile-to-tar mapping that complements the standardized EROFS metadata work, letting a client regenerate byte-identical tar layers (e.g. to verify diff_id) from EROFS + object store alone. Drop the previously-proposed composefs-chunked/incremental-pull sketch for now; it can be revisited separately once the canonical tar format itself has settled. Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
force-pushed
the
spec-enhancements-more
branch
from
97a86cb to
5badac9
Compare
Done! Now this PR is only about the canonical tar spec.
We could do that today without canonical tar, it's just that conceptually different implementations might generate a different tar layer right? I mean, it's a good idea to make that canonical, but nothing stops this project from synthesizing OCI images without it, it's what we do in the test suite with https://github.com/bootc-dev/ocidir-rs/ |
cgwalters
changed the title
|
True, there is no real need for commit to produce guaranteed reproducible results. Its just that I do need to produce tar splitstreams from a Filesystem (while converting real whiteouts to .wh.* files), and having a pre-existing implementation for this would mean not having to duplicate such code. |
|
I think this will also play well with the idea of also using composefs to efficiently store generic non-os-image content, while doing deduplication. You'd have |
|
Or maybe the better approach is to always store as image, but allow to easily export it as a tar. |
3 participants
Sketch two speculative follow-on proposals that build toward incremental container image pulls: a canonical, byte-reproducible tar serialization, and a composefs-chunked layer format with an external offset-map artifact for range-based fetching.
--
Moved from #224