Figure out how to remove platform specific files from docker images

[jsturtevant]

Ideally, WASM images should be built to contain pretty much only the wasm module and maybe a configuration file for the runtime (i.e. slightfile.toml) and not much else. This would keep the OCI packages cross platform with out relying on multi-arch manifests improving the ability to use them everywhere, without modifications.

This might be tricky for a few reasons:

• The current slight dockerfile contains linux specific logic for getting and install certs:

  containerd-wasm-shims/images/slight/Dockerfile

  Line 5 in a662b07

  RUN apt-get update && apt-get install ca-certificates -y

  . When porting to windows these files are not useful as is.
• For Windows there are some platform specific files (some files that are required by the Windows snapshotter in Contianerd) that are need but don't make sense on Linux. If we use buildx on Windows, then these files and a few other files get added to the image.
• other OS specific deps (please list below if you have come across any)

There is a crate in runwasi that helps with the creation of images like this without the need for buildx: https://github.com/containerd/runwasi/tree/main/crates/oci-tar-builder

There will need to be some investigation on how to provide things like ssl certs and likely other components as well. The end goal would be to have purely "wasm" based images but that might not be practical, in those cases how do we deal with OS specific deps?

[jsturtevant]

Discussed a bit at Runwasi office hours and offline with a few folks, A couple options we came up with (there might be more):

• Using a tool like tar-builder to build images in OCI image format
  • This works for Linux mostly. The ssl specific files wouldn't work well or would be in the image whether it was needed or not.
  • There was also a concern raise in Prevent wasm functions from being able to access their source. containerd/runwasi#106 (comment) around the use of OCI image formats.
  • would require a windows specific snapshotter that doesn't have the file system requirements that it
• Using a tool to build OCI artifacts that contain WASM Modules.
  • There is a proposal for a storing wasm modules in a registry proposal for OCI v1.1 content source bytecodealliance/registry#87
    • The challenge with the proposal is that containerd doesn't necessarily know about OCI artifacts only OCI image manifests.
  • One solution to explore is using Stream Processors to translate the unknown media types to formats that Containerd does know: https://github.com/containerd/containerd/blob/main/docs/stream_processors.md
    • This possibly has a couple benefits:
      • work directly with modules instead of image formats (possibly address the concern around file systems mentioned above)
      • no need to write/adopt new snapshotters.
      • could potentially handle OS specific files via annotations
      • don't need to modify core contianerd, which means we can iterate faster than needing a containerd release atleast initially.

Prototyping the second solution using stream processors would be a good way to determine if it is feasible.

[jsturtevant]

@cpuguy83 @Mossaka Incase I am missing something

[cpuguy83]

So today we should be able to use stream processors, however it was suggested is to handle this at the "unpacker" level (via the transfer service) and essentially choose an unpacker passed on the image platform. So we could have a wasi/wasm unpacker and have different implementations of the unpacker when running on linux vs windows.

This (transfer service) is not wired up to CRI today, but it is being discussed.

[jsturtevant]

Is there any docs/issue/pr on the transfer service or unpacker that I could look at? What would be the differences and pros/cons of using stream processor vs transfer service?

[cpuguy83]

I think the main thing is you can select an unpacker based on image platform and not have to do anything special with new types.

Using a stream processor requires a new type, a backchannel to containerd to create the content in, and installing more binaries on a node.

[jsturtevant]

found containerd/containerd#7592 and containerd/containerd#8227

[jsturtevant]

The stream processor approach didn't work out since (see prototype)

• The stream processors don't get called on the manifest lists or manifests. They are used during "Apply" and so only called for each layers:
  https://github.com/containerd/containerd/blob/6a5b4c9c242a0278033b20f7f5eca4fdbbea231a/diff/windows/windows.go#L118
  https://github.com/containerd/containerd/blob/06e085c8b50a4953c6a9ea636b459ce3a18964e4/diff/apply/apply.go#L77

• Modifing the content streams creates a different Diff hash which fails

[jsturtevant]

I have since done a few prototypes and have a proposal in review at https://docs.google.com/document/d/11shgC3l6gplBjWF1VJCWvN_9do51otscAm0hBDGSSAc