We discussed this during the OCI meeting at KubeCon today and decided it makes more sense to add this to image-spec instead of a separate repo. I'm closing this and we can coordinate with PR's to image-spec.
https://hackmd.io/nZzK_4AfRz-xgya6KkqseA

Reopening to continue the discussion from the mailing list: https://groups.google.com/a/opencontainers.org/g/dev/c/PpzsQMOnda4

cc @vbatts @Jamstah @sajayantony

Thanks for reopening this @sudo-bmitch. I think clarifying meaning of reference in distribution and OCI layout is valuable since parsing that regex depending on context and language choice would make it easier if there was a clear definition in image spec (on which distribution indirectly depends on anyway). e.g. write up (but doesn't include the regex) - https://oras.land/docs/concepts/reference

cc @lumjjb

There's not a lot in the minutes explaining the decision to include it in image-spec, can anyone remember the reasoning?

Personally, I think defining references separately makes sense. The image spec as written is very cleanly encapsulated around what an image is and not where an image is. It also gives more flexibility for the future, if we need to refer to things other than images. Putting it in the image spec would also tie up the lifecycles, and I'm not sure that's a good thing.

There's not a lot in the minutes explaining the decision to include it in image-spec, can anyone remember the reasoning?

I believe @imjasonh weighed in on that. The impression I got was that creating a new spec is more effort than extending an existing one, and they felt that image spec was the closest fit. Given that I originally opened this to create a new spec myself, I tend to agree with you that it doesn't feel like a great fit.

Personally, I think defining references separately makes sense. The image spec as written is very cleanly encapsulated around what an image is and not where an image is. It also gives more flexibility for the future, if we need to refer to things other than images. Putting it in the image spec would also tie up the lifecycles, and I'm not sure that's a good thing.

We've muddied the definition, particularly with defining the usage for artifacts, and the image layout. The spec also uses "image" for a lot of things that aren't necessarily an image, like the image index.

Linking my thoughts on what the spec could include here: https://github.com/Jamstah/reference-spec/blob/main/spec.md

I think such a codification of existing practice would make sense. (I’m a tiny bit worried about a separate spec working group finding opportunities to invent new features with little previous precedent.)

https://github.com/Jamstah/reference-spec/blob/main/spec.md
Worth referencing different transports? Even if it's just to say they exist and are client specific?

(As the maintainer of the linked Skopeo tool), at best as a warning. Strings are notoriously ambiguous. IMHO every string field / syntax should strictly define whether it is an “image reference” in the sense of the rest of that text, or a transport:… string.

E.g. consider dir:foo. That could be Skopeo’s docker://docker.io/library/dir:foo when interpreted by Docker CLI, or it could be Skopeo’s dir:$(pwd)/foo when interpreted as a transport:… syntax. Hence Skopeo requires the user to always include a transport (OTOH Podman does rely on heuristics).

Accepting both syntaxes in the same field forces the consumer to resort to heuristics to disambiguate. And while it would certainly be more useful for every software to use the same heuristics, it seems to me to be very hard to design and standardize heuristics that will stand the test of time, as other registry features, and new c/image transports (or transports in other image codebases?), are added.

Kubernetes contributor here.

We would benefit significantly from having this spec. We're current dealing with a proposal to improve our validation support for image references (kubernetes/kubernetes#130834) and it's quite risky to offer guarantees on our API surface without an actual spec.

Note that this PR is currently held in draft state waiting on a sufficient number of volunteers to be a proposed owners and stakeholders in the working group.

Do volunteers need any particular credential?

I'm happy to be involved.

Do volunteers need any particular credential?

I'm happy to be involved.

Mostly a willingness to roll up their sleeves and do the work. Submit a review with your name/project to add if you have the bandwidth. See the referrers WG for an example of what it took to push through a large change.

The auth WG hit a wall when it came time to write the spec changes, and I still need to spend a lot of time doing the heavy lifting over there. I think we had more people interested in seeing the result than we had people with the time to lead the effort. So I'm holding off opening up new working groups where I'd end up being the only author.

See the referrers WG for an example of what it took to push through a large change.

The successfulness of that WG is also not universally held (I'd say there's at least as much cautionary tale there as blueprint, if not more).

The goal here is essentially to take the Grammar from https://pkg.go.dev/github.com/distribution/reference@v0.6.0#pkg-overview and write slightly more words around it, especially describing the docker.io, docker.io/library/, and :latest defaults, right?

(Perhaps to be even more clear, if this working group ends up designing something new, I think we've missed the mark.)

In other words, I'm not strictly opposed to this working group, but I don't think "extension" should be explicitly part of the initial charter. I also think this "spec" probably belongs in either the existing image-spec or distribution-spec. It's got a little bit of overlap between them, but that's the nature of those two specs already, and IMO this probably falls more on the distribution-spec than the image (as these "references" are describing objects as referenced via that API).

In other words, I'm not strictly opposed to this working group, but I don't think "extension" should be explicitly part of the initial charter. I also think this "spec" probably belongs in either the existing image-spec or distribution-spec. It's got a little bit of overlap between them, but that's the nature of those two specs already, and IMO this probably falls more on the distribution-spec than the image (as these "references" are describing objects as referenced via that API).

I agree with this. It seems to me that a reasonable deliverable for the WG (if this really needs a WG) would be a PR against distribution-spec.

I agree with this. It seems to me that a reasonable deliverable for the WG (if this really needs a WG) would be a PR against distribution-spec.

The community has gone back and forth on this a few times. It was originally proposed as a new spec, then there was a decision that it belonged as part of image-spec since it wasn't a registry API change, then it was back to making it a separate spec because image-spec should focus on the image json schemas and layer formats, and now it's over to distribution-spec where I'd suggest it prematurely limits it so that a reference to anything outside of a registry is permanently out of scope.

Define how the syntax can be extended to support other use cases

The goal with that section of the scope is to avoid defining a string that cannot be extended. There's a lot of external knowledge we can leverage for this, like the syntax of a URI, but that depends on ensuring character sets for various fields are limited from the start so that characters are available for future extensions. This doesn't mean those extensions need to be added as part of the working group, just that we acknowledge that groups will be doing this and to avoid rushing into a quick decision that locks us into a syntax that can never by extended without breaking changes.