Add a policy on where to implement new codecs

[dstansby]

There is a new codec currently being implemented in #2863 that proposes to be included directly inside zarr-python, instead of the historical home of all codecs, numcodecs. This is a break from past behaviour where numcodecs was the home of all codecs.

We should decide, and document on a policy for where new codecs should go.

While numcodecs exists, it's my opinion that all user-facing codecs should be defined there. This is because it means:

• A clear separation of responsibilities for zarr-python and numcodecs
• A single place for users to discover and use codecs from

[d-v-b]

I don't have a strong opinion on where codecs are defined, and I don't think users care too much about this either. While numcodecs has proven generally useful beyond zarr, I don't think that was its intended purpose. IIRC numcodecs came into being because the cython build process required for some codecs was onerous for an otherwise pure-python library like zarr-python.

[dstansby]

and I don't think users care too much about this either

I think "give me a list of all codecs I can use with zarr-python" is a pretty important use case we should solve. I agree though that from a technical POV it doesn't really matter where they are.

Would your preference then be

• All new codecs implemented directly in zarr-python, unless they require a non-trivial build system
• Import and surface numcodecs codecs in the zarr-python API, and have zarr-python as the 'single source' of codecs

?

[d-v-b]

There are three places to consider for a codec:

1. a codec is defined in zarr-python (e.g., we could do this with gzip without any extra imports)
2. a codec is defined in numcodecs
3. a codec is defined in its own package

And there's a major complicating factor: the v2 codec API is effectively frozen, but I am 99% sure the v3 codec API is going to change. I'm saying this because I have very concrete (this week) changes in mind for the v3 codec API.

So if a v3-only codec was defined in a package outside of zarr python (including numcodecs), that package would need to carefully track the evolution of the codec API in zarr python. We already saw how a change to zarr python dtypes broke numcodecs. That should not happen again, and IMO the solution there is weaker coupling between numcodecs and zarr python.

A v3-native codec is a lot easier to develop inside zarr python, provided the codec doesn't have a complex build process that interferes with basic zarr python development. So, for v3-native codecs that can be effectively maintained in zarr python, zarr python seems like a good place for them. The codec could always be spun off into its own package if we feel like it has general use and we feel that the v3 codec API is completely done.

[dstansby]

#3272 (comment) all sounds good to me. I'll leave some time (a week?) for other devs to chime in, but at the moment lets assume we're going to put new codecs in zarr-python if they don't have a complex build process 👍

Maybe a world we can move towards is:

• V3 codecs defined in zarr-python
• Leave numcodecs for just V2 codecs?

[d-v-b]

Maybe a world we can move towards is:

* V3 codecs defined in `zarr-python`

* Leave `numcodecs` for just V2 codecs?

Yes, I think this is the right outcome. the v2 codec API is extremely simple and stable. While you can wrap a v2 codec with a v3 adapter, code for doing this should not live in numcodecs. To fix our current problems, I think we need to move all the v3 codecs into zarr python itself; importing them from numcodecs will require zarr python to be installed, and will re-direct the import to zarr python in that case.

[TomAugspurger]

There is a new codec currently being implemented in #2863 that proposes to be included directly inside zarr-python

Just to be clear, #2863 isn't proposing a new codec. It's adding the zarr-python-specific wrapper around an existing codec implementation. That code has to live somewhere (zarr-python or a third-party package; I don't think numcodecs is an option because numcodecs shouldn't depend on zarr-python).

[dstansby]

It's adding the zarr-python-specific wrapper around an existing codec implementation. That code has to live somewhere (zarr-python or a third-party package; I don't think numcodecs is an option because numcodecs shouldn't depend on zarr-python).

Unless I'm misunderstanding, the status quo is that zarr-python-specific wrappers live in numcodecs: https://github.com/zarr-developers/numcodecs/blob/main/numcodecs/zarr3.py. Of course, it's not great that numcodecs depends on zarr-python as you say, so moving forward we need to work out how to untangle this.

[TomAugspurger]

#2863 is proposing something analogous to https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/codecs/zstd.py: the zarr-python wrapper around numcodecs.zstd. My understanding is that zarr.codecs.zstd is used:

In [1]: import zarr

In [2]: store = zarr.storage.MemoryStore()

In [3]: z = zarr.create_array(store, name='a', shape=(4, 4), chunks=(2, 2), dtype='int32', overwrite=True)

In [4]: type(z.metadata.codecs[1]).__module__
Out[4]: 'zarr.codecs.zstd'

I don't know what numcodecs.zarr3 does but I don't think it's involved here..

[TomAugspurger]

FWIW, I wasn't aware of numcodecs.zarr3 until Davis mentioned it on a dev call week or two ago. I don't have a ton of context here, but I'd prefer that the core codecs be implemented in other libraries (like numcodecs) and that the zarr-specific (or at least zarr-python specific) pieces be implemented here:

• from a general cleanliness standpoint, having bidirectional dependencies between libraries isn't great.
• the API of the core codec will be relatively stable; the zarr / zarr-python specific details seems to be a bit more in flux.

[TomAugspurger]

@dstansby was this closed by #3273?

[dstansby]

It wasn't, that PR just documented the current state. Closing this issue requires adding a policy/guide on where to implement new codecs to both:

• https://zarr.readthedocs.io/en/stable/developers/contributing.html
• https://numcodecs.readthedocs.io/en/stable/contributing.html

so it's obvious to developers and contributors where new stuff should go.

[d-v-b]

I do think we should reflect on the zarr python 2.x experience a bit before we start adding more optional codecs / stores to this library. Zarr Python 2.x supported a bunch of storage backends (sqlite, mongodb, lmbd, etc). While adding these stores individually was easy, in sum these packages became tedious to maintain. It can become a slippery slope very easily -- when you added N-1th extra feature, it's hard to say no the the Nth.

We intended to learn from this experience, but I worry that we are moving in the wrong direction (e.g., with obstore integration). IMO our default should be that any proposed codec or store that brings in its own set of dependencies is developed in a separate package. We should aim to keep zarr-python as narrowly scoped as possible, while also ensuring that we define APIs that external packages can plug into.

[dstansby]

Agreed about the need to avoid creeping maintenance burden here. We do need a set of core codecs to function as a zarr implementation, and for codecs in v3 there is a specific set that is recommended by the specification: https://zarr-specs.readthedocs.io/en/latest/v3/codecs/index.html

So we could say for v3:

• Any codecs recommended by the v3 spec can (and should) be implemented in zarr-python, they cannot have compiled code but they can depend on other libraries that have widely available Python distributions on PyPI and conda-forge (including numcodecs)
• We will maintain a single CPU implementation of each 'core' codec
• Any codecs not recommended by the v3 spec should live in other packages, but zarr-python should provide tooling for them to be used.

for v2:

• The supported v2 codecs are what are currently in numcodecs, we will not be adding support for any more v2 codecs or for v2 codecs defined outside of numcodecs

In terms of where this leaves the numcodecs <> zarr-python dependency, it means we end up in a world where numcodecs' only job is to directly provide v2 codecs, and zarr-python depends only on numcodecs (not the other way round).