Add n5_varlen codec for N5 varlength block format

[mkitti]

Summary

Registers the n5_varlen codec, the varlength-mode counterpart to the existing n5_default codec.

• Handles N5 blocks with mode = 0x0001 (varlength)
• Wraps an inner codec pipeline (array-to-bytes + optional bytes-to-bytes) with the N5 varlength block header
• Intended for variable-width data types where the per-chunk byte size cannot be derived from the chunk shape alone (e.g. label_multiset)

N5 varlength header format

Offset    Size    Endian  Field
------    ------  ------  ----------------------------------------
0         2       BE      mode      uint16  = 0x0001 (varlength)
2         2       BE      ndim      uint16  number of dimensions
4         4·ndim  BE      dims[]    uint32 each  block shape
4+4·ndim  4       BE      num_bytes uint32  byte count of payload

Codec chain (example with label_multiset)

{
  "name": "n5_varlen",
  "configuration": {
    "codecs": [
      {"name": "n5_label_multiset"},
      {"name": "gzip", "configuration": {"level": 6}}
    ]
  }
}

Test plan

• Confirm codecs/n5_varlen/schema.json validates correctly
• Verify the annotated binary layout against a real N5 varlength block (see README)

🤖 Generated with Claude Code

[mkitti]

@clbarnes here is the variable length N5 codec. Could you review?

[clbarnes]

default-mode... header... contains the number of array elements.

I don't think this is correct, default mode blocks just have a shorter header (according to the spec anyway).

[mkitti]

I see how this can be confusing. We were referring to the number of array elements in each dimension described in the third field.

[clbarnes]

Is there a canonical description of this somewhere or is it gleaned from the implementation? Either way a link would be great.

[clbarnes]

Not strictly related to this schema, but what is the num_bytes field actually for? You're reading to the end of the block anyway because you don't know where each value is.

[clbarnes]

I know that this is the same pattern used by the n5_default codec, where the inner codecs describes the whole pipeline for the payload, but in this case we're having to invent (and specify and implement) a second new codec n5_label_multiset which will only ever be used in this n5_varlen context, and n5_varlen will always use this n5_label_multiset.

With n5_default, I went back and forth on whether to explicitly include the transpose and bytes codecs or whether to more closely match N5 by handling them in the outer codec and just having an optional compressor field which would contain one bytes-to-bytes codec. I settled on including the whole chain because it meant the whole thing could be delegated to core-spec codecs. In this case, I'd say there's more argument for having more of the logic in the n5_varlen codec and just an optional compressor in the configuration.

But I'm ambivalent about it.

[mkitti]

I'd lean toward keeping the current design with a separate n5_label_multiset codec for a few reasons:

1. Consistency with n5_default: Both use an inner codecs array for the full payload pipeline. Changing n5_varlen to a compressor-only config would make the two codecs asymmetric.

2. n5_varlen stays data-type agnostic: The varlength block mode isn't specific to label_multiset. zarr-developers/zarr-specs#3 mentions per-chunk histograms (of the values in the ROI corresponding to that chunk) as another N5 varlength use case — a type where different chunks can have different sizes. If other variable-length types are added in the future they can slot in as a different inner array-to-bytes codec without needing a new outer codec.

3. Discoverability: Having n5_label_multiset as an explicit named codec in the codecs array means it can be referenced, documented, and implemented independently.

Happy to discuss further if you'd prefer the embedded approach.

Drafted with assistance from Claude (Anthropic).