Skip to content

chunk_grids.md

Latest commit

 

History

History
154 lines (115 loc) · 5.24 KB

chunk_grids.md

File metadata and controls

154 lines (115 loc) · 5.24 KB
Raw
:tocdepth: 3

Chunk Grid Models

:author: Altay Sansal
:date: "{sub-ref}`today`"
:read-time: "{sub-ref}`wordcount-minutes` min read"
:class-container: sd-p-0 sd-outline-muted sd-rounded-3 sd-font-weight-light

The variables in MDIO data model can represent different types of chunk grids. These grids are essential for managing multi-dimensional data arrays efficiently. In this breakdown, we will explore four distinct data models within the MDIO schema, each serving a specific purpose in data handling and organization.

MDIO implements data models following the guidelines of the Zarr v3 spec and ZEPs:

Regular Grid

The regular grid models are designed to represent a rectangular and regularly paced chunk grid.

.. autosummary::
   RegularChunkGrid
   RegularChunkShape

For 1D array with size = 31{l=python}, we can divide it into 5 equally sized chunks. Note that the last chunk will be truncated to match the size of the array.

{ "name": "regular", "configuration": { "chunkShape": [7] } }{l=json}

Using the above schema resulting array chunks will look like this:

 ←─ 7 ─→ ←─ 7 ─→ ←─ 7 ─→ ←─ 7 ─→  ↔ 3
┌───────┬───────┬───────┬───────┬───┐
└───────┴───────┴───────┴───────┴───┘

For 2D array with shape rows, cols = (7, 17){l=python}, we can divide it into 9 equally sized chunks.

{ "name": "regular", "configuration": { "chunkShape": [3, 7] } }{l=json}

Using the above schema, the resulting 2D array chunks will look like below. Note that the rows and columns are conceptual and visually not to scale.

 ←─ 7 ─→ ←─ 7 ─→  ↔ 3
┌───────┬───────┬───┐
│       ╎       ╎   │  ↑
│       ╎       ╎   │  3
│       ╎       ╎   │  ↓
├╶╶╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
│       ╎       ╎   │  ↑
│       ╎       ╎   │  3
│       ╎       ╎   │  ↓
├╶╶╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
│       ╎       ╎   │  ↕ 1
└───────┴───────┴───┘

Rectilinear Grid

The RectilinearChunkGrid model extends the concept of chunk grids to accommodate rectangular and irregularly spaced chunks. This model is useful in data structures where non-uniform chunk sizes are necessary. RectilinearChunkShape specifies the chunk sizes for each dimension as a list allowing for irregular intervals.

.. autosummary::
   RectilinearChunkGrid
   RectilinearChunkShape

:::{note} It's important to ensure that the sum of the irregular spacings specified in the chunkShape matches the size of the respective array dimension. :::

For 1D array with size = 39{l=python}, we can divide it into 5 irregular sized chunks.

{ "name": "rectilinear", "configuration": { "chunkShape": [[10, 7, 5, 7, 10]] } }{l=json}

Using the above schema resulting array chunks will look like this:

 ←── 10 ──→ ←─ 7 ─→ ← 5 → ←─ 7 ─→ ←── 10 ──→
┌──────────┬───────┬─────┬───────┬──────────┐
└──────────┴───────┴─────┴───────┴──────────┘

For 2D array with shape rows, cols = (7, 25){l=python}, we can divide it into 12 rectilinear (rectangular bur irregular) chunks. Note that the rows and columns are conceptual and visually not to scale.

{ "name": "rectilinear", "configuration": { "chunkShape": [[3, 1, 3], [10, 5, 7, 3]] } }{l=json}

 ←── 10 ──→ ← 5 → ←─ 7 ─→  ↔ 3
┌──────────┬─────┬───────┬───┐
│          ╎     ╎       ╎   │  ↑
│          ╎     ╎       ╎   │  3
│          ╎     ╎       ╎   │  ↓
├╶╶╶╶╶╶╶╶╶╶┼╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
│          ╎     ╎       ╎   │  ↕ 1
├╶╶╶╶╶╶╶╶╶╶┼╶╶╶╶╶┼╶╶╶╶╶╶╶┼╶╶╶┤
│          ╎     ╎       ╎   │  ↑
│          ╎     ╎       ╎   │  3
│          ╎     ╎       ╎   │  ↓
└──────────┴─────┴───────┴───┘

Model Reference

:::{dropdown} RegularChunkGrid :animate: fade-in-slide-down

.. autopydantic_model:: RegularChunkGrid

----------

.. autopydantic_model:: RegularChunkShape

::: :::{dropdown} RectilinearChunkGrid :animate: fade-in-slide-down

.. autopydantic_model:: RectilinearChunkGrid

----------

.. autopydantic_model:: RectilinearChunkShape

:::