diff --git a/docs/architecture/zarr.md b/docs/architecture/zarr.md index 8b011d0..4775cf1 100644 --- a/docs/architecture/zarr.md +++ b/docs/architecture/zarr.md @@ -26,9 +26,9 @@ current supported type mappings. | `float16` | `Float16` | | | `float32` | `Float32` | | | `float64` | `Float64` | | -| `bytes` | `BinaryView` | When the field name is `bbox`, mapped to WKB with EPSG:4326 CRS via the GeoArrow extension type instead | +| `VariableLengthBytes` | `BinaryView` | When the field name is `bbox`, mapped to WKB with EPSG:4326 CRS via the GeoArrow extension type instead | | `r` (raw bits) | `BinaryView` | | -| `string` | `Utf8View` | | +| `VariableLengthUTF8` | `Utf8View` | | | `numpy.datetime64[s]` | `Timestamp(Second, None)` | | | `numpy.datetime64[ms]` | `Timestamp(Millisecond, None)` | | | `numpy.datetime64[us]` | `Timestamp(Microsecond, None)` | | @@ -53,3 +53,15 @@ group called `indexes` and search for an array of the same name as the predicate array. Currently the only index type supported are R-tree indexes generated by the [geo-index](https://github.com/georust/geo-index) library but we will continue to expand index support. + +### Decoding performance +Whenever possible it is worthwhile to avoid including VariableLengthBytes fields +in your query's projection. Variable length decoding incurs a significantly +higher cost than decoding fixed width types and can drastically affect query +performance. This is especially evident when an index is present and the actual +chunks don't need to be materialized for filter operations. See the results of +``` +cargo bench --bench bbox_zarrs_raw +``` +as an example. + diff --git a/docs/index.md b/docs/index.md index 7ac0a7f..cb0db2f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,4 +21,5 @@ This can present problems as systems require complex, fragile orchestration to m - **[User Guide](user-guide/installation.md)** — Installation, quickstart - **[API Reference](api/zarr-table.md)** — `ZarrTable` class documentation +- **[Examples](examples/README.md)** - Notebook examples - **[Developer Docs](DEVELOP.md)** — Contributing and running benchmarks diff --git a/docs/user-guide/quickstart.md b/docs/user-guide/quickstart.md index ab11582..fd7bec3 100644 --- a/docs/user-guide/quickstart.md +++ b/docs/user-guide/quickstart.md @@ -44,7 +44,9 @@ bbox[:] = shapely.to_wkb([ shapely.box( 30.0, 30.0, 50.0, 50.0), ]) ``` - +Note that the chunk size and shape must be the same for all of our `meta` arrays. See +[zarr structure](../architecture/zarr.md) for more details on why. +: ## 2. Register the store Use `ZarrTable.from_obstore` to open the store through