Skip to content

Commit 9336998

Browse files
Doc updates (#49)
* Add note about chunk size and shape. * Add note about variable length decoding performance.
1 parent 394d38e commit 9336998

3 files changed

Lines changed: 18 additions & 3 deletions

File tree

docs/architecture/zarr.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,9 @@ current supported type mappings.
2626
| `float16` | `Float16` | |
2727
| `float32` | `Float32` | |
2828
| `float64` | `Float64` | |
29-
| `bytes` | `BinaryView` | When the field name is `bbox`, mapped to WKB with EPSG:4326 CRS via the GeoArrow extension type instead |
29+
| `VariableLengthBytes` | `BinaryView` | When the field name is `bbox`, mapped to WKB with EPSG:4326 CRS via the GeoArrow extension type instead |
3030
| `r<N>` (raw bits) | `BinaryView` | |
31-
| `string` | `Utf8View` | |
31+
| `VariableLengthUTF8` | `Utf8View` | |
3232
| `numpy.datetime64[s]` | `Timestamp(Second, None)` | |
3333
| `numpy.datetime64[ms]` | `Timestamp(Millisecond, None)` | |
3434
| `numpy.datetime64[us]` | `Timestamp(Microsecond, None)` | |
@@ -53,3 +53,15 @@ group called `indexes` and search for an array of the same name as the
5353
predicate array. Currently the only index type supported are R-tree indexes
5454
generated by the [geo-index](https://github.com/georust/geo-index)
5555
library but we will continue to expand index support.
56+
57+
### Decoding performance
58+
Whenever possible it is worthwhile to avoid including VariableLengthBytes fields
59+
in your query's projection. Variable length decoding incurs a significantly
60+
higher cost than decoding fixed width types and can drastically affect query
61+
performance. This is especially evident when an index is present and the actual
62+
chunks don't need to be materialized for filter operations. See the results of
63+
```
64+
cargo bench --bench bbox_zarrs_raw
65+
```
66+
as an example.
67+

docs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,5 @@ This can present problems as systems require complex, fragile orchestration to m
2121

2222
- **[User Guide](user-guide/installation.md)** — Installation, quickstart
2323
- **[API Reference](api/zarr-table.md)**`ZarrTable` class documentation
24+
- **[Examples](examples/README.md)** - Notebook examples
2425
- **[Developer Docs](DEVELOP.md)** — Contributing and running benchmarks

docs/user-guide/quickstart.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ bbox[:] = shapely.to_wkb([
4444
shapely.box( 30.0, 30.0, 50.0, 50.0),
4545
])
4646
```
47-
47+
Note that the chunk size and shape must be the same for all of our `meta` arrays. See
48+
[zarr structure](../architecture/zarr.md) for more details on why.
49+
:
4850
## 2. Register the store
4951

5052
Use `ZarrTable.from_obstore` to open the store through

0 commit comments

Comments
 (0)