Add benchmarks for sharded + local store indexing

[aldenks]

Parameterize with shards and no shards. Also parameterize with local + memory store to have an example of a store which has some modest latency.

When testing performance of optimizing sharded zarr reads I found that the memory store is too fast to show any difference, but the local store makes performance improvements there very visible. The extra parameterizations do expand the test space; test_slice_indexing with benchmarking on now takes ~30 seconds vs ~7 seconds on main.

[codspeed-hq]

Merging this PR will not alter performance

✅ 24 untouched benchmarks
🆕 24 new benchmarks
⏩ 6 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
🆕	WallTime	test_slice_indexing[None-(slice(None, None, None), slice(None, None, None), slice(None, None, None))-memory]	N/A	502.6 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory]	N/A	498.2 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory]	N/A	272.5 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory]	N/A	286.6 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory]	N/A	4.7 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory]	N/A	7.5 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(0, 0, 0)-memory]	N/A	1.9 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(None, 10, None), slice(None, 10, None), slice(None, 10, None))-memory]	N/A	932.5 µs	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(None, None, None), slice(None, None, None), slice(None, None, None))-memory]	N/A	546.6 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(None, 10, None), slice(None, 10, None), slice(None, 10, None))-memory]	N/A	2 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory]	N/A	541.4 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory_get_latency]	N/A	5.1 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency]	N/A	301.4 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(None, None, None), slice(None, None, None), slice(None, None, None))-memory_get_latency]	N/A	556.1 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(None, 10, None), slice(None, 10, None), slice(None, 10, None))-memory_get_latency]	N/A	4.3 ms	N/A
🆕	WallTime	test_slice_indexing[None-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory_get_latency]	N/A	552.2 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(0, 0, 0)-memory_get_latency]	N/A	4.2 ms	N/A
🆕	WallTime	test_slice_indexing[None-(0, 0, 0)-memory]	N/A	907.1 µs	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(10, -10, 4), slice(10, -10, 4), slice(10, -10, 4))-memory_get_latency]	N/A	314.1 ms	N/A
🆕	WallTime	test_slice_indexing[(50, 50, 50)-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory_get_latency]	N/A	8.3 ms	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---

_{Comparing aldenks:shard-slice-indexing-benchmark (6271050) with main (3fa19d2)}

Footnotes

1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[d-v-b]

if we want more control over the latency, we could wrap memory storage with a LatencyStore here. This would keep the benchmark a bit more isolated from stuff the OS is doing than localstore. Happy to implement this if you don't mind me pushing to this branch

[aldenks]

@d-v-b go ahead! If you make it I can't be accused of juicing my numbers by picking high latencies ;p

In all seriousness, that would be quite useful for picking a good default coalesce_max_gap_bytes.

[d-v-b]

with .0001s of latency on get requests against a MemoryStore, we get performance that's ~6x slower than a regular MemoryStore. This is actually much slower than local storage, but it should be a more durable metric, if we assume that local storage devices get faster, faster than system memory

[aldenks]

Thank you @d-v-b!