Add ObjectStoreRegistry to support default stores

[maxrjones]

This PR makes it possible to have an empty dict in ManifestStores._stores and automatically create an S3Store or LocalStore from a filepath.

Part of #498

• Closes #xxxx
• Tests added
• Tests passing
• Full type hint coverage
• Changes are documented in docs/releases.rst
• New functions/methods are listed in api.rst
• New functionality has documentation

[sharkinsspatial]

@maxrjones Can you include the drop_variables arg in each of these overloads. It needs to be passed down to the _construct_manifest_group call (see https://github.com/zarr-developers/VirtualiZarr/pull/542/files#diff-e89ebbf07b078f47cd46e9b829b2513b7f56a8193af6d09cf045e2669c61a1dbR170)

[maxrjones]

I'd rather address this comment separately if that's alright to keep this PR isolated to supporting default store instances

[maxrjones]

Thanks for the reviews, all. FYI I want to revise this to follow the same design as https://docs.rs/datafusion/latest/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html. So a ManifestStore would have a ObjectStoreRegistry which would implement the register_store and get_store methods. But I've been a bit busy with other tasks and probably won't be able to return to this PR until tomorrow afternoon.

[maxrjones]

I think this PR is ready for a round of reviews if anyone has some extra time 🙏

[TomNicholas]

Thanks for doing this @maxrjones !

FYI I want to revise this to follow the same design as https://docs.rs/datafusion/latest/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html

Cool idea. Nice to make that connection.

The main thing that's missing from this PR is documentation of how to instantiate ManifestStore in different scenarios. That could either be done now (perhaps as examples in the ManifestStore.__init__ docstring) or it could be left until later. We are building up technical debt in this refactor in the sense that we will need to go back through and document how everything now works.

[maxrjones]

The main thing that's missing from this PR is documentation of how to instantiate ManifestStore in different scenarios. That could either be done now (perhaps as examples in the ManifestStore.init docstring) or it could be left until later. We are building up technical debt in this refactor in the sense that we will need to go back through and document how everything now works.

sounds good, I can do this now.

[maxrjones]

@sharkinsspatial with using the scheme + bucket for the ObjectStoreRegistry keys, we no longer need the prefix argument for _create_manifest_store since it can be inferred from the filepath. 🙇 to @kylebarron for this suggested approach

[maxrjones]

@TomNicholas I'm actually going to use your approval to merge this an add the additional docs separately as a new "How to build a reader" section of the contrib guide.

[TomNicholas]

add the additional docs separately as a new "How to build a reader" section of the contrib guide.

This is actually tracked already in #452 FYI