Developing new providers

[axelboc]

The need for developing new data providers is starting to become a priority. I'm opening this discussion thread in response notably to this request for a new provider for Carteva2 (ping @FrancescAlted).

Should new providers be developed inside the H5Web monorepo or outside?

Integrating a provider into the monorepo means that we, as core maintainers, agree to take on the responsibility of maintaining the provider in the long term (since external contributors may come and go). For this to happen, we would expect for the provider to benefit a significant number of users and also meet a certain level of maturity and quality: reproducible test environment (c.f support folder), basic smoke tests against the sample.h5 file (c.f packages/app/src/providers/h5grove/h5grove-api.test.ts), etc.

Our preference is therefore for new providers to first be developed outside of the H5Web monorepo until they reach a good level of maturity and are proven to work reliably in a production environment (with security and performance concerns in mind, notably).

What does it involve to develop a new provider outside of the H5Web monorepo?

You would typically need to:

1. Set up a fresh git repository for front-end React development — see for instance the repository of myHDF5.
2. Set up a testing environment of the back-end solution that the new provider is supposed to interface with. Ideally anyone cloning/forking the repo should be able to quickly set up this testing environment. If this is not possible, then it would be good to at least expose a test instance publicly with some test data. Our very own sample.h5 file, which we use for testing the providers, can be downloaded here and used as test data. (This server has many other test files to play with too.)
3. Install the @h5web/app package and follow the Getting started guide to have a simple React app to start with.
4. Develop and test the new provider by following the model of our most mature, server-based provider, H5GroveProvider.

We really don't recommend forking the H5Web monorepo and developing the new provider into it, as this may become a maintenance nightmare for you if, for some reason we decide not to include the provider in the monorepo or if this decision takes a long time. Our recommendation is to develop the provider in a fresh repository.

Since we have not yet developed providers outside of the monorepo, it is likely that some of the pieced needed (utilities, types, etc.) won't be exposed through the @h5web/app package. If this happens, feel free to ping us in this thread and we will do our best to expose the missing pieces and release a beta version of @h5web/app (i.e. without release notes) as quickly as possible, especially if duplicating the required code in your own repo doesn't work.

We have recently released a beta version, v15.0.0-beta.0, that includes breaking changes to the providers that have not yet been officially released. Make sure to keep an eye on PRs and on the tags page to stay up to date.

[FrancescAlted]

We have been having a look into your instructions for building a new provider, and even that it is clear that this is the more flexible way to go, that would take us quite a bit of effort to tackle that (React knowledge is scarce among us). On the other hand, emulating the OpenAPI in h5grove would be much quicker to implement, and that would be enough for allowing Caterva2 users to access h5web functionality. So, we probably will try the latter first and see how it goes.

Thanks!

[axelboc]

Sounds great! The complexity might come when optimising the data transfer, as you won't have as much wiggle room to work with on the H5GroveProvider side. But it's definitely worth seeing how far you can go with this solution, for sure. Keep us in the loop!