Guidance for Sourcemeta One usage/Assorted Questions

[egnwd]

Hi!

Firstly, Sourcemeta One looks like it will be well suited for my use case for hosting some JSON Schemas for services and teams, I'm excited to trial it.

I've taken a look at some of the documentation and this Github to gain some inspiration, but I've got a few questions around best practices and any tips for hosting an instance of Sourcemeta one.

For the examples used in questions, I've put up this repo: https://github.com/egnwd/jsonschema-one-testing as (fairly basic) examples in case it's useful.

1. Based on this I'm assuming @sourcemeta/std/v0 is only a built-in extension for enterprise licenses, is this correct? I don't see this information on the commercial licensing page about this, also it's used in the examples which makes it slightly confusing.
2. I'm looking for any guidance/wisdom on how to structure the schemas. My example shows a common folder (e.g. eventbridge envelopes) and then service folders for services to define their event schemas/data schemas etc. This works well, but seems fragile as I have to ensure my one collections match my folder structure as otherwise I cannot build the docker image. For example, changing the collection name in schemas/common/one.json to common2 gives me the error:

0.551 error: Could not resolve the reference to an external schema
0.551   at identifier https://schemas.company.co/common/aws/eventbridge
0.551   at path /source/schemas/service-a/new-data-event.schema.json
0.551
0.551 Did you forget to register a schema with such URI?

which can be resolved by updating the path in service-a's schema but this then breaks local usages of jsonschema validate on the local schemas.

error: Could not resolve the reference to an external schema
  at identifier file:///Users/elliot/projects/playground/jsonschema-one-testing/schemas/common2/aws/eventbridge.json
  at file path /Users/elliot/projects/playground/jsonschema-one-testing/schemas/service-a/new-data-event.schema.json

is there a best practice for how to make this less fragile or do I just need to ensure my local directories match the collections I wish to host? I've seen for the self/v1 example you somehow copy some the schemas for the @sourcemeta/std/v0 rather than just referencing the original.
3. For multi-private-repo set-ups, do you have any suggestions on how best to collate the schemas? It seems intuitive to me that the schemas should live close to the services that developers are writing, but that means I need to somehow bump the schemas repo on changes. It feels like this is how the catalog becomes stale fairly quickly, at least while still trying to gain adoption. I wondered if you had any insight on solving this, as using JSON schemas in such a manner is fairly new to me!
4. I'm interested in the roadmap item "Schema evolution and transforms" - I think one thing I'd like for my use case is the ability to version the schemas so developers can safely add features without breaking downstream systems and it looks as this could be beneficial.
5. It looks as though having a base URL of something like https://catalog.company.co/schemas means that a lot of references are breaking, is this a known limitation or should such a base URL be supportable within Sourcemeta One?

Also, I think "Includes" is a mistake in the docs? It should be include as per the schema?

I'm happy to split this out into separate questions/issues if it's easier, but I thought I'd start with them all in one place.
Thanks

[jviotti]

Hey @egnwd ,

Thanks for reaching out! Let me unpack the comment!

Based on this I'm assuming @sourcemeta/std/v0 is only a built-in extension for enterprise licenses, is this correct? I don't see this information on the commercial licensing page about this,

Yes, the standard library is part of the Enterprise offering. We typically offer it to enterprises as part of their governance initiatives, so that they can start with a solid foundation on day 1. Not sure what's your use case but, the standard library is also free to use for non-commercial / open-source projects and specifications.

Technically, nothing the standard library offers is impossible to do yourself. The benefit of getting it from us is that we pay for the purchasing many of the standards, we do extensive unit testing of every schema, performance tuning, compliance checks, and on-going maintenance. Additionally, for customers on the Enterprise plan, we extend the standard library with coverage for any standard they need, which often saves companies a lot of schema writing work.

also it's used in the examples which makes it slightly confusing.

Good point about the docs. Thanks for noticing. We'll fix it 🙏🏻

is there a best practice for how to make this less fragile or do I just need to ensure my local directories match the collections I wish to host?

I would advise having multiple one.json files and extends unless your setup is a lot bigger and complex (which in the vast majority of cases its not). For inspiration, see https://github.com/sourcemeta/one/blob/main/public/one.json, which is the single one.json we use to power https://schemas.sourcemeta.com. Does that help?

I've seen for the self/v1 example you somehow copy some the schemas for the @sourcemeta/std/v0 rather than just referencing the original.

That's a bit of an internal workaround, just to not ship the entire Enterprise standard library on the open-source version :) We just pick and ship for free the ones that the registry API itself uses. Probably not a good idea to copy that!

3. For multi-private-repo set-ups, do you have any suggestions on how best to collate the schemas? It seems intuitive to me that the schemas should live close to the services that developers are writing, but that means I need to somehow bump the schemas repo on changes. It feels like this is how the catalog becomes stale fairly quickly, at least while still trying to gain adoption. I wondered if you had any insight on solving this, as using JSON schemas in such a manner is fairly new to me!

I think there are two ways of doing it that we've seen in the wild:

• You can have a master git repo with one.json and include each project as a git submodule or something like that. If each of your service has a schemas folder, then you configure collections to pull from that. You could also script it out on the Dockerfile. For example, we saw people with a Dockerfile with several git clone statements, copying the schemas, and then building against the master one.json

• That said, we usually find that services maintaining their own schemas leads to the schema silos that a registry is supposed to prevent. The most successful teams we have seen focus on centrally maintaining company wide schemas in a single repo (usually maintained by a platform team), which services consume as a foundation instead. On consuming, you can use jsonschema install, or just directly $ref the schemas from i.e. OpenAPI specs and bundle with redocly bundle, etc.

The latter is largely the "governance" recipe, so I would definitely suggest keeping it in mind. It works great in our experience, and services can always have "custom" schemas on top as needed, while periodically upstream them to the central schema/governance repository.

4. I'm interested in the roadmap item "Schema evolution and transforms" - I think one thing I'd like for my use case is the ability to version the schemas so developers can safely add features without breaking downstream systems and it looks as this could be beneficial.

Definitely. We have some tooling already and this year's Google Summer of Code under the official JSON Schema organisation (starting next month) is running a project on integration this tooling on Sourcemeta offerings. So hopefully ready very soon!

5. It looks as though having a base URL of something like https://catalog.company.co/schemas means that a lot of references are breaking, is this a known limitation or should such a base URL be supportable within Sourcemeta One?

This is a good point. I don't recall anybody mounting the schema on a specific path. To be honest, I'm not sure and we don't run this on the suite, but we'll test it out today and make any fixes, if needed.

Also, I think "Includes" is a mistake in the docs? It should be include as per the schema?

Ah, another great catch. You are quite an attentive reader! 🙏🏻 I wish we could run CI checks on the docs examples somehow!

[egnwd]

Yes, the standard library is part of the Enterprise offering.

👍🏻

I would advise having multiple one.json files and extends unless your setup is a lot bigger and complex

I can stick to one to start with, I think though the same principle applies however, the collections within one.json would need to match the folder structure in the schemas folder in order to be able to also compile the schemas locally. That might be fine to start with, I'll see if I run into any issues with this in practice.

The most successful teams we have seen focus on centrally maintaining company wide schemas in a single repo

Interesting, I'll explore this - thank you

[jviotti]

As a quick update, we fixed the documentation things you raised, and the base URL including a path component is now fully supported in https://github.com/sourcemeta/one/releases/tag/v5.3.0! We added significant tests for the latter, but if you still find anything odd, just let us know!

[egnwd]

Definitely builds now, the self/v1 schemas no longer appear in the "Special directories". Whilst I don't mind necessarily I thought I'd let you know in case this was not intentional

[jviotti]

Ah, good catch. See https://github.com/sourcemeta/one/releases/tag/v5.3.1. Just fixed that little "Special directories" case.

Also, we kickstarted https://one.sourcemeta.com/guide/ as a longer guide on the problem we are trying to solve and the approach we suggest. It is not complete, but any feedback would be very welcomed. Hopefully it answers some of your questions too.

[jviotti]

@egnwd Also, I saw the you consume EventBridge schemas: https://github.com/egnwd/jsonschema-one-testing/blob/main/schemas/common/aws/eventbridge.json. Would it be helpful to provide these as built-in collections you can directly extend?

[egnwd]

For sure, I could see various envelope types being useful as built ins

[jviotti]

Looks like AWS does not publish authoritative schemas for EventBridge that we can pre-package here. Or am I missing something? Did you manually create the one in there? Is there any other one than EventBridge that you use or plan to use from any upstream sources?

[egnwd]

AWS publish their schemas in Eventbridge schema registry, they are bundled versions so you get the envelope and the details in one schema, I abstracted that to this envelope schema so it's easier for devs to reuse.

SNS, SQS all have similar envelopes when you consume them so I imagine the whole package would be useful, though eventbridge is our main use case

[jviotti]

Right. I guess the challenge is that they don't easily publish the schemas somewhere else we can pull them. Sounds like we would need to run the AWS services to extract them, and continue doing so if the schemas change? It'd be nice if they also published the schemas in a git repo, or something.