[DOC] [RFC] Add Schema Documentation for Migration Assistant for OpenSearch

[gregschohn]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request.

Background

Users configure the Migration Assistant for OpenSearch with json/yaml files that need to match a specific schema. If the documents are not well-formed to the schema, the workflow won't run and the user cannot proceed.

The Migration Assistant applications use schema verification and the project release artifacts include a schema. Each of these schema files also includes in-depth documentation for each field, making it an invaluable reference. However, schemas aren't easy for humans to digest and the schema is too big (~500KB) for humans to read without tools since it includes complete definitions from dependencies that some users may need to be aware of. Users need a reference to go to figure out how to specify their migration. Having deep links to specific fields provides a great way to help others. Since the Migration Assistant doesn't maintain an HTTP service to provide web-content, it doesn't have a natural way to vend rich content to its users.

As the Migration Assistant evolves, its schema changes too. Users will be distributed across a variety of users. The Migration Assistant maintainers want to make sure that each user can see exactly how the schema is defined for the version that they're using. It would also be helpful to know what's changed between versions of the Migration Assistant.

The Migration Assistant's release cadence is not tied to the OpenSearch project's. As such, the projects have different version numbers and release at different times and frequencies.

Proposal

This proposal is to add a new page under https://docs.opensearch.org/latest/migration-assistant/ to show the schema. Users should see the latest version by default and should have the choice to see older versions. It would be nice to also have the ability to see differences between the schemas.

We propose to implement the schema view page by dynamically loading the schema in the browser and using javascript to render it. Javascript allows the user to collapse and expand specific sections, easily inline $refs, and search through an otherwise formidable tree.

We're open to how this is integrated into the page and want to maintain the same look and feel. We'd like to see some new patterns though, which can be summarized in this sample frame, which we're hoping to embed some form of into the documentation website.

Implementation

There are a variety of ways we can deliver the schema content from the opensearch-migrations GitHub artifacts through the documentation website. We can open a PR with the new schema for every Migration-Assistant release, though we'd like to see that eventually improve. Our release cadence has been up to weekly.

Version: List the OpenSearch version to which this issue applies, e.g. 2.14, 2.12--2.14, or all.
Migration Assistant v3.0+

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
The raw schema can be found here

[kolchfa-aws]

Thank you for raising this proposal, @gregschohn. I have some follow-up questions about this addition:

1. Currently, the doc site has minimal JavaScript and is mostly static HTML and CSS. How would a 500-KB schema and a rendering library affect load times of the doc site, especially on mobile devices?
2. JavaScript is difficult to navigate for screen readers so leads to less than ideal accessibility. An alternative would be generating static HTML at build time (for example, using a details block).
3. This is an interactive reference tool. Search, version comparison, tree navigation, and detail panels are features of a standalone app that has significant scope beyond documentation. Should we host this interactive page on GitHub pages instead and automatically update it with each release and link to the page from the doc site?
4. The left navigation of the new JavaScript tool can be confusing because we already have a left navigation on the doc site. How will it look on mobile screens?
5. How would the search bar be related to the search bar on the doc site and how would the search be implemented? We currently have a search indexer that runs at build time. Would the new search integrate with the doc site search or be independent? Also, content rendered on the client side by JavaScript is not indexed by the site's build-time search indexer, so schema field names would not appear in the doc site search results.

[wrigleyDan]

I think this addition can be useful not only for the Migration Assistant workflow schemas but also for the User Behavior Insights schema, currently maintained under https://github.com/o19s/ubi and also with a release cadence independent from the OpenSearch project's.

If runtime is a concern to parse and dereference this could be done as part of an workflow after delivering the schema. That would remove one dependency and some latency loading the schema.

[wrigleyDan]

We built a small PoC last week so I'll try and address some of the questions you raised @kolchfa-aws

Currently, the doc site has minimal JavaScript and is mostly static HTML and CSS. How would a 500-KB schema and a rendering library affect load times of the doc site, especially on mobile devices?

The schema only loads when a user navigates specifically to the schema page so it has no effect on the rest of the doc site. The schema compresses from 568 KB on disk to 41.5 KB over the wire (gzip). I did some basic performance measurements with Chrome Dev Tools against the current implementation:

• No throttling: 176 ms total page load, 12 ms schema fetch
• Fast 4G: 1.09 s, 212 ms
• Slow 4G: 3.67 s, 801 ms

These numbers reflect an unoptimised state from a quickly built PoC where the viewer fetches a rendering library from a CDN and resolves $ref pointers in the browser on every load. One optimisation (already indicated in my last comment) is a pre-resolution step where schema $ref pointers are resolved at publish time removing the CDN dependency. This can reduce load times. The schema file itself is the only meaningful payload and it can be compressed efficiently.

JavaScript is difficult to navigate for screen readers so leads to less than ideal accessibility. An alternative would be generating static HTML at build time (for example, using a details block).

I'm imagining this static approach generating one page per schema version (e.g. /workflow-schema/3.2.1/). A user only downloads the page they navigate to and other versions are separate URLs that are loaded on demand.
I assume the build time grows with each new version because Jekyll renders the full schema tree for each but I cannot estimate the actual impact this might have. It may well be negligible considering the total build time. Probably more importantly, (live) search is lost helping users navigate the tree quickly because that is a JavaScript feature.

What I could imagine: generate the static <details> tree as the accessible baseline, then layer the JavaScript viewer on top for users who have JS enabled. This gives screen reader users a fully readable page while preserving the interactive experience for everyone else.

This is an interactive reference tool. Search, version comparison, tree navigation, and detail panels are features of a standalone app that has significant scope beyond documentation. Should we host this interactive page on GitHub pages instead and automatically update it with each release and link to the page from the doc site?

I see the main disadvantage being the context switch when users have to leave the doc site to consult the schema reference. If users look at the schema reference as part of the documentation this might feel very disconnected and keeping it on the same domain would be preferable. If users are expected to look at the schema reference without reading any other documentation pages Github Pages can indeed be a valid alternative. Could a Github Pages version be somehow embedded in the documentation?

The left navigation of the new JavaScript tool can be confusing because we already have a left navigation on the doc site. How will it look on mobile screens?

I'll follow up with mockups showing the look and feel of the schema viewer in the docs.

How would the search bar be related to the search bar on the doc site and how would the search be implemented? We currently have a search indexer that runs at build time. Would the new search integrate with the doc site search or be independent? Also, content rendered on the client side by JavaScript is not indexed by the site's build-time search indexer, so schema field names would not appear in the doc site search results.

The search bar in the schema viewer is intentionally independent: it is a live tree filter that narrows the sidebar as the user types, matching against field names. So the search function in the schema viewer has no relation to the doc's build-time search index.

I assume users searching the doc site are more likely looking for conceptual documentation ("how do I configure authentication") and not expect field names. However it would be interesting to see if property names are already searched for and it might make sense to watch future search traffic for field names.

[wrigleyDan]

Here's a mockup showing how mobile devices could render the schema viewer. The view is split horizontally, not vertically which makes sense for mobile devices I think: