General Questions

[buildgreatthings]

I want to check my understanding of this proposed schema:

The spec spans model design, model deployment, and model monitoring.

The json file originates when the PyTorch, XGB, or TF completes a model training. Typically a model artifact is created by doing xgb.save(). A complementary function xgb.save_data_schema() could be implemented that saves this JSON output.

When the model is deployed on an inference server, data could be validated against this. /metadata URI on a model server would also return metadata in using subset of this structure.

Within a model monitoring utility, the schema could be read from the provider to type the data attributes to inference outputs.

[ukclivecox]

Yes I think that all makes sense and is correct. The tools to create the schema would need to be added or it would need to be created by hand by the data scientist. The connection with V2 schema is whether we want to extend it to allow this to be returned from /metadata but even then it would need to be in a form that allows a client to correctly use this to create a V2 Schema request which might need help in V2 client libraries. Others @SachinVarghese @jklaise @axsaucedo @adriangonz may want to comment.

[buildgreatthings]

I agree that a metadata format file can inform the API calls and protobuf/JSON body construction of a user.

Without doling out work to anyone, I propose we think about stabilizing the V2 spec and design the client experience. I suggest using a good template for open source collaboration that encourages and documents conversation amongst many different stakeholders.

There are two parts (maybe more?). First, is design of the metadata schema. We should start reaching out to the PyTorch, FastAI and XGB contributors as we formalize this into a spec for feedback and implementation (I can take this one). Second is a client experience. The bugs I discovered came from understandable and minor deltas between described behavior and actual. The client SDK should also serve an integral part of the testing suite for model servers that desire compliance.

The simplest user story is that summarizes this is:

As an application developer, I can be given the metadata file, endpoint URI, and body type (GRPC, JSON), and I don't have to write boilerplate code to run an inference invocation.

A number of enterprise customers have something similar in heavier forms. Their internal SDKs also handle CRUD interaction with the model registry, model training, and model deployment infrastructure.

[buildgreatthings]

GCP has a concept around this called Schemata

https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.models#predictschemata

[buildgreatthings]

MLFLow refers this as a Model Signature

https://www.mlflow.org/docs/latest/_modules/mlflow/models/signature.html