Are there any libs that implement schema validation in a streaming fashion, e.g., not loading the whole object in memory?

[mezhaka]

I have put a 400 experience point bounty on that question (https://stackoverflow.com/q/79389053/18775) with some valuable comments from @jeremyfiel (btw. I could not join slack channel for some reason). I just wanted to cross-post it here in hope to get more details on that topic. I appologize if this is not the place to discuss such matters.

I have a large JSON that I do not want to load into memory. I would like to validate it against a JSON schema in a streaming fashion. All libraries I could find so far, only validate completely loaded JSON objects (like Pydantic or https://github.com/python-jsonschema/jsonschema). What I rather need is some way to validate it feeding the original JSON chunk by chunk, i.e, control the size of the buffer.

This could look like this:

import pydantic  # I use V2
import ijson
import pathlib

class A(pydantic.BaseModel):
    i: int
    a: list[int]
    s: str

jsonpath = pathlib.Path("some.json")
validator = MyValidator(schema=A.model_json_schema())
with jsonpath.open("rb") as file:
    for prefix, event, value in ijson.parse(file, use_float=True):
        validator.event((prefix, event, value))

print(validator.errors)

Imagine some.json file is ~50 MB large A instance with a very long array. I do not want to load the whole object into memory (this is what Pydantic would do), but I want to make sure that some.json complies with the schema of A. The validator.errors could give me a list of errors which is going to be empty in cases none where discovered.

[EDIT 2025-01-31] The term "streaming fashion" means that event is seen exactly once and there is no way to see it again. However, I am willing to accept an answer if there is a way to do the validation with multiple scans, i.e., in my example above multiple file scans are fine with me.

[jeremyfiel]

Try here. https://join.slack.com/t/json-schema/shared_invite/zt-1rptj13ch-ZygpGVq7jUxQYpct0vwsGQ

[jdesrosiers]

@awwright wrote a streaming validator a while back. It's in JavaScript, so you probably won't be able to use it. But, it shows that it's possible. I don't remember if there were any limitations.

[jounii]

Depending on your use case, you should also take look at jsonl format if it suits your needs. Especially of your json object is top level array. That would allow each individual line to be streamed and validated individually. Not sure if the JSON schema specifically takes note on jsonl, but it should work.

[jviotti]

Not in a library, but https://github.com/sourcemeta/jsonschema does this with JSONL. When validating a JSONL against a schema, each entry is loaded in memory and parsed, validated, and deconstructed. Internally with https://github.com/sourcemeta/blaze. I presume you can hook in most libraries to do it in this way?