Support use of JSON Pointers (RFC 6901) for serializing object graphs (Object Id)

[scottslewis]

Is your feature request related to a problem? Please describe.

RFC 6901 specifies the syntax for json pointers.

Currently, upon serialization of an object graph, Jackson inserts a separate copy for each reference. When serializing a large data structure (e.g. a tree) such copying can come at a significant size and transmission cost.

Jackson has JsonIdentityInfo currently for identifying object instances in the serialized stream, but this mechanism is unique to Jackson, and so cannot work for cross-language or cross-library interoperability. Only a specification like RFC 6901 can provide such interoperability.

There exist multiple implementations of the RFC 6901 json pointer syntax for multiple languages, and here are three relatively new ones coming from the jref project:

Java, Typescript,
Python

We (the authors of these implementations) will offer to contribute any/all of them as they are, or with any modifications (or rewrite) as needed to fit into the appropriate Jackson layer or layers. The architecture of these implementations is quite simple...i.e. a depth-first traversal of the given object graph to find and construct json pointers in place of object references).

Describe the solution you'd like

An optional means for Jackson to serialize an object graph and use json pointer syntax to represent a more efficient/compact form for transmission.

Usage example

No response

Additional context

For context relative to the JsonIdentityInfo Jackson annotation, see this discussion

[cowtowncoder]

Quick note: Jackson already has its own mechanisms for handling (possibly cyclic) Object Graphs, with @JsonIdentityInfo. Mechanism proposed here would be complementary approach, ideally within same framework.

[scottslewis]

Quick note: Jackson already has its own mechanisms for handling (possibly cyclic) Object Graphs, with @JsonIdentityInfo. Mechanism proposed here would be complementary approach, ideally within same framework.

From issue description

JsonIdentityInfo as it behaves now...cannot work for cross-language or cross-library interoperability.

AFAICT, only a specification like RFC 6901 can provide interoperability between python/pydantic, java/Jackson, typescript/zod...i.e across prog language and serialization libraries.

The original authors of jref (which in another form was a proposed specification) are/have been participating in json schema specification, and we are looking to get jref 'local' (leveraging RFC 6901 for efficient object graph serialization) into some formal specification status.

We've also proposed a project with the new Linux Foundation Agentic AI Foundation (aaif.io), called MCP Tools that includes the jref/RFC 6901 work. I'm hopeful that that can also lead to support for interoperability efforts like jref across library projects, as json object serialization/json schema in agentic communication becomes more widespread.

with @JsonIdentityInfo. Mechanism proposed here would be complementary approach, ideally within same framework.

If JsonIdentityInfo++ can put/get RFC 6901 serialized object graphs on/off the wire that's completely fine with me. The jref implementations are just that...a standalone impl of the technique. They are architecturally straightforward and the source is all there for the community, but there's no hard requirement that impls use any of the existing code.

More info and detail (with code pointers is great) is needed about how this could be implemented within the JsonIdentityInfo mechanism.

[scottslewis]

One point: the interoperability of object graph json serialization and efficiency for data structures may not appear to be a huge issue for many use cases.

With agentic AI, however, and the use of wire protocols based upon json schema and json serialization (e.g. MCP/jsonrpc), and the the increasing use of data structures such as sets, groups, trees and other graphs for inter-process context/data exchange...e.g. it seem likely that interoperability and efficiency of json serialization will need to increase.

[pjfanning]

@scottslewis any interest in creating a PR, Agentic or otherwise?

[scottslewis]

@scottslewis any interest in creating a PR, Agentic or otherwise?

Yes absolutely. As mentioned, I've created a java impl here that uses the json pointer spec to serialize and deserialize and object graph. This impl was based upon the Typescript and Python impls. My hope is that some of it can be used for Jackson integration as I know it works to spec, but that's not a hard requirement for me.

So yes, I'm willing to produce a PR(s).

What I need to do that is greater understanding of the various Jackson extension mechanisms at the various layers/extension mechanisms (e.g. custom serialization, JsonIdentityInfo mechanisms, ObjectMapper) and where in the existing json serialization/dev ui this could fit...optionally I expect...and without breaking backward compatibility.

[scottslewis]

Oh...I didn't answer your 'agentic' part. I'm not sure about that. I've thought about giving that a go, but I'm a little uncomfortable with trying it on an established codebase (Jackson) that I'm architecturally not very familiar with. I was comfortable doing so with the Java jref impl since I wrote the python impl by hand to get a full understanding of how it worked and why (depth first traversal of object graph). If those of you more familiar with Jackson want to give that a try...as you are better able to evaluate the outcome, I have no objections.

[cowtowncoder]

JsonIdentityInfo as it behaves now...cannot work for cross-language or cross-library interoperability.

AFAICT, only a specification like RFC 6901 can provide interoperability between python/pydantic, java/Jackson,
typescript/zod...i.e across prog language and serialization libraries.

You misunderstood my point here: what I mean is the integration of JSON Pointer based approach via machinery that is used by Jackson databind. Not a separate feature, integrated one.

But beyond this, I have thought of couple of points wrt how to possibly make this happen: will outline in separate comment.

EDIT: added comments/questions on FasterXML/jackson#311.

[cowtowncoder]

@scottslewis Quick question: would it be possible to include a link to actual reference spec(s)? 6901 is only about JSON Pointer, for which support exists in Jackson (see https://github.com/FasterXML/jackson-core/blob/3.x/src/main/java/tools/jackson/core/JsonPointer.java ) -- but that is not what what we are taking about here but rather specifying of Object References using JSON Pointer notation.

Also, a simple sample document would be useful to have inline, here (as opposed to embedded in reference Java project).

EDIT: I suppose JRef spec (https://github.com/hyperjump-io/json-reference/blob/main/spec.md) is sort of the basis.

And a trivial sample doc is:

{
  "foo": { "$ref": "#/bar" },
  "bar": 42
}

although that is more of a toy example.

[scottslewis]

@scottslewis Quick question: would it be possible to include a link to actual reference spec(s)? 6901 is only about JSON Pointer, for which support exists in Jackson (see https://github.com/FasterXML/jackson-core/blob/3.x/src/main/java/tools/jackson/core/JsonPointer.java ) -- but that is not what what we are taking about here but rather specifying of Object References using JSON Pointer notation.

Also, a simple sample document would be useful to have inline, here (as opposed to embedded in reference Java project).

EDIT: I suppose JRef spec (https://github.com/hyperjump-io/json-reference/blob/main/spec.md) is sort of the basis.

Here's the typescript jref version with a link to the Jref specification

The Jref specification originally included the whole uri syntax for referencing remote json objects...e.g.

{ "$ref": "https://example.com/example.json#/foo/bar" }

What I/we are requesting with this feature is only support for the fragment (local) portion of the JSON Pointer specification.

{ "$ref": "#/foo/bar" }

We are not looking to support the full uri (remote) ref.

And a trivial sample doc is:

{
"foo": { "$ref": "#/bar" },
"bar": 42
}

although that is more of a toy example.

Here's a more complicated local-only sample

{
    "foo": { 

                 "bar": { 
                               "baz": { "tool": 5, "data": "45AC" }
                 },

                 "a": [ "first", "second", "third"]
    },
    "blah": { "$ref": "#/foo"},   // foo object 
    "num": { "$ref": "#/foo/bar/baz" },   // baz object 
    "sec": { "$ref": "#/foo/a/1"},   // second string in a array
}

There are typescript samples in test test code and the java test code.

[cowtowncoder]

Thank you @scottslewis. One other thing I was not sure about -- and asked on Jref repo (hyperjump-io/json-reference#2) is whether actual cycles are allowed (full Object graphs or just DAGs).

[scottslewis]

Thank you @scottslewis. One other thing I was not sure about -- and asked on Jref repo (hyperjump-io/json-reference#2) is whether actual cycles are allowed (full Object graphs or just DAGs).

Jason is the author of jref spec/hyperjump and the ts implementation at jref_typescript. I have communicated with him directly and hopefully he will answer your question directly and publicly. My guess is that he may not monitor or communicate very much on hyperjump repo any longer and that maybe the OpenMCPTools repo issues might be better for jref local (as opposed to the complete jref spec), but I'm not sure about that.

What I can tell you is that all three implementations (typescript, python, java) that we have currently do support cyclic object graphs as per jref...in a similar manner. My own understanding is that the jref spec simply requires impls to deal with the cyclic case, but doesn't specify how.