Can `ResumableParser` return whether the last `partial_value` container is completed or not?

[kou]

This is not a critical feature request.

I'm trying to migrate to json gem from json-stream gem in https://github.com/groonga/groonga-command-parser/ by JSON::ResumableParser.

It parses records to be loaded to Groonga (a full text search engine with column store). The records use the following format:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},
{"column_name1": "record2's column_value1", "column_name2": 2, ...},
...
]

groonga-command-parser wants to process a record as soon as it's completed. For example, it wants to process the first record when the following chunk is processed:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},

We can do it by the following script:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"}

But it doesn't work with the following chunk:

[
{"column_name1": "record1's column_value1",

We can't know whether the record is completed or not:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1",
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"} # This is not completed yet

groonga-command-parser has a workaround for it. It ignores the last record:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => []

parser << <<-CHUNK
{"column1": "value2"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => [{"column1" => "value1"}] # 1st record

If we can detect whether a record in this example is completed or not, it's useful. But this is not a critical feature request because there is a workaround for this case.

[byroot]

I see. Extracting complete elements of a top level array, is indeed a use case that makes a lot of sense and that I would like to support.

Not quite certain of the best way yet, but I will think about it.

The question though is whether only supporting it for a top-level array is enough, or if we'd need something more flexible.

Perhaps:

parser = JSON::ResumableParser.new
parser << <<~CHUNK
  [
  {"column1": "value1"},
CHUNK
parser.skip_array
while parser.parse
  p parser.value # => {"column1" => "value2"}
end

But what if you want to do the same for an array that is nested inside an object?, e.g.

{
  "events": [
    {...},
    {...}
  ]
}

But either way, I feel like having a way to tell the parser to ignore some levels of the structure is probably the way to go.

NB: I'm at a conference, unclear when I'll have time to experiment with this.

[kou]

Ah, good point.

In the groonga-command-parser use case, the [...] pattern is only needed.

But there is another use case that use the {"XXX": [...]} pattern. It's Groonga's logical_range_filter response: https://github.com/groonga/groonga/blob/05a7fc1ec296ca31c7e4b2f47348032103f13b9d/test/command/suite/sharding/logical_range_filter/command_version/3/json.expected#L28-L102

It has the following structure:

{
  "body": {
    "columns": [
      {...column1...},
      {...column2...},
      ...
    ],
    "records": [
      [...record1...],
      [...record2...],
      ...
    ]
  },
  "header": {...}
}

It's helpful that the records part can be processed as soon as it's parsed. But this is a minor use case...

I also checked an OpenAI API: https://developers.openai.com/api/docs/guides/migrate-to-responses#messages-vs-items

It has the following structure:

{
  ...,
  "output": [
    {...output1...},
    {...output2...},
    ...
  ],
  ...
}

But it's not returned as a stream.

OpenAI API uses server-sent events for streamed response: https://developers.openai.com/api/docs/guides/streaming-responses

It returns small complete JSONs as events. So we don't need to care about OpenAI API case.

[byroot]

On mobile so can’t give a lot of details, but I wonder if the on_load callback already solves the need, as it yields all values as they are completed.

Of course in some case it may not be easy to identify the values you care about