Skip to content

Reject unpaired JSON surrogate escapes#1085

Draft
He-Pin wants to merge 1 commit into
databricks:masterfrom
He-Pin:fix/parsejson-surrogates
Draft

Reject unpaired JSON surrogate escapes#1085
He-Pin wants to merge 1 commit into
databricks:masterfrom
He-Pin:fix/parsejson-surrogates

Conversation

@He-Pin

@He-Pin He-Pin commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Motivation

Jsonnet strings are defined in Unicode-codepoint terms, and sjsonnet already rejects unpaired surrogate escapes in Jsonnet source strings. std.parseJson and the strict .json import fast path could still let JSON strings containing lone UTF-16 surrogate halves through as Jsonnet strings or defer the failure until a later string operation.

The better long-term fix belongs in ujson, where surrogate-pair validation can happen while decoding \uXXXX escapes without a downstream pass. This PR stays draft as a downstream guard while the upstream issue is tracked.

Modification

  • Add ValVisitor.rejectUnpairedSurrogates for decoded JSON string values and object keys.
  • Enable the check for std.parseJson, returning a normal Invalid JSON: unpaired surrogate in string error.
  • Apply the same guard in the strict .json import fast path and preloader so imported JSON cannot expose malformed strings.
  • Preserve valid surrogate pairs and escaped literal text such as "\\uD800".
  • Keep the import fast path allocation-conscious with a private nullable parseJsonImportOrNull contract; a reusable OptionVal abstraction is intentionally left for a separate PR.
  • Add regression coverage for lone high/low surrogates, surrogate object keys, valid surrogate pairs, escaped literal text, and Jsonnet fallback comments.

Result

std.parseJson now aligns with cpp-jsonnet and jrsonnet by rejecting unpaired surrogate escapes, while still accepting valid surrogate pairs. go-jsonnet currently normalizes these cases to U+FFFD.

Case sjsonnet before this PR sjsonnet after this PR cpp-jsonnet go-jsonnet jrsonnet
std.codepoint(std.parseJson("\"\\uD800\"")) 55296 error error 65533 error
std.codepoint(std.parseJson("\"\\uDC00\"")) 56320 error error 65533 error
std.codepoint(std.parseJson("\"\\uD83D\\uDE00\"")) 128512 128512 128512 128512 128512
std.codepoint(std.objectFields(std.parseJson("{\"\\uD800\":1}"))[0]) 55296 error error 65533 error
strict .json import {"s":"\\uD800"} then std.codepoint((import "surrogate.json").s) not rejected by JSON import; later std.codepoint errors on a malformed string Invalid JSON: unpaired surrogate in string n/a n/a n/a
strict .json import {"s":"\\uD83D\\uDE00"} then std.codepoint((import "pair.json").s) 128512 128512 n/a n/a n/a

Import rows are sjsonnet-specific because this PR changes sjsonnet's .json import fast path.

References

@He-Pin He-Pin force-pushed the fix/parsejson-surrogates branch from 6aee07c to 8974a1e Compare July 4, 2026 11:52
Motivation:
std.parseJson and strict .json import fast paths accepted or normalized JSON strings containing unpaired UTF-16 surrogate escapes. That allowed invalid JSON Unicode to become Jsonnet strings, diverging from cpp-jsonnet and jrsonnet behavior and from the intent of rejecting malformed JSON input.

Modification:
Add a shared surrogate validator for JSON visitors, enable it for std.parseJson, and convert strict .json import surrogate failures into normal Jsonnet errors instead of falling back or producing raw invalid strings. Keep the import fast path allocation-conscious with a private nullable return contract, leaving any reusable OptionVal abstraction for a separate PR.

Result:
Lone high/low surrogate values and keys now fail with Invalid JSON: unpaired surrogate in string, valid surrogate pairs still materialize as codepoint strings, escaped literal text such as \\uD800 remains ordinary text, and loose Jsonnet .json files can still fall back through the Jsonnet parser.

References:
https://jsonnet.org/ref/stdlib.html#std-parseJson
google/go-jsonnet#885
com-lihaoyi/upickle#722
@He-Pin He-Pin force-pushed the fix/parsejson-surrogates branch from 8974a1e to 07ed891 Compare July 4, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant