feat: return unsafe integers inside JSON columns as exact strings with supportBigNumbers#4388
Conversation
…h supportBigNumbers
JSON column values (MySQL JSON, and MariaDB JSON identified via extended
metadata) are parsed with plain JSON.parse, which silently rounds
integers beyond Number.MAX_SAFE_INTEGER to the nearest double: a stored
{"id": 9007199254740993} comes back as 9007199254740992 with no error.
When the existing supportBigNumbers option is enabled, such integers are
now returned as exact String objects, mirroring the option's documented
behaviour for BIGINT and DECIMAL columns; safe integers, floats and
numeric strings are unaffected, and the default behaviour without the
option is unchanged. jsonStrings continues to return the raw (always
exact) document text.
Implemented as a packet.parseJson() method shared by all four row
parsers. Exactness uses JSON.parse source access (proposal-json-parse-
with-source, Node.js 22+); older runtimes fall back to plain JSON.parse.
A 16-digit pre-scan keeps the reviver off the hot path for documents
that cannot contain unsafe integers. supportBigNumbers is already part
of the parser cache key, so no cache changes are needed.
| // The shortest numeral able to exceed Number.MAX_SAFE_INTEGER has 16 | ||
| // digits; documents without such a digit run parse exactly with plain | ||
| // JSON.parse, keeping the reviver off the hot path |
There was a problem hiding this comment.
When comments are exactly what the logic already does next, I think they become unnecessary. The same goes for other comments that explain the implementation. What do you think about using comments only for things that the logic alone doesn't explain, or for the reasoning behind the decisions? 🙋🏻♂️
There was a problem hiding this comment.
Happy to prune the comments.
There was a problem hiding this comment.
Thanks! It looks like GitHub Actions is down (unrelated to the PR). I'm going to restart the workflow (Done).
There was a problem hiding this comment.
Pruned in f3294cd - dropped everything that narrated the control flow, kept three short rationale notes (the TC39 proposal name/availability behind the feature detection, why the regex is 16 digits, and that parseJson reuses supportBigNumbers' existing BIGINT out-of-range contract rather than adding a new option).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4388 +/- ##
==========================================
+ Coverage 91.49% 91.52% +0.02%
==========================================
Files 91 91
Lines 14726 14768 +42
Branches 1934 1948 +14
==========================================
+ Hits 13474 13516 +42
Misses 1252 1252
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Problem
JSON column values are parsed with plain
JSON.parse, which silently rounds integers beyondNumber.MAX_SAFE_INTEGERto the nearest double, with no error and no way to detect it after the fact:This affects MySQL
JSONcolumns and, since #4373, MariaDBJSONcolumns identified via extended metadata.BIGINT/DECIMALcolumns already have an opt-out from this class of corruption (supportBigNumbers), but values inside JSON documents had none short ofjsonStrings: true(which gives up parsing entirely).Change
When the existing
supportBigNumbersoption is enabled, integers inside parsed JSON documents that cannot be accurately represented as JavaScript Numbers are now returned as exact String objects - mirroring the option's long-documented behaviour forBIGINT/DECIMALcolumns:JSON.parse).jsonStrings: truecontinues to return the raw (always exact) document text and takes precedence.Implementation
packet.parseJson(encoding, supportBigNumbers)method now backs all eight JSON parse sites across the four row parsers (text/binary codegen + static variants), alongside the existingparseVector()/parseGeometryValue()helpers.JSON.parsesource access (proposal-json-parse-with-source, Node.js 22+ / V8 12.2), feature-detected once at module load. Older runtimes keep plainJSON.parsebehaviour, so nothing changes for them andengines: node >= 8.0is unaffected.Number.MAX_SAFE_INTEGER) take the plainJSON.parsepath with no reviver.supportBigNumbersis already part of the cache key.ConnectionOptionsandQueryOptionstypings.Tests
test/unit/parsers/support-big-numbers-json.test.mtscovers all four parsers x both JSON flavours (MySQLTypes.JSONand MariaDBextendedFormat === 'json'): unsafe positive/negative integers, safe integers, floats, unsafe-magnitude decimals, exponent notation, numerals inside strings, JSONnull, bare root-level scalars, the option-off default, andjsonStringsprecedence. The expectations self-adjust via the same feature detection, so CI lanes on Node 18/20 assert the (unchanged) legacy behaviour while 22/24 assert exactness.Full suite passes locally (214 files) on Node 22 against MySQL 8.4 and MariaDB 11.8; also verified end-to-end over both text and binary protocols on both servers.