fix(read): preserve Int64 precision for unsafe Longs in tool outputs#1220
fix(read): preserve Int64 precision for unsafe Longs in tool outputs#1220STiFLeR7 wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Introduces a custom EJSON stringifier that preserves 64-bit integer precision by converting BSON Long values to standard JS numbers when within safe-integer range and to {$numberLong: "..."} otherwise, then wires it into the MongoDB read tools that previously used EJSON.stringify directly.
Changes:
- New
src/helpers/ejson.tswithserializeSafeLongsandstringifyEJSONhelpers, plus unit tests. - Replaced direct
EJSON.stringify(...)usage infind,aggregate,aggregateDB, anddbStatstools with the newstringifyEJSON.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/helpers/ejson.ts | New helper that serializes safe Longs as numbers and unsafe Longs as $numberLong strict EJSON. |
| tests/unit/helpers/ejson.test.ts | Unit tests covering primitives, safe/unsafe Longs, and nested structures. |
| src/tools/mongodb/read/find.ts | Switched to stringifyEJSON for result serialization. |
| src/tools/mongodb/read/aggregate.ts | Switched to stringifyEJSON for result serialization. |
| src/tools/mongodb/read/aggregateDB.ts | Switched to stringifyEJSON for result serialization. |
| src/tools/mongodb/metadata/dbStats.ts | Switched to stringifyEJSON for stats serialization. |
| const longObj = obj as unknown as Long; | ||
| const num = longObj.toNumber(); | ||
| if (num >= Number.MIN_SAFE_INTEGER && num <= Number.MAX_SAFE_INTEGER) { | ||
| return num; | ||
| } |
| if (typeof obj === "object") { | ||
| const proto: unknown = Object.getPrototypeOf(obj); | ||
| if (proto === null || proto === Object.prototype) { | ||
| const result: Record<string, unknown> = {}; | ||
| for (const key of Object.keys(obj)) { | ||
| result[key] = serializeSafeLongs((obj as Record<string, unknown>)[key]); | ||
| } | ||
| return result; | ||
| } | ||
| } |
| replacer?: ((this: unknown, key: string, value: unknown) => unknown) | (string | number)[] | null, | ||
| space?: string | number | ||
| ): string { | ||
| return EJSON.stringify(serializeSafeLongs(value), replacer as Parameters<typeof EJSON.stringify>[1], space); |
There was a problem hiding this comment.
Any reason to not just do EJSON.stringify(value, { relaxed: false })? Yes that gives you verbose EJSON always which we may or may not want, but then we don't need custom code at all.
There was a problem hiding this comment.
Using { relaxed: false }\ would force all BSON values (including safe \Long\s, \Int32\s, \Double\s, etc.) to serialize into their verbose strict EJSON representations (e.g. {\count: {\: \42}}\ or {\value: {\: \123}}).
This significantly reduces output readability for downstream LLMs and increases client payload sizes. By using this targeted preprocessing helper, we retain the highly readable relaxed representations for all safe numbers and other BSON types, while strictly preventing precision loss for numbers that exceed JS safe integer limits.
There was a problem hiding this comment.
Compass has this concept called "semi-relaxed" which does a variation of what you're doing here and that's probably what we should be using everywhere we output JSON. ie. do relaxed: true except for these edge cases. But it does have the problem that something that reads the output could work fine for years and then suddenly encounter an int greater than 2^^53 for the first time which suddenly changes the output format and then their code blows up.
And it still has the readability problem because those values are suddenly in EJSON. So if LLMs don't deal well with seeing { $numberLong: "value" } then it still applies in that case - it just goes from common to rare. Which as I explained above isn't necessarily better.
I think the team should probably deal with all all content and structuredContent in one go. Make a decision and apply it everywhere consistently. Also make sure that output always contains structuredContent while at it.
There was a problem hiding this comment.
Here are some code-pointers for Compass' solution for reference: https://github.com/mongodb-js/compass/blob/42405c4f6f3fef390e0a511ad1913a6ce865b6d6/packages/hadron-document/src/utils.ts#L48-L82
So there's a little more to it but not much.
Introduce a custom EJSON stringifier helper that pre-processes objects to convert BSON Long values exceeding safe JavaScript integer limits into strict numberLong string representations, while keeping safe Long values as standard relaxed numbers. Use this custom stringifier across find, aggregate, aggregate-db, and db-stats tool outputs to prevent precision round-off errors in downstream LLM clients.
…mit check Compare the Long object directly rather than calling toNumber() first, preventing precision loss for boundary values (e.g. MAX_SAFE_INTEGER + 2) that round back to safe limits as floats. Add clarifying JSDoc documentation about the prototype recursion constraint.
STiFLeR7
force-pushed
the
fix/int64-precision-loss
branch
from
dc47721 to
31b183d
Compare
|
Thanks for the context @dudaschar — closing this in favor of #1249 since it covers the same Int64/Long precision issue with a broader, more consistent approach across BSON types. Happy to help test/review #1249 if useful. |
Description
In relaxed mode serialization (
EJSON.stringifydefault), BSONLongvalues are directly serialized to standard JSON numbers. When these numbers exceed JavaScript's safe integer boundaries (Number.MIN_SAFE_INTEGERandNumber.MAX_SAFE_INTEGER), they lose precision (for example,7583362298413593073gets serialized as7583362298413593000). This leads to round-off errors and tool failures when downstream clients/LLMs execute queries using these values.Changes
stringifyEJSONinsrc/helpers/ejson.tsto pre-process objects, converting unsafeLonginstances (or those identified by_bsontype === "Long") into strict EJSON format (i.e.{ $numberLong: "value" }) to preserve exact 64-bit precision. SafeLongs are kept as normal numbers to maintain relaxed mode readability.find,aggregate,aggregate-db, anddb-statstools to usestringifyEJSONinstead of defaultEJSON.stringify.tests/unit/helpers/ejson.test.tsensuring correct precision behavior for safe/unsafe values and deeply nested objects or arrays.Closes #728