Commit 8520d57
Fix precision loss for large bigint PKs in table repair JSON roundtrip
OrderedMap.UnmarshalJSON used Go's default JSON number decoding (float64),
which silently truncates integers exceeding 2^53. For tables with large
bigint primary keys (e.g. snowflake IDs like 415588913294348289), this
caused:
- PK corruption: 415588913294348289 → 415588913294348288 (off by 1)
- PK collisions: adjacent PKs mapped to the same float64, causing rows
to silently overwrite each other in repair maps
- Wrong upserts/deletes: repairs targeted wrong rows or missed them
- Growing diffs after repair: corrupted values replicated via spock
Fix: add dec.UseNumber() to OrderedMap.UnmarshalJSON so JSON numbers are
preserved as json.Number strings. Update ConvertToPgxType to handle
json.Number for integer types (lossless Int64 parse), numeric/decimal
(exact string via pgtype.Numeric), and float types (Float64 parse).
Replace the three duplicate lossy numeric helpers (toFloat64, asFloat,
asNumber) with a single precision-safe CompareNumeric in pkg/common that
uses an int64 fast path and falls back to math/big.Float (256-bit) for
json.Number decimals, large uint64, and native floats. Also handle
json.Number in repair plan when-expression literals (scanNumber),
pk_in matching, and origin timestamp extraction.
The v4 pgtype dependency will be removed in a follow-on PR.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 005e1e0 commit 8520d57
8 files changed
Lines changed: 545 additions & 133 deletions
File tree
- internal/consistency/repair
- plan/parser
- pkg
- common
- types
- tests/integration
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
286 | 286 | | |
287 | 287 | | |
288 | 288 | | |
289 | | - | |
290 | | - | |
291 | | - | |
292 | | - | |
| 289 | + | |
| 290 | + | |
293 | 291 | | |
294 | 292 | | |
295 | 293 | | |
| |||
308 | 306 | | |
309 | 307 | | |
310 | 308 | | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
| 309 | + | |
| 310 | + | |
315 | 311 | | |
316 | 312 | | |
317 | 313 | | |
| |||
353 | 349 | | |
354 | 350 | | |
355 | 351 | | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
361 | 356 | | |
362 | 357 | | |
363 | 358 | | |
| |||
369 | 364 | | |
370 | 365 | | |
371 | 366 | | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | 367 | | |
404 | 368 | | |
405 | 369 | | |
| |||
780 | 744 | | |
781 | 745 | | |
782 | 746 | | |
783 | | - | |
784 | | - | |
785 | | - | |
786 | | - | |
787 | | - | |
788 | | - | |
789 | | - | |
790 | | - | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
791 | 752 | | |
| 753 | + | |
792 | 754 | | |
793 | 755 | | |
794 | 756 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | | - | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
227 | 229 | | |
228 | 230 | | |
229 | 231 | | |
230 | | - | |
231 | | - | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
232 | 236 | | |
233 | 237 | | |
234 | | - | |
| 238 | + | |
235 | 239 | | |
236 | 240 | | |
237 | 241 | | |
| |||
612 | 616 | | |
613 | 617 | | |
614 | 618 | | |
615 | | - | |
616 | | - | |
617 | | - | |
| 619 | + | |
618 | 620 | | |
619 | 621 | | |
620 | | - | |
| 622 | + | |
621 | 623 | | |
622 | | - | |
| 624 | + | |
623 | 625 | | |
624 | | - | |
| 626 | + | |
625 | 627 | | |
626 | | - | |
| 628 | + | |
627 | 629 | | |
628 | | - | |
| 630 | + | |
629 | 631 | | |
630 | | - | |
| 632 | + | |
631 | 633 | | |
632 | 634 | | |
633 | 635 | | |
| |||
670 | 672 | | |
671 | 673 | | |
672 | 674 | | |
673 | | - | |
674 | | - | |
675 | | - | |
676 | | - | |
677 | | - | |
678 | | - | |
679 | | - | |
680 | | - | |
681 | | - | |
682 | | - | |
683 | | - | |
684 | | - | |
685 | | - | |
686 | | - | |
687 | | - | |
688 | | - | |
689 | | - | |
690 | | - | |
691 | | - | |
692 | | - | |
693 | | - | |
694 | | - | |
695 | | - | |
696 | | - | |
697 | | - | |
698 | | - | |
699 | | - | |
700 | | - | |
701 | | - | |
702 | | - | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2258 | 2258 | | |
2259 | 2259 | | |
2260 | 2260 | | |
| 2261 | + | |
| 2262 | + | |
| 2263 | + | |
| 2264 | + | |
| 2265 | + | |
| 2266 | + | |
| 2267 | + | |
2261 | 2268 | | |
2262 | 2269 | | |
2263 | 2270 | | |
| |||
0 commit comments