Commit 2133067
committed
Fix encode_phys precision loss for int64 with factor=1
Skip the float division in encode_phys when factor is 1 (the default).
Previously, `value /= self.factor` always performed float division,
causing up to 10 bits of precision loss for large int64 values
(e.g. 0x55554444AAAABBBB became 0x55554444AAAABC00).
Use `!= 1` for the comparison since we are comparing values, not
identity. Also clean up the redundant int(round(...)) to just round(),
since round() already returns int when ndigits is omitted.
Add unit tests covering:
- int64 roundtrip with factor=1 (the original bug)
- int type preservation with factor=1
- rounding with integer factor != 1 (existing behavior preserved)
- float factor (existing behavior preserved)
Based on the discussion in #611.1 parent 4e789fe commit 2133067
2 files changed
Lines changed: 31 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
484 | 484 | | |
485 | 485 | | |
486 | 486 | | |
487 | | - | |
488 | | - | |
| 487 | + | |
| 488 | + | |
489 | 489 | | |
490 | 490 | | |
491 | 491 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
182 | 182 | | |
183 | 183 | | |
184 | 184 | | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
185 | 214 | | |
186 | 215 | | |
187 | 216 | | |
| |||
0 commit comments