Skip to content

Carry the division remainder into rounding as a sticky bit (#236)#237

Open
b-erdem wants to merge 1 commit into
ericmj:mainfrom
b-erdem:fix-div-sticky-rounding
Open

Carry the division remainder into rounding as a sticky bit (#236)#237
b-erdem wants to merge 1 commit into
ericmj:mainfrom
b-erdem:fix-div-sticky-rounding

Conversation

@b-erdem

@b-erdem b-erdem commented Jul 5, 2026

Copy link
Copy Markdown

Fixes the :half_even/:ceiling/:floor rounding half of #236.

Problem

Decimal.div/2 computes the quotient to precision + 1 digits and then discards the long-division remainder, so the rounding step cannot distinguish an exact tie from a value with more nonzero digits below the guard digit. This rounds roughly 5% of inexact divisions the wrong way:

  • :half_even / :half_down — a guard digit of 5 with a nonzero tail is not a tie and must round up. 4.7460 / -5522 at precision 28 truly ends …1405287…; decimal returned …140 instead of …141.
  • :ceiling / :floor / :up — a guard digit of 0 with a nonzero tail is still nonzero. :ceiling could return a result below the true quotient (and :floor above it), violating the modes' defining guarantee. Example: Decimal.div(Decimal.new("1.0000001"), Decimal.new("1")) at precision 5 :ceiling returned 1.0000 (below the true value) instead of 1.0001.

Fix

div_calc/5 already returns the remainder; this passes rem != 0 to context/3 as the sticky bit, which every rounding mode's increment?/5 already consumes (the same mechanism add/2 uses since the CVE‑2026‑32686 mitigation). One line.

The :inexact flag

The same remainder also feeds the :inexact condition. context_test.exs asserted [:rounded] (not :inexact) for 10^106 / 17 at precision 111, because :inexact was suppressed whenever the remainder happened to be a power of ten. That division is non-terminating, so the rounded result is inexact — Python's decimal flags both Rounded and Inexact. The test is updated to assert both.

Verification

Differential fuzz against Python's decimal (which passes the official GDA decarith suite): all 7 rounding modes × 6,000 random divisions each (42,000 total), 0 mismatches after the fix — every result adjudicated against the exact quotient with rational arithmetic. Before the fix, ~5% per mode diverged.

Tests

  • decimal_test.exs — new div/2 regression test covering the guard‑5 sticky case, genuine ties still rounding to even, and the :ceiling/:floor directional cases.
  • context_test.exs — the 10^106 / 17 flag assertion now expects :rounded and :inexact.

All existing tests pass (101 doctests, 27 properties, 102 tests).

Decimal.div/2 computed the quotient to precision+1 digits and then
discarded the long-division remainder, so the rounding step could not
tell an exact tie from a value with more nonzero digits below the guard
digit. This rounded ~5% of inexact divisions the wrong way:

  * :half_even / :half_down — a guard digit of 5 with a nonzero tail is
    not a tie and must round up, e.g. 4.7460 / -5522 at precision 28
    truly ends …1405287…; it returned …140 instead of …141.
  * :ceiling / :floor / :up — a guard digit of 0 with a nonzero tail is
    still nonzero, so :ceiling could return a result below the true
    quotient (and :floor above it), violating the modes' guarantee.

Pass `rem != 0` as the sticky bit to context/3, which every rounding
mode's increment?/5 already consumes. This also corrects the :inexact
flag, which context_test.exs pinned as suppressed when the remainder was
a power of ten; both :rounded and :inexact are now signalled, matching
the spec and Python's decimal.

Verified with a differential fuzz against Python's decimal: all 7
rounding modes, 42,000 random divisions, 0 mismatches after the fix
(previously ~5% per mode).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant