Skip to content

issue 59: Fix decade boundary detection in find_exponent and mantissa for long double#60

Open
MitchellThompkins wants to merge 2 commits into
kthohr:masterfrom
MitchellThompkins:fix-long-double-find-exponent
Open

issue 59: Fix decade boundary detection in find_exponent and mantissa for long double#60
MitchellThompkins wants to merge 2 commits into
kthohr:masterfrom
MitchellThompkins:fix-long-double-find-exponent

Conversation

@MitchellThompkins
Copy link
Copy Markdown

@MitchellThompkins MitchellThompkins commented Apr 13, 2026

Description

This corrects #59 (and additionally #44), which discusses in detail the bug corrected by this PR.

There are 4 fixes here:

  1. A simple change to allow pipelines to actually fail.
  2. An update to gcem::find_exponent to correctly calculate the exponent for long doubles for values that lie at the boundary used to find the exponent.
  3. An update to gcem::mantissa that does the same as (2).
  4. An update to gcem::find_exponent and gcem::mantissa to correctly handle exact decade boundaries (10.0L, 100.0L).

I also updated 3 tests:

  1. Tests added to cover gcem::find_exponent.
  2. Tests added to cover gcem::mantissa.
  3. Tests added to gcem::log to cover edge cases.

Changes

The fix here is two-fold. First, values are scaled up so that the comparison threshold becomes an exact integer T(1), rather than the imprecise T(1e-03) literal. This is the x * T(10) and x * T(1000) portions. Because T(1) is exactly represnteable, the rounding issue of T(1e-03) goes away. Second, they are compared against 1 - machine epsilon, so that values whose scaled representation falls just below 1 due to rounding are still guaranteed to fall into the correct decade. This is tabulated below:

x range condition jump snap zone
x < 0.001 x * 1000 < 1 - eps x10000 x * 1000 in [1-eps, 1), 1 ULP wide
0.001 <= x < 0.1 x * 10 < 1 - eps x100 x * 10 in [1-eps, 1), 1 ULP wide
0.1 <= x < 1 x < 1 - eps x10 x in [1-eps, 1), 1 ULP wide

This gives a "snap zone" of 1 ULP below each scaled boundary. Real numbers that fall within this zone cannot be distinguished from the boundary at long double precision, so they snap "up" to T(1).

A similar technique is implemented in gcem::mantissa, although sans the scaling because that function on checks for bounding cases between 1 and 10.

x < T(1) - GCLIM<T>::epsilon()  ->  mantissa(x * T(10))
"x is genuinely below 1, scale up and recurse"

x < T(1) -> T(1)
"x is within one epsilon of 1 from below. Close enough, snap to 1"

x >= T(10) ->  mantissa(x / T(10))
"x is 10 or above, scale down and recurse"

x
"x is in [1, 10], this is the mantissa, return it"

After a lot of thought and testing, I'm pretty convinced that gcem::find_exponent originally returned incorrect values for every odd negative of power 10 (i.e. 10^-1, 10^-3, 10^-5, etc...) for long double. I think the reason for the alternating pattern is that the value cascades through both broken thresholds (T(1e-03) and T(1e-01)), scaling through each in turn until the final product lands just below 10.0, where the > T(10) branch that would otherwise fix the exponent cannot be taken (b/c of the rounding). So even powers of 10 are rounded but in a way that they take the correct branch.

I also fixed a separate simpler issue: both gcem::find_exponent and gcem::mantissa used a strict x > T(10) comparison for the downscaling branch. Since 10.0L and 100.0L are exactly representable, they fell through to the base case, returning an exponent one too low. Changing > to >= fixes this. Unlike the odd-power bug, this did not produce an incorrect log output.

slightly larger than the corresponding long double values 0.1L and
0.001L (which was prevoiusly used). This caused find_exponent to return
the wrong exponent for those exact boundary values, e.g. log(0.1L)
returned 2*ln(0.1).

This actually effected all calls to gcem::log; it just only manifests
when log lands exactly on 0.1 or 0.001 in long double.

add test case for failing log case

add 8 epsilon of error

add tests for find_exponent

make epsilon ULP choice configurable and add mantissa.hpp tests

no reason to use T(8)

wording updates

fix fall-through for 10.0L and 100.0L
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant