Commit 36794ad
committed
fix(format): correct fp16 subnormal decode bias (113-shifts, not 112-exp)
The fp16 subnormal decoder had an incorrect exponent calculation.
When decoding subnormal values (exp=0), the biased exponent was
computed as (112 - exp) where exp wraps negatively. This produced
values 131072x too large for small subnormals.
Fix: use (113 - shifts) where shifts counts the normalization steps.
Before: fp16(0x0002) = 1.5625e-2 (wrong)
After: fp16(0x0002) = 1.1921e-7 (correct: 2/1024 * 2^(-14))
Closes #631 parent ae8ffa2 commit 36794ad
1 file changed
Lines changed: 41 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
155 | | - | |
156 | | - | |
| 155 | + | |
| 156 | + | |
157 | 157 | | |
158 | 158 | | |
159 | | - | |
| 159 | + | |
| 160 | + | |
160 | 161 | | |
161 | 162 | | |
162 | 163 | | |
| |||
647 | 648 | | |
648 | 649 | | |
649 | 650 | | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
0 commit comments