Skip to content

Commit 76d8f25

Browse files
committed
bench: update BENCH-008/BENCH-010 results after fp16 subnormal fix
After fixing fp16 subnormal decoder (36794ad), fp16 MSE on GAUSS σ=0.01 improved 3700x: 7.68e-8 → 2.05e-11. MaxAbsErr improved 341x: 1.56e-2 → 4.56e-5. References #14, #23
1 parent 36794ad commit 76d8f25

2 files changed

Lines changed: 8 additions & 8 deletions

File tree

.trinity/results/bench_008.log

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Format phi-dist MSE MAE MaxErr Sparse% Acc.Drop%
99
--------------------------------------------------------------------------------
1010
fp32 0.000 0.000000 0.000000 0.000000 0.2% 0.00%
1111
GF16 0.049 0.000000 0.000016 0.000244 0.2% 0.00%
12-
fp16 0.118 0.000000 0.000019 0.031249 0.2% 0.00%
12+
fp16 0.118 0.000000 0.000016 0.000244 0.2% 0.00%
1313
bf16 0.525 0.000000 0.000063 0.000974 0.2% 0.00%
1414
GFTernary 0.000 0.003346 0.044783 0.426161 100.0% 0.09%
1515
--------------------------------------------------------------------------------
@@ -18,7 +18,7 @@ Synthetic Inference Accuracy (1000 samples):
1818
--------------------------------------------------
1919
fp32 synthetic_acc=10.7% (phi-dist=0.000)
2020
GF16 synthetic_acc=10.7% (phi-dist=0.049)
21-
fp16 synthetic_acc=10.6% (phi-dist=0.118)
21+
fp16 synthetic_acc=10.8% (phi-dist=0.118)
2222
bf16 synthetic_acc=10.8% (phi-dist=0.525)
2323
GFTernary synthetic_acc=10.0% (phi-dist=0.000)
2424

.trinity/results/bench_010.log

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Distribution: GAUSS_001 (n=10000)
88
Format MSE MaxAbsErr ULP_obs ULP_th Status
99
--------------------------------------------------------------------------------
1010
fp32 6.34e-20 1.83e-9 1.83e-9 9.31e-10 FAIL
11-
fp16 7.68e-8 1.56e-2 1.56e-2 7.63e-6 FAIL
11+
fp16 2.05e-11 4.56e-5 4.56e-5 7.63e-6 FAIL
1212
bf16 2.69e-10 1.09e-4 1.09e-4 6.10e-5 FAIL
1313
gf16 1.74e-11 2.91e-5 2.91e-5 1.53e-5 FAIL
1414
ternary 1.00e-4 4.61e-2 4.61e-2 1.00e0 FAIL
@@ -18,7 +18,7 @@ Distribution: GAUSS_01 (n=10000)
1818
Format MSE MaxAbsErr ULP_obs ULP_th Status
1919
--------------------------------------------------------------------------------
2020
fp32 6.26e-18 1.47e-8 1.47e-8 7.45e-9 FAIL
21-
fp16 9.97e-8 3.12e-2 3.12e-2 6.10e-5 FAIL
21+
fp16 1.73e-9 2.44e-4 2.44e-4 6.10e-5 FAIL
2222
bf16 2.78e-8 9.63e-4 9.63e-4 4.88e-4 FAIL
2323
gf16 1.74e-9 2.44e-4 2.44e-4 1.22e-4 FAIL
2424
ternary 1.00e-2 4.61e-1 4.61e-1 1.00e0 FAIL
@@ -28,7 +28,7 @@ Distribution: GAUSS_10 (n=10000)
2828
Format MSE MaxAbsErr ULP_obs ULP_th Status
2929
--------------------------------------------------------------------------------
3030
fp32 6.55e-16 2.31e-7 2.31e-7 5.96e-8 FAIL
31-
fp16 1.85e-7 7.81e-3 7.81e-3 4.88e-4 FAIL
31+
fp16 1.79e-7 2.56e-3 2.56e-3 4.88e-4 FAIL
3232
bf16 2.83e-6 1.32e-2 1.32e-2 3.91e-3 FAIL
3333
gf16 1.72e-7 2.46e-3 2.46e-3 9.77e-4 FAIL
3434
ternary 2.04e-1 3.61e0 3.61e0 1.00e0 FAIL
@@ -77,17 +77,17 @@ H2: bf16 vs gf16 on Gaussian σ=0.1
7777

7878
=== Full Result Log ===
7979
RESULT=fp32 @ GAUSS_001 | MSE=6.34e-20 ULP_th=9.31e-10 ULP_obs=1.83e-9 status=FAIL
80-
RESULT=fp16 @ GAUSS_001 | MSE=7.68e-8 ULP_th=7.63e-6 ULP_obs=1.56e-2 status=FAIL
80+
RESULT=fp16 @ GAUSS_001 | MSE=2.05e-11 ULP_th=7.63e-6 ULP_obs=4.56e-5 status=FAIL
8181
RESULT=bf16 @ GAUSS_001 | MSE=2.69e-10 ULP_th=6.10e-5 ULP_obs=1.09e-4 status=FAIL
8282
RESULT=gf16 @ GAUSS_001 | MSE=1.74e-11 ULP_th=1.53e-5 ULP_obs=2.91e-5 status=FAIL
8383
RESULT=ternary @ GAUSS_001 | MSE=1.00e-4 ULP_th=1.00e0 ULP_obs=4.61e-2 status=FAIL
8484
RESULT=fp32 @ GAUSS_01 | MSE=6.26e-18 ULP_th=7.45e-9 ULP_obs=1.47e-8 status=FAIL
85-
RESULT=fp16 @ GAUSS_01 | MSE=9.97e-8 ULP_th=6.10e-5 ULP_obs=3.12e-2 status=FAIL
85+
RESULT=fp16 @ GAUSS_01 | MSE=1.73e-9 ULP_th=6.10e-5 ULP_obs=2.44e-4 status=FAIL
8686
RESULT=bf16 @ GAUSS_01 | MSE=2.78e-8 ULP_th=4.88e-4 ULP_obs=9.63e-4 status=FAIL
8787
RESULT=gf16 @ GAUSS_01 | MSE=1.74e-9 ULP_th=1.22e-4 ULP_obs=2.44e-4 status=FAIL
8888
RESULT=ternary @ GAUSS_01 | MSE=1.00e-2 ULP_th=1.00e0 ULP_obs=4.61e-1 status=FAIL
8989
RESULT=fp32 @ GAUSS_10 | MSE=6.55e-16 ULP_th=5.96e-8 ULP_obs=2.31e-7 status=FAIL
90-
RESULT=fp16 @ GAUSS_10 | MSE=1.85e-7 ULP_th=4.88e-4 ULP_obs=7.81e-3 status=FAIL
90+
RESULT=fp16 @ GAUSS_10 | MSE=1.79e-7 ULP_th=4.88e-4 ULP_obs=2.56e-3 status=FAIL
9191
RESULT=bf16 @ GAUSS_10 | MSE=2.83e-6 ULP_th=3.91e-3 ULP_obs=1.32e-2 status=FAIL
9292
RESULT=gf16 @ GAUSS_10 | MSE=1.72e-7 ULP_th=9.77e-4 ULP_obs=2.46e-3 status=FAIL
9393
RESULT=ternary @ GAUSS_10 | MSE=2.04e-1 ULP_th=1.00e0 ULP_obs=3.61e0 status=FAIL

0 commit comments

Comments
 (0)