You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Revise theory content for exponential function optimization
Updated the title and improved the clarity of the text throughout the document, including mathematical explanations and descriptions of techniques for optimizing exponential function computations.
title: Learn exponential function optimization techniques
3
3
weight: 2
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
9
## The exponential function
10
-
The exponential function is a fundamental mathematical function used across a wide range of algorithms for signal processing, High-Performance Computing and Machine Learning. Optimizing its computation has been the subject of extensive research for decades. The precision of the computation depends both on the selected approximation method and on the inherent rounding errors associated with finite-precision arithmetic, and it is directly traded off against performance when implementing the exponential function.
10
+
The exponential function is a fundamental mathematical function used across a wide range of algorithms for signal processing, High-Performance Computing and Machine Learning. Researchers have extensively studied optimizing its computation for decades. The precision of the computation depends both on the selected approximation method and on the inherent rounding errors associated with finite-precision arithmetic, and it is directly traded off against performance when implementing the exponential function.
11
11
12
12
## Range reduction
13
-
Polynomial approximations are among the most widely used methods for software implementations of the exponential function. The accuracy of a Taylor series approximation for exponential function can be improved with the polynomial’s degree but will always deteriorate as the evaluation point moves further from the expansion point. By applying range reduction techniques, the approximation of the exponential function can however be restricted to a very narrow interval where the function is well-conditioned. This approach consists in reformulating the exponential function in the following way:
13
+
Polynomial approximations are among the most widely used methods for software implementations of the exponential function. The accuracy of a Taylor series approximation for exponential function can be improved with the polynomial's degree but deteriorates as the evaluation point moves further from the expansion point. By applying range reduction techniques, you can restrict the approximation of the exponential function to a very narrow interval where the function is well-conditioned. This approach reformulates the exponential function in the following way:
14
14
15
15
$$e^x=e^{k×ln2+r}=2^k \times e^r$$
16
16
@@ -22,16 +22,16 @@ Since k is an integer, the evaluation of 2^k can be efficiently performed using
22
22
23
23
$$e^x \approx 2^k \times p(r)$$
24
24
25
-
It is important to note that the polynomial p(r) is evaluated exclusively over the interval [-ln2/2, +ln2/2]. So, the computational complexity can be optimized by selecting the polynomial degree based on the required precision of p(r) within this narrow range. Rather than relying on a Taylor polynomial, a minimax polynomial approximation can be used to minimize the maximum approximation error over the considered interval.
25
+
The polynomial p(r) is evaluated exclusively over the interval [-ln2/2, +ln2/2]. So, the computational complexity can be optimized by selecting the polynomial degree based on the required precision of p(r) within this narrow range. Rather than relying on a Taylor polynomial, a minimax polynomial approximation can be used to minimize the maximum approximation error over the considered interval.
26
26
27
27
## Decomposition of the input
28
-
The decomposition of an input value as x = k × ln2 + r can be done in 2 steps:
28
+
Decompose an input value as x = k × ln2 + r in two steps:
29
29
- Compute k as: k = round(x⁄ln2), where round(.) is the round-to-nearest function
30
30
- Compute r as: r = x - k × ln2
31
31
32
-
Rounding of k is performed by adding an adequately chosen large number to a floating-point value and subtracting it just afterward (the original value is rounded due to the finite precision of floating-point representation). Although explicit rounding instructions are available in both SVE and SME, this method remains advantageous as the addition of the constant can be fused with the multiplication by the reciprocal of ln2. This approach assumes however that the floating-point rounding mode is set to round-to-nearest, which is the default mode in Armv9-A. By integrating the bias into the constant, 2^k can also be directly computed by shifting the intermediate value.
32
+
Rounding of k is performed by adding an adequately chosen large number to a floating-point value and subtracting it just afterward (the original value is rounded because of the finite precision of floating-point representation). Although explicit rounding instructions are available in both SVE and SME, this method remains advantageous because the addition of the constant can be fused with the multiplication by the reciprocal of ln2. This approach assumes that the floating-point rounding mode is set to round-to-nearest, which is the default mode in Armv9-A. By integrating the bias into the constant, 2^k can also be directly computed by shifting the intermediate value.
33
33
34
-
Rounding error during the second step will introduce a global error as we will have:
34
+
A rounding error during the second step introduces a global error as you have:
where s is the sign bit and 1.fraction represents the significand.
46
46
47
-
The value 2^k can be encoded by setting both the sign and fraction bits to zero and assigning the exponent field the value k + bias. If k is an 8-bits integer, 2^k can be efficiently computed by adding the bias value and positioning the result into the exponent bits of a 32-bit floating-point number using a logical shift.
47
+
The value 2^k can be encoded by setting both the sign and fraction bits to zero and assigning the exponent field the value k + bias. If k is an 8-bit integer, 2^k can be efficiently computed by adding the bias value and positioning the result into the exponent bits of a 32-bit floating-point number using a logical shift.
48
48
49
-
Taking this approach a step further, a fast approximation of exponential function can be achieved using bits manipulation techniques alone. Specifically, adding a bias to an integer k and shifting the result into the exponent field can be accomplished by computing an integer i as follows:
49
+
Taking this approach a step further, you can achieve a fast approximation of exponential function using bit manipulation techniques alone. Specifically, adding a bias to an integer k and shifting the result into the exponent field can be accomplished by computing an integer i as follows:
This formulation assumes a 23-bit significand, but the method can be generalized to other floating-point precisions.
54
54
55
-
Now, consider the case where k is a real number. The fractional part of k will propagate into the significand bits of the resulting 2^k approximation. However, this side effect is not detrimental, it effectively acts as a form of linear interpolation, thereby improving the overall accuracy of the approximation. To approximate the exponential function, the following identity can be used:
55
+
Now, consider the case where k is a real number. The fractional part of k propagates into the significand bits of the resulting 2^k approximation. However, this side effect isn't detrimental; it effectively acts as a form of linear interpolation, thereby improving the overall accuracy of the approximation. To approximate the exponential function, use the following identity:
56
56
57
57
$$e^x = 2^{x⁄ln2}$$
58
58
59
59
As previously discussed, this value can be approximated by computing a 32-bit integer:
60
60
61
61
$$i = 2^{23} \times x⁄ln2 + 2^{23} \times bias = a \times x + b $$
62
62
63
-
Continue to the next section to make a C-based implementation of the exponential function.
63
+
## What you've accomplished and what's next
64
+
65
+
In this section, you learned the mathematical foundations for optimizing exponential functions:
66
+
- Range reduction techniques that narrow the evaluation interval
67
+
- How to decompose inputs using k × ln2 + r reformulation
68
+
- Bit manipulation techniques for computing scaling factors
69
+
70
+
Next, you'll implement these concepts in C using SVE intrinsics.
0 commit comments