Skip to content

Commit f10281e

Browse files
committed
fix image in doc
Signed-off-by: Hao Wu <skyw@nvidia.com>
1 parent 7f44f58 commit f10281e

2 files changed

Lines changed: 1 addition & 1 deletion

File tree

docs/assets/adam.update.vs.eps.png

-16.1 KB
Loading

docs/primer/epsilon.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ When $g_t$ is small enough so that $g_t^2$ underflows single precision, it is po
152152

153153
When $g_t$ is comparable to $\epsilon$, the impact of $\epsilon$ depends on the magnitude of $g_t$. Let's look at a simple example: $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$, constant $g_t$, and bias correction ignored for simplicity. We plot the update with and without $\epsilon$.
154154

155-
<img src="../assets/adam.update.vs.eps.png" alt="eps" width="500" />
155+
![eps](../assets/adam.update.vs.eps.png)
156156

157157
A much smaller update is applied because $\epsilon$ makes the denominator meaningfully larger than $\sqrt{\widehat{v_t}}$ alone. The effect wears off once the gradient is more than three orders of magnitude larger than $\epsilon$.
158158

0 commit comments

Comments
 (0)