deploy: 3e5fb87

Author: msneubauer

_sources/lectures/Attention.html

@@ -1100,8 +1100,8 @@ <h3><span style="color:LightGreen">Sequence Padding and Attention Masking</span>
 <section id="span-style-color-orange-computing-the-reweighted-padded-attention-mask-span">
 <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</span><a class="headerlink" href="#span-style-color-orange-computing-the-reweighted-padded-attention-mask-span" title="Permalink to this heading">#</a></h2>
 <p>Lets create some numbers so we can get a better idea of how this works. Let the tokens be <span class="math notranslate nohighlight">\(X = [10, 2, \text{&lt;pad&gt;}]\)</span>, so the third token is a padding token. Lets then also pretend, we pass this to our model, and when we go to compute our attention <span class="math notranslate nohighlight">\(QK^T\)</span>. The raw output before the Softmax is below:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-a230ab2d-044f-4351-929f-ea68f6ce11ea">
-<span class="eqno">(1)<a class="headerlink" href="#equation-a230ab2d-044f-4351-929f-ea68f6ce11ea" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-d5241a6d-9f72-48fc-8054-0a38b162b56d">
+<span class="eqno">(1)<a class="headerlink" href="#equation-d5241a6d-9f72-48fc-8054-0a38b162b56d" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
   -3       &amp; 2   &amp; 4   \\
@@ -1114,8 +1114,8 @@ <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</s
 \text{Softmax}(\vec{x}) = \frac{e^{x_i}}{\sum_{j=1}^N{e^{x_j}}}
 \]</div>
 <p>If we ignore padding and everything right now, we can compute softmax for row of the matrix above:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-2d164eb8-340f-4ce3-bd4c-e2f2d7925d74">
-<span class="eqno">(2)<a class="headerlink" href="#equation-2d164eb8-340f-4ce3-bd4c-e2f2d7925d74" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-48203bcc-acc5-41e4-8587-c0a6c10607fb">
+<span class="eqno">(2)<a class="headerlink" href="#equation-48203bcc-acc5-41e4-8587-c0a6c10607fb" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
@@ -1134,17 +1134,17 @@ <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</s
 \end{bmatrix}
 \end{equation}\]</div>
 <p>But what we need is to mask out all the tokens in this matrix related to padding. Just like we did in <a class="reference external" href="https://github.com/priyammaz/HAL-DL-From-Scratch/tree/main/PyTorch%20for%20NLP/GPT">GPT</a>, we will fill in the indexes of the that we want to mask with <span class="math notranslate nohighlight">\(-\infty\)</span>. If only the last token was a padding token in our sequence, then the attention before the softmax should be written as:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-0253ffa2-35b1-488c-bcf4-d9752f37cee9">
-<span class="eqno">(3)<a class="headerlink" href="#equation-0253ffa2-35b1-488c-bcf4-d9752f37cee9" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-579f0937-3f63-4f69-a64b-235de2e41a33">
+<span class="eqno">(3)<a class="headerlink" href="#equation-579f0937-3f63-4f69-a64b-235de2e41a33" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; -\infty  \\
   -3       &amp; 2   &amp; -\infty   \\
   1       &amp; 6  &amp; -\infty  \\
 \end{bmatrix}
 \end{equation}\]</div>
 <p>Taking the softmax of the rows of this matrix then gives:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-4bfa0c8b-4c04-4693-ae6d-2206c57b9264">
-<span class="eqno">(4)<a class="headerlink" href="#equation-4bfa0c8b-4c04-4693-ae6d-2206c57b9264" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-c2a4a3f2-727f-4823-9678-0d2e45b64251">
+<span class="eqno">(4)<a class="headerlink" href="#equation-c2a4a3f2-727f-4823-9678-0d2e45b64251" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
  7       &amp; -8   &amp; -\infty  \\
@@ -1186,8 +1186,8 @@ <h3><span style="color:LightGreen">Repeating to Match Attention Matrix Shape</sp
 <p><code class="docutils literal notranslate"><span class="pre">attn.shape</span></code> - (Batch x seq_len x seq_len)</p>
 <p><code class="docutils literal notranslate"><span class="pre">mask.shape</span></code> - (Batch x seq_len)</p>
 <p>It is clear that our mask is missing a dimension, and we need to repeat it. Lets take sequence_1 for instance that has a mask of [True, True, True, False]. Because the sequence length here is 4, lets repeat this row 4 times:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-ed607000-2e59-4f14-a321-36f1bd479965">
-<span class="eqno">(5)<a class="headerlink" href="#equation-ed607000-2e59-4f14-a321-36f1bd479965" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
+<div class="amsmath math notranslate nohighlight" id="equation-a60b1a61-35b1-4942-a0de-f3d82ca3bce0">
+<span class="eqno">(5)<a class="headerlink" href="#equation-a60b1a61-35b1-4942-a0de-f3d82ca3bce0" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
@@ -1447,8 +1447,8 @@ <h3><span style="color:LightGreen">Enforcing Causality</span><a class="headerlin
 <section id="span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span">
 <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mask</span><a class="headerlink" href="#span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span" title="Permalink to this heading">#</a></h3>
 <p>Lets pretend the raw outputs of <span class="math notranslate nohighlight">\(QK^T\)</span>, before the softmax, is below:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-b19cc58c-07a4-493d-9726-af9cfb39ab99">
-<span class="eqno">(6)<a class="headerlink" href="#equation-b19cc58c-07a4-493d-9726-af9cfb39ab99" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-8ad62b77-687e-4290-8db7-aaa61c8b985f">
+<span class="eqno">(6)<a class="headerlink" href="#equation-8ad62b77-687e-4290-8db7-aaa61c8b985f" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
   -3       &amp; 2   &amp; 4   \\
@@ -1459,8 +1459,8 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
 <div class="math notranslate nohighlight">
 \[\text{Softmax}(\vec{x}) = \frac{e^{x_i}}{\sum_{j=1}^N{e^{x_j}}}\]</div>
 <p>Then, we can compute softmax for row of the matrix above:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-3b9abc65-eeb7-427f-a4aa-5d72ba8cf814">
-<span class="eqno">(7)<a class="headerlink" href="#equation-3b9abc65-eeb7-427f-a4aa-5d72ba8cf814" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-9ba80b99-5751-4c89-aeb1-1417f3ffc534">
+<span class="eqno">(7)<a class="headerlink" href="#equation-9ba80b99-5751-4c89-aeb1-1417f3ffc534" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
@@ -1499,17 +1499,17 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
 \text{Softmax}(x_2) = [\frac{e^{-3}}{e^{-3}+e^{2}+0}, \frac{e^{2}}{e^{-3}+e^{2}+0}, \frac{0}{e^{-3}+e^{2}+0}] = [\frac{e^{-3}}{e^{-3}+e^{2}+0}, \frac{e^{2}}{e^{-3}+e^{2}+0}, \frac{0}{e^{-3}+e^{2}+0}] = [0.0067, 0.9933, 0.0000]
 \]</div>
 <p>So we have exactly what we want! The attention weight of the last value is set to 0, so when we are on the second vector <span class="math notranslate nohighlight">\(x_2\)</span>, we cannot look forward to the future value vectors <span class="math notranslate nohighlight">\(v_3\)</span>, and the remaining parts add up to 1 so its still a probability vector! To do this correctly for the entire matrix, we can just substitute in the top triangle of <span class="math notranslate nohighlight">\(QK^T\)</span> with <span class="math notranslate nohighlight">\(-\infty\)</span>. This would look like:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-760a21b3-e8a1-4aec-b493-d61ffeb78f22">
-<span class="eqno">(8)<a class="headerlink" href="#equation-760a21b3-e8a1-4aec-b493-d61ffeb78f22" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-aab30ec1-747d-4c44-9c19-e3a142972185">
+<span class="eqno">(8)<a class="headerlink" href="#equation-aab30ec1-747d-4c44-9c19-e3a142972185" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -\infty   &amp; -\infty  \\
   -3       &amp; 2   &amp; -\infty   \\
   1       &amp; 6  &amp; -2   \\
 \end{bmatrix}
 \end{equation}\]</div>
 <p>Taking the softmax of the rows of this matrix then gives:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-629fd4eb-96a4-4077-a5ff-4765234df0f9">
-<span class="eqno">(9)<a class="headerlink" href="#equation-629fd4eb-96a4-4077-a5ff-4765234df0f9" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-6e0ab1b6-62e9-4b8d-8a64-7e80ef4002c4">
+<span class="eqno">(9)<a class="headerlink" href="#equation-6e0ab1b6-62e9-4b8d-8a64-7e80ef4002c4" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -\infty   &amp; -\infty  \\